Bug#479681: Perl symbol problem - release critical (Re: Bug#489132)

[Niko Tyni]

On Sat, May 21, 2011 at 06:23:25PM +0100, Dominic Hargreaves wrote:
> On Thu, Aug 21, 2008 at 12:52:44AM +0300, Niko Tyni wrote:
> > Post-lenny, I see two options that don't involve changing the module path:
> > 
> > - mandate that ABI changes in the Perl XS module interface
> >   will always be accompanied with a symbol rename caught by
> >   PERL_DL_NONLAZY, and artificially do that for Debian if needed in the
> >   future. This practically means "just carry on and hope we don't have
> >   to deviate from perl upstream".
> > 
> > - integrate Locale::gettext in perl-base (#479681) and mandate that
> >   Essential:yes programs may not load non-Essential XS modules even
> >   opportunistically (inside an eval block) because PERL_DL_NONLAZY
> >   can't be trusted.  This seems to be the safer option of the two.
> 
> I think this bug probably represents the same issues as #479711 now.
> I wonder whether they should be merged.

I'm not sure about the merge either. Probably yes.

Reviewing the discussion, I think that Ian had a valid point and that
we should revisit the vendorarch directory choice. Brendan (cc'd):
I'd love to have your thoughts on this.

First, I see all the dpkg commands in /usr/bin have been rewritten in C
for squeeze so I don't think this issue concerns Essential functionality
at all anymore.

As a consequence, I think Locale::gettext is no longer quite as special as
it used to be. The remaining cases that I'm aware of where XS modules get
loaded opportunistically during upgrades (without package dependencies
fulfilled) are 'preinst upgrade' maintainer scripts and debconf config
scripts.

The debconf case is sourcing /usr/share/debconf/confmodule.sh, which
currently tries to load three XS modules: Locale::Gettext, Text::Iconv
and Text::CharWidth.

Therefore, if any problems remain, they are not going to be solved
just by integrating Locale::gettext into perl-base.

Now, as long as PERL_DL_NONLAZY catches ABI skew problems, I think we're
safe. However, as Ian pointed out [1] there may well be ABI changes
that don't involve missing or renamed symbols. A prime example of those
would be backing out the -Duse64bitint change we made for 5.12. [2]

Ideally, the XS modules would have an ABI signature that would be checked
by DynaLoader first. I doubt this would fly upstream as they already
rejected making the PERL_DL_NONLAZY=1 behaviour the default in eval{}
blocks (see  #479711) because some people might be relying on partially
loading ABI incompatible modules. Also, I believe they encode some
Configure options into the default @INC [3], which helps to avoid
situations like this.

Lacking such a signature, switching to a versioned @INC for
$Config{vendorarch} (currently /usr/lib/perl5) while keeping
$Config{vendorlib} as /usr/share/perl5 seems to me the way to go.

It would be lovely to only have one vendorarch directory per ABI,
but I think upstream's policy [4] of not breaking the ABI in minor
releases doesn't cover extending it. Such extensions would break forward
compatibility (i.e. 5.12.3 would not be able load XS modules built
for 5.12.4.)

That would leave us with something like (say)
 /usr/lib/perl5.12/5.12.3
 /usr/lib/perl5.12/5.12.4
which could be renamed after a hypothetical -Duse64bitint reversion to
(for example)
 /usr/lib/perl5.12/5.12.4-32bitint

Given the upstream new schedule of yearly major releases, I'd hope
there would not be too many subdirectories.

The 5.14 transition is probably a good time to do this change if we decide
to go forward with it. Naturally we need a discussion on the debian-perl /
debian-devel lists before pushing changes to the policy.

I suppose the alternative is to carry on as usual and worry about
these things when they actually become a problem. Hopefully we aren't
in too much of a hurry at that point so that the necessary changes can
be reviewed properly.

One real problem I'm seeing now is that building a Debian 5.14.0 on a
system with 5.12.3 hits test failures if certain XS module packages are
installed. This is because the modules get opportunistically loaded from
/usr/lib/perl5 but fail to work due to ABI incompatibilities. This is
not a big thing but it's rather annoying and may need Build-Conflicts
that would otherwise be unnecessary.

[1] http://lists.debian.org/debian-devel/2008/07/msg00067.html 
[2] observed behaviour in such a situation:
    # perl -MDevel::Refcount=refcount -le 'print refcount($a=[])'
    583208145925439490
[3] for instance 'x86_64-linux-thread-multi' or 'i686-linux-64int';
    I don't think there's enough information to uniquely define the ABI though
[4] see perlpolicy.pod
-- 
Niko Tyni   ntyni at debian.org