Bug#925179: perl: autopkgtest failures on arm64, ppc64el and s390x: NDBM backwards binary compatibility

[Niko Tyni]

(Copying Adam as he expressed some interest in this on IRC.)

On Wed, Mar 20, 2019 at 10:51:13PM +0200, Niko Tyni wrote:
> Package: perl
> Version: 5.28.1-5
> Severity: important
> 
> As seen at https://autopkgtest.ubuntu.com/packages/perl the NDBM
> autopkgtest checks are failing on several 64-bit architectures.
> 
> These checks bundle binary NDBM databases generated on current sid/buster
> and stretch versions of perl, and the purpose is to check that those
> databases remain readable with the current version of perl.
> 
> We used to test this with just amd64 databases, but it turned out that
> they are architecture specific, so we now ship separate test data files
> for the different architectures.
> 
> All the test failures are issues with reading the stretch NDBM.
> 
>   autopkgtest [05:12:53]: test ndbm: [-----------------------
>   testing NDBM reading...
>   buster-ndbm OK
>   opening NDBM file failed: No such file or directory at -e line 1.
>   autopkgtest [05:12:54]: test ndbm: -----------------------]
>   ndbm                 FAIL non-zero exit status 1
> 
> The "No such file or directory" is a misleading diagnostic here;
> the files get opened correctly. The issue seems to be an actual
> incompatibility with the stretch databases.
> 
> I can reproduce these on Debian porter machines. Clearly I should
> have tested this better before including the data. Apologies for
> any inconvenience caused.

Looking deeper, it seems more likely that I have goofed up the test
files than this being an issue with actual compatibility.

While the current test data files from stretch indeed cannot be read
on sid, I cannot make new test data files on stretch that reproduce the
behaviour. I've created 100 such databases on each affected architecture
and all of those are readable on sid.

I note that the test data files we're shipping for stretch versions of
ppc64el and arm64 are identical, which seems improbable. My best guess
is that I've mixed up the test data files from different architectures
when copying them around and committing them to git. How embarrassing.

I'll try to get this fixed for buster by regenerating the test data files.

Somewhat interestingly, investigating this I noticed that just testing the
contents of the data files on sid changes the database file format. The
stretch Perl NDBM bindings create two identical files, db.dir and db.pag
with 'file' reporting

 GNU dbm 1.x or ndbm database, little endian, old

but reading those in on sid converts db.dir to a much smaller file
which is reported as

 GNU dbm 2.x database

while not touching the .pag file in any way. The resulting files are
still readable on stretch, so even this doesn't seem to be the core of
the issue. It looks like we're shipping such "converted" test data on
amd64 though, and this should be fixed.
-- 
Niko Tyni   ntyni at debian.org