[Reproducible-builds] Bug#813052: Bug#813052: Bug#813052: Bug#813052: diffoscope takes more than an hour on foreign arch libc6

[Steven Chamberlain]

Hi!

Jérémy Bobbio wrote:
> Helmut Grohne:
> > What I mean is that diffoscope takes the directory that contains the
> > first debian package and then recursively looks at all contained files.
> > If that tree happens to be big, bad things can happen.

I'm observing exactly the same behaviour, giving diffoscope just two
.debs to compare.  Right after:

| DEBUG Looking for a dbgsym package for Build Id 614f9b04792aee7656821afdc52b2a23a387d424 (debuglink: 4f9b04792aee7656821afdc52b2a23a387d424.debug

diffoscope would stat() each file in the containing directory, but also
recurse into subdirectories, including the whole build trees in my case.
This is extremely slow on real disks, not having tmpfs.

> diffoscope will try to locate a package with matching debug symbols when
> it's comparing ELF files inside two .deb. It will indeed look at the
> files in the parent container (in your case a directory), but it's only
> a quick look: looking for .deb files and looking at the control file.
> 
> Still, it was indeed looking at all files in the tree. Could you try the
> attached patch and see if it helps?

With your patch, it doesn't recurse any more.  But it will still stat()
everything in the containing directory, looking for .debs.  It also
opens some files and reads them - even decompressing random .gz files
along the way!  It does some processing on .buildinfo files too:

| DEBUG Looking for a dbgsym package for Build Id 614f9b04792aee7656821afdc52b2a23a387d424 (debuglink: 4f9b04792aee7656821afdc52b2a23a387d424.debug)
| DEBUG Using TextFile for a/build/.bash_logout
| DEBUG Using TextFile for a/build/.bashrc
| DEBUG Using TextFile for a/build/.profile
| DEBUG Using DebFile for a/build/cpp-4.9-dbgsym_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/cpp-4.9_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/g++-4.9-dbgsym_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/g++-4.9-multilib_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/g++-4.9_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-base_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-dbgsym_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-locales_4.9.3-11_all.deb
| DEBUG Using DebFile for a/build/gcc-4.9-multilib_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-plugin-dev-dbgsym_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-plugin-dev_4.9.3-11_kfreebsd-amd64.deb
| DEBUG Using DebFile for a/build/gcc-4.9-source_4.9.3-11_all.deb
| DEBUG Using GzipFile for a/build/gcc-4.9_4.9.3-11.diff.gz
| DEBUG Using DotDscFile for a/build/gcc-4.9_4.9.3-11.dsc
| DEBUG Using DotBuildinfoFile for a/build/gcc-4.9_4.9.3-11_kfreebsd-amd64.buildinfo
| DEBUG validating sha1 checksums
| DEBUG sha1 Checksum for file a/build/cpp-4.9-dbgsym_4.9.3-11_kfreebsd-amd64.deb matches
| DEBUG sha1 Checksum for file a/build/cpp-4.9_4.9.3-11_kfreebsd-amd64.deb matches
| DEBUG sha1 Checksum for file a/build/g++-4.9-dbgsym_4.9.3-11_kfreebsd-amd64.deb matches

It only fully unpacks the -dbgsym .debs corresponding to the .debs
it was asked to compare.

| DEBUG returning a <class 'diffoscope.comparators.deb.DebContainer'> for <<class 'abc.DebFile'> a/build/cpp-4.9-dbgsym_4.9.3-11_kfreebsd-amd64.deb>
| DEBUG unpacking data.tar.xz

The time taken to do all that is now quite small compared to the time
for objdump to run.

I see this maybe still being a problem where, on some machines, my home
directory is littered with hundreds of .debs, and thousands of other
files, some of them possibly gzipped.

Regards,
-- 
Steven Chamberlain
steven at pyro.eu.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/reproducible-builds/attachments/20160216/ef7b608f/attachment.sig>