[Reproducible-builds] Bug#807997: diffoscope: CbfsFile checks make diffoscope extra slow

[Mike Hommey]

On Tue, Dec 15, 2015 at 02:43:26PM +0900, Mike Hommey wrote:
> Source: diffoscope
> Version: ada1a1dcdc19217fb611e0a1e57bc3744399aefa
> Severity: important
> 
> Dear Maintainer,
> 
>    * What led up to the situation?
> 
> Comparing
> http://archive.mozilla.org/pub/firefox/try-builds/mh@glandium.org-93eecb52fcadc562b0f52cb22a8971770446a12e/try-linux64/firefox-45.0a1.en-US.linux-x86_64.tar.bz2
> with
> https://queue.taskcluster.net/v1/task/Y1kcxbpQQcOQhIHDp7oRxA/runs/0/artifacts/public/build/target.tar.bz2
> 
>    * What exactly did you do (or not do) that was effective (or
>      ineffective)?
> 
> Ran `diffoscope --html output.html firefox-45.0a1.en-US.linux-x86_64.tar.bz2 target.tar.bz2`
> 
>    * What was the outcome of this action?
> 
> It took 9 minutes and 15 seconds on my machine.
> 
>    * What outcome did you expect instead?
> 
> It taking much less time. In fact, after doing some profiling and seeing that
> CbfsFile was a hotspot, applying the following patch made the command take 80
> seconds instead, most of which is spent bunzip2'ing the two archives
> sequentially, and objdump --disassemble'ing libxul.so in the two archives in
> parallel (so there could be a win with bunzip2'ing the two archives in
> parallel, too).
> 
> diff --git a/diffoscope/comparators/__init__.py b/diffoscope/comparators/__init__.py
> index b4615c9..c4cffbb 100644
> --- a/diffoscope/comparators/__init__.py
> +++ b/diffoscope/comparators/__init__.py
> @@ -129,7 +129,6 @@ FILE_CLASSES = (
>      DebDataTarFile,
>      TextFile,
>      Bzip2File,
> -    CbfsFile,
>      CpioFile,
>      DebFile,
>      DexFile,

Another way to make it faster here is to put CbfsFile last in this list,
although it will still make things slow for large files that contain
unknown content.

Mike