[Reproducible-builds] Adding a 'c' build to test with DEB_BUILD_OPTIONS=nocheck ?

[Antonio Terceiro]

On Wed, Sep 23, 2015 at 03:14:10AM +0200, Holger Levsen wrote:
> Hi,
> 
> adding Antonio to the loop, keeping full quote for his benefit.
> 
> On Samstag, 19. September 2015, Chris Lamb wrote:
> > I just ran into yet another package where the contents can vary
> > depending on whether the tests are run or not.
> 
> I do think this is an interesting thing to test, but I'm not sure whether the 
> reproducible builds test setup is the right place for it.

For sure it's something interesting to test. IMO a source package
producing different binaries based on whether the tests are run or not
is a bug.

> > As an example, without tests a given Python package entirely vanilla and
> > is thus reproducible in our toolchain. However, executing the tests
> > creates various intermediary files that are genuinely useful (eg.
> > compiled versions of grammars, not .pyc files). These files are then
> > installed to the binary package.
> > 
> > I'm only really discovering these when these files themselves are
> > unreproducible/non-deterministic, otherwise they are completely
> > invisible.
> > 
> > So, does this matter to us? It's strictly more of a general QA issue if
> > we are declare that DEB_BUILD_OPTIONS does not contain nocheck from an
> > reproducibility PoV.. but on the other hand, our testing framework would
> > make this almost trivial to detect.
> > 
> > (Why another build? Whilst adding `nocheck` to our current `b` build
> > could work, it would be a bad regression as I would dearly miss having
> > the tests run in an exotic locale and timezone, etc., hence proposing a
> > `c` build).
> 
> Antonio, do you think this is something for ci.d.o?

I don't think this is something that makes much sense to do in ci
because:

- it does not do builds, and will take binaries from the archive
- the test suite is obtained from source packages also in the archive
- even then, the test suite is not executed in the context of a build,
  but against the installed binaries, so there would be no way to check
  the effects of running the test suite on the produced binaries

> Chris, should we forward this as a bug report for qa.d.o so someone can 
> implement this some day? OTOH, if you have an idea how to meaningful integrate 
> this into the current reproducible setup, I'd be all ears! From the above I 
> can gather that this could be possible, maybe, but I'm not sure how much 
> interference this will cause…

I think reproducibility should include requiring producing the same
binaries regardless of whether tests were executed during the build.

-- 
Antonio Terceiro <terceiro at debian.org>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/reproducible-builds/attachments/20150923/7f9a613e/attachment.sig>