[Pkg-javascript-devel] testing during build is useless now with autopkgtests

[Pirate Praveen]

On Saturday 26 November 2016 11:23 PM, Jérémy Lal wrote:
> Side question - Do you know why we're not there yet ?

I guess its because autopkgtest is new service and we don't want to
disrupt this release with more rc bugs. It may change in next release
cycle and autopkgtest failure can be rc.

> Having an RC bug because of a test failure is wrong too ! Sometimes
> the failure comes from the test itself, so it's not so bad as it looks.

We are adding tests to detect problems early and many times it is really
a bug because some dependencies are changed and functionality that
worked before is no longer working. If a bug is wrongly tagged rc, we
can just reduce the severity, but missing a real bug is worse.

> Also some tests are meant to run in a build tree - in which case of course,
> they need to run during build, and some tests are meant to run after
> the package is installed. Some tests are "in-between" they can run
> in the source tree without being built.
> 
> What i want to say is that defining a "standard" choice for tests is not
> as obvious as it seems. We can agree upon some strategy though.

Yes, lets build consensus inside team about that strategy and document it.

> Maybe tests that need source tree (be it built or not) must run during build,
> and autopkgtests should only be for testing if the nodejs module can
> be required (which is already a default test from autopkgtest) ?
> What do you think ?

Having tests in autopkgtest helps us find if reverse dependencies are
broken when updating. Tools like build-and-upload from pkg-ruby-extras
repository (which I use) depend on autopkgtest to verify updates don't
break reverse dependencies.

I suggest we do both until autopkgtest can report failures as bugs
automatically.

> All right, but between mistakes and wrong-doings, when it "just
> works", please don't
> change what has been done before - especially without testing it
> before uploading.
> Also discussing it or at least notifying it, would be a good idea.

This specific case of node-rimraf was caused by a clear bug and I did
not think enabling tests during build was done purposefully. none-rimraf
did not declare dependency on node-glob, autopkgtest worked because
node-tap pulled it in, when testing node-cpr, which used mocha, it
failed. I did run build-and-upload (script from pkg-ruby-extras) before
uploading. I wonder why pristine-tar was failing to generate correct
orig.tar.gz.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/pkg-javascript-devel/attachments/20161127/05d18fb6/attachment.sig>