[Pkg-javascript-devel] testing during build is useless now with autopkgtests

[Jérémy Lal]

2016-11-26 18:41 GMT+01:00 Jérémy Lal <kapouer at melix.org>:
> 2016-11-26 17:15 GMT+01:00 Pirate Praveen <praveen at onenetbeyond.org>:
>>
>>
>> On 2016, നവംബർ 26 6:34:26 PM IST, "Jérémy Lal" <kapouer at melix.org> wrote:
>>>Hi,
>>>
>>>i've been doing in all my latest uploads this setup:
>>>- autopkgtest runs the package test suite
>>>- build does not run tests
>>>
>>>Because:
>>>1) it feels wrong to add Depends to Build-Depends,
>>>and debian/tests/control already have all the info to properly run the
>>>tests
>>>2) when an autopkgtest fails, archive-wide testing reports the failure
>>>as a package bug,
>>>rendering tests during build useless anyway
>>
>> No, that is not correct. If tests during build fails, it is rc and it is found in archive wide rebuilds. But autopkgtest failures are not rc and the bug reporting is not automatic. I know this for certain because autopkgtest for gitlab has been failing for some time. So at least in the current situation, autopkgtest is not a replacement for tests during build, it may reach that point in future, but we are not there yet.

Side question - Do you know why we're not there yet ?

Having an RC bug because of a test failure is wrong too ! Sometimes
the failure comes from the test itself, so it's not so bad as it looks.

Also some tests are meant to run in a build tree - in which case of course,
they need to run during build, and some tests are meant to run after
the package is installed. Some tests are "in-between" they can run
in the source tree without being built.

What i want to say is that defining a "standard" choice for tests is not
as obvious as it seems. We can agree upon some strategy though.

Maybe tests that need source tree (be it built or not) must run during build,
and autopkgtests should only be for testing if the nodejs module can
be required (which is already a default test from autopkgtest) ?
What do you think ?

>>>I know that in some cases, one really need a built tree to run the
>>>tests,
>>>but it's not the majority. Usually a tree with patches applied is
>>>enough.
>>>
>>>So please Team, don't re-add to all my packages "run tests during
>>>build".
>>>(and, of course, do whatever you see best when you're packaging it
>>>yourself).
>>
>> This is not the right way to approach team maintained packages. If we end up differentiating "my package" and "your package", we defeat the purpose of team maintenance. Instead, we should be setting standards for the whole team, document it as team policy and implement it across all team maintained packages.

All right, but between mistakes and wrong-doings, when it "just
works", please don't
change what has been done before - especially without testing it
before uploading.
Also discussing it or at least notifying it, would be a good idea.

Jérémy