Bug#834682: mina2: FTBFS too much often (failing tests)

[Markus Koschany]

Control: severity -1 normal
Control: tags -1 unreproducible

On Wed, 17 Aug 2016 23:50:03 +0000 Santiago Vila <sanvila at debian.org> wrote:
> Package: src:mina2
> Version: 2.0.13-1
> Severity: serious
> 
> Dear maintainer:
> 
> I tried to build this package with "dpkg-buildpackage -A"
> (which is what the "Arch: all" autobuilder would do to build it)
> but it failed:
> 
[...]
> 
> Tests in error: 
>   NioFileRegionTest.testSendLargeFile ?? IO Broken pipe
> 
> Tests run: 201, Failures: 0, Errors: 1, Skipped: 5


I have rebuilt mina2 ten times in a row now but I can't reproduce the
failing test or the build failure in general. It appears this issue is
not random and does not affect the majority of our users. In my opinion
it is not release critical hence I'm going to lower the severity unless
there is a compelling reason to do otherwise.

Since we have seen similar bug reports against packages with enabled
test suites in the past, I suggest to following:

If the tests fail randomly, the test(s) or the whole test suite should
be disabled or the tests changed because otherwise they cause more
maintenance work. The reasoning for enabling the test suite is that
regressions should be quickly detected but if the test results are
random we won't benefit from them.

The test suite is not the only way to spot regressions in a
distribution. Most of the time we are aware of build failures and
receive respective reports and if there is an important behavioral
change users will notice it.

On the other hand if we can choose between enabling or disabling a test
suite I'm all for enabling it because when the results are reliable,
they are a helpful mechanism to spot errors. So there is a trade-off
between the usefulness of the tests and the cost of triaging this kind
of bug reports and causing build failures.

Markus

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 949 bytes
Desc: OpenPGP digital signature
URL: <http://lists.alioth.debian.org/pipermail/pkg-java-maintainers/attachments/20160826/fcafaf39/attachment.sig>