Bug#834682: mina2: FTBFS in testing (failing tests)

Wed Sep 7 12:33:06 UTC 2016

retitle 834682 mina2: FTBFS in testing (failing tests)
thanks

On Fri, 26 Aug 2016, Markus Koschany wrote:

> I have rebuilt mina2 ten times in a row now but I can't reproduce the
> failing test or the build failure in general.

Congratulations. I attach five build logs in five different autobuilders.

Apparently, it fails for me all the time.

[ So, instead of you asking me for a way to reproduce it, perhaps I
  should be asking you for a way *not* to reproduce it... ]

> It appears this issue is not random

After my last five builds yesterday, that's what it seems, yes.
Now it fails all the time, so it's not random anymore.

I'm changing the subject accordingly.

> and does not affect the majority of our users.

Whether it affects the user is not really the issue because the usual
severity definitions do not apply here. In this case, we have the
"packages must build" in release policy:

https://release.debian.org/stretch/rc_policy.txt

"Packages must autobuild" means "packages must autobuild", not
"packages must autobuild almost always", or "packages must autobuild
in the maintainer's machine".

> In my opinion it is not release critical hence I'm going to lower the
> severity unless there is a compelling reason to do otherwise.

Well, there is indeed a compelling reason, and it's the fact that it's
a FTBFS bug and release policy says those bugs are RC:

https://release.debian.org/stretch/rc_policy.txt

so once again I feel that you are downgrading too happily.

> Since we have seen similar bug reports against packages with enabled
> test suites in the past, I suggest to following:
> 
> If the tests fail randomly, the test(s) or the whole test suite should
> be disabled or the tests changed because otherwise they cause more
> maintenance work. The reasoning for enabling the test suite is that
> regressions should be quickly detected but if the test results are
> random we won't benefit from them.
> 
> The test suite is not the only way to spot regressions in a
> distribution. Most of the time we are aware of build failures and
> receive respective reports and if there is an important behavioral
> change users will notice it.
> 
> On the other hand if we can choose between enabling or disabling a test
> suite I'm all for enabling it because when the results are reliable,
> they are a helpful mechanism to spot errors. So there is a trade-off
> between the usefulness of the tests and the cost of triaging this kind
> of bug reports and causing build failures.

I fully agree with your proposal that tests that fail randomly
(in the sense that we don't know why they fail) should be disabled.

That's precisely the idea behind some of the bugs that I've filed
lately with subjects like "FTBFS too much often".

By providing several different failed build logs on different
autobuilders I want to make the point that the failures, if random,
happen often enough that we might better disable those tests.

When I reported this the first time, I believed that the fact that it
happened in my autobuilders *and* the autobuilders from reproducible
builds was enough evidence that the failures happened often enough.

Now I am providing five different failed build logs in five different
autobuilders. Is this still not enough?

I would appreciate if you or anybody else in the team could really and
honestly try to reproduce this in your own system the same way I do,
so that we can compare build logs. This is what I'm using:

sbuild
A minimal stretch chroot
eatmydata (but I've disabled for the build logs attached to this message)

Everything inside a KVM virtual machine with memory ranging from 768 MB to 6 GB.

Please note that I don't discard the possibility that this is a bug in
sbuild, but after building more than 15000 different source packages,
whenever a build does not end successfully and sbuild hangs, it often
happens that the test suite was doing something not very "orthodox"
(like, for example, starting a daemon and not killing it before
finishing the tests).

In other words: I have yet to see an example where sbuild is really to
blame and not the package for doing "weird things".

Please note also that sbuild is the same software that it's running in
the official build daemons, so using sbuild is the best way to do
things "as close as possible" to the way official buildds do things.

Thanks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mina2_2.0.13-1_amd64-20160906T234602Z.gz
Type: application/gzip
Size: 38588 bytes
Desc: 
URL: <http://lists.alioth.debian.org/pipermail/pkg-java-maintainers/attachments/20160907/77c591cd/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mina2_2.0.13-1_amd64-20160906T234603Z.gz
Type: application/gzip
Size: 40598 bytes
Desc: 
URL: <http://lists.alioth.debian.org/pipermail/pkg-java-maintainers/attachments/20160907/77c591cd/attachment-0006.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mina2_2.0.13-1_amd64-20160906T234610Z.gz
Type: application/gzip
Size: 39856 bytes
Desc: 
URL: <http://lists.alioth.debian.org/pipermail/pkg-java-maintainers/attachments/20160907/77c591cd/attachment-0007.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mina2_2.0.13-1_amd64-20160906T234611Z.gz
Type: application/gzip
Size: 40056 bytes
Desc: 
URL: <http://lists.alioth.debian.org/pipermail/pkg-java-maintainers/attachments/20160907/77c591cd/attachment-0008.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mina2_2.0.13-1_amd64-20160906T234618Z.gz
Type: application/gzip
Size: 40611 bytes
Desc: 
URL: <http://lists.alioth.debian.org/pipermail/pkg-java-maintainers/attachments/20160907/77c591cd/attachment-0009.bin>