[debhelper-devel] dh_auto_test - Integration with CTest/CDash

[Kyle Edwards]

On 04/10/2018 03:27 PM, Niels Thykier wrote:
> How would you normally do this without the Debian packaging stack?  Is
> it something like:
>
>    cmake ...
>    make/ninja build
>    make test/ninja test
>    cdash ... ?
>
> Note that cmake is not a very strong suit for me so rather assume I know
> less rather than more.

Sort of. The dashboard script does all of those steps, so all you have 
to do is set up a cronjob that calls ctest -S /.../my_dashboard.cmake. 
The dashboard script looks vaguely like this:

ctest_start()
ctest_configure()
ctest_build()
ctest_test()
ctest_submit()

...with a bunch of extra options added to it. However, I have spoken to 
our CDash expert, who told me I can write a simpler dashboard script 
which omits the ctest_configure() and ctest_build() steps, since those 
have already been completed by the Debian packaging stack. So it looks 
like my strategy would be to have the Debian packaging stack call the 
CDash dashboard script in the dh_auto_test rule. But that brings me to 
the next problem...

> If it does not contain passwords or sensitive material, then it could be
> copied into the debian dir before the source package was built and have
> the build use the package from debian/.  But given the nature of what it
> is doing, I have a feeling that you will shoot down that assumption.

The dashboard scripts typically do not contain passwords - Kitware's own 
VTK CDash server is open to the public, so anyone can anonymously submit 
test results without authentication. However, the scripts DO contain 
site-specific configuration such as hostnames, and I would rather not 
pollute the Debian packaging with this information. I would like to do 
something like:

$ VTK_DASHBOARD_SCRIPT=path/to/dashboard/script.cmake gbp buildpackage 
--git-builder=sbuild

and then debian/rules would detect the presence of a dashboard script 
and change its behavior accordingly. But from my understanding, this 
goes against the Debian philosophy that builds should be reproducible 
and not be polluted by the external environment, and it would also 
potentially present a problem when the source package and binary package 
are being built on different machines. It also violates the rule of no 
network access, but that's why it would be configurable, so that it 
would only be turned on if the server admin explicitly allows it.

Has anyone else done something similar to this? Uploading test results 
somewhere from dh_auto_test? Or perhaps on our end we could change CMake 
to provide better support for testing inside Debhelper?

Kyle