Testing packages [was Re: RFS: cl-asdf/3.3.1 - Another System Definition Facility]

[Sébastien Villemot]

[I'm putting back the Common Lisp team in CC]

On Wed, Nov 15, 2017 at 10:52:27AM +0100, Kambiz Darabi wrote:
> > Please push your changes in the git repository for cl-asdf packaging,
> > and I will make the upload based on that.
> 
> done.

Thanks, I have made the upload. Note that it's not necessary to go through
mentors.debian.net, the git is enough for internal team work.

> Do you have a good resource regarding autopkgtest other than its README?
> 
> https://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/plain/doc/README.package-tests.rst
> 
> And looking at your changes for 3.3.0, I wonder where I can learn those checks
> which you have performed on the package. I would appreciate any hint or links
> you might have.

I basically run three tools before every upload:

- lintian, at the info level (which you already know)

- piuparts, which does install, upgrade and purge tests of the package; this
  one is useful for detecting file conflicts between packages, problems in
  maintainer scripts… It's not likely to detect many issues in ASDF, since the
  packaging is rather simple (only one package, no maintainer scripts) and does
  not change over time

- autopkgtest, which tests the *installed* package (and not the built package
  as the previous ones). In practice, autopkgtest installs the package through
  apt/dpkg, and then run various tests that you have to design.

(these three tools are run automatically as part of my build setup, see https://wiki.debian.org/sbuild)

It would indeed be great to add autopkgtest support to cl-asdf. For example,
you could create tests to verify that ASDF loads correctly in some or in all CL
implementations that are in Debian (SBCL, CMUCL, ECL, CLISP). Or you could
design more complicated tests that verify some precise functionnality. There is
virtually no limit to what can be implemented, it's up to you, depending on
your quality requirements and on your time. Of course, if ASDF provides its own
testsuite (I did not verify), a natural test would be to run that testsuite in
all supported implementations.

The documentation for autopkgtest indeed consists of the the various README
(installed under /usr/share/doc/autopkgtest/). It's exhaustive but not very
user-friendly.

Basically you have to ship a debian/tests/ directory with a "control" file
(which lists the tests and give dependencies for each of them), and then
possibly other files depending on the specific tests that you want to run.

You can then run the tests locally using the autopgktest command, which
provides many ways of running the tests (on the current system, in a chroot, in
a container…)

You can also take inspiration from other packages.  For example, in some other
packages that I maintain:

https://anonscm.debian.org/cgit/debian-science/packages/lapack.git/tree/debian/tests
https://anonscm.debian.org/cgit/debian-science/packages/glpk.git/tree/debian/tests
https://anonscm.debian.org/cgit/pkg-octave/dynare.git/tree/debian/tests
https://anonscm.debian.org/cgit/debian-science/packages/r-cran-statmod.git/tree/debian/tests

Don't hesitate to ask if you have more questions.

Best,

-- 
⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
⣾⠁⢠⠒⠀⣿⡁  Debian Developer
⢿⡄⠘⠷⠚⠋⠀  http://sebastien.villemot.name
⠈⠳⣄⠀⠀⠀⠀  http://www.debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-common-lisp-devel/attachments/20171115/32217721/attachment.sig>