Testing packages [was Re: RFS: cl-asdf/3.3.1 - Another System Definition Facility]

[Kambiz Darabi]

Thanks! I'll check autopkgtest.

----- Original Message -----
> [I'm putting back the Common Lisp team in CC]
> 
> On Wed, Nov 15, 2017 at 10:52:27AM +0100, Kambiz Darabi wrote:
> > > Please push your changes in the git repository for cl-asdf
> > > packaging,
> > > and I will make the upload based on that.
> > 
> > done.
> 
> Thanks, I have made the upload. Note that it's not necessary to go
> through
> mentors.debian.net, the git is enough for internal team work.
> 
> > Do you have a good resource regarding autopkgtest other than its
> > README?
> > 
> > https://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/plain/doc/README.package-tests.rst
> > 
> > And looking at your changes for 3.3.0, I wonder where I can learn
> > those checks
> > which you have performed on the package. I would appreciate any
> > hint or links
> > you might have.
> 
> I basically run three tools before every upload:
> 
> - lintian, at the info level (which you already know)
> 
> - piuparts, which does install, upgrade and purge tests of the
> package; this
>   one is useful for detecting file conflicts between packages,
>   problems in
>   maintainer scripts… It's not likely to detect many issues in ASDF,
>   since the
>   packaging is rather simple (only one package, no maintainer
>   scripts) and does
>   not change over time
> 
> - autopkgtest, which tests the *installed* package (and not the built
> package
>   as the previous ones). In practice, autopkgtest installs the
>   package through
>   apt/dpkg, and then run various tests that you have to design.
> 
> (these three tools are run automatically as part of my build setup,
> see https://wiki.debian.org/sbuild)
> 
> It would indeed be great to add autopkgtest support to cl-asdf. For
> example,
> you could create tests to verify that ASDF loads correctly in some or
> in all CL
> implementations that are in Debian (SBCL, CMUCL, ECL, CLISP). Or you
> could
> design more complicated tests that verify some precise
> functionnality. There is
> virtually no limit to what can be implemented, it's up to you,
> depending on
> your quality requirements and on your time. Of course, if ASDF
> provides its own
> testsuite (I did not verify), a natural test would be to run that
> testsuite in
> all supported implementations.
> 
> The documentation for autopkgtest indeed consists of the the various
> README
> (installed under /usr/share/doc/autopkgtest/). It's exhaustive but
> not very
> user-friendly.
> 
> Basically you have to ship a debian/tests/ directory with a "control"
> file
> (which lists the tests and give dependencies for each of them), and
> then
> possibly other files depending on the specific tests that you want to
> run.
> 
> You can then run the tests locally using the autopgktest command,
> which
> provides many ways of running the tests (on the current system, in a
> chroot, in
> a container…)
> 
> You can also take inspiration from other packages.  For example, in
> some other
> packages that I maintain:
> 
> https://anonscm.debian.org/cgit/debian-science/packages/lapack.git/tree/debian/tests
> https://anonscm.debian.org/cgit/debian-science/packages/glpk.git/tree/debian/tests
> https://anonscm.debian.org/cgit/pkg-octave/dynare.git/tree/debian/tests
> https://anonscm.debian.org/cgit/debian-science/packages/r-cran-statmod.git/tree/debian/tests
> 
> Don't hesitate to ask if you have more questions.
> 
> Best,
> 
> --
> ⢀⣴⠾⠻⢶⣦⠀  Sébastien Villemot
> ⣾⠁⢠⠒⠀⣿⡁  Debian Developer
> ⢿⡄⠘⠷⠚⠋⠀  http://sebastien.villemot.name
> ⠈⠳⣄⠀⠀⠀⠀  http://www.debian.org
>