Bug#786039: add debconf testing

[Wouter Verhelst]

On Wed, May 20, 2015 at 02:12:19PM -0300, Antonio Terceiro wrote:
> On Wed, May 20, 2015 at 11:49:45AM +0200, Wouter Verhelst wrote:
> > Package: autopkgtest
> > Severity: wishlist
> > 
> > Hi,
> > 
> > It would be great if autopkgtest were to include a method to test
> > debconf scripts. This could be done by creating a custom debconf
> > frontend that takes answers to questions from autopkgtest rather than
> > from the user. The system could automatically permutate values over
> > various runs (e.g., install the package once while answering "yes" to a
> > boolean, and once while answering "no" to a boolean)
> > 
> > For templates which could have a potentially infinite number of answers,
> > either the maintainer could enter the possible range in the autopkgtest
> > configuration (or, better yet, in the template, though that would
> > require changes to the debconf spec), or the system could use a limited
> > number of random values. Also, for templates which have a limited but
> > large number of answers (e.g., the locales package), it might be good to
> > have a way to specify that the system doesn't need to try *all* possible
> > permutations.
> 
> Would adding a Preseed: field to the test definition be a decent
> solution to this? Something like:
> 
> ----------------8<----------------8<----------------8<-----------------
> Test: check-installed-package
> Preseed: path/to/scenario1.preseed
> 
> Test: check-installed-package
> Preseed: path/to/scenario2.preseed
> ----------------8<----------------8<----------------8<-----------------
> 
> Then that preseed could be loaded in the testbed's debconf database
> before installing any packages, and the maintainer would have complete
> control of what is being tested.
> 
> (the test programs listed in Test: might be the same or not, that's an
> orthogonal concern)

I suppose that could be helpful, too, but would necessarily be somewhat
more limited.

Imagine a frontend which detects that the first question is a multiple
choice one, selects the first answer, and moves on. It receives no more
answers, so uninstalls the package, and reinstalls it; then it selects
the second answer, and moves on. This time it receives a second
question, which is a boolean, so answers "true", and moves on, etc. A
frontend like that could test all possible answers with no user input,
and with no need for the maintainer to maintain a (possibly very large)
set of preseed files.

A Preseed: field would certainly work for things like the locales
package in this context, though, since otherwise the number of
permutations to test all possibilities would be extremely large[1]

Maybe this is out of scope for autopkgtest though, and might be a
somewhat more reasonable thing to do for something like piuparts, or
some such. Not sure.

[1] n+(n-1)!+(n-2)!+(n-3)!+...+1, if I'm not mistaken (with n being the
    number of available language codes).

-- 
It is easy to love a country that is famous for chocolate and beer

  -- Barack Obama, speaking in Brussels, Belgium, 2014-03-26