specifying extra requirements on testbeds

[Martin Pitt]

Hey Antonio,

Antonio Terceiro [2014-01-28 22:38 -0300]:
> > Indeed. The first thing that comes to my mind is that we need
> > something better in the DEP-8 control fields to describe what kind of
> > machines or runners the test works with. Just yesterday I evaluated
> > all our autopkgtests (in Ubuntu, but as we push them back to Debian as
> > much as possible it shouldn't be totally different) in LXC [1], and
> > there are some tests where schroot and even LXC just don't cut it, and
> > a full VM is needed.
> 
> For these cases we could use a field called "Supported-Testbeds" or
> something like it, that would take a list of supported testbeds, "any"
> meaning there is no restriction.
> 
> Supported-Testbeds: any
> 
> Supported-Testbeds: schroot lxc
> 
> any could/should be the default

I think we shouldn't use the names of adt-virt-* here; DEP-8 is a
specification, and autopkgtest "just" its (reference) implementation,
but e. g. there is devscripts' new "sadt" too. Also, adt-virt-chroot
and -schroot are very similar in terms of "how much isolation does a
test need", as does a runner in qemu and -null.

So I don't think we should specify implementations, we'd have to
update them with each new runner (e. g. someone could write an
adt-virt-docker), but instead abstract them into isolation levels:

 * isolation-fs: testbed provides its own file system space
   adt-virt-{chroot,schroot}

 * isolation-container: fs + provides its own network space and you
   can start all services
   adt-virt-lxc, or e. g. a hyopthetical adt-virt-docker

 * isolation-machine: complete machine including full kernel access
   adt-virt-null, adt-virt-qemu [1]

I don't even think that we explicitly need isolation-fs; all runners
provide that, and pretty much all sensible tests need to install some
files, test dependencies, and so on. Their "intrusiveness" is already
described with needs-root, breaks-testbed, etc.

So we would be down to "isolation-container", which tests need to
specify which rely heavily on services to get started (chroots might
have a policy-rc.d so that things like apache don't come up) or on
having their own network space, and "isolation-machine" for tests
which meddle with the kernel (like udisks2, bluez, crash,
network-manager, gvfs). These are simple and static enough to become
new restrictions, instead of an entirely new field.

adt-virt-* then need to be updated to provide their isolation level in
their capabilities.

Opinions?

> > Also, we soon want to move our other kinds of
> > tests to use the autopkgtest machinery, then we'll need something like
> > "this test needs to run on a machine with an NVidia card". I. e. some
> > kind of tags that you can assign test execution environmens with and
> > match them against the requirements in debian/tests/control.>
> 
> How would we mark each testbed with these tags?

These tags are more free-form, to describe particular hardware and
such. At some point we'll need those, but it's not quite clear yet on
which level it makes sense to implement those. E. g. in Jenkins you
can already tag execution slaves in that manner, and it would actually
seem redundant having to add an e. g. /etc/autopkgtest/tags.txt file
to them for the runner to read. So it might actually turn out to be
impractical to implement this in autopkgtest and the spec, so maybe
let's put that idea on the backburner for now.

Thanks,

Martin


[1] I am currently working on a QEMU runner which does things
"properly", unlike the "run -null in QEMU" setup that we currently use
in Ubuntu. That one will have full "revert" capability and such.

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.alioth.debian.org/pipermail/autopkgtest-devel/attachments/20140205/24db8b07/attachment-0001.sig>