[Freedombox-discuss] Building from the bottom up, using automated tests

[Lars Wirzenius]

I believe it is best to start building the FreedomBox from the bottom
up: start with something simple that is easy to get to work, and create
the tools and processes to do that in a sensible way. The main
deliverable should be system images, possibly with installers, that 
can be installed on suitable target hardware. The two main targets I
can think of are the DreamPlug device, and any i386-compatible machine
that can boot off a USB memory stick. The latter is important because
many people have such a system available to them, and this makes it
easier to develop things.

I don't have a DreamPlug, and I'm still trying to wrap my head around
how to generate working ARM images, particularly on a non-ARM host.
However, I do have a preliminary tool for generating i386 images
easily. See http://liw.fi/vmdebootstrap/ for mine. There are others,
but I like mine because of its extreme simplicity. It may, however,
be too simple. I've talked a bit with Bdale, and he's got some scripts
around the multistrap tool to generate ARM images, it seems.

However, just generating images is not enough. We also need to test
them, and as much of the testing as possible should be automated. 
Automatic tests give us a great deal of confidence that whatever
changes we make to the FBX, we don't break stuff that used to work.

There are several levels of tests involved:

* unit tests, which test individual functions and classes
* integration and functional tests, which test larger parts of programs, or
  entire programs, or collections of co-operating programs
* system tests, which test the entire system

Unit, integration, and functional tests are expected to be provided
by each program. If they're missing (as they usually are), working
with the project producing the program to add the missing tests is
a good idea. It is, however, outside the scope of the FBX project,
or else we'll never get anything done.

What we need to assume is that the individual pieces we get from
other projects work sufficiently well. There will always be bugs,
and no amount of testing will find all bugs, but we'll deal with 
those when we find them.

Instead, we'll worry about making sure that the system we're building
works as a whole. In other words, rather than writing tests for, say,
the Apache web server, and trying to prove that it has no bugs that
concern us, we'll take it as a given that it works well, but that any
configuration and setup we do needs to be tested. Also, the interaction
of Apache and, say, ikiwiki, needs to be tested. Further, system level
configuration, such as bringing up the network interface and configuring
DNS, needs testing.

For this, I've written a prototype of a tool called systest
(see http://liw.fi/systest/). It takes some tests, written similarly
to unit tests using the Python unittest library, and runs them
against an existing, running system.

The idea is to write tests for the system level functionality, as a
form of executable specification. Some tests are relevant to all forms
of the FBX:

* does the box respond to ping?
* can one log in via ssh?
* can one gain root access via sudo, after logging in via sudo?
* can the box do DNS lookups?

Other tests are relevant to specific services provided by the box.
For example, when used as a NAS, there might be tests like these:

* can one mount disk shares over SMB?
* can one modify data in shares mounted over SMB?

Here are tests I've written for testing a Debian base install for
the FBX:

    def test_only_ssh_port(self):
        self.assertEqual(self.find_open_ports(), ['22/tcp'])

    def test_ssh_login(self):
        user = self.settings['user']
        out = self.targetcmd(['id'])
        self.assertMatches(r'^uid=\d+\(%s\)' % user, out)

    def test_simple_dns_lookup(self):
        out = self.targetcmd(['host', 'www.debian.org'])
        self.assert_('www.debian.org' in out)
        
    def test_ping_localhost(self):
        self.targetcmd(['ping', '-c1', 'localhost'])
        
    def test_ping6_localhost(self):
        self.targetcmd(['ping6', '-c1', 'ip6-localhost'])
        
    def test_cat(self):
        out = self.targetcmd(['cat'], stdin='foo')
        self.assertEqual(out, 'foo')

    def test_sudo(self):
        out = self.targetcmd(['sudo', '-S', 'id'], 
                             stdin='%s\n' % self.settings['user-password'])
        self.assertMatches(r'^uid=0\(root\)', out)

The systest tool provides the systest.TestCase class, as an extension
of the unittest.TestCase class. The extensions are:

* hostcmd runs a process ("shell command") on the host running the tests
* targetcmd runs a process on the host being tested
* assertMatches tests whether a regular expression matches a string
* find_open_ports scans the target system (with nmap) and reports any open
  ports

Additionally there is a systest command line tool for running the
tests:

    systest --target=192.168.122.99 --user=tomjon \
        --user-ssh-private-key=tomjon.secret tests*.py

There's a need for some further tools, but this should get us started.
As an example of a missing tool, note that systest does not itself
create a virtual machine, or start one, so there needs to be a tool
that does that. I haven't yet written that, since it will be highly
dependent on the virtualization solution that is chosen, and that in
turn depends on the server where it gets set up, etc, so it's hard
to write it in a way that is fully generic.

Now, where do we go from here? Before I write any more code, I'd like
to get feedback from those willing to work on this part of FBX
development whether they're interested in this test driven approach,
and how it meshes with what they've done so far, and would like to do
next.

If we do want to do the test driven stuff, in the first iteration, 
we should generate an image that provides no services, but passes the 
tests for the base install. In further iterations, we can start adding 
services (such as Samba for NAS functionality, or Apache+ikiwiki for a 
wiki/blog instance, etc). However, before iterating further, I would
like to write the missing tools so that we can, with a single command,
perhaps run from cron or incron, generate and test an image.

Anyone?

-- 
Freedom-based blog/wiki/web hosting: http://www.branchable.com/