[pkg-lxc-devel] Bug#911806: Please provide a way to opt out of AppArmor confinement when running tests

[intrigeri]

Hi,

Michael Biebl:
> E.g. if I try to run autopkgtest for systemd on sid in a AA-confined LXC
> container, it explodes left and right, whereas it passes with
> AA-confinement turned off.

Thanks for reporting this!

Disclaimer: I don't use LXC.

I've spent some time reading the various related systemd upstream
issues and commits and my understanding is that we have two problems:

1. Recent systemd sets up namespacing with calls to mount that are not
   allowed under the default AppArmor policy shipped in the lxc
   package (or rather: it has always done so but previously, failures
   to set up said namespacing were non-fatal). If I got it right, this
   means that lots of services will be broken when run in LXC once
   systemd v240 is released and uploaded to Debian, i.e. the problem
   is broader than our CI: most typical LXC use cases will be affected
   as well.

   It's an instance of a classical problem when nesting security
   systems:

    - Either the outside one (in this case, AppArmor) allows the
      inside one (in this case, systemd) whatever it takes so the
      latter can do its job. Ideally this cooperation gives us the
      best of both worlds. But in practice, this sometimes requires
      the outside security system to give permissions that are so
      relaxed that it becomes essentially useless (which might be the
      case here, I don't know); when this happens, we basically can't
      rely on the outside security system anymore, and instead we rely
      mostly on the inside one. When the inside security system is
      controlled by untrusted parties, such as an untrusted
      container's root user, that's a risky approach, to say the
      least. In other words, there's a risk to break isolation between
      containers in order to improve isolation between services inside
      a given container.

    - Or the outside security system is restrictive to the point that
      it breaks the inside one, and then it's the opposite: weaker
      service isolation inside the container but stronger isolation
      between containers.

   Ideally, picking the best strategy and implementing it would be
   a matter of coordination between LXC and systemd (ideally upstream,
   but quite possibly distro maintainers will need to be involved
   here). I'll raise this issue to the Ubuntu LXC and AppArmor folks.

   Worst case, if this does not get resolved in time for Buster, we
   may have to disable AppArmor confinement for LXC containers in
   Buster; that would be sad but it may be the best we can do for
   Buster, and it would not be a regression vs. Stretch for most users
   (the Stretch kernel lacks the AppArmor features that LXC needs for
   meaningful confinement so even if one opts in for AppArmor on
   Stretch, they get pretty weak isolation). And then whoever
   particularly cares about hardening LXC on Debian has a couple more
   years to sort this out for Bullseye :)

2. Regardless of #1, some software is broken when run inside
   a container, be it due to the way it the container technology uses
   Linux namespaces or due to additional LSM-based sandboxing. In this
   case the best approach is probably to run these tests inside a KVM
   runner instead. During the autopkgtest at last DebConf, with
   another similar issue in mind, I've asked about KVM runners for
   ci.debian.net and I was told this was WIP and would have it
   soonish, which will provide a tool to solve this class of problems:
   any affected autopkgtest can declare the isolation-machine restriction.

   A special sub-case is when the software would run just fine in
   a container but its test suite exercises it in a way that requires
   different privileges than normal usage of said software inside
   a container, e.g. by setting up stuff in super custom way, which
   might break under the default sandboxing rules. In this case,
   either I would argue that the test suite should instead exercise
   something closer to what actual users will run under that
   confinement: that's a guiding design principle of autopkgtests.
   Whenever fixing such a test suite is not possible for some reason,
   then running the affected tests in KVM could be an
   acceptable workaround.

> It is my understanding, that Ubuntu itself turns off AA-confinement in
> their LXC based CI or at least ship a AA policy which is much less
> strict [1].

If I got it right, that's correct for armhf but for every other
architecture, they run tests in standard Ubuntu KVM cloud instances
and not in LXC. Advantage: apart of armhf, no need to deal with the
peculiarities of LXC+AppArmor, and AppArmor-wise they're testing
exactly what users will experience outside of containers. Drawback: to
exercise software that would be broken under the default LXC+AppArmor
sandboxing, they need to relax the AppArmor sandboxing for _all_ tests
on armhf, meaning they don't test software in exactly the same sandbox
that will be used by actual users of Ubuntu LXC containers.

On the Debian side, currently we run tests only in LXC containers,
which is probably not the most common use case of Debian. In most
cases it does not matter because the software under test behaves in
exactly the same way regardless of whether it runs in a container or
not. In some cases it does matter.

> Looking ahead at the buster release, we either need to turn off AA
> confinement in our Debian CI as well, ship a custom AA policy or provide
> a way for individual packages to opt out of AA confinement. In the
> latter case, this will probably need some changes to autopkgtest as
> well.

I hope the problem will be solved in time for Buster with
a combination of a) fixing the cooperation between LXC + AppArmor and
systemd for the general LXC use case and incidentally for most
autopkgtests too; b) having KVM runners available, for cases that
won't run fine in LXC (possibly due to AppArmor).

If that does not happen, then my guiding principles would be "in
doubt, do no harm" and the idea that autopkgtests should be run in an
environment that's as close as possible as the production one, which
you seem to agree with
(https://github.com/systemd/systemd/issues/10166#issuecomment-432743680).
So turning off AppArmor entirely in our CI *only* would feel very
wrong. The remaining options are:

 - Shipping a custom AppArmor policy for adt runners. This would feel
   wrong too, for the same reason, if that policy is different from
   the production one is important ways, e.g. if it lets systemd do
   stuff that a production container would block.

 - Disabling AppArmor support for LXC by default (not only for our
   CI). As I've discussed above, this would be sad but acceptable.
   At least we would run our CI in an environment that matches what
   production usage of Debian in a container.

 - Introducing a new autopkgtest restriction like
   isolation-container-without-lsm is probably doable, but I'm wary
   that this is a slippery slope towards getting requests for having
   all kinds of less restricted containers, which may lead to
   introducing more low-level autopkgtest restrictions, which does not
   fit into the abstraction level at which they seem to be designed.

(As a side note, when running autopkgtests in a KVM runner,
AppArmor-wise one is testing in the exact environment that people
running Debian on bare metal or in a VM use, so we're all good.
But when running autopkgtests in containers, AppArmor-wise things are
very different; ideally we want policy stacking, i.e. there's the
policy applied to the container as a whole by the host system to
provide container isolation and on top of this, if the container is
sufficiently privileged, it may load its own AppArmor policy that will
be stacked on top of the host system's one; ideally we want
a container that has done the equivalent of "systemctl start
apparmor", to be closer to what actual users will do. AFAIK this
policy stacking setup is used mostly for LXC/LXD-based Ubuntu systems
and I'm not sure what's the status of AppArmor policy stacking support
in the kernel + userspace we will ship in Buster. I gave it a quick
try (using the lxc-ubuntu template) and was not able to load AppArmor
policy inside a LXC container running on sid.)

Cheers,
-- 
intrigeri