[buildd-tools-devel] Bug#775423: schroot: Better tools for tidying old chroot sessions needed.

[Wookey]

Package: schroot
Version: 1.6.10-1+b1
Severity: wishlist

I find that I often end up with a pile of old schroot sessions, some
of which could be 'recovered' and used, some of which are just empty,
and maybe some half-deleted/otherwise buggered.

As these all end up taking up space in /var/lib (which is often in /
partition by default these days), it's not hard to get in s state
where / is more than 90% full and things break. 

That's what I have right now after upgrading to jessie on a machine. I
have 88G of 92G in / used: '100%', and everything breaks and I find
this pile of schroot sessions:

session:jessie-amd64-sbuild-f5feb6e0-d4b2-41e2-8294-c8f46cadef34
session:jessie-amd64-sbuild-f68448f6-1ff2-4bad-8a75-1ce9d9d17ba7
session:saucy-amd64-sbuild-920165d2-b58d-4905-b42d-e4963a80439a
session:squeeze-amd64-sbuild-44f6c0ca-66e0-4dc5-9b4d-b196367ebc3a
session:unstable-amd64-sbuild-00d235ba-330a-4944-ab06-a98cac91d1a7
session:unstable-amd64-sbuild-8be61ddd-a5a9-41e6-860a-0cf3438b890e
session:unstable-amd64-sbuild-94c7df83-4162-49a5-bd22-fda4c719b716
session:unstable-amd64-sbuild-954c1dc4-901e-42d2-a7fa-f83a1aaae9bd
session:unstable-amd64-sbuild-d3cf51ab-c41f-4ad4-a40c-5f192a5d520d
session:unstable-amd64-sbuild-fa9bcdf7-81d8-4905-8488-5d20c25acd4f
session:wheezy-amd64-a640551a-0217-451e-b331-e3969f98dc39
session:wheezy-amd64-sbuild-0ef7a391-3131-40ea-aa52-359db22cac39
session:wheezy-amd64-sbuild-5edd6847-572f-4a68-918d-d4dbee2703c9
session:wheezy-amd64-sbuild-70455b7e-33f4-4e37-af8d-341618e31ba9
session:wheezy-amd64-sbuild-a755722b-ffea-46d3-975c-a6d151a6f13c

And that's after I tidied a few.

I find these schroots painful to manage with the existing
tools. Perhaps there are better way I don't know about, or perhaps we
could use better tidying-tools?

So quite often one can tidy up _everything_ with 
schroot -e --all-sessions
 
However that's annoying if you have a few chroots in use and just want
to tidy up the 10 or so other old ones.

Now you have to look through which long UIDs are mounted, then match them up with session: identifiers then run 
schroot -e -c session:<long delete, long paste>
and sometimes (like now), you get told that something is still in use:

e.g trying to tidy now I get:
schroot -e -c session:jessie-amd64-sbuild-2f0f8275-26ea-4445-b980-f6b726c9c380
E: 15binfmt: update-binfmts: unable to open /var/lib/schroot/mount/jessie-amd64-sbuild-2f0f8275-26ea-4445-b980-f6b726c9c380/bin/sh: No such file or directory
E: 10mount: rmdir: failed to remove '/var/lib/schroot/mount/jessie-amd64-sbuild-2f0f8275-26ea-4445-b980-f6b726c9c380': Device or resource busy
E: jessie-amd64-sbuild-2f0f8275-26ea-4445-b980-f6b726c9c380: Chroot setup failed: stage=setup-stop

And it's not clear what's causing that:
lsof  /var/lib/schroot/mount/jessie-amd64-sbuild-2f0f8275-26ea-4445-b980-f6b726c9c380
mount | grep  jessie-amd64-sbuild-2f0f8275-26ea-4445-b980-f6b726c9c380

Also, if you start cleaning up manually you can easily end up deleting
your /dev or /home or otherwise making a nasty mess. The massive list
of mounts (typically 5 per session) with huge long UIDs is not easy to
deal with manually.

(and by the way why is update-binfmts trying to run things inside the
chroot before tidying it up in the above case?)

So, the point here is that I could certainly use better tools for
'schroot tidying'. Do others agree? Does something already exist?
Any opinions on how that should be best integrated into schroot/sbuild?

I'm imagining something that would look at the session list and report
corresponding empty mounts, or could be told to clean up all 'empty'
sessions or all sessions with no /bin dir (or other probably fatal
flaws), or remove all sessions with no mounts set up, or report which
processes were still using a chroot.

I don't know how hard it is to detect if an schroot session is 'in
use'? The automatic session-recovery on boot makes this harder. Can we
get last-access info which might provide clues?

Presumably there is already some code in schroot for some of this, so
it may not need much on top to have a couple of levels of 'please tidy up'.

Feedback welcome on how best to do this. I'm sufficiently irritated
after a couple of years of this sort of faff to write some code
sometime if that's what's needed.