[Buildd-tools-devel] Bug#477942: git storage backend for chroots
martin f krafft
madduck at debian.org
Fri Apr 25 20:23:01 UTC 2008
Package: schroot
Version: 1.1.6-1
Severity: wishlist
Please find attached the files 05git and a diff to 00check, which
together should enable Git to be used as chroot backend. The
rationale is that
- Git is itself a filesystem, really.
- Because it snapshots the tree, it gives the same functionality as
copy-on-write.
- If you make a change in a temporary session which you'd actually
like to keep, Git allows you to create the appropriate patch
(unless the change was binary).
- Git's in-place branches could be really cool for testing stuff in
temporary chroots.
- Git's branches allow multiple flavours of a chroot to be served
from the same repository
- Git's store is highly efficient. While operations take about the
same time as with tgz files, the packed repository takes up less
space than the tgz. For instance, a buildd-flavour chroot of sid
on i386 creates a 109M tgz file, but the .git object store is 104M
in size. a tbz2 file is smaller, but operations take significantly
longer than with Git (bzip2 is slow).
Obviously, when a chroot is updated a couple of times, the
history is kept and the size increases. While the history might be
a feature, I have yet to find a way to properly prune it.
Nevertheless, operation speeds are not affected.
- ...
When using type=git, the configuration must also specify
repository=/path/to/repo.git or repository=/path/to/repo/.git,
depending on whether the source is a bare repo or not.
The repository is set up so that the object store of the source is
used and thus the copying is minimal. Since existing objects cannot
be changed with Git, only new ones created, this is effectively
copy-on-write and safe. Even if source and destination are on the
same filesystem, Git does not use hardlinks to prevent involuntary
changes to the source.
git-core is currently not installed into the checkout. I am
currently ambivalent of whether this should happen as part of the
setup, or whether Git-enabled chroots just expect the admin to
install git-core into the source chroot. It's *not* needed in the
chroot for normal operation, but might be needed for -source
handling (see below).
Simple use of the repo already works, but -source handling does not
work at all yet. Right now, -source chroots set CHROOT_FILE_REPACK
(I'm abusing the file type, see below) and that's not handled at all
in 05git, simply because it's not the way things should happen.
Instead, 05git should check whether the source repository is bare or
not.
- for bare repositories, it should clone the repository just as
before, but on close, it should push all commits (from outside the
chroot, or else the source repository is unreachable). It remains
to be seen whether it makes sense to prevent schroot from leaving
the chroot if the tree has uncommitted changes (see separate bug).
- for non-bare repositories, it can chroot straight into the
worktree of the repository. In this case, uncommitted changes in
the tree should prevent schroot from leaving the chroot (see
separate bug).
19devices, which is also attached, creates all devices, since those
cannot be stored in Git. This takes quite a while and could be
optimised by using a tarball with the devices. 19devices includes
a similarly ugly hack to work around the aforementioned type
limitation (it uses type=file and the .git/config file).
There's a problem related to empy directories, which Git does not
checkout. Until mount mkdir's mountpoints before trying to mount
onto them, 05git creates a number of standard directories, but not
all. I may have to figure out a way to recreate all empty
directories found in the source, which might be possible with
a simple find call.
Right now, schroot also checks for the chroot type in the binary
(see other bug) and thus I am using the following at the top of
05file:
# hackery until I can add my own types easily
if [ "${CHROOT_FILE#*/.git/}" = "config" ]; then
CHROOT_TYPE=git CHROOT_REPOSITORY="${CHROOT_FILE%/.git/*}" \
exec ${0%/*}/05git "$@"
fi
this allows me to set file=../path/to/repo/.git/config and have the
git method invoked.
Note that 05git and 19devices are GPLv2 explicitly. I don't approve
of the (philosophy behind) GPLv3 and thus prefer to stay with v2. If
that's a problem, let me know and we can surely find a way.
-- System Information:
Debian Release: lenny/sid
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1, 'experimental')
Architecture: i386 (i686)
Kernel: Linux 2.6.24-1+scoflowctrl.1-686 (SMP w/1 CPU core)
Locale: LANG=en_GB, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages schroot depends on:
ii libboost-program-options1.34. 1.34.1-11 program options library for C++
ii libboost-regex1.34.1 1.34.1-11 regular expression library for C++
ii libc6 2.7-10 GNU C Library: Shared libraries
ii libgcc1 1:4.3.0-3 GCC support library
ii liblockdev1 1.0.3-1.2 Run-time shared library for lockin
ii libpam0g 0.99.7.1-6 Pluggable Authentication Modules l
ii libstdc++6 4.3.0-3 The GNU Standard C++ Library v3
ii libuuid1 1.40.8-2 universally unique id library
ii schroot-common 1.1.6-1 common files for schroot
schroot recommends no packages.
-- no debconf information
--
.''`. martin f. krafft <madduck at debian.org>
: :' : proud Debian developer, author, administrator, and user
`. `'` http://people.debian.org/~madduck - http://debiansystem.info
`- Debian - when you have better things to do than fixing systems
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 00check.diff
Type: text/x-diff
Size: 948 bytes
Desc: not available
Url : http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20080426/7e1b8b83/attachment-0001.diff
-------------- next part --------------
#!/bin/sh
# Copyright ? 2005-2007 Roger Leigh <rleigh at debian.org>
# Copyright ? 2008 martin f. krafft <madduck at debian.org>
#
# This file is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
# published by the Free Software Foundation.
#
# This file is based on setup.d/05file / schroot 1.0.6-1.
#
# This file is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see
# <http://www.gnu.org/licenses/>.
#
#####################################################################
set -e
if [ -f "$CHROOT_SCRIPT_CONFIG" ]; then
. "$CHROOT_SCRIPT_CONFIG"
fi
if [ "$CHROOT_TYPE" = "git" ]; then
if [ $1 = "setup-start" ]; then
if [ -d "$CHROOT_MOUNT_LOCATION" ]; then
if ! ls "$CHROOT_MOUNT_LOCATION"/* 2>/dev/null; then
rmdir "$CHROOT_MOUNT_LOCATION"
else
echo "$CHROOT_MOUNT_LOCATION exists and is non-empty"
exit 1
fi
fi
QUIET=
if [ "$AUTH_VERBOSITY" != "verbose" ]; then
QUIET=--quiet
fi
git clone $QUIET --shared --no-hardlinks \
"$CHROOT_REPOSITORY" "$CHROOT_MOUNT_LOCATION"
for d in boot home media mnt opt proc root srv sys tmp dev/pts dev/shm; do
# TODO: hackery until schroot mkdir's again before mounting, since
# Git does not checkout empty directories.
mkdir --parent "$CHROOT_MOUNT_LOCATION"/$d
done
elif [ "$1" = "setup-stop" ]; then
# almost verbatim from 05file
"$LIBEXEC_DIR/schroot-listmounts" -m "$CHROOT_MOUNT_LOCATION" |
while read mountloc; do
if [ "$AUTH_VERBOSITY" = "verbose" ]; then
echo "Not purging $CHROOT_MOUNT_LOCATION; filesystems are mounted:"
"$LIBEXEC_DIR/schroot-listmounts" -m "$CHROOT_MOUNT_LOCATION"
fi
exit 1
done || exit 1
if [ "$CHROOT_SESSION_PURGE" = "true" ]; then
if [ "$AUTH_VERBOSITY" = "verbose" ]; then
echo "Purging $CHROOT_MOUNT_LOCATION"
fi
rm -rf "$CHROOT_MOUNT_LOCATION"
fi
fi
fi
-------------- next part --------------
#!/bin/sh
# Copyright ? 2008 martin f. krafft <madduck at debian.org>
#
# This file is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
# published by the Free Software Foundation.
#
# This file is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see
# <http://www.gnu.org/licenses/>.
#
#####################################################################
set -e
if [ -f "$CHROOT_SCRIPT_CONFIG" ]; then
. "$CHROOT_SCRIPT_CONFIG"
fi
if [ "$CHROOT_TYPE" = file ] && [ "${CHROOT_FILE#*/.git/}" = "config" ]; then
if [ "$1" = setup-start ]; then
[ "$AUTH_VERBOSITY" = "verbose" ] &&
echo "Creating devices..."
chroot "$CHROOT_MOUNT_LOCATION" sh -c \
'cd /dev && /sbin/MAKEDEV generic'
fi
fi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
Url : http://lists.alioth.debian.org/pipermail/buildd-tools-devel/attachments/20080426/7e1b8b83/attachment-0001.pgp
More information about the Buildd-tools-devel
mailing list