[Vmdebootstrap-devel] I'm adopting vmdebootstrap

Sun Sep 4 19:51:07 UTC 2016

On Sun, Sep 4, 2016 at 9:28 AM Lars Wirzenius <liw at liw.fi> wrote:

> Hi, all.
>
> I am the original author vmdebootstrap. It started as a quick hack to
> let me create disk images for virtual machines without doing a manual
> install with debian-installer. I didn't want to do it manually,
> because I needed six images that only varied with Debian release and
> CPU architecture: I was setting up a CI system test Obnam on Debian
> oldstable, stable, and unstable, on i386 and amd64 targets.
>
> Much to my surprise, a few other people seemed to like vmdebootstrap.
> This led to some feature requests and other maintenance burden I
> wasn't ready for, and as a result I turned over the project to Neil
> Williams. Neil took good care of it for a couple of years, adding things
> like grub support, and foreign architecture support, and other things I
> haven't yet found. Many thanks to Neil.
>
> Neil has now asked me to take vmdebootstrap back, and I have agreed to
> do so, partly because I need it at work and at home.
>
> I would like to see vmdebootstrap grow into a tool that adequately
> handles all of Debian's image creation needs. I am currently thinking
> particularly of the following use cases:
>
> * Create VM images for libvirt + kvm (which is what I use at home).
>
> * Create VM images for all the cloud services: OpenStack, AWS, etc. I
>   don't know what they all are (I've only used OpenStack and AWS), and
>   I'm not on expert on any of them (but we use OpenStack at work). As
>   such, I'll certainly need help with them.
>
> * Create disk images for actual hardware. I have installed two of my
>   servers at home like this: I generate a disk image with
>   vmdebootstrap, write that to a USB stick with dd, and then boot off
>   the stick. The initial image is very barebones, and I use Ansible to
>   do any additional configuration (and specifically I don't use the
>   vmdebootstrap customization script).
>
>   I imagine this would also be useful to many who want to hack on ARM
>   devboards, and such, but again, that isn't something I'm an expert
>   on.
>
> These are the overall classes, but each specific use may have its
> unique requirements. For example, there's been some recent discussion
> of how customizable partitioning should vmdebootstrap support. My
> opinion is that it should support anything users need, even if the
> tool becomes more complex. As an example, I have a personal need to be
> able to create a bootable image that has /boot in cleartext and
> everything else encrypted, with LVM.
>
> There may be other use cases that I haven't thought of yet. Please
> tell me. There may be features vmdebootstrap currently lacks that you
> really, really want. Tell me.
>
> I wish to maintain vmdebootstrap so that it serves its users and that
> requires it to be quite flexible. The challenge for me is to come up
> with a way to be flexible while keeping complexity at a manageable
> level.
>
> I also want vmdebootstrap to be pleasant to use. What hinders you from
> enjoying vmdebootstrap? Tell me.
>
> The other challenge is that I think the code quality is not great. It
> started as a quick hack, without automated tests, and that's how it
> was when I gave the project to Neil. Some tests have since been added,
> but not enough, I feel. This is not Neil's fault in any way: the mess
> he inherited was something I feel ashamed of. I intend to improve the
> code quality massively. I want a thorough unit test suite, and I want
> to extend the integration tests so that the images they create also
> get tested.
>
> This is all a lot of work. I would like help from others. A lot of
> things are mysterious to me. Would you like to help? What are concrete
> things you yourself could do or would like to do? Tell me.
>
> I am aware that there are other tools than vmdebootstrap. The most
> common other choice seems to be bootstrap-vz. I haven't used it; it's
> possible that it could be enhanced to replace vmdebootstrap and that
> my project should thus be dropped. From a cursory skim of the
> documentation, it seems to not target bare hardware at all, and that's
> one of the things I need myself. If you have opinions on this, tell
> me.
>
> My own next actions with regards to vmdebootstrap is to review the
> currently open bugs, the code base, and find enough small changes to
> make to warrant an upload to change Maintainer/Uploaders.
>
> --
> I want to build worthwhile things that might last. --joeyh
>

Hello Lars and all

I am the original author of bootstrap-vz.
Just this morning (around half an hour before you wrote your mail)
I decided that I would no longer be able to maintain the project,
since I simply don't have enough free time to allot to it
(or rather, I have other projects I'd rather spend my time on).
Additionally we are switching to a dockerized setup at work, which means
the amount of real-world problems I encounter,
whose solutions could improve bootstrap-vz, is reduced.

The co-maintainer of bootstrap-vz is Tiago Ileve, whom I wrote
this morning to offer the project.
Regretfully he declined, because of no free time in general.

Would you be interested in taking over bootstrap-vz instead?
I was planning on making an open offer to the mailing list,
but I think you'd be the perfect fit.

I'd like to think that the code quality, maintainability and
documentation of bootstrap-vz is top notch, which would ease the transition.
You mentioned hardware support - bootstrap-vz was originally
intended for virtualized environments, but I have myself been
close to implementing support for e.g. RaspberryPi mostly because
I believed it would be pretty easy (though I didn't get around to doing it).
I think general hardware support would be fairly trivial as well, there
already is support
for both grub and extlinux bootloaders. Disks can be partitioned with
msdos, gpt or not at all.

> As an example, I have a personal need to be able to create a bootable
> image that has /boot in cleartext and everything else encrypted, with LVM.

I never got around to implementing LVM support, but I am fairly certain
that the
partition abstraction code
<https://github.com/andsens/bootstrap-vz/tree/b9d63b701df3e7757a6f3c3336ce2cc71af6b4b2/bootstrapvz/base/fs>
could be extended without too much trouble.

> Create VM images for libvirt + kvm (which is what I use at home).

bootstrap-vz has kvm support
<https://github.com/andsens/bootstrap-vz/tree/b9d63b701df3e7757a6f3c3336ce2cc71af6b4b2/bootstrapvz/providers/kvm>
.

> The initial image is very barebones, and I use Ansible to do any
additional configuration

bootstrap-vz has an ansible plugin
<https://github.com/andsens/bootstrap-vz/tree/b9d63b701df3e7757a6f3c3336ce2cc71af6b4b2/bootstrapvz/plugins/ansible>
for running playbooks (and for puppet
<https://github.com/andsens/bootstrap-vz/tree/b9d63b701df3e7757a6f3c3336ce2cc71af6b4b2/bootstrapvz/plugins/puppet>
)

Regarding maintainability: bootstrap-vz is around 7k lines of code,
most of that code is part of the common architecture and the common tasks.
Provider specific code averages at about 200 lines, so there is loads of
reuse going on.

> I want a thorough unit test suite

Here you go
<https://github.com/andsens/bootstrap-vz/tree/b9d63b701df3e7757a6f3c3336ce2cc71af6b4b2/tests>

> and I want to extend the integration tests so that the images they create
also get tested.

Done
<https://github.com/andsens/bootstrap-vz/tree/b9d63b701df3e7757a6f3c3336ce2cc71af6b4b2/tests/system>...
well,
the testing harness is done and you can bootstrap machines
from scratch and start them automatically.
I never got around to SSHing into the machines, and running test commands,
which is like... running a marathon and then sitting down and
drinking a beer right in front of the finish line.
I have distilled the bootstrapping, image copying, booting and teardown
into to this line:
    *with boot_manifest(manifest_data, boot_vars) as instance:*
For EC2 this includes a temporary VPC and subnet creation,
for virtualbox it means creation of a virtual machine through VBoxManage
and on docker
the image is automatically imported to the repo.

I hope I am not trying to sell bootstrap-vz too hard, I just believe that
the
reproducability and ease of configuration of images offers something
that other bootstrappers do not.

Also: Please know that I will not just abruptly stop my involvment in
bootstrap-vz,
I will most certainly be available for questions and would love to help the
transition
process as much as possible.

So, what do you think? :-)
-- 
Anders Ingemann
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.alioth.debian.org/pipermail/vmdebootstrap-devel/attachments/20160904/bb3fb6b6/attachment.html>