Bug#594472: grub-pc: scary messages and very long boot time after upgrade: proposal for solution

[Alexander Kurtz]

Hi everybody,

I spent the last hour re-reading the entire bug report, testing the new
mdadm version and testing a way to restore the system using a live CD.
Here's what I found:

 * I am really sure that this is not a GRUB bug. You all get to the 
   initramfs. That's where problems start. GRUB is innocent here.

 * I am quite sure now that you all have been bitten by mdadm bug 
   #583917[1] and friends. The symptoms are quite unique (loads of 
   "/sys/devices/virtual/block/md0" messages and high/endless disk 
   activity). Look here[2] for a good explanation of what happens.

   This bug was fixed in mdadm 3.1.4-1+8efb9d1 which just migrated to 
   testing/squeeze[3]. I verified that with this version my system 
   boots fine, even without the workaround[4].

So the important part is how to get your systems back online.
Here's what you can do, if your system is still not booting (you might
want to print this out):

1. Get the current Debian Live CD[5]
	32 bit: http://cdimage.debian.org/cdimage/squeeze_live_alpha2/i386/iso-hybrid/debian-live-60alpha2-i386-standard.iso
	64 bit: http://cdimage.debian.org/cdimage/squeeze_live_alpha2/amd64/iso-hybrid/debian-live-60alpha2-amd64-standard.iso
   burn it to a CD/DVD or write it to a USB drive using dd (if your 
   machine can boot from USB) and boot the live system (note that 
   you'll get a US keyboard layout first, this[6] might be helpful)

2. Get root and install the required tools
	sudo su
	apt-get update
	apt-get install console-data cryptsetup less lvm2 mdadm

3. Configure the keyboard layout (if necessary)
	dpkg-reconfigure console-data

4. Start your RAID arrays if not yet done. See "man mdadm".

5. Start your LVM VGs (if any). See "man lvm".

6. Load the dm_crypt kernel module using
	modprobe dm_crypt
   and open your crypto devices (see "man cryptsetup"), e.g.
	cryptsetup luksOpen /dev/mapper/lv0 root

By now all your systems block devices should be available and useable.

7. Mount your system to /mnt, e.g.
	mount /dev/mapper/root /mnt
	mount /dev/md0         /mnt/boot
	...

8. Bind-mount the virtual file systems:
	mount -o bind /dev     /mnt/dev
	mount -o bind /dev/pts /mnt/dev/pts
	mount -o bind /proc    /mnt/proc
	mount -o bind /sys     /mnt/sys

9. Chroot into your main system
	chroot /mnt

10. Update the required tools
	apt-get update
	apt-get install cryptsetup lvm2 mdadm
    Ensure that you have mdadm v3.1.4:
	mdadm --version

11. Check your mdadm configuration
	/usr/share/mdadm/mkconf
    If the output seems sane, write it to disk
	/usr/share/mdadm/mkconf > /etc/mdadm/mdadm.conf

12. Carefully check whether the contents of /etc/fstab 
    and /etc/crypttab are correct and up to date.

13. On my system mdadm didn't create the necessary links 
    under /dev/md/. Check if they are *all* there
	ls -l /dev/md/
    If some links are missing, create them manually, e.g.
	ln -s /dev/md0 /dev/md/0
    See the mdadm.conf you just generated for the exact names.

14. Now you should be able to recreate the initramfs and update grub
	update-initramfs -u
	update-grub
    This should work now without any errors.

15. Exit the chroot and reboot
	exit
	init 6

16. Check if your system boots now. If so, then run
	update-initramfs -u
	update-grub
    again from the main system and reboot one last time.

I hope that you'll be able to repair your systems with that. Please
report any success or failure (Which part of the above fails? What error
messages are shown?)!

Hope this helps!

Alexander Kurtz

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=583917
[2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=585015#45
[3] http://release.debian.org/migration/testing.pl?package=mdadm
[4] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=594418#25
[5] http://live.debian.net/ 
[6] http://upload.wikimedia.org/wikipedia/commons/5/51/KB_United_States-NoAltGr.svg

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.alioth.debian.org/pipermail/pkg-grub-devel/attachments/20100917/e4184b69/attachment.pgp>