[Pkg-xen-devel] Bug#640500: Bug#640500: xen-hypervisor-4.0-amd64: xend invokes oomkiller and reboots machine when creating DomU's

[Thomas Goirand]

On 09/05/2011 06:48 PM, Wiebe Cazemier wrote:
> Package: xen-hypervisor-4.0-amd64
> Version: 4.0.1-2
> Severity: normal
> Tags: upstream
> 
> When creating Xen DomU's, at some point xend invokes the oom-killer and
> the entire machine restarts:
> 
> Sep  5 12:04:59 arbiter kernel: [259697.101212] __ratelimit: 136
> callbacks suppressed
> Sep  5 12:04:59 arbiter kernel: [259697.101218] xend invoked oom-killer:
> gfp_mask=0x200da, order=0, oom_adj=0
> Sep  5 12:04:59 arbiter kernel: [259697.101225] xend cpuset=/
> mems_allowed=0
> Sep  5 12:04:59 arbiter kernel: [259697.101230] Pid: 1841, comm: xend
> Not tainted 2.6.32-5-xen-amd64 #1
> Sep  5 12:06:23 arbiter kernel: imklog 4.6.4, log source = /proc/kmsg
> started.
> Sep  5 12:06:23 arbiter rsyslogd: [origin software="rsyslogd"
> swVersion="4.6.4" x-pid="1409" x-info="http://www.rsyslog.com"]
> (re)start
> Sep  5 12:06:23 arbiter kernel: [    0.000000] Initializing cgroup
> subsys cpuset
> 
> xm info says the machine has 10231 MB of RAM. These are the VMs that
> normally run:
> 
> Name                     ID   Mem VCPUs      State Time(s)
> Domain-0                 0  2177     1     r----- 127.1
> one                      1  1536     2     r----- 192.4
> two                      2   512     1     -b---- 2.7
> three                    3  1024     1     -b---- 188.6
> four                     4   512     1     -b---- 12.7
> five                     5  2048     2     -b---- 151.7
> six                      6  1024     2     r----- 64.9
> seven                    7  1024     2     -b---- 65.9
> eight                    8  1024     2     -b---- 100.7

Funny domU names! :)

> That means 8704 MB of memory is used. 
> 
> Over the last couple of days, I've experienced that when creating two
> VM's with 512 MB RAM, the oom-killer is invoked. This should not happen
> because Dom0 can shrink and because the machine should not restart when
> it is out of memory.
> 
> This is a production Xen host, which is becomming a liability, because
> it's so unpredictable.
> 
> Current Dom0 kernel: Linux arbiter 2.6.32-5-xen-amd64 

Hi,

I wouldn't recommend to just let your dom0 shrink. Set it directly to a
much lower memory footprint, something like 512 MB of RAM, so that the
Linux kernel sees directly that it doesn't have so much memory to deal
with, and it will not allocate too big buffers and so on. Here's how to
do this with the Squeeze grub:

                echo "
# Start dom0 with less RAM
GRUB_CMDLINE_XEN_DEFAULT=\"dom0_mem=512M\"
" >>/etc/default/grub

Then either run update-grub2 or dpkg-reconfigure grub-pc (and of course
reboot...). If you do that, I'm sure your troubles will go away. You
shouldn't be running big software in the dom0 anyway. I always done that
because of habits from running older Xen (since 2.0.7), and because I've
seen issues with the dom0 ballooning out. To me, ballooning the memory
of dom0 is more a desktop feature than for a server.

I believe this bug should be closed, because that's not really an issue
with Xen (but more with the way the Linux dom0 kernel works). I'll let
Bastian decide though.

Thomas Goirand (zigo)