[Pkg-xen-devel] Bug#640500: Bug#640500: xen-hypervisor-4.0-amd64: xend invokes oomkiller and reboots machine when creating DomU's

[Wiebe Cazemier]

Op 05-09-11 18:54, Thomas Goirand schreef:
> On 09/05/2011 06:48 PM, Wiebe Cazemier wrote:
>> Package: xen-hypervisor-4.0-amd64
>> Version: 4.0.1-2
>> Severity: normal
>> Tags: upstream
>>
>> When creating Xen DomU's, at some point xend invokes the oom-killer and
>> the entire machine restarts:
>>
>> Sep  5 12:04:59 arbiter kernel: [259697.101212] __ratelimit: 136
>> callbacks suppressed
>> Sep  5 12:04:59 arbiter kernel: [259697.101218] xend invoked oom-killer:
>> gfp_mask=0x200da, order=0, oom_adj=0
>> Sep  5 12:04:59 arbiter kernel: [259697.101225] xend cpuset=/
>> mems_allowed=0
>> Sep  5 12:04:59 arbiter kernel: [259697.101230] Pid: 1841, comm: xend
>> Not tainted 2.6.32-5-xen-amd64 #1
>> Sep  5 12:06:23 arbiter kernel: imklog 4.6.4, log source = /proc/kmsg
>> started.
>> Sep  5 12:06:23 arbiter rsyslogd: [origin software="rsyslogd"
>> swVersion="4.6.4" x-pid="1409" x-info="http://www.rsyslog.com"]
>> (re)start
>> Sep  5 12:06:23 arbiter kernel: [    0.000000] Initializing cgroup
>> subsys cpuset
>>
>> xm info says the machine has 10231 MB of RAM. These are the VMs that
>> normally run:
>>
>> Name                     ID   Mem VCPUs      State Time(s)
>> Domain-0                 0  2177     1     r----- 127.1
>> one                      1  1536     2     r----- 192.4
>> two                      2   512     1     -b---- 2.7
>> three                    3  1024     1     -b---- 188.6
>> four                     4   512     1     -b---- 12.7
>> five                     5  2048     2     -b---- 151.7
>> six                      6  1024     2     r----- 64.9
>> seven                    7  1024     2     -b---- 65.9
>> eight                    8  1024     2     -b---- 100.7
> Funny domU names! :)

Well, I actually obfuscated the names. Personally, I don't mind revealing such 
things, but I don't know what 'the company' would say...

>
>> That means 8704 MB of memory is used.
>>
>> Over the last couple of days, I've experienced that when creating two
>> VM's with 512 MB RAM, the oom-killer is invoked. This should not happen
>> because Dom0 can shrink and because the machine should not restart when
>> it is out of memory.
>>
>> This is a production Xen host, which is becomming a liability, because
>> it's so unpredictable.
>>
>> Current Dom0 kernel: Linux arbiter 2.6.32-5-xen-amd64
> Hi,
>
> I wouldn't recommend to just let your dom0 shrink. Set it directly to a
> much lower memory footprint, something like 512 MB of RAM, so that the
> Linux kernel sees directly that it doesn't have so much memory to deal
> with, and it will not allocate too big buffers and so on. Here's how to
> do this with the Squeeze grub:
>
>                  echo "
> # Start dom0 with less RAM
> GRUB_CMDLINE_XEN_DEFAULT=\"dom0_mem=512M\"
> ">>/etc/default/grub
>
> Then either run update-grub2 or dpkg-reconfigure grub-pc (and of course
> reboot...). If you do that, I'm sure your troubles will go away. You
> shouldn't be running big software in the dom0 anyway. I always done that
> because of habits from running older Xen (since 2.0.7), and because I've
> seen issues with the dom0 ballooning out. To me, ballooning the memory
> of dom0 is more a desktop feature than for a server.
>
> I believe this bug should be closed, because that's not really an issue
> with Xen (but more with the way the Linux dom0 kernel works). I'll let
> Bastian decide though.
>
> Thomas Goirand (zigo)

I agree that the dom0 shouldn't do heavy work and it doesn't. I will try your 
suggestion; I had already looked for an option to set the max dom0 memory, but 
couldn't find it (only a minimum one). I didn't think about kernel params.

I wonder though, if creating VM's when memory runs out even with a small dom0 
will the still trigger the OOM killer. A nice warning about being out of memory 
would be nicer...