[Pkg-xen-devel] kernel crashes after soft lockups in xen domU
Jonas Meurer
jonas at freesources.org
Wed Nov 5 16:56:15 UTC 2014
And some more information ...
Am 2014-10-13 12:04, schrieb Jonas Meurer:
> Am 2014-08-19 12:26, schrieb Jonas Meurer:
>> I encounter kernel crashes on an up-to-date Debian/Wheezy Xen domU
>> with the
>> stock kernel. The dom0 runs the same linux kernel and
>> xen/4.1.4-3+deb7u1.
>
> the bug is still reproducible with the latest kernel and Xen packages
> from
> Debian Wheezy. Unfortunately it seems like a corner case, somehow
> related
> to the hardware setup. I'm unable to reproduce the crash on another Xen
> system with same kernel an Xen versions but different CPU and
> motherboard.
> The VM runs in production mode on the second Xen system since several
> weeks
> without one single crash.
>
> I've got a test system now where I'm able to reproduce the bug by
> putting the
> VM (a webserver) under heavy load with the help of siege and pylot. The
> VM
> crashes every time I put the webserver under heavy load, everytime with
> the
> same backtrace.
I replaced the full system (all hardware components except the
harddisks)
with a new one in the meantime - and still the bug is repoducible. I'll
try to describe the setup:
Two Xen virtualization servers (xen1 and xen2), mirroring one block
device with DRBD using a primary/secondary setup. The DRBD device
holds LVM with the LVs for one single virtual machine (the webserver).
This webserver runs an Apache2 daemon.
The first virtualization server (xen1, the one that's live) runs rock
stable, same for the webserver VM on top. No crashes, no exploding
load. The second virtualization server (xen2) runs well as long as
it's only secondary (i.e. no virtual machine started). As soon as I
switch the DRBD primary to xen2 and start the webserver VM there,
load on the webserver is unusual high, it becomes laggy and after
some hours (sometimes minutes) it crashes like described in earlier
mails.
Now I created a test-VM on xen2 that is not on top of the DRBD device
in order to factor out DRBD as reason. As already written, if I fire
some HTTP requests against the Apache daemon on the test-VM, the VM
crashes the same way.
I first replaced memory modules and CPU by similar ones without
results. Now I replaced the whole hardware (except harddisks) by
another one - still the same crashes.
So the question is: why does the VM run stable on xen1 while it
crashes all the time on xen2. If I compare xen1 and xen2, only
real difference is mainboard (Supermicro X8 on xen1; Supermicro
X9 on xen2) and CPU (Xeon L5939 on xen1; E5-2609 on xen2)
As a next step I'll put the harddisks into another X8/Xeon L5639
server system and try to reproduce the crashes there. My bet is
that this system will not crash anymore. In other words, I guess
that this very bug is only triggered with the X9 + E-2609
combination.
> Can I do anything additional to help debugging the bug? Shall I report
> it
> to Xen upstream or send it to lkml?
Still the same question. Shall I send the bugreport to upstream?
Unfortunately nobody from Debian Linux kernel and/or Xen team seems
to care :-/
Cheers,
jonas
> Regarding the hardware: the RAM was checked with memtest86+ and no
> errors
> found and the CPU has been replaced by a new one (same model). Still,
> the
> VM crash is reproducible.
>
> The hardware on the crashing system is:
> CPU: Intel Xeon E5-2609v2/4x2,5GHz
> Motherboard: Supermicro X9SRI-F
>
> For information, the hardware on non-crashing system is:
> CPU: Intel XEON L5639/6x2,13 GHz
> Motherboard: Supermicro X8STi
>
>> It seems like the crashes are related to a RT process, even though no
>> sched_fifo/rr processes are started on this system intentionally.
>> Also, the
>> CPU usage is low all the time, no peaks at all. But the kernel
>> reports:
>>
>> kernel: [39101.461586] sched: RT throttling activated
>
> This is still valid, even though I no longer think that it's related to
> a
> RT process at all, as no sched_fifo/rr processes are started.
>
>> Usually, a few minutes later, soft lockups start to happen, and then
>> the
>> system crashes:
>
> The backtrace is slightly different now due to kernel and Xen updates:
>
> [353013.384931] sched: RT throttling activated
> [354008.100835] BUG: soft lockup - CPU#5 stuck for 22s! [apache2:24848]
> [354008.100846] Modules linked in: evdev coretemp crc32c_intel
> ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
> aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
> mbcache dm_mod xen_netfront xen_blkfront
> [354008.100872] CPU 5
> [354008.100874] Modules linked in: evdev coretemp crc32c_intel
> ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
> aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
> mbcache dm_mod xen_netfront xen_blkfront
> [354008.100894]
> [354008.100898] Pid: 24848, comm: apache2 Not tainted 3.2.0-4-amd64 #1
> Debian 3.2.60-1+deb7u3
> [354008.100904] RIP: e030:[<ffffffff8100122a>] [<ffffffff8100122a>]
> hypercall_page+0x22a/0x1000
> [354008.100914] RSP: e02b:ffff8802f0b41c00 EFLAGS: 00000246
> [354008.100918] RAX: 0000000000040001 RBX: ffff8802ffff0200 RCX:
> ffffffff8100122a
> [354008.100922] RDX: ffffffffffffffc8 RSI: 0000000000000000 RDI:
> 0000000000000000
> [354008.100927] RBP: 000000000000000e R08: 0000000000000200 R09:
> dead000000100100
> [354008.100931] R10: dead000000200200 R11: 0000000000000246 R12:
> ffff88028e261da0
> [354008.100935] R13: ffff8802ffffbe00 R14: ffffea00099905e8 R15:
> 000000000000000d
> [354008.100944] FS: 00007f7b66cd2740(0000) GS:ffff8802ffd40000(0000)
> knlGS:0000000000000000
> [354008.100949] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [354008.100953] CR2: 00007f7b68c2b000 CR3: 00000002855b7000 CR4:
> 0000000000002660
> [354008.100958] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [354008.100962] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [354008.100967] Process apache2 (pid: 24848, threadinfo
> ffff8802f0b40000, task ffff8802ef49c8c0)
> [354008.100972] Stack:
> [354008.100974] 0000000000000000 0000000000000200 ffffffff81006790
> ffffffff81006d22
> [354008.100981] 0000000000000000 dead000000200200 dead000000100100
> 0000000000000200
> [354008.100988] ffff8802ffff0200 ffff8802ffff0200 ffffffffffffffc8
> 0000000000000200
> [354008.100995] Call Trace:
> [354008.101000] [<ffffffff81006790>] ?
> xen_force_evtchn_callback+0x9/0xa
> [354008.101006] [<ffffffff81006d22>] ? check_events+0x12/0x20
> [354008.101011] [<ffffffff81006d0f>] ?
> xen_restore_fl_direct_reloc+0x4/0x4
> [354008.101017] [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8
> [354008.101024] [<ffffffff8135049f>] ?
> _raw_spin_unlock_irqrestore+0xe/0xf
> [354008.101031] [<ffffffff810be895>] ? release_pages+0xf4/0x14d
> [354008.101038] [<ffffffff810de78b>] ?
> free_pages_and_swap_cache+0x48/0x60
> [354008.101045] [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
> [354008.101049] [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
> [354008.101054] [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
> [354008.101060] [<ffffffff81044b82>] ? mmput+0x56/0xf8
> [354008.101064] [<ffffffff81049d07>] ? exit_mm+0x117/0x122
> [354008.101069] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354008.101074] [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
> [354008.101078] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354008.101083] [<ffffffff81049f57>] ? do_exit+0x245/0x713
> [354008.101088] [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
> [354008.101093] [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
> [354008.101098] [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
> [354008.101102] [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
> [354008.101107] [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
> [354008.101112] [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
> [354008.101115] Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc
> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00
> 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
> cc cc
> [354008.101157] Call Trace:
> [354008.101160] [<ffffffff81006790>] ?
> xen_force_evtchn_callback+0x9/0xa
> [354008.101165] [<ffffffff81006d22>] ? check_events+0x12/0x20
> [354008.101170] [<ffffffff81006d0f>] ?
> xen_restore_fl_direct_reloc+0x4/0x4
> [354008.101175] [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8
> [354008.101180] [<ffffffff8135049f>] ?
> _raw_spin_unlock_irqrestore+0xe/0xf
> [354008.101185] [<ffffffff810be895>] ? release_pages+0xf4/0x14d
> [354008.101190] [<ffffffff810de78b>] ?
> free_pages_and_swap_cache+0x48/0x60
> [354008.101195] [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
> [354008.101199] [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
> [354008.101204] [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
> [354008.101209] [<ffffffff81044b82>] ? mmput+0x56/0xf8
> [354008.101213] [<ffffffff81049d07>] ? exit_mm+0x117/0x122
> [354008.101217] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354008.101221] [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
> [354008.101226] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354008.101230] [<ffffffff81049f57>] ? do_exit+0x245/0x713
> [354008.101234] [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
> [354008.101239] [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
> [354008.101243] [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
> [354008.101247] [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
> [354008.101251] [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
> [354008.101256] [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
> [354036.127528] BUG: soft lockup - CPU#5 stuck for 22s! [apache2:24848]
> [354036.127539] Modules linked in: evdev coretemp crc32c_intel
> ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
> aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
> mbcache dm_mod xen_netfront xen_blkfront
> [354036.127564] CPU 5
> [354036.127567] Modules linked in: evdev coretemp crc32c_intel
> ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
> aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
> mbcache dm_mod xen_netfront xen_blkfront
> [354036.127587]
> [354036.127590] Pid: 24848, comm: apache2 Not tainted 3.2.0-4-amd64 #1
> Debian 3.2.60-1+deb7u3
> [354036.127597] RIP: e030:[<ffffffff8100122a>] [<ffffffff8100122a>]
> hypercall_page+0x22a/0x1000
> [354036.127606] RSP: e02b:ffff8802f0b41c00 EFLAGS: 00000246
> [354036.127610] RAX: 0000000000040001 RBX: ffff8802ffff0200 RCX:
> ffffffff8100122a
> [354036.127615] RDX: ffffffffffffffc8 RSI: 0000000000000000 RDI:
> 0000000000000000
> [354036.127619] RBP: 000000000000000e R08: 0000000000000200 R09:
> dead000000100100
> [354036.127623] R10: dead000000200200 R11: 0000000000000246 R12:
> ffff880265e01cc0
> [354036.127627] R13: ffff8802ffffbe00 R14: ffffea00099748c0 R15:
> 000000000000000d
> [354036.127637] FS: 00007f7b66cd2740(0000) GS:ffff8802ffd40000(0000)
> knlGS:0000000000000000
> [354036.127642] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [354036.127646] CR2: 00007f7b68c2b000 CR3: 00000002855b7000 CR4:
> 0000000000002660
> [354036.127650] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [354036.127655] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [354036.127660] Process apache2 (pid: 24848, threadinfo
> ffff8802f0b40000, task ffff8802ef49c8c0)
> [354036.127665] Stack:
> [354036.127667] 0000000000000000 0000000000000200 ffffffff81006790
> ffffffff81006d22
> [354036.127674] 0000000000000000 dead000000200200 dead000000100100
> 0000000000000200
> [354036.127680] ffff8802ffff0200 ffff8802ffff0200 ffffffffffffffc8
> 0000000000000200
> [354036.127687] Call Trace:
> [354036.127693] [<ffffffff81006790>] ?
> xen_force_evtchn_callback+0x9/0xa
> [354036.127699] [<ffffffff81006d22>] ? check_events+0x12/0x20
> [354036.127704] [<ffffffff81006d0f>] ?
> xen_restore_fl_direct_reloc+0x4/0x4
> [354036.127710] [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8
> [354036.127717] [<ffffffff8135049f>] ?
> _raw_spin_unlock_irqrestore+0xe/0xf
> [354036.127723] [<ffffffff810be895>] ? release_pages+0xf4/0x14d
> [354036.127730] [<ffffffff810de78b>] ?
> free_pages_and_swap_cache+0x48/0x60
> [354036.127736] [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
> [354036.127741] [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
> [354036.127746] [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
> [354036.127752] [<ffffffff81044b82>] ? mmput+0x56/0xf8
> [354036.127756] [<ffffffff81049d07>] ? exit_mm+0x117/0x122
> [354036.127760] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354036.127765] [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
> [354036.127769] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354036.127773] [<ffffffff81049f57>] ? do_exit+0x245/0x713
> [354036.127779] [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
> [354036.127784] [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
> [354036.127788] [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
> [354036.127793] [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
> [354036.127797] [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
> [354036.127802] [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
> [354036.127805] Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc
> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00
> 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
> cc cc
> [354036.127846] Call Trace:
> [354036.127849] [<ffffffff81006790>] ?
> xen_force_evtchn_callback+0x9/0xa
> [354036.127854] [<ffffffff81006d22>] ? check_events+0x12/0x20
> [354036.127859] [<ffffffff81006d0f>] ?
> xen_restore_fl_direct_reloc+0x4/0x4
> [354036.127864] [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8
> [354036.127868] [<ffffffff8135049f>] ?
> _raw_spin_unlock_irqrestore+0xe/0xf
> [354036.127873] [<ffffffff810be895>] ? release_pages+0xf4/0x14d
> [354036.127879] [<ffffffff810de78b>] ?
> free_pages_and_swap_cache+0x48/0x60
> [354036.127883] [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
> [354036.127888] [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
> [354036.127893] [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
> [354036.127897] [<ffffffff81044b82>] ? mmput+0x56/0xf8
> [354036.127901] [<ffffffff81049d07>] ? exit_mm+0x117/0x122
> [354036.127905] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354036.127909] [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
> [354036.127914] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354036.127918] [<ffffffff81049f57>] ? do_exit+0x245/0x713
> [354036.127922] [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
> [354036.127927] [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
> [354036.127931] [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
> [354036.127935] [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
> [354036.128001] [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
> [354036.128007] [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
> [354044.085361] BUG: soft lockup - CPU#3 stuck for 23s! [apache2:20278]
> [354044.085370] Modules linked in: evdev coretemp crc32c_intel
> ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
> aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
> mbcache dm_mod xen_netfront xen_blkfront
> [354044.085395] CPU 3
> [354044.085397] Modules linked in: evdev coretemp crc32c_intel
> ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
> aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
> mbcache dm_mod xen_netfront xen_blkfront
> [354044.085417]
> [354044.085420] Pid: 20278, comm: apache2 Not tainted 3.2.0-4-amd64 #1
> Debian 3.2.60-1+deb7u3
> [354044.085426] RIP: e030:[<ffffffff8100122a>] [<ffffffff8100122a>]
> hypercall_page+0x22a/0x1000
> [354044.085436] RSP: e02b:ffff8802ef743c00 EFLAGS: 00000246
> [354044.085440] RAX: 0000000000040001 RBX: ffff8802ffff0200 RCX:
> ffffffff8100122a
> [354044.085444] RDX: ffffffffffffffc8 RSI: 0000000000000000 RDI:
> 0000000000000000
> [354044.085449] RBP: 000000000000000e R08: 0000000000000200 R09:
> dead000000100100
> [354044.085453] R10: dead000000200200 R11: 0000000000000246 R12:
> ffff880265c53320
> [354044.085457] R13: ffff8802ffffbe00 R14: ffffea0009cec350 R15:
> 000000000000000d
> [354044.085466] FS: 00007f7b66cd2740(0000) GS:ffff8802ffcc0000(0000)
> knlGS:0000000000000000
> [354044.085471] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [354044.085475] CR2: 00007f7b6906cff0 CR3: 00000002efbfb000 CR4:
> 0000000000002660
> [354044.085479] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [354044.085484] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [354044.085489] Process apache2 (pid: 20278, threadinfo
> ffff8802ef742000, task ffff8802efa5b880)
> [354044.085493] Stack:
> [354044.085495] 0000000000000000 0000000000000200 ffffffff81006790
> ffffffff81006d22
> [354044.085502] 0000000000000000 dead000000200200 dead000000100100
> 0000000000000200
> [354044.085508] ffff8802ffff0200 ffff8802ffff0200 ffffffffffffffc8
> 0000000000000200
> [354044.085746] Call Trace:
> [354044.085753] [<ffffffff81006790>] ?
> xen_force_evtchn_callback+0x9/0xa
> [354044.085758] [<ffffffff81006d22>] ? check_events+0x12/0x20
> [354044.085763] [<ffffffff81006d0f>] ?
> xen_restore_fl_direct_reloc+0x4/0x4
> [354044.085769] [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8
> [354044.085776] [<ffffffff8135049f>] ?
> _raw_spin_unlock_irqrestore+0xe/0xf
> [354044.085783] [<ffffffff810be895>] ? release_pages+0xf4/0x14d
> [354044.085790] [<ffffffff810de78b>] ?
> free_pages_and_swap_cache+0x48/0x60
> [354044.085796] [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
> [354044.085800] [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
> [354044.085806] [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
> [354044.085811] [<ffffffff81044b82>] ? mmput+0x56/0xf8
> [354044.085815] [<ffffffff81049d07>] ? exit_mm+0x117/0x122
> [354044.085820] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354044.085824] [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
> [354044.085829] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354044.085833] [<ffffffff81049f57>] ? do_exit+0x245/0x713
> [354044.085838] [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
> [354044.085843] [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
> [354044.085848] [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
> [354044.085852] [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
> [354044.085856] [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
> [354044.085861] [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
> [354044.085865] Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc
> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00
> 00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
> cc cc
> [354044.085905] Call Trace:
> [354044.085909] [<ffffffff81006790>] ?
> xen_force_evtchn_callback+0x9/0xa
> [354044.085914] [<ffffffff81006d22>] ? check_events+0x12/0x20
> [354044.085919] [<ffffffff81006d0f>] ?
> xen_restore_fl_direct_reloc+0x4/0x4
> [354044.085924] [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8
> [354044.085928] [<ffffffff8135049f>] ?
> _raw_spin_unlock_irqrestore+0xe/0xf
> [354044.085933] [<ffffffff810be895>] ? release_pages+0xf4/0x14d
> [354044.085939] [<ffffffff810de78b>] ?
> free_pages_and_swap_cache+0x48/0x60
> [354044.085944] [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
> [354044.085948] [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
> [354044.085953] [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
> [354044.085957] [<ffffffff81044b82>] ? mmput+0x56/0xf8
> [354044.085961] [<ffffffff81049d07>] ? exit_mm+0x117/0x122
> [354044.085965] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354044.085970] [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
> [354044.085974] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354044.085978] [<ffffffff81049f57>] ? do_exit+0x245/0x713
> [354044.085983] [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
> [354044.085987] [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
> [354044.085992] [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
> [354044.085996] [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
> [354044.086000] [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
> [354044.086004] [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
> [354047.219103] INFO: rcu_sched detected stall on CPU 5 (t=15039
> jiffies)
> [354047.224054] INFO: rcu_sched detected stalls on CPUs/tasks: { 5}
> (detected by 1, t=15039 jiffies)
> [354047.224054] sending NMI to all CPUs:
> [354047.224054] BUG: unable to handle kernel paging request at
> ffffffffff5fb310
> [354047.224054] IP: [<ffffffff81027b9a>] native_apic_mem_write+0x2/0x9
> [354047.224054] PGD 1607067 PUD 1608067 PMD 172d067 PTE 0
> [354047.224054] Oops: 0002 [#1] SMP
> [354047.224054] CPU 1
> [354047.224054] Modules linked in: evdev coretemp crc32c_intel
> ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
> aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
> mbcache dm_mod xen_netfront xen_blkfront
> [354047.224054]
> [354047.224054] Pid: 24843, comm: apache2 Not tainted 3.2.0-4-amd64 #1
> Debian 3.2.60-1+deb7u3
> [354047.224054] RIP: e030:[<ffffffff81027b9a>] [<ffffffff81027b9a>]
> native_apic_mem_write+0x2/0x9
> [354047.224054] RSP: e02b:ffff8802ffc43c90 EFLAGS: 00010086
> [354047.224054] RAX: 0000000000000000 RBX: ffffffff816800e0 RCX:
> 00000000000007c2
> [354047.224054] RDX: 0000000000000000 RSI: 00000000ff000000 RDI:
> 0000000000000310
> [354047.224054] RBP: 0000000000000002 R08: 0000000000000002 R09:
> 0000000000000000
> [354047.224054] R10: 0000000000000000 R11: ffff8802f2863d40 R12:
> 0000000000000800
> [354047.224054] R13: 00000000000000ff R14: ffffffff81624400 R15:
> 0000000000000000
> [354047.224054] FS: 00007f7b66cd2740(0000) GS:ffff8802ffc40000(0000)
> knlGS:0000000000000000
> [354047.224054] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> [354047.224054] CR2: ffffffffff5fb310 CR3: 00000002f06c2000 CR4:
> 0000000000002660
> [354047.224054] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [354047.224054] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [354047.224054] Process apache2 (pid: 24843, threadinfo
> ffff8802ef78a000, task ffff8802f1476800)
> [354047.224054] Stack:
> [354047.224054] ffffffff81027dd3 0000000000000000 0000000000002710
> ffffffff81622300
> [354047.224054] ffffffff81622400 0000000000000001 ffffffff81024d16
> ffff8802ffc4edc0
> [354047.224054] ffffffff81096308 0000000000000048 ffffffff81624400
> ffffffff81098d57
> [354047.224054] Call Trace:
> [354047.224054] <IRQ>
> [354047.224054] [<ffffffff81027dd3>] ? _flat_send_IPI_mask+0x4b/0x78
> [354047.224054] [<ffffffff81024d16>] ?
> arch_trigger_all_cpu_backtrace+0x4d/0x7b
> [354047.224054] [<ffffffff81096308>] ? __rcu_pending+0x21a/0x358
> [354047.224054] [<ffffffff81098d57>] ? arch_local_irq_restore+0x7/0x8
> [354047.224054] [<ffffffff8106c35c>] ? tick_nohz_handler+0xd0/0xd0
> [354047.224054] [<ffffffff81096773>] ? rcu_check_callbacks+0x90/0xcc
> [354047.224054] [<ffffffff81052cbe>] ? update_process_times+0x31/0x63
> [354047.224054] [<ffffffff8106c3c6>] ? tick_sched_timer+0x6a/0x90
> [354047.224054] [<ffffffff81062632>] ? __run_hrtimer+0xac/0x135
> [354047.224054] [<ffffffff81062d1c>] ? hrtimer_interrupt+0xd7/0x1b1
> [354047.224054] [<ffffffff8103b087>] ? check_preempt_curr+0x52/0x5f
> [354047.224054] [<ffffffff810068b9>] ? xen_timer_interrupt+0x28/0xfc
> [354047.224054] [<ffffffff81244771>] ? get_cycles+0x5/0x8
> [354047.224054] [<ffffffff8124561e>] ?
> add_interrupt_randomness+0x38/0x155
> [354047.224054] [<ffffffff8109124d>] ?
> handle_irq_event_percpu+0x50/0x17d
> [354047.224054] [<ffffffff8121ca3a>] ? disable_pirq+0x2/0x2
> [354047.224054] [<ffffffff8121c624>] ? info_for_irq+0x7/0x17
> [354047.224054] [<ffffffff81093847>] ? handle_percpu_irq+0x3a/0x4f
> [354047.224054] [<ffffffff8121c866>] ?
> __xen_evtchn_do_upcall+0xd3/0x287
> [354047.224054] [<ffffffff8104b780>] ? __local_bh_enable+0x40/0x77
> [354047.224054] [<ffffffff813576ac>] ? call_softirq+0x1c/0x30
> [354047.224054] [<ffffffff81095239>] ? arch_local_irq_save+0x11/0x15
> [354047.224054] [<ffffffff8121dd98>] ? xen_evtchn_do_upcall+0x22/0x32
> [354047.224054] [<ffffffff813576fe>] ?
> xen_do_hypervisor_callback+0x1e/0x30
> [354047.224054] <EOI>
> [354047.224054] [<ffffffff810be219>] ? add_page_to_lru_list+0x64/0x64
> [354047.224054] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
> [354047.224054] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
> [354047.224054] [<ffffffff81006790>] ?
> xen_force_evtchn_callback+0x9/0xa
> [354047.224054] [<ffffffff81006d22>] ? check_events+0x12/0x20
> [354047.224054] [<ffffffff81006d0f>] ?
> xen_restore_fl_direct_reloc+0x4/0x4
> [354047.224054] [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8
> [354047.224054] [<ffffffff8135049f>] ?
> _raw_spin_unlock_irqrestore+0xe/0xf
> [354047.224054] [<ffffffff810be97d>] ? pagevec_lru_move_fn+0x8f/0xb5
> [354047.224054] [<ffffffff810beb8a>] ? __lru_cache_add+0x4a/0x51
> [354047.224054] [<ffffffff810d1537>] ? handle_pte_fault+0x224/0x79f
> [354047.224054] [<ffffffff810ceacb>] ? pmd_val+0x7/0x8
> [354047.224054] [<ffffffff810ceb49>] ? pte_offset_kernel+0x16/0x35
> [354047.224054] [<ffffffff813533ee>] ? do_page_fault+0x320/0x345
> [354047.224054] [<ffffffff81003223>] ? xen_end_context_switch+0xe/0x1c
> [354047.224054] [<ffffffff81003ba5>] ?
> xen_mc_issue.constprop.23+0x31/0x49
> [354047.224054] [<ffffffff8100d750>] ? __switch_to+0x1e5/0x258
> [354047.224054] [<ffffffff81035bd7>] ? arch_local_irq_enable+0x7/0x8
> [354047.224054] [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
> [354047.224054] [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
> [354047.224054] [<ffffffff813509f5>] ? page_fault+0x25/0x30
> [354047.224054] Code: 00 74 18 48 8d 74 24 0c bf 1b 00 00 00 e8 ab fb
> ff ff f6 c4 04 0f 95 c0 0f b6 c0 48 83 c4 10 c3 90 ff 14 25 d8 57 61
> 81 c3 89 ff <89> b7 00 b0 5f ff c3 89 ff 8b 87 00 b0 5f ff c3 48 8b 07
> 25 ff
> [354047.224054] RIP [<ffffffff81027b9a>] native_apic_mem_write+0x2/0x9
> [354047.224054] RSP <ffff8802ffc43c90>
> [354047.224054] CR2: ffffffffff5fb310
> [354047.224054] ---[ end trace 94d691dcc7253fa7 ]---
> [354047.224054] Kernel panic - not syncing: Fatal exception in
> interrupt
> [354047.224054] Pid: 24843, comm: apache2 Tainted: G D
> 3.2.0-4-amd64 #1 Debian 3.2.60-1+deb7u3
> [354047.224054] Call Trace:
> [354047.224054] <IRQ> [<ffffffff81349af6>] ? panic+0x95/0x1a2
> [354047.224054] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354047.224054] [<ffffffff8135049f>] ?
> _raw_spin_unlock_irqrestore+0xe/0xf
> [354047.224054] [<ffffffff81351286>] ? oops_end+0xa9/0xb6
> [354047.224054] [<ffffffff8134943f>] ? no_context+0x1ff/0x20e
> [354047.224054] [<ffffffff81348ccd>] ? pmd_val+0x7/0x8
> [354047.224054] [<ffffffff81348d0c>] ? pte_offset_kernel+0x16/0x35
> [354047.224054] [<ffffffff81353284>] ? do_page_fault+0x1b6/0x345
> [354047.224054] [<ffffffff811b2c7f>] ? vsnprintf+0x7c/0x427
> [354047.224054] [<ffffffff811b2c7f>] ? vsnprintf+0x7c/0x427
> [354047.224054] [<ffffffff8102bb7c>] ?
> pvclock_clocksource_read+0x42/0xb2
> [354047.224054] [<ffffffff811b3070>] ? sprintf+0x46/0x4b
> [354047.224054] [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
> [354047.224054] [<ffffffff8107116d>] ? arch_local_irq_save+0x11/0x17
> [354047.224054] [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8
> [354047.224054] [<ffffffff8135049f>] ?
> _raw_spin_unlock_irqrestore+0xe/0xf
> [354047.224054] [<ffffffff810637e0>] ? down_trylock+0x20/0x29
> [354047.224054] [<ffffffff813509f5>] ? page_fault+0x25/0x30
> [354047.224054] [<ffffffff81027b9a>] ? native_apic_mem_write+0x2/0x9
> [354047.224054] [<ffffffff81027dd3>] ? _flat_send_IPI_mask+0x4b/0x78
> [354047.224054] [<ffffffff81024d16>] ?
> arch_trigger_all_cpu_backtrace+0x4d/0x7b
> [354047.224054] [<ffffffff81096308>] ? __rcu_pending+0x21a/0x358
> [354047.224054] [<ffffffff81098d57>] ? arch_local_irq_restore+0x7/0x8
> [354047.224054] [<ffffffff8106c35c>] ? tick_nohz_handler+0xd0/0xd0
> [354047.224054] [<ffffffff81096773>] ? rcu_check_callbacks+0x90/0xcc
> [354047.224054] [<ffffffff81052cbe>] ? update_process_times+0x31/0x63
> [354047.224054] [<ffffffff8106c3c6>] ? tick_sched_timer+0x6a/0x90
> [354047.224054] [<ffffffff81062632>] ? __run_hrtimer+0xac/0x135
> [354047.224054] [<ffffffff81062d1c>] ? hrtimer_interrupt+0xd7/0x1b1
> [354047.224054] [<ffffffff8103b087>] ? check_preempt_curr+0x52/0x5f
> [354047.224054] [<ffffffff810068b9>] ? xen_timer_interrupt+0x28/0xfc
> [354047.224054] [<ffffffff81244771>] ? get_cycles+0x5/0x8
> [354047.224054] [<ffffffff8124561e>] ?
> add_interrupt_randomness+0x38/0x155
> [354047.224054] [<ffffffff8109124d>] ?
> handle_irq_event_percpu+0x50/0x17d
> [354047.224054] [<ffffffff8121ca3a>] ? disable_pirq+0x2/0x2
> [354047.224054] [<ffffffff8121c624>] ? info_for_irq+0x7/0x17
> [354047.224054] [<ffffffff81093847>] ? handle_percpu_irq+0x3a/0x4f
> [354047.224054] [<ffffffff8121c866>] ?
> __xen_evtchn_do_upcall+0xd3/0x287
> [354047.224054] [<ffffffff8104b780>] ? __local_bh_enable+0x40/0x77
> [354047.224054] [<ffffffff813576ac>] ? call_softirq+0x1c/0x30
> [354047.224054] [<ffffffff81095239>] ? arch_local_irq_save+0x11/0x15
> [354047.224054] [<ffffffff8121dd98>] ? xen_evtchn_do_upcall+0x22/0x32
> [354047.224054] [<ffffffff813576fe>] ?
> xen_do_hypervisor_callback+0x1e/0x30
> [354047.224054] <EOI> [<ffffffff810be219>] ?
> add_page_to_lru_list+0x64/0x64
> [354047.224054] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
> [354047.224054] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
> [354047.224054] [<ffffffff81006790>] ?
> xen_force_evtchn_callback+0x9/0xa
> [354047.224054] [<ffffffff81006d22>] ? check_events+0x12/0x20
> [354047.224054] [<ffffffff81006d0f>] ?
> xen_restore_fl_direct_reloc+0x4/0x4
> [354047.224054] [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8
> [354047.224054] [<ffffffff8135049f>] ?
> _raw_spin_unlock_irqrestore+0xe/0xf
> [354047.224054] [<ffffffff810be97d>] ? pagevec_lru_move_fn+0x8f/0xb5
> [354047.224054] [<ffffffff810beb8a>] ? __lru_cache_add+0x4a/0x51
> [354047.224054] [<ffffffff810d1537>] ? handle_pte_fault+0x224/0x79f
> [354047.224054] [<ffffffff810ceacb>] ? pmd_val+0x7/0x8
> [354047.224054] [<ffffffff810ceb49>] ? pte_offset_kernel+0x16/0x35
> [354047.224054] [<ffffffff813533ee>] ? do_page_fault+0x320/0x345
> [354047.224054] [<ffffffff81003223>] ? xen_end_context_switch+0xe/0x1c
> [354047.224054] [<ffffffff81003ba5>] ?
> xen_mc_issue.constprop.23+0x31/0x49
> [354047.224054] [<ffffffff8100d750>] ? __switch_to+0x1e5/0x258
> [354047.224054] [<ffffffff81035bd7>] ? arch_local_irq_enable+0x7/0x8
> [354047.224054] [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
> [354047.224054] [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
> [354047.224054] [<ffffffff813509f5>] ? page_fault+0x25/0x30
>
> Cheers,
> jonas
More information about the Pkg-xen-devel
mailing list