[pkg-lxc-devel] Bug#887787: lxc: CentOS 7 amd64 container can't be stopped
Toni Mueller
toni at debian.org
Fri Jan 19 21:51:13 UTC 2018
Package: lxc
Version: 1:2.0.7-2+deb9u1
Severity: normal
Hi,
I installed an unprivileged CentOS 7 container with
$ lxc-create -n centos -t download
after setting my system up according to the instructions given here:
https://wiki.debian.org/LXC
The resulting container starts as expected, but any attempt to shut it
down again, fails. The container remains responsive, but just does not
stop. Inside the container, the process list looks as follows:
$ lxc-attach -n centos
bash-4.2# ps auwwx
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 42544 2332 ? Ds Jan20 0:00 /sbin/init
root 132 0.0 0.1 113380 13416 ? Ss Jan20 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H centos eth0
root 191 0.0 0.0 115396 3160 ? Ss Jan20 0:00 /bin/bash
root 192 0.0 0.0 107912 616 ? D+ Jan20 0:00 sync
root 193 0.0 0.0 115396 3064 ? Ss Jan20 0:00 /bin/bash
root 194 0.0 0.0 151072 3564 ? R+ Jan20 0:00 ps auwwx
bash-4.2#
In a different shell, things look like this:
$ lxc-attach -n centos
bash-4.2# halt -n -f
c^C^Z^C
(ie, no reaction)
In the kernel log of the host, I get a lot of these:
Jan 19 22:20:37 debian kernel: [39269.678133] INFO: task systemd:6381 blocked for more than 120 seconds.
Jan 19 22:20:37 debian kernel: [39269.678144] Tainted: G O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:20:37 debian kernel: [39269.678149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:20:37 debian kernel: [39269.678154] systemd D 0 6381 6371 0x00000104
Jan 19 22:20:37 debian kernel: [39269.678163] ffff976453d38000 ffff976553415800 ffff976504ffafc0 ffff97655fc98940
Jan 19 22:20:37 debian kernel: [39269.678170] ffff976500564480 ffffa7a7040b3bc0 ffffffffa6802923 ffff976504ffafc0
Jan 19 22:20:37 debian kernel: [39269.678176] 00ff976504ffafc0 ffff97655fc98940 ffff976504161dc0 ffff976504ffafc0
Jan 19 22:20:37 debian kernel: [39269.678181] Call Trace:
Jan 19 22:20:37 debian kernel: [39269.678194] [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:20:37 debian kernel: [39269.678200] [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:20:37 debian kernel: [39269.678207] [<ffffffffa6805999>] ? rwsem_down_write_failed+0x1f9/0x360
Jan 19 22:20:37 debian kernel: [39269.678214] [<ffffffffa64801d0>] ? kernfs_sop_show_options+0x30/0x30
Jan 19 22:20:37 debian kernel: [39269.678220] [<ffffffffa6538213>] ? call_rwsem_down_write_failed+0x13/0x20
Jan 19 22:20:37 debian kernel: [39269.678225] [<ffffffffa6805039>] ? down_write+0x29/0x40
Jan 19 22:20:37 debian kernel: [39269.678231] [<ffffffffa64055ab>] ? grab_super+0x2b/0x90
Jan 19 22:20:37 debian kernel: [39269.678237] [<ffffffffa6405b43>] ? sget_userns+0x163/0x490
Jan 19 22:20:37 debian kernel: [39269.678242] [<ffffffffa6480240>] ? kernfs_sop_show_path+0x40/0x40
Jan 19 22:20:37 debian kernel: [39269.678246] [<ffffffffa648044a>] ? kernfs_mount_ns+0x7a/0x220
Jan 19 22:20:37 debian kernel: [39269.678252] [<ffffffffa630dba4>] ? cgroup_mount+0x334/0x810
Jan 19 22:20:37 debian kernel: [39269.678259] [<ffffffffa6406ae6>] ? mount_fs+0x36/0x150
Jan 19 22:20:37 debian kernel: [39269.678264] [<ffffffffa6423f32>] ? vfs_kern_mount+0x62/0x100
Jan 19 22:20:37 debian kernel: [39269.678268] [<ffffffffa64263ff>] ? do_mount+0x1cf/0xc80
Jan 19 22:20:37 debian kernel: [39269.678273] [<ffffffffa64271de>] ? SyS_mount+0x7e/0xd0
Jan 19 22:20:37 debian kernel: [39269.678279] [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:20:37 debian kernel: [39269.678285] [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jan 19 22:20:37 debian kernel: [39269.678294] INFO: task sync:6958 blocked for more than 120 seconds.
Jan 19 22:20:37 debian kernel: [39269.678300] Tainted: G O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:20:37 debian kernel: [39269.678303] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:20:37 debian kernel: [39269.678307] sync D 0 6958 6951 0x00000104
Jan 19 22:20:37 debian kernel: [39269.678312] ffff976553fc4800 0000000000000000 ffff97653103a140 ffff97655fcd8940
Jan 19 22:20:37 debian kernel: [39269.678318] ffff976556666f00 ffffa7a706043df0 ffffffffa6802923 ffff9765500a5760
Jan 19 22:20:37 debian kernel: [39269.678324] 0000000000000286 ffff97655fcd8940 ffffa7a706043df8 ffff97653103a140
Jan 19 22:20:37 debian kernel: [39269.678329] Call Trace:
Jan 19 22:20:37 debian kernel: [39269.678335] [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:20:37 debian kernel: [39269.678340] [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:20:37 debian kernel: [39269.678345] [<ffffffffa6805740>] ? rwsem_down_read_failed+0xf0/0x150
Jan 19 22:20:37 debian kernel: [39269.678350] [<ffffffffa641f86e>] ? iput+0x7e/0x210
Jan 19 22:20:37 debian kernel: [39269.678356] [<ffffffffa6436870>] ? SyS_tee+0x390/0x390
Jan 19 22:20:37 debian kernel: [39269.678361] [<ffffffffa65381e4>] ? call_rwsem_down_read_failed+0x14/0x30
Jan 19 22:20:37 debian kernel: [39269.678366] [<ffffffffa6804ffc>] ? down_read+0x1c/0x30
Jan 19 22:20:37 debian kernel: [39269.678371] [<ffffffffa64064ac>] ? iterate_supers+0x9c/0x100
Jan 19 22:20:37 debian kernel: [39269.678376] [<ffffffffa6436bc2>] ? sys_sync+0x42/0xb0
Jan 19 22:20:37 debian kernel: [39269.678380] [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:20:37 debian kernel: [39269.678386] [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jan 19 22:20:37 debian kernel: [39269.678401] INFO: task halt:7352 blocked for more than 120 seconds.
Jan 19 22:20:37 debian kernel: [39269.678406] Tainted: G O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:20:37 debian kernel: [39269.678409] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:20:37 debian kernel: [39269.678412] halt D 0 7352 7211 0x00000100
Jan 19 22:20:37 debian kernel: [39269.678417] ffff976554b10000 0000000000000000 ffff97650de10200 ffff97655fc18940
Jan 19 22:20:37 debian kernel: [39269.678422] ffffffffa6e11500 ffffa7a7059bfdf0 ffffffffa6802923 ffff9765500a5760
Jan 19 22:20:37 debian kernel: [39269.678428] 0000000000000286 ffff97655fc18940 ffffa7a7059bfdf8 ffff97650de10200
Jan 19 22:20:37 debian kernel: [39269.678433] Call Trace:
Jan 19 22:20:37 debian kernel: [39269.678438] [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:20:37 debian kernel: [39269.678443] [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:20:37 debian kernel: [39269.678448] [<ffffffffa6805740>] ? rwsem_down_read_failed+0xf0/0x150
Jan 19 22:20:37 debian kernel: [39269.678452] [<ffffffffa641f86e>] ? iput+0x7e/0x210
Jan 19 22:20:37 debian kernel: [39269.678458] [<ffffffffa6436870>] ? SyS_tee+0x390/0x390
Jan 19 22:20:37 debian kernel: [39269.678463] [<ffffffffa65381e4>] ? call_rwsem_down_read_failed+0x14/0x30
Jan 19 22:20:37 debian kernel: [39269.678468] [<ffffffffa6804ffc>] ? down_read+0x1c/0x30
Jan 19 22:20:37 debian kernel: [39269.678473] [<ffffffffa64064ac>] ? iterate_supers+0x9c/0x100
Jan 19 22:20:37 debian kernel: [39269.678478] [<ffffffffa6436bc2>] ? sys_sync+0x42/0xb0
Jan 19 22:20:37 debian kernel: [39269.678482] [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:20:37 debian kernel: [39269.678487] [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jan 19 22:22:38 debian kernel: [39390.499861] INFO: task systemd:6381 blocked for more than 120 seconds.
Jan 19 22:22:38 debian kernel: [39390.499872] Tainted: G O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:22:38 debian kernel: [39390.499877] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:22:38 debian kernel: [39390.499881] systemd D 0 6381 6371 0x00000104
Jan 19 22:22:38 debian kernel: [39390.499890] ffff976453d38000 ffff976553415800 ffff976504ffafc0 ffff97655fc98940
Jan 19 22:22:38 debian kernel: [39390.499897] ffff976500564480 ffffa7a7040b3bc0 ffffffffa6802923 ffff976504ffafc0
Jan 19 22:22:38 debian kernel: [39390.499903] 00ff976504ffafc0 ffff97655fc98940 ffff976504161dc0 ffff976504ffafc0
Jan 19 22:22:38 debian kernel: [39390.499908] Call Trace:
Jan 19 22:22:38 debian kernel: [39390.499922] [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:22:38 debian kernel: [39390.499927] [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:22:38 debian kernel: [39390.499934] [<ffffffffa6805999>] ? rwsem_down_write_failed+0x1f9/0x360
Jan 19 22:22:38 debian kernel: [39390.499941] [<ffffffffa64801d0>] ? kernfs_sop_show_options+0x30/0x30
Jan 19 22:22:38 debian kernel: [39390.499947] [<ffffffffa6538213>] ? call_rwsem_down_write_failed+0x13/0x20
Jan 19 22:22:38 debian kernel: [39390.499952] [<ffffffffa6805039>] ? down_write+0x29/0x40
Jan 19 22:22:38 debian kernel: [39390.499958] [<ffffffffa64055ab>] ? grab_super+0x2b/0x90
Jan 19 22:22:38 debian kernel: [39390.499963] [<ffffffffa6405b43>] ? sget_userns+0x163/0x490
Jan 19 22:22:38 debian kernel: [39390.499968] [<ffffffffa6480240>] ? kernfs_sop_show_path+0x40/0x40
Jan 19 22:22:38 debian kernel: [39390.499973] [<ffffffffa648044a>] ? kernfs_mount_ns+0x7a/0x220
Jan 19 22:22:38 debian kernel: [39390.499978] [<ffffffffa630dba4>] ? cgroup_mount+0x334/0x810
Jan 19 22:22:38 debian kernel: [39390.499985] [<ffffffffa6406ae6>] ? mount_fs+0x36/0x150
Jan 19 22:22:38 debian kernel: [39390.499990] [<ffffffffa6423f32>] ? vfs_kern_mount+0x62/0x100
Jan 19 22:22:38 debian kernel: [39390.499995] [<ffffffffa64263ff>] ? do_mount+0x1cf/0xc80
Jan 19 22:22:38 debian kernel: [39390.500000] [<ffffffffa64271de>] ? SyS_mount+0x7e/0xd0
Jan 19 22:22:38 debian kernel: [39390.500006] [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:22:38 debian kernel: [39390.500012] [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jan 19 22:22:38 debian kernel: [39390.500021] INFO: task sync:6958 blocked for more than 120 seconds.
Jan 19 22:22:38 debian kernel: [39390.500026] Tainted: G O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:22:38 debian kernel: [39390.500030] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:22:38 debian kernel: [39390.500033] sync D 0 6958 6951 0x00000104
Jan 19 22:22:38 debian kernel: [39390.500038] ffff976553fc4800 0000000000000000 ffff97653103a140 ffff97655fcd8940
Jan 19 22:22:38 debian kernel: [39390.500045] ffff976556666f00 ffffa7a706043df0 ffffffffa6802923 ffff9765500a5760
Jan 19 22:22:38 debian kernel: [39390.500050] 0000000000000286 ffff97655fcd8940 ffffa7a706043df8 ffff97653103a140
Jan 19 22:22:38 debian kernel: [39390.500055] Call Trace:
Jan 19 22:22:38 debian kernel: [39390.500061] [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:22:38 debian kernel: [39390.500066] [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:22:38 debian kernel: [39390.500071] [<ffffffffa6805740>] ? rwsem_down_read_failed+0xf0/0x150
Jan 19 22:22:38 debian kernel: [39390.500076] [<ffffffffa641f86e>] ? iput+0x7e/0x210
Jan 19 22:22:38 debian kernel: [39390.500082] [<ffffffffa6436870>] ? SyS_tee+0x390/0x390
Jan 19 22:22:38 debian kernel: [39390.500088] [<ffffffffa65381e4>] ? call_rwsem_down_read_failed+0x14/0x30
Jan 19 22:22:38 debian kernel: [39390.500092] [<ffffffffa6804ffc>] ? down_read+0x1c/0x30
Jan 19 22:22:38 debian kernel: [39390.500098] [<ffffffffa64064ac>] ? iterate_supers+0x9c/0x100
Jan 19 22:22:38 debian kernel: [39390.500103] [<ffffffffa6436bc2>] ? sys_sync+0x42/0xb0
Jan 19 22:22:38 debian kernel: [39390.500107] [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:22:38 debian kernel: [39390.500112] [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Jan 19 22:22:38 debian kernel: [39390.500127] INFO: task halt:7352 blocked for more than 120 seconds.
Jan 19 22:22:38 debian kernel: [39390.500132] Tainted: G O 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Jan 19 22:22:38 debian kernel: [39390.500135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 19 22:22:38 debian kernel: [39390.500138] halt D 0 7352 7211 0x00000100
Jan 19 22:22:38 debian kernel: [39390.500143] ffff976554b10000 0000000000000000 ffff97650de10200 ffff97655fc18940
Jan 19 22:22:38 debian kernel: [39390.500149] ffffffffa6e11500 ffffa7a7059bfdf0 ffffffffa6802923 ffff9765500a5760
Jan 19 22:22:38 debian kernel: [39390.500154] 0000000000000286 ffff97655fc18940 ffffa7a7059bfdf8 ffff97650de10200
Jan 19 22:22:38 debian kernel: [39390.500159] Call Trace:
Jan 19 22:22:38 debian kernel: [39390.500165] [<ffffffffa6802923>] ? __schedule+0x233/0x6d0
Jan 19 22:22:38 debian kernel: [39390.500169] [<ffffffffa6802df2>] ? schedule+0x32/0x80
Jan 19 22:22:38 debian kernel: [39390.500174] [<ffffffffa6805740>] ? rwsem_down_read_failed+0xf0/0x150
Jan 19 22:22:38 debian kernel: [39390.500179] [<ffffffffa641f86e>] ? iput+0x7e/0x210
Jan 19 22:22:38 debian kernel: [39390.500184] [<ffffffffa6436870>] ? SyS_tee+0x390/0x390
Jan 19 22:22:38 debian kernel: [39390.500189] [<ffffffffa65381e4>] ? call_rwsem_down_read_failed+0x14/0x30
Jan 19 22:22:38 debian kernel: [39390.500194] [<ffffffffa6804ffc>] ? down_read+0x1c/0x30
Jan 19 22:22:38 debian kernel: [39390.500199] [<ffffffffa64064ac>] ? iterate_supers+0x9c/0x100
Jan 19 22:22:38 debian kernel: [39390.500204] [<ffffffffa6436bc2>] ? sys_sync+0x42/0xb0
Jan 19 22:22:38 debian kernel: [39390.500208] [<ffffffffa6203b1c>] ? do_syscall_64+0x7c/0xf0
Jan 19 22:22:38 debian kernel: [39390.500213] [<ffffffffa68076ee>] ? entry_SYSCALL64_slow_path+0x25/0x25
Trying to 'lxc-stop -n centos' (name of the container) also does not
work. A KVM machine with CentOS 7 on the same host works without
problems. From the host machine, the container's processes look like
this:
$ ps auwwx|grep 231072
231072 6381 0.0 0.0 42544 2100 ? Ds 22:10 0:00 /sbin/init
231072 6708 0.0 0.1 113380 14640 ? Ss 22:12 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /var/run/dhclient-eth0.pid -H centos eth0
231072 6951 0.0 0.0 115396 3160 pts/17 Ss 22:15 0:00 /bin/bash
231072 6958 0.0 0.0 107912 616 pts/17 D+ 22:15 0:00 sync
231072 7211 0.0 0.0 115396 3168 pts/18 Ss 22:16 0:00 /bin/bash
231072 7352 0.0 0.0 128432 2576 pts/18 D+ 22:18 0:00 halt -f
231072 7525 0.0 0.0 115396 3116 pts/20 Ss 22:19 0:00 /bin/bash
231072 7589 0.0 0.0 128432 2796 pts/20 D+ 22:19 0:00 halt -n -f
toni 10162 0.0 0.0 12720 980 pts/5 S+ 22:49 0:00 grep --color=auto 231072
231072 27515 0.0 0.0 1508 8 ? Ss 18:55 0:00 /sbin/init
231072 27827 0.0 0.0 1508 48 ? Ss 18:55 0:00 /sbin/syslogd -Z
231072 27848 0.0 0.0 1512 48 ? Ss 18:55 0:00 /usr/sbin/crond -c /etc/crontabs
231072 27917 0.0 0.0 1508 48 ? Ss 18:56 0:00 udhcpc -b -p /var/run/udhcpc.eth0.pid -i eth0 -x hostname:alpine
231072 27944 0.0 0.0 7320 308 ? S 18:56 0:00 /usr/sbin/sshd
231072 27948 0.0 0.0 1508 4 pts/0 Ss+ 18:56 0:00 /sbin/getty 38400 tty1
231072 27949 0.0 0.0 1508 4 pts/1 Ss+ 18:56 0:00 /sbin/getty 38400 tty2
231072 27950 0.0 0.0 1508 4 pts/2 Ss+ 18:56 0:00 /sbin/getty 38400 tty3
231072 27951 0.0 0.0 1508 4 pts/3 Ss+ 18:56 0:00 /sbin/getty 38400 tty4
231072 27952 0.0 0.0 1508 4 pts/9 Ss+ 18:56 0:00 /sbin/getty 38400 console
It would be great if lxc could prevent a container misbehaving like
that. There's a side effect on the host machine: Since the 'init'
process of the container hangs, the host machine cannot shut down
properly anymore.
Cheers,
Toni
-- System Information:
Debian Release: 9.3
APT prefers stable
APT policy: (990, 'stable'), (90, 'testing'), (70, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 4.9.0-5-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.utf8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages lxc depends on:
ii init-system-helpers 1.48
ii libapparmor1 2.11.0-3
ii libc6 2.24-11+deb9u1
ii libcap2 1:2.25-1
ii libgnutls30 3.5.8-5+deb9u3
ii liblxc1 1:2.0.7-2+deb9u1
ii libseccomp2 2.3.1-2.1
ii libselinux1 2.6-3+b3
ii lsb-base 9.20161125
ii python3 3.5.3-1
ii python3-lxc 1:2.0.7-2+deb9u1
Versions of packages lxc recommends:
ii bridge-utils 1.5-13+deb9u1
ii debootstrap 1.0.89
ii dirmngr 2.1.18-8~deb9u1
ii dnsmasq-base 2.76-5+deb9u1
ii gnupg 2.1.18-8~deb9u1
ii iptables 1.6.0+snapshot20161117-6
ii libpam-cgfs 2.0.7-1
ii lxcfs 2.0.7-1
ii openssl 1.1.0f-3+deb9u1
ii rsync 3.1.2-1+deb9u1
ii uidmap 1:4.4-4.1
Versions of packages lxc suggests:
pn apparmor <none>
ii btrfs-tools 4.7.3-1
ii lvm2 2.02.168-2
-- Configuration Files:
/etc/lxc/default.conf changed:
lxc.network.type = veth
lxc.network.link = virbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx
-- no debconf information
More information about the Pkg-lxc-devel
mailing list