[Pkg-iscsi-maintainers] Bug#809334: Bug#809334: Bug#809320: open-iscsi: no session termination on stop

Sun Feb 7 09:41:35 UTC 2016

On Wed, 2015-12-30 at 13:34 +0100, Christian Seiler wrote:
> On 12/30/2015 12:35 PM, Ritesh Raj Sarraf wrote:
> > But the change you've proposed has other repercussion. It breaks
> > shutdown, because now, iscsi service/sessions are terminated way
> > early.
> > This leaves, effectively, a hung sanboot system.
> > 
> > Please see attached images.
> > 
> > I also didn't like that iscsi shutdown, even thought my rootfs was
> > running.
> > 
> > I'll keep digging.
> 
> Ok, so I think there's two things going on here: a problem in a
> change I made in that doesn't work in your specific configuration
> (which we should debug) and a general question about how to
> handle shutdown.
> 
> First of all: the fact that iSCSI volumes are shut down is a
> change I made on purpose. There are two parts to this: first of
> all, iscsid is going to be killed anyway,[*] but the kernel will
> keep running sessions open regardless (unless there's an error
> with the connection / the target). Secondly, which is the change
> I did make, is that iSCSI sessions are logged out on shutdown
> regardless of whether the rootfs is on iSCSI. I did that because
> I think it is still correct to log out of sessions that the
> rootfs is NOT on, because we want to cleanly logout of anything
> that is possible to log out of. There is a flag in
> /etc/default/open-iscsi to return to the old behavior, this is
> also mentioned in the NEWS file, which is shown on updates, but
> not on new installs. Note that this does not include the rootfs
> sessions - they SHOULD be excluded. (Of course, if you disagree
> with this change and think ALL sessions, not just the rootfs
> sessions, should be kept open by default, we could discuss this
> separately.)
> 

This is what is happening right now.

root at debian-sanboot:~# systemctl stop open-iscsi
root at debian-sanboot:~# systemctl status open-iscsi
● open-iscsi.service - Login to default iSCSI targets
   Loaded: loaded (/lib/systemd/system/open-iscsi.service; enabled;
vendor preset: enabled)
   Active: inactive (dead) since Sun 2016-02-07 14:45:11 IST; 7s ago
     Docs: man:iscsiadm(8)
           man:iscsid(8)
  Process: 971 ExecStop=/lib/open-iscsi/logout-all.sh (code=exited,
status=0/SUCCESS)
  Process: 968 ExecStop=/bin/sync (code=exited, status=0/SUCCESS)
  Process: 788 ExecStop=/lib/open-iscsi/umountiscsi.sh (code=exited,
status=0/SUCCESS)
  Process: 601 ExecStart=/lib/open-iscsi/activate-storage.sh
(code=exited, status=0/SUCCESS)
  Process: 567 ExecStart=/sbin/iscsiadm -m node --loginall=automatic
(code=exited, status=15)
  Process: 559 ExecStartPre=/bin/systemctl --quiet is-active
iscsid.service (code=exited, status=0/SUCCESS)
 Main PID: 601 (code=exited, status=0/SUCCESS)

Feb 07 14:45:08 debian-sanboot umountiscsi.sh[788]: ls: cannot access
/dev/sdd?*: No such ...ry
Feb 07 14:45:09 debian-sanboot logout-all.sh[971]: Logging out of
session [sid: 1, target:...0]
Feb 07 14:45:09 debian-sanboot logout-all.sh[971]: Logout of [sid: 1,
target: iqn.2003-01....l.
Feb 07 14:45:10 debian-sanboot logout-all.sh[971]: Logging out of
session [sid: 2, target:...0]
Feb 07 14:45:10 debian-sanboot logout-all.sh[971]: Logout of [sid: 2,
target: iqn.2003-01....l.
Feb 07 14:45:10 debian-sanboot logout-all.sh[971]: Logging out of
session [sid: 3, target:...0]
Feb 07 14:45:11 debian-sanboot logout-all.sh[971]: Logout of [sid: 3,
target: iqn.2003-01....l.
Feb 07 14:45:11 debian-sanboot logout-all.sh[971]: Logging out of
session [sid: 4, target:...0]
Feb 07 14:45:11 debian-sanboot logout-all.sh[971]: Logout of [sid: 4,
target: iqn.2003-01....l.
Feb 07 14:45:11 debian-sanboot systemd[1]: Stopped Login to default
iSCSI targets.
Hint: Some lines were ellipsized, use -l to show in full.

It terminated all the connections, which it should not.

root at debian-sanboot:~# systemctl stop open-iscsi
root at debian-sanboot:~# iscsiadm -m session
iscsiadm: No active sessions.

> Now for the problem in your case: there is code in the open-iscsi
> package in unstable that tries to make sure we don't close
> sessions where the rootfs is on. This consists of two steps:
> 
> 1) /lib/open-iscsi/umountiscsi.sh [#] will try to umount
> filesystems on iSCSI. At the same time it detects sessions that
> contain the root filesystem and writes them out into the file
> /run/open-iscsi/shutdown-keep-sessions.
> 

Looks like this invocation is problematic

root at debian-sanboot:~# /lib/open-iscsi/umountiscsi.sh 
ls: cannot access /dev/sda?*: No such file or directory
ls: cannot access /dev/sdb?*: No such file or directory
ls: cannot access /dev/sdc?*: No such file or directory
ls: cannot access /dev/sdd?*: No such file or directory

> 2) /lib/open-iscsi/logout-all.sh [~] will detect if the file
> /run/open-iscsi/shutdown-keep-sessions exists - and if it does
> it will *NOT* run the default iscsiadm -m node --logoutall=all
> but rather log out of all sessions that were not excluded.
> 
> So what happens in your case is that umountiscsi.sh likely
> doesn't properly detect your setup where your rootfs is on
> iSCSI - and therefore doesn't exclude the relevant sessions
> from logout.
> 
> To test this, you should run the /lib/open-iscsi/umountiscsi.sh
> script manually - that should *not* try to umount anything
> (because it should detect that it's the rootfs) and write out
> /run/open-iscsi/shutdown-keep-sessions. You should check that
> that file contains all the sessions that your rootfs relies
> on.
> 
> Hope that gives you enough pointers to debug this.
> 

From what you've explained, this is what I've gotten below.

root at debian-sanboot:~# bash /lib/open-iscsi/umountiscsi.sh ^C
root at debian-sanboot:~# cat /run/open-iscsi/shutdown-keep-sessions 
/sys/devices/platform/host2/session1
/sys/devices/platform/host3/session2
/sys/devices/platform/host4/session3
/sys/devices/platform/host5/session4
root at debian-sanboot:~# iscsiadm -m session
tcp: [1] 172.16.20.40:3260,1 iqn.2003-01.org.linux-iscsi.debian.sanboot 
(non-flash)
tcp: [2] 172.16.20.41:3260,1 iqn.2003-01.org.linux-iscsi.debian.sanboot 
(non-flash)
tcp: [3] 172.16.20.42:3260,1 iqn.2003-01.org.linux-iscsi.debian.sanboot 
(non-flash)
tcp: [4] 172.16.20.43:3260,1 iqn.2003-01.org.linux-iscsi.debian.sanboot 
(non-flash)

But with all this setting, it hangs the shutdown, because all session
are then terminated.

> Regards,
> Christian
> 
> [*] systemd tracks processes, so it's going to be killed. There
> is a possibility to avoid this, but this requires changes to
> iscsid and will be rather complicated to get the integration
> right. Not saying we shouldn't consider this, but just saying
> that's complicated, and is something that's much lower on my
> TODO list.
> 
> [#] http://anonscm.debian.org/cgit/pkg-iscsi/open-iscsi.git/tree/debi
> an/extra/umountiscsi.sh
> 
> [~] http://anonscm.debian.org/cgit/pkg-iscsi/open-iscsi.git/tree/debi
> an/extra/logout-all.sh
> 

root at debian-sanboot:~# systemctl stop open-iscsi

root at debian-sanboot:~# cat /run/open-iscsi/shutdown-keep-sessions 
/sys/devices/platform/host2/session1
/sys/devices/platform/host3/session2
/sys/devices/platform/host4/session3
/sys/devices/platform/host5/session4

root at debian-sanboot:~# iscsiadm -m session
iscsiadm: No active sessions.

From the workflow you've explained so far, the mention of certain sessions in /run/o-i/s-k-s should have been honored.

-- 
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: This is a digitally signed message part
URL: <http://lists.alioth.debian.org/pipermail/pkg-iscsi-maintainers/attachments/20160207/094477e4/attachment-0003.sig>