Bug#776276: unblock: open-iscsi/2.0.873+git0.3b4b4500-4

[Christian Seiler]

(Since I didn't get cc'd in the original reply, I'm replying here. Sorry
about that.)

Am 26.01.2015 um 16:12 schrieb Ritesh Raj Sarraf:
> On 01/26/2015 07:54 PM, Michael Biebl wrote:
>>> unblock open-iscsi/2.0.873+git0.3b4b4500-4
>> That patch doesn't look right.
>>
>> Calling systemctl from an init script is a big no-go.
>>
>> Second, shipping a generated unit file which does run /etc/init.d/foo is
>> a hack at best.
>>
>> I'd be really unhappy, if this was accepted.
>>
> 
> Well, this is just an interim fix, for Jessie. For sid and later, we'll
> also ship the systemd files that'll be independent.

Just for reference: I proposed the original patch, and I spent a lot of
time looking at different solutions.

In the end I think the whole startup logic of iSCSI should be reworked.
There are lots of corner cases that are not properly handled currently,
even on sysvinit. HOWEVER, that would require quite a bit of effort and
even a lot more testing, and possibly coordinated changes to quite a lot
of other pieces of software. And I don't think this is something that
would fit the current release policy.

On the other hand, currently there are the following problems due to
this init script:

 - 90s delay on a default Jessie installation (systemd as init) with
   the most basic iSCSI configuration you can think of (so nearly
   everyone using iSCSI initiator on Jessie will run into this)

 - startup ordering is botched on systemd, i.e. if you have services
   that require filesystems on iSCSI to be present (file servers, ...)
   you might have a race on your hand

   (This is not the case in sysvinit, because rcS.d scripts are always
   executed before rc[2-5].d scripts, so there's an additional sync
   point that doesn't exist with systemd.)

 - conversely, shutdown ordering is also botched, because open-iscsi
   has no Before= ordering, so systemd thinks it can stop it
   immediately, when stuff is potentially still using iSCSI filesystems
   (and if you sever an iSCSI connection that's in use, the kernel
   will simply do that)

 - but even if the filesystems weren't used by any services: since the
   call-out to umountiscsi.sh doesn't work (systemd keeps track of
   service state and thinks it's already stopped).

   That means that iSCSI connections might be severed before
   filesystems are unmounted, potentiall causing data loss.

   I couldn't trigger this on my system the couple of times I tried
   it because systemd also unmounts all network filesystems on its own,
   and that was always faster on my box, but those FS were empty and
   no cache had to be written to them, so my guess is that this will
   occur at least sometimes in the wild.

So then I thought: what's the least-invasive way to fix the problems
with this so it works at least as well as with sysvinit? And that was
the attached debdiff that Ritesh was kind enough to request a
pre-approval for.

This is a targeted fix for a specific issue, without making too many
changes this close to the release. Since the open-iscsi init script
never worked well with systemd (in Wheezy it's kind of broken), but lots
of sysvinit systems are using it, my goal was to make sure that nothing
changes for people that continue using sysvinit this late in the release
process and just fix it so systemd systems work at least in principle
(with a cleanup post-Jessie).

Of course, if you have a better idea of how to do this?

>> A few questions:
>> - If open-iscsi is supposed to provide remote file systems, how can it
>> have # Required-Start: $remote_fs

As far as I can tell, the historic reason is probably that open-iscsi is
installed in /usr and if /usr is on NFS on sysvinit, open-iscsi won't
work unless remote filesystems have been mounted.

Since sysvinit ignores mount errors and has an additional sync point
between runlevel S and runlevels 2-5, this always kind-of worked on
sysvinit.

Since /usr on NFS without premounting it in initramfs doesn't really
work with systemd anyway, this dependency doesn't have to be there for
systemd.

But changing the init script would probably break some sysvinit setups.

>> That's a classic circular dep and most likely the reason for the issue

Yes, obviously, which is why for systemd the dependencies changed to
make it Before=remote-fs-pre.target.

>> - Why is umountiscsi.sh in a separate init script, when the open-iscsi
>> init script calls it via invoke-rc.d on stop?

To be honest, no idea. If I were do create a new package from scratch,
there shouldn't be a umountiscsi.sh, but everything should simply be
done in open-iscsi directly. And for post-Jessie that may well be the
way to go, BUT I'm not sure it would be wise to do this this late in the
Jessie release cycle.

But if you don't like the systemctl start, perhaps one could just do

_SYSTEMCTL_SKIP_REDIRECT=true /etc/init.d/umountiscsi.sh stop

in the stop portion to make sure that the script is called directly even
with systemd?

(Although that gets rid of policy-rc.d handling, but I don't know if
anybody actually uses it for this purpose or if invoke-rc.d is just in
there because of some 'best practices for invoking init scripts'.)

Christian