Bug#798625: systemd-networkd: Runs arbitrary inappropriate scripts on network changes
josh at joshtriplett.org
josh at joshtriplett.org
Fri Sep 11 18:58:39 BST 2015
On Fri, Sep 11, 2015 at 09:41:34AM +0200, Martin Pitt wrote:
> Josh Triplett [2015-09-10 23:54 -0700]:
> > * Make networkd call if-up.d/ scripts when it brings up interfaces, to
> > become compatible with ifupdown and NetworkManager for packages shipping
> > hooks. (LP: #1492129)
> >
> > (Along with various other changes related to these hooks.)
> >
> > This is an *extremely* bad idea; please revert it before any package
> > incorrectly starts to rely on it. And this should have been discussed
> > on at least pkg-systemd-maintainers, if not systemd-devel, before being
> > implemented.
>
> FTR, this was discussed a few months ago with Tom Gundersen (the
> author of networkd) at the UOS discussion for this:
>
> https://blueprints.launchpad.net/ubuntu/+spec/foundations-w-networkd-vs-ifupdown
What was his response? I'd be quite surprised to hear any upstream
systemd or networkd developers favoring this approach.
In any case, I'd really appreciate this being discussed more broadly,
ideally on systemd-devel, to get some feedback there.
The proposal there also seems somewhat different than the one
implemented here.
For that matter, it seems like there's a race condition here: if
resolveconf is going to pull in systemd-resolved's resolv.conf, networkd
hooks can't guarantee that systemd-resolved actually creates that
*before* those hooks run. Perhaps resolveconf could start using inotify
(along with a list of places to look for generated resolv.conf files)?
> > Several reasons why this is a bad idea:
> >
> > - networkd is intended to bring up interfaces *quickly*, on the order of
> > microseconds (not milliseconds) even with DHCP, let alone without.
> > Spawning arbitrary processes, and especially shell scripts, is not and
> > will never be compatible with networkd's performance requirements.
>
> The hooks are run asynchronously and don't block networkd.
Interesting; as far as I know, that's a change compared to how other as
far as I know those hooks are normally run synchronously by other
software. The changelog didn't give any indication about asynchronous
invocation. I'd still be concerned about whether this adds any delay to
the fast-path, though a bit less so. Moving this to use networkd's
existing event mechanism seems preferable to spawning a process from
networkd.
> > - These hooks don't exist upstream. Packages shipping if-up.d hooks are
> > thus still broken anywhere other than Debian, and even *in* Debian
> > they're broken with dynamic network configuration. Those package need
> > fixing (upstream) to handle dynamic network configuration, and once
> > they do, the Debian-specific hooks become obsolete. Allowing these
> > hooks makes it less obvious that the packages themselves need fixing.
>
> I do agree that it would be better to fix openssh, postfix, ntpdate,
> avahi-autoipd, and the zillion other packages that ship if-up.d hooks
> (and other if-*.d/ hooks, but these are much less important).
> However, IMHO this shouldn't be a blocker for adopting networkd -- I'd
> like networkd to work in Debian as well as ifupdown and
> NetworkManager; providing if* hook and resolvconf integration should
> bring it up to par with NetworkManager, and thus I believe people can
> now actually use this. Without these, a lot of things would suddenly
> not work as expected any more, like openssh not listening to newly
> brought up interfaces or postfix not flushing the mail queue when the
> system gets online (in a hotplug environment).
Hardly a zillion packages; here's a complete list of Debian packages:
$ zgrep '^etc/network/if-up\.d' Contents-amd64.gz
etc/network/if-up.d/000resolvconf net/openresolv,net/resolvconf
etc/network/if-up.d/00README.linkdir net/ifupdown-scripts-zg2
etc/network/if-up.d/00check-network-cable admin/ifupdown-extra
etc/network/if-up.d/10check-duplicate-ip admin/ifupdown-extra
etc/network/if-up.d/20static-routes admin/ifupdown-extra
etc/network/if-up.d/30check-gateway admin/ifupdown-extra
etc/network/if-up.d/51guidedog net/guidedog
etc/network/if-up.d/arping net/arping
etc/network/if-up.d/avahi-autoipd net/avahi-autoipd
etc/network/if-up.d/avahi-daemon net/avahi-daemon
etc/network/if-up.d/bind9 net/bind9
etc/network/if-up.d/clamav-freshclam-ifupdown utils/clamav-freshclam
etc/network/if-up.d/controlaula.ifup admin/controlaula
etc/network/if-up.d/epoptes-client admin/epoptes-client
etc/network/if-up.d/ethtool net/ethtool
etc/network/if-up.d/gogoc net/gogoc
etc/network/if-up.d/htpdate net/htpdate
etc/network/if-up.d/ifenslave net/ifenslave
etc/network/if-up.d/ifmetric net/ifmetric
etc/network/if-up.d/ifupdown-multi admin/ifupdown-multi
etc/network/if-up.d/ip misc/vlan
etc/network/if-up.d/isatapd net/isatapd
etc/network/if-up.d/lprng net/lprng
etc/network/if-up.d/miredo net/miredo
etc/network/if-up.d/mountnfs admin/initscripts
etc/network/if-up.d/nslcd admin/nslcd
etc/network/if-up.d/ntpdate net/ntpdate
etc/network/if-up.d/openntpd net/openntpd
etc/network/if-up.d/openssh-server net/openssh-server
etc/network/if-up.d/openvpn net/openvpn
etc/network/if-up.d/postfix mail/postfix
etc/network/if-up.d/sendmail mail/sendmail-base
etc/network/if-up.d/shorewall net/shorewall-init
etc/network/if-up.d/slrn news/slrn
etc/network/if-up.d/slrnpull news/slrnpull
etc/network/if-up.d/tinc net/tinc
etc/network/if-up.d/ucarp net/ucarp
etc/network/if-up.d/uml-utilities otherosfs/uml-utilities
etc/network/if-up.d/upstart admin/ifupdown
etc/network/if-up.d/uruk net/uruk
etc/network/if-up.d/vzifup-post admin/vzctl
etc/network/if-up.d/wide-dhcpv6-client net/wide-dhcpv6-client
etc/network/if-up.d/wpasupplicant net/wpasupplicant
This seems quite manageable, especially once you rule out the ones that
make no sense at all with networkd (or with anything other than
ifupdown). I'd be happy to help evaluate these further.
In the absence of the wireless support and desktop integration that
NetworkManager has, networkd seems unlikely to get used in
non-server environments anytime soon. As far as I know, wireless
support is being worked on, and desktop integration will likely wait for
that.
I'd like to see networkd more widely used too, but not at the expense of
diverging this drastically from upstream in an area where doing so
creates an API and an expectation for future support of that API. If we
start doing this, trying to remove it in the future will *hurt*. (As
will dealing with portability between distributions.)
For OpenSSH, and for any software that just re-listens when new
interfaces come up, they should just listen to INADDR_ANY, rather than
caring what specific interfaces are up or down. Any software that
listens on INADDR_ANY shouldn't need any changes. And even with
specific ListenAddress lines (or equivalent for other software), OpenSSH
and similar daemons could use IP_FREEBIND (see "man 7 ip") to start
listening before any interface has that address, and it would again Just
Work without ever reloading the daemon. (Preempting the obvious issue:
IP_FREEBIND may not work on arbitrary non-Linux systems, but then
neither will networkd ;) On any system networkd can run on, dynamically
reloading OpenSSH and other listening daemons shouldn't be needed.)
ntpd handles dynamic network configurations just fine, as does
timesyncd; ntpdate *never* handled dynamic network configurations, and I
don't think it makes sense to work around that with scripts when real
NTP daemons do the right thing automatically. ntpdate (and other
software like tlsdate) makes sense as a standalone tool, but invoking it
automatically seems less sensible. (Plus, it makes more sense to
continuously keep time up to date, rather than only at boot or network
change.)
As far as I know, avahi-autoipd is entirely obsoleted by networkd, which
assigns link-local addresses itself; see the "LinkLocalAddressing"
option in "man systemd.network".
> > - Network configuration can change at any time, and networkd is not
> > stateful; state lives in the kernel, not in networkd. These hooks
> > break that assumption.
>
> Can you please explain this? How is this different from ifupdown and NM?
ifupdown is stateful; it remembers whether it thinks the state is up or
down, independently from the hardware. (That sometimes causes prolems.)
NetworkManager, similarly, has quite a bit of state remembered in the
NetworkManager daemon; to the extent the hardware changes "out from
under it", NetworkManager reacts by updating its state and possibly
taking other actions, but fundamentally NetworkManager considers itself
the canonical location of network state, rather than just something that
manipulates network state.
networkd intentionally *doesn't* maintain such state. In
particular, if networkd has no ongoing work to do, it can exit. And on
systems that need networking in the initramfs (such as systems that need
the network for the root filesystem), networkd can run in the initramfs,
configure those interfaces, *exit*, and then re-run after switching to
the root filesystem. Finally, networkd has never had the problem that
NetworkManager has of assuming it should manage every interface on the
system; it manages what you tell it to.o
Any daemon or other software that has /etc/network/if-*.d/* hooks will
also break if someone manually brings an interface up or down, yanks out
a network cable, etc. Sysadmins shouldn't have to run such scripts.
Network software should just say "oh, look, stuff changed, I'll adapt".
> > (This will also likely break with future
> > changes to networkd and other packages integrating with it, as well as
> > with other types of interfaces or virtual networks networkd can
> > configure.) Among other things, as the systemd-networkd manpage
> > documents, "Network configurations applied before networkd is started
> > are not removed, and static configuration applied by networkd is not
> > removed when networkd exits. Dynamic configuration applied by networkd
> > may also optionally be left in place on shutdown. This ensures
> > restarting networkd does not cut the network connection, and, in
> > particular, that it is safe to transition between the initrd and the
> > real root, and back."
>
> Right, and consequently it won't call if-post-down.d/ when you stop
> networkd.
And what happens if you then re-start networkd? Will it call if-up.d/*
? How confident are you that every such script in Debian is idempotent?
Or if it doesn't run those scripts when re-started, then those scripts
won't run at all if the network was already set up (such as in virtual
environments, when networkd ran in the initramfs, or if some other
software configured the network interface and handed it to networkd,
which is currently the plan for how some other types of software will
extend networkd).
Will those scripts work if networkd brings up a bridge? Or a bond? Or
a virtual network interface? Or a network with no non-link-local IP?
> > - Several of the existing if-up.d and if-post-down.d hooks should not
> > run under networkd. Among others: wpasupplicant's hooks shouldn't run
> > at all under anything but ifupdown
>
> Why not? If it's not meant to run under NM or other mechanisms, it
> should check $METHOD. But right now the hook doesn't do anything under
> !ifupdown anyway as none of the $WPA_* and $IF_* vars is supplied.
Sure it does: it runs, and checks environment variables. :)
But that's a convenient way to filter out scripts from the above list
that we don't need to find replacements for.
> That said, as networkd doesn't have wifi support by itself, it would
> certainly be *nice* to make wpa_supplicant work OOTB with networkd :-)
Not that way, though; not least of which because the ordering would be
backward, since the wifi link layer needs to run (and complete) *before*
networkd can set up an interface. There's a plan to do that upstream;
as I understand it, the plan involves using a wifi configuration daemon
that will set up the link layer and then hand the interface to networkd
to configure at the IP layer. Hooks aren't going to suffice there.
> > , mountnfs's hooks shouldn't run (because they conflict with several
> > other approaches to nfs handling that integrate properly with
> > systemd)
>
> That sounds like a real issue indeed, do you have some details? Again,
> how is that not affecting NetworkManager?
See some of the comments at the top of mountnfs. mountnfs calls into
various /etc/init.d/* scripts, and mounts filesystems from /etc/fstab.
systemd already handles mounting filesystems, and has its own logic for
making sure network filesystems don't get mounted before bringing up the
network.
Because of NetworkManager's target use case and system type, I doubt
many people run NetworkManager on systems that have NFS filesystems
listed in /etc/fstab, let alone critical ones. :)
Also see bugs filed against systemd and nfs about the integration
between the two and various failure modes. We don't need to throw
*another* complex mechanism into the mix when people are already working
on properly integrated solutions.
> > , avahi-daemon's hook is responsible for numerous problems and
> > slowdowns even under ifupdown
>
> This just disables avahi if there is a "real" .local domain. This
> seems rather independent of the mechanism used to bring up networking?
It can cause *long* delays bringing up and configuring interfaces (including
under NetworkManager).
Also see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=785325 .
> > and wireless-tools' hook shouldn't run under anything but
> > ifupdown.
>
> Same question/issues as for wpasupplicant?
Same answer.
> > - Calling if-up.d and if-post-down.d, but not calling if-down.d or
> > if-pre-up.d, may well break assumptions that a family of scripts in
> > those directories have about when they'll be called and what state
> > machine they'll go through.
>
> NetworkManager has done just that for many years. if-pre-up.d/ is
> rather specific to ifupdown indeed (configuring device drivers and
> stuff), and if-down.d/ has been a promise which we've never been able
> to keep anyway since we have hotplugging and wifi.
NetworkManager doing it was a bad idea too. :)
> (FTR, I consider NM now calling if-pre-up.d/ and down.d/ not an
> improvement; but that's another story)
It's an argument that networkd shouldn't go down the same road, though.
:)
> > Packages shipping if-up.d or if-post-down.d scripts are not compatible
> > with networkd. Primarily because they aren't compatible with
> > dynamically changing network configurations, and secondarily because
> > they tend to do the kind of really silly things that happen with
> > arbitrary shell-script hooks available. This is not the right way to
> > fix that problem.
>
> Well, we have two options: Make networkd work with Debian packages
> today, or wait an infinite time until all packages listen to dynamic
> network changes by themselves. For some this totally should be done
> (openssh), for others it's impractical as they don't have a
> permanently running daemon.
There's a third option: Fix critical packages like OpenSSH (which I'd be
happy to help with) and file bugs on the rest.
> > What specific problem is this trying to solve?
>
> See above -- making use of Debian's integration logic for a lot of
> packages, to be on par with ifupdown and NetworkManager.
I'm asking to find out if there's some specific package or packages
whose lack of such integration motivated the change, primarily to scope
the problem of fixing those packages.
This problem is not unbounded; I provided a complete list of Debian
packages above, and the list of Ubuntu universe packages is likely
similar, with the list of non-universe packages even shorter.
> That said: If Michael, or a majority on debian-devel@ or the TC or
> whatever disagree with this, I'm okay with dropping this from Debian
> again (and just keep it in the Ubuntu package for now). I don't feel
> like having a big fight over this. :-)
I really don't want this to be acrimonious, and I certainly have no
intention of invoking the TC. Let's talk this through and work out a
solution. And I'm happy to help.
What I'm asking, though, is that before any packages start *relying* on
networkd invoking these scripts (which thus becomes an API with
potential backward-compatibility commitments), this change be reverted
(or at the very least stuck in experimental or hidden behind a
disabled-by-default configuration option), and we figure out how much
work it would be to fix the critical packages that ship these scripts to
do the right thing on *all* systems.
- Josh Triplett
More information about the Pkg-systemd-maintainers
mailing list