<div dir="ltr">Huh? I don't think you read the bug. You seem to be confusing bug symptoms with intent.<div><br></div><div>Why would anyone *try* to run two dhcpds off the same config file and corrupt their lease database and confuse a failover partner? The intent here is to run a perfectly normal dhcpd setup.<div><br></div><div>But the pid file is used for concurrency protection by dhcpd. That "rm -f" of the pid file outside of dhcpd interferes with this locking and introduces a race condition that allows two dhcpds to start, which is a *bad thing*, hence the bug report.</div><div><br></div><div>My guess is it's almost never safe to remove a pid file like this, though for daemons that have other concurrency protection (such as bind() failures), it would not be as important as it is for dhcpd.</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Sep 6, 2015 at 7:30 AM, Debian Bug Tracking System <span dir="ltr"><<a href="mailto:owner@bugs.debian.org" target="_blank">owner@bugs.debian.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This is an automatic notification regarding your Bug report<br>
which was filed against the isc-dhcp-server package:<br>
<br>
#704175: isc-dhcp-server: init script removes dhcpd.pid<br>
<br>
It has been closed by Michael Gilbert <<a href="mailto:mgilbert@debian.org">mgilbert@debian.org</a>>.<br>
<br>
Their explanation is attached below along with your original report.<br>
If this explanation is unsatisfactory and you have not received a<br>
better one in a separate message then please contact Michael Gilbert <<a href="mailto:mgilbert@debian.org">mgilbert@debian.org</a>> by<br>
replying to this email.<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
704175: <a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=704175" rel="noreferrer" target="_blank">http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=704175</a><br>
Debian Bug Tracking System<br>
Contact <a href="mailto:owner@bugs.debian.org">owner@bugs.debian.org</a> with problems<br>
</font></span><br><br>---------- Forwarded message ----------<br>From: Michael Gilbert <<a href="mailto:mgilbert@debian.org">mgilbert@debian.org</a>><br>To: <a href="mailto:704175-close@bugs.debian.org">704175-close@bugs.debian.org</a><br>Cc: <br>Date: Sat, 5 Sep 2015 19:27:21 -0400<br>Subject: Re: [pkg-dhcp-devel] Bug#704175: isc-dhcp-server: init script removes dhcpd.pid<br>On Thu, Mar 28, 2013 at 5:55 PM, Alan Sundell wrote:<br>
> In debian/isc-dhcp-server.init.d, there is the following code:<br>
><br>
> stop)<br>
> log_daemon_msg "Stopping $DESC" "$NAME"<br>
> start-stop-daemon --stop --quiet --pidfile "$DHCPD_PID"<br>
> log_end_msg $?<br>
> rm -f "$DHCPD_PID"<br>
> ;;<br>
><br>
> So, you can end up in a situation like this:<br>
> process a: stops dhcpd<br>
> process b: starts dhcpd, which writes pid file<br>
> process a: removes pid file<br>
> process a: starts dhcpd, which writes another pid file<br>
<br>
I think you want to use different DHCPD_CONF files for your two server<br>
processes, each with different pid-file-name settings, to avoid this<br>
problem.<br>
<br>
Best wishes,<br>
Mike<br><br>---------- Forwarded message ----------<br>From: Alan Sundell <<a href="mailto:sundell@gmail.com">sundell@gmail.com</a>><br>To: Debian Bug Tracking System <<a href="mailto:submit@bugs.debian.org">submit@bugs.debian.org</a>><br>Cc: <br>Date: Thu, 28 Mar 2013 17:55:55 -0400<br>Subject: isc-dhcp-server: init script removes dhcpd.pid<br>Package: isc-dhcp-server<br>
Version: 4.2.4-5<br>
Severity: important<br>
<br>
Dear Maintainer,<br>
<br>
In debian/isc-dhcp-server.init.d, there is the following code:<br>
<br>
stop)<br>
log_daemon_msg "Stopping $DESC" "$NAME"<br>
start-stop-daemon --stop --quiet --pidfile "$DHCPD_PID"<br>
log_end_msg $?<br>
rm -f "$DHCPD_PID"<br>
;;<br>
<br>
So, you can end up in a situation like this:<br>
process a: stops dhcpd<br>
process b: starts dhcpd, which writes pid file<br>
process a: removes pid file<br>
process a: starts dhcpd, which writes another pid file<br>
<br>
Here's an strace from when I removed the pidfile and started dhcpd<br>
immediately afterwards, while another dhcpd was running:<br>
<br>
10698 13:47:26 bind(5, {sa_family=AF_PACKET, proto=0x6574, if12392, pkttype=PACKET_HOST, addr(0)={0, }, 16) = 0<br>
10698 13:47:26 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 7<br>
10698 13:47:26 ioctl(7, SIOCGIFHWADDR, {ifr_name="eth0", ifr_hwaddr=aa:00:00:15:cb:1d}) = 0<br>
10698 13:47:26 close(7) = 0<br>
10698 13:47:26 setsockopt(5, SOL_SOCKET, SO_ATTACH_FILTER, "\v\0\0\0\0\0\0\0\340\373\237/\35\177\0\0", 16) = 0<br>
10698 13:47:26 fcntl(5, F_SETFD, FD_CLOEXEC) = 0<br>
10698 13:47:26 socket(PF_PACKET, SOCK_PACKET, 768) = 7<br>
10698 13:47:26 bind(7, {sa_family=AF_PACKET, proto=0x6c6f, if0, pkttype=PACKET_HOST, addr(0)={0, }, 16) = 0<br>
10698 13:47:26 socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 8<br>
10698 13:47:26 ioctl(8, SIOCGIFHWADDR, {ifr_name="lo", ifr_hwaddr=00:00:00:00:00:00}) = 0<br>
10698 13:47:26 close(8) = 0<br>
10698 13:47:26 setsockopt(7, SOL_SOCKET, SO_ATTACH_FILTER, "\v\0\0\0\0\0\0\0\340\373\237/\35\177\0\0", 16) = 0<br>
10698 13:47:26 fcntl(7, F_SETFD, FD_CLOEXEC) = 0<br>
10698 13:47:26 socket(PF_INET, SOCK_DGRAM, IPPROTO_UDP) = 8<br>
10698 13:47:26 setsockopt(8, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0<br>
10698 13:47:26 bind(8, {sa_family=AF_INET, sin_port=htons(67), sin_addr=inet_addr("0.0.0.0")}, 16) = 0<br>
10698 13:47:26 fcntl(8, F_SETFD, FD_CLOEXEC) = 0<br>
10698 13:47:26 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 9<br>
10698 13:47:26 fcntl(9, F_SETFD, FD_CLOEXEC) = 0<br>
10698 13:47:26 setsockopt(9, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0<br>
10698 13:47:26 bind(9, {sa_family=AF_INET, sin_port=htons(7911), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)<br>
10698 13:47:26 close(9) = 0<br>
10698 13:47:26 close(9) = -1 EBADF (Bad file descriptor)<br>
10698 13:47:26 sendto(3, "<163>May 14 13:47:26 dhcpd: Can'"..., 77, MSG_NOSIGNAL, NULL, 0) = 77<br>
10698 13:47:26 write(6, "\nfailover peer \"xxxxxx\" stat"..., 131) = 131<br>
10698 13:47:26 fsync(6) = 0<br>
10698 13:47:26 sendto(3, "<166>May 14 13:47:26 dhcpd: fail"..., 83, MSG_NOSIGNAL, NULL, 0) = 83<br>
10698 13:47:26 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 9<br>
10698 13:47:26 bind(9, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("172.30.192.69")}, 16) = 0<br>
10698 13:47:26 fcntl(9, F_SETFD, FD_CLOEXEC) = 0<br>
10698 13:47:26 setsockopt(9, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0<br>
10698 13:47:26 fcntl(9, F_SETFL, O_RDONLY|O_NONBLOCK) = 0<br>
10698 13:47:26 connect(9, {sa_family=AF_INET, sin_port=htons(520), sin_addr=inet_addr("172.24.154.70")}, 16) = -1 EINPROGRESS (Operation now in progress)<br>
10698 13:47:26 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 10<br>
10698 13:47:26 fcntl(10, F_SETFD, FD_CLOEXEC) = 0<br>
10698 13:47:26 setsockopt(10, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0<br>
10698 13:47:26 bind(10, {sa_family=AF_INET, sin_port=htons(520), sin_addr=inet_addr("172.30.192.69")}, 16) = -1 EADDRINUSE (Address already in use)<br>
10698 13:47:26 close(10) = 0<br>
10698 13:47:26 close(10) = -1 EBADF (Bad file descriptor)<br>
10698 13:47:26 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f1d2f72a9d0) = 10699<br>
10698 13:47:26 exit_group(0) = ?<br>
10699 13:47:26 open("/var/run/dhcp-server/dhcpd.pid", O_RDONLY) = -1 ENOENT (No such file or directory)<br>
10699 13:47:26 open("/var/run/dhcp-server/dhcpd.pid", O_WRONLY|O_CREAT|O_TRUNC, 0644) = 10<br>
10699 13:47:26 write(10, "10699\n", 6) = 6<br>
<br>
dhcpd has some checks for bind() failures, but they don't seem to be triggered<br>
here (the bind() to port 67 succeeds because no listen() has been called,<br>
because it uses recvfrom(); the preceeding ones are raw sockets, and dhcpd<br>
doesn't seem to treat the other failures as fatal).<br>
<br>
Now we have two dhcpds, and there is no locking around dhcpd.leases, so<br>
they get to fight over writing data there, which causes loss of lease data.<br>
<br>
Things get even more confusing if (as in this case), this is part of a<br>
failover pair. One of the dhcpds will have the TCP ports for inbound<br>
connections from the peer (the other will indefinitely call add_timeout()<br>
to reschedule a bind attempt). Both will be trying to connect to the peer<br>
and resolve conflicts, which will confuse the peer about the state.<br>
<br>
Obviously, there are some serious problems upstream with lack of concurrency<br>
protection in dhcpd (checking if the pid in the pidfile is alive is not<br>
very reliable).<br>
<br>
But I'm not sure why that "rm -f" was added, or what problem it solves,<br>
and it certainly interacts badly with the limited checks that dhcpd does<br>
to prevent multiple copies from running. Hence this bug.<br>
<br>
[Note: I'm filing this from a system that has nothing to do with dhcpd, and<br>
I've marked the version in sid, but AFAIK, this 'rm -f' has been there a long<br>
time. Also not sure about severity -- there is data loss potential, but it's<br>
lease data. Eventually, in failover pairs, the state conflicts will result<br>
in a non-serving pair.]<br>
<br>
-- System Information:<br>
Debian Release: wheezy/sid<br>
APT prefers precise-updates<br>
APT policy: (600, 'precise-updates'), (600, 'precise-security'), (600, 'precise'), (400, 'precise-backports')<br>
Architecture: amd64 (x86_64)<br>
<br>
Kernel: Linux 3.5.0-26-generic (SMP w/4 CPU cores)<br>
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)<br>
Shell: /bin/sh linked to /bin/bash<br>
<br></blockquote></div><br></div>