[pkg-netfilter-team] Bug#1023844: nftables: firewalld segfaults in libnftables.so rule.c cache_init_objects()

[Bernhard Übelacker]

Hello Kayim,
I am not involved in packaging or development of nftables or firewalld,
just trying to help debugging random crashes in Debian.

And want just to mention a maybe possible way of debugging this.
As this is a kind of rare issue and when the core is produced it is
hard to find the place that created the wrong pointers,
it might be possible to record the whole life of the firewalld
process with rr-debugger, and when it hits a segfault replay that
recording to the segfault and then step backwards.

In a test VM I could get this by:
- apt install rr
- echo 1 > /proc/sys/kernel/perf_event_paranoid
- modify /lib/systemd/system/firewalld.service like in [2]
- systemctl daemon-reload
- systemctl stop firewalld
- systemctl start firewalld

This would produce a recording below /tmp/rr/.
And in case some interesting event is inside this recording,
it can be replayed by e.g. "rr replay /tmp/rr/firewalld-0".
(Usually recordings are placed in $HOME/.local/share/rr.)


There are a few points that could make that attempt fail:
- the CPU needs to support recording by rr - just test if "rr record true" is working.
- the recording might increase too fast in size to be usable.
- the bug might not manifest because of the presence of rr.
- rr might not be compatible enough, e.g. firewalld might use some ioctl's that are not yet covered by rr.


As said, this might be a possible way - I don't know if this
bug is important enough to justify this effort.


Last you mention a reload time of 30 minutes - maybe you could
lower that value for the recording machine to make it more likely
to hit it in less time.


Because you say just one machine has the core creation enabled,
I want to mention the package systemd-coredump - that should
handle cores and would print a very basic backtrace to the journal,
without much configuration effort.


Kind regards,
Bernhard




[1] https://github.com/rr-debugger/rr

[2]
--- firewalld.service.orig     2021-02-01 13:54:28.000000000 +0100
+++ firewalld.service       2022-11-27 11:25:39.835585418 +0100
@@ -8,11 +8,11 @@ Conflicts=iptables.service ip6tables.ser
  Documentation=man:firewalld(1)
  
  [Service]
-ExecStart=/usr/sbin/firewalld --nofork --nopid
-ExecReload=/bin/kill -HUP $MAINPID
+ExecStart=/usr/bin/rr record /usr/sbin/firewalld --nofork --nopid
+#ExecReload=/bin/kill -HUP $MAINPID
  # supress to log debug and error output also to /var/log/messages
-StandardOutput=null
-StandardError=null
+#StandardOutput=null
+#StandardError=null
  Type=dbus
  BusName=org.fedoraproject.FirewallD1
  KillMode=mixed