[Pkg-cacti-maint] Bug#1031116: autopkgtest for cacti/1.2.23+ds1-2 fails as dbconfig-common attempts to create database before any server is up

[Otto Kekäläinen]

Package: cacti

(Filing against cacti for now for visibility, actual root cause might
be elsewhere.)

After uploading mariadb 1:10.11.1-4 the autopkgtests for cacti started
failing on all architectures. I suspect the problem is that the
mariadbd server is not running at the time when dbconfig-common
attempts to create credentials for cacti.

Note in log https://ci.debian.net/data/autopkgtest/testing/s390x/d/dbconfig-common/31204959/log.gz
section:

Setting up mariadb-server (1:10.11.1-4) ...
Created symlink
/etc/systemd/system/multi-user.target.wants/mariadb.service →
/lib/systemd/system/mariadb.service.
Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.
...
Determining localhost credentials from /etc/mysql/debian.cnf: succeeded.
...
dbconfig-common: writing config to /etc/dbconfig-common/db-test-multidbtype.conf
debconf (developer): <-- X_LOADTEMPLATEFILE /var/lib/dpkg/info/ucf.templates ucf
debconf (developer): --> 0

Creating config file /etc/dbconfig-common/db-test-multidbtype.conf
with new version
dbc_read_package_config() .
dbc_detect_installed_dbtype() mysql.
_dbc_detect_installed_dbtype() mysql.
dbc_get_admin_pass() .
debconf (developer): <-- FGET db-test-multidbtype/mysql/admin-pass seen
debconf (developer): --> 0 true
debconf (developer): <-- GET db-test-multidbtype/mysql/admin-pass
debconf (developer): --> 0
_dbc_detect_installed_dbtype() mysql.
ERROR 2002 (HY000): Can't connect to local server through socket
'/run/mysqld/mysqld.sock' (2).
unable to connect to mysql server.

Line 145 in /usr/bin/deb-systemd-invoke is:

            system('systemctl', '--quiet', @instance_args, $action,
@start_units) == 0 or die("Could not execute systemctl: $!");



Looking at previous log
https://ci.debian.net/data/autopkgtest/testing/amd64/c/cacti/31113621/log.gz
(with MariaDB 1:10.11.1-3) this section has:

Setting up mariadb-server (1:10.11.1-3) ...
Created symlink
/etc/systemd/system/multi-user.target.wants/mariadb.service →
/lib/systemd/system/mariadb.service.

There is nothing about deb-systemd-invoke.


When I run this manually inside failed cacti autopkgtest shell I get:

root at fb9c1752c1c7:/tmp/autopkgtest.n2vt3p/build.b5I/real-tree#
/usr/bin/deb-systemd-invoke restart mariadb
sh: 1: systemctl: not found
Can't exec "systemctl": No such file or directory at
/usr/bin/deb-systemd-invoke line 133.
mariadb is a disabled or a static unit not running, not starting it.


Reproducing this manually:
# /usr/bin/deb-systemd-invoke restart mariadb
ERROR:systemctl:the ExecStartPre control process exited with error code
Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.

After applying some debugging seems the script attempts to run:
systemctl --quiet --system restart mariadb

Checking what the systemd service file tries to run:
# grep ExecStartPre /lib/systemd/system/mariadb.service
ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld
ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION"
ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \

If systemctl is used from package 'systemctl' (not real, but systemd
shim) the start is bound to fail:
# systemctl unset-environment
ERROR:systemctl:Unknown operation unset-environment.

However, nothing about this changed in MariaDB 10.11.1-4.

# diff -u mariadb.service-10.6.11-2 mariadb.service-10.11.1-4
--- mariadb.service-10.6.11-2 2023-02-12 00:22:18.522820677 +0000
+++ mariadb.service-10.11.1-4 2023-02-12 00:25:01.013942479 +0000
@@ -19,7 +19,7 @@
 # (at your option) any later version.

 [Unit]
-Description=MariaDB 10.6.11 database server
+Description=MariaDB 10.11.1 database server
 Documentation=man:mariadbd(8)
 Documentation=https://mariadb.com/kb/en/library/systemd/
 After=network.target

# systemctl restart mariadb
ERROR:systemctl:the ExecStartPre control process exited with error code
# dpkg -S /lib/systemd/system/mariadb.service
mariadb-server-10.6: /lib/systemd/system/mariadb.service


Did cacti perhaps change how it uses systemd/systemctl?

Probably not, the same version of cacti has been in Debian for a long
time. Latest upload of dbconfig-common was on Feb 2nd, and after that
MariaDB 10.11.1-3 was still working on Feb 3rd..


Suggestions on what to debug next welcome.