[debian-mysql] Bug#1064464: mariadb 10.3.39 crash at startup and can't be recovered

[Brice Waegeneire]

Package: mariadb-server
Version: 1:10.3.39-0+deb10u2

Hello there,

We had a crash with MariaDB server 10.3.39 on a Debian 10 server, which 
we weren't able to recover from. We know this is an old version of 
MariaDB, so we don't expect this bug to be fixed, but we want to 
document it in case other person are impacted.

What happened was: the instance crashed, and we weren't able to start it 
up again (see the attached partial-backtrace.txt), then we managed to 
have it start with the additional options "innodb_force_recovery = 3" 
and "innodb_purge_threads = 0". However, when we remove these options, 
the server crash during startup again. So the bug is reproducible on our 
hand,

The following is the environment of the crashed instance:

# cat /etc/debian_version
10.13
# dpkg -l mariadb-server-10.3
Desired=Unknown/Install/Remove/Purge/Hold
| 
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version             Architecture Description
+++-===================-===================-============-=================================
ii  mariadb-server-10.3 1:10.3.34-0+deb10u1 amd64        MariaDB 
database server binaries
# mariadb --version
mariadb  Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu (x86_64) 
using readline 5.2
# mysqld --print-defaults or SHOW VARIABLES
mysqld would have been started with the following arguments:
--user=mysql --pid-file=/run/mysqld/mysqld.pid 
--socket=/run/mysqld/mysqld.sock --basedir=/usr --datadir=/var/lib/mysql 
--tmpdir=/tmp --lc-messages-dir=/usr/share/mysql 
--bind-address=127.0.0.1 --query_cache_size=16M 
--log_error=/var/log/mysql/error.log --expire_logs_days=10 
--character-set-server=utf8mb4 --collation-server=utf8mb4_general_ci 
--max_connections=250 --back_log=100 --max_connect_errors=10 
--slow_query_log=1 --slow_query_log_file=/var/log/mysql/mysql-slow.log 
--long_query_time=10 --key_buffer_size=512M --max_allowed_packet=64M 
--thread_stack=192K --thread_cache_size=1 --max_heap_table_size=64M 
--table_open_cache=4096 --table_definition_cache=4096 
--query_cache_limit=8M --query_cache_size=256M --query_cache_type=1 
--max_heap_table_size=128M --tmp_table_size=128M --innodb_file_per_table 
--innodb_buffer_pool_size=512M --innodb_thread_concurrency=16 
--character-set-server=utf8 --collation-server=utf8_general_ci 
--secure-file-priv= --bind-address=0.0.0.0 --thread_cache_size=1 
--innodb_buffer_pool_size=4815M --tmpdir=/home/mysqltmp 
--innodb_log_file_size=256M --log_bin=/srv/mysql_binlogs/mysql-bin.log 
--expire_logs_days=5 --max_binlog_size=100M --binlog_format=mixed 
--sync_binlog=1 --gtid_domain_id=1 --server-id=1 
--auto-increment-increment=10 --auto-increment-offset=2 
--gtid_strict_mode=On --report-host=10.0.14.154 --report-port=3306 or 
SHOW VARIABLES

Note, the exact version of the mariadb-server package may not be 
consistent in the different logs (10.3.34-0+deb10u1 and 
10.3.39-MariaDB-0+deb10u2) because I wasn't able to get the debug 
symbols for the initial version (10.3.39-MariaDB-0+deb10u2) and ended up 
downgrading the package to get a full back trace. The crash occurred 
with both package version.

Note the attached innodb-status.txt if from the instance with 
innodb_force_recovery set.

I have already reported this bug to upstream: 
https://jira.mariadb.org/browse/MDEV-33528

If I've forgotten to share some information, I'll be happy to provide it.

Cheers,
- Brice Waegeneire
-- 
Brice WAEGENEIRE <bwaegeneire at evolix.fr> - SysAdmin support Evolix
Evolix - Hébergement et Infogérance Open Source
Marseille (37 rue Guibal, Pôle Média, 13003) / Paris / Montréal
http://evolix.com | Twitter: @Evolix @EvolixNOC | http://blog.evolix.com