[debian-mysql] Bug#447028: mysql-server-5.0: crashes on unknown condition, is directly restarted by mysqld_safe

Gertjan Oude Lohuis gertjan at byte.nl
Wed Oct 17 15:12:46 UTC 2007


Package: mysql-server-5.0
Version: 5.0.32-7etch1
Severity: important

One of our MySQL servers has crashed 3 times during the last 5 days.
The server is heavily used, but there is no visble relationship
between the load on the server and the crashes, since one of the
crashes occurred at 1 AM. After every crash mysqld_safe restarted the
server, so for a little time there were no connections possible. Each
crash gives more or less the same errorlog. I've made a stacktrace
using resolve_stack_dump, and have attached the combined logfile and
the combined stacktraces of the 3 crashes. They are in chronological
order: 2007-10-12, 2007-10-15, 2007-10-17.
All three stacktraces are very similar, which leads me to think that
this problem must be reproducible. However, over 40 GB of databases
and 1000 queries/second on average doesn't make it easy to isolate the
problem :-). 
Only the last errorlog mentions a hint of a query. Could
that be the problem?


-- System Information:
Debian Release: 4.0
  APT prefers stable
  APT policy: (600, 'stable')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.20.4-fwsh-byte
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)

Versions of packages mysql-server-5.0 depends on:
ii  adduser                3.102             Add and remove users and groups
ii  debconf [debconf-2.0]  1.5.11            Debian configuration management sy
ii  libc6                  2.3.6.ds1-13etch2 GNU C Library: Shared libraries
ii  libdbi-perl            1.53-1            Perl5 database interface by Tim Bu
ii  libgcc1                1:4.1.1-21        GCC support library
ii  libmysqlclient15off    5.0.32-7etch1     mysql database client library
ii  libncurses5            5.5-5             Shared libraries for terminal hand
ii  libreadline5           5.2-2             GNU readline and history libraries
ii  libstdc++6             4.1.1-21          The GNU Standard C++ Library v3
ii  libwrap0               7.6.dbs-13        Wietse Venema's TCP wrappers libra
ii  lsb-base               3.1-23.2etch1     Linux Standard Base 3.1 init scrip
ii  mysql-client-5.0       5.0.32-7etch1     mysql database client binaries
ii  mysql-common           5.0.32-7etch1     mysql database common files (e.g. 
ii  passwd                 1:4.0.18.1-7      change and administer password and
ii  perl                   5.8.8-7           Larry Wall's Practical Extraction 
ii  psmisc                 22.3-1            Utilities that use the proc filesy
ii  zlib1g                 1:1.2.3-13        compression library - runtime

Versions of packages mysql-server-5.0 recommends:
ii  mailx            1:8.1.2-0.20050715cvs-1 A simple mail user agent

-- debconf information:
  mysql-server/root_password: (password omitted)
  mysql-server-5.0/really_downgrade: false
  mysql-server-5.0/need_sarge_compat: false
  mysql-server-5.0/start_on_boot: true
  mysql-server/error_setting_password:
  mysql-server-5.0/nis_warning:
  mysql-server-5.0/postrm_remove_databases: false
  mysql-server-5.0/need_sarge_compat_done: true
-------------- next part --------------
mysqld got signal 8;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=357
max_connections=750
threads_connected=16
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1894138 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x77f308e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x78cbb6b8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81c0659
0x817a812
0x817b767
0x8143be8
0x814d400
0x8208784
0x821440c
0x8209e85
0x820fc37
0x8209e85
0x820fc37
0x8209e85
0x820fc37
0x8209e85
0x820fc37
0x8213997
0x821fefd
0x8221c20
0x822248e
0x81d74a5
0x81dba17
0x81dbed0
0x81dd198
0x81ddba4
0xb7f55240
0xb7d903de
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x785100a0  is invalid pointer
thd->thread_id=16722517
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
pure virtual method called
terminate called without an active exception
Fatal signal 11 while backtracing
Fatal signal 6 while backtracing
071012 20:00:38  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
071012 20:01:38  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 4048875227.
InnoDB: Doing recovery: scanned up to log sequence number 0 4048877244
071012 20:01:38  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 324, file name mysql-bin.000025
InnoDB: Last MySQL binlog file position 0 9944799, file name /srv/db/mysql-bin.000316
071012 20:01:39  InnoDB: Started; log sequence number 0 4048877244
071012 20:01:39 [Note] Recovering after a crash using /srv/db/mysql-bin
071012 20:01:39 [Note] Starting crash recovery...
071012 20:01:39 [Note] Crash recovery finished.
071012 20:01:40 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.32-Debian_7etch1-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Debian etch distribution



mysqld got signal 8;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=689
max_connections=750
threads_connected=10
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1894138 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x7b76aa50
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x7ac796b8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81c0659
0x817a812
0x817b767
0x8143be8
0x814d400
0x8208784
0x821440c
0x8209e85
0x820fc37
0x8209e85
0x820fc37
0x8209e85
0x820fc37
0x8209e85
0x820fc37
0x8213997
0x821fefd
0x8221c20
0x822248e
0x81d74a5
0x81dba17
0x81dbed0
0x81dd198
0x81ddba4
0xb7f10240
0xb7d4b4ae
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x7bfaa7d8  is invalid pointer
thd->thread_id=7348090
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
071017  0:49:32  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
071017  0:50:37  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 1 1783333500.
InnoDB: Doing recovery: scanned up to log sequence number 1 1783333500
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 324, file name mysql-bin.000025
InnoDB: Last MySQL binlog file position 0 28884411, file name /srv/db/mysql-bin.000415
071017  0:50:37  InnoDB: Started; log sequence number 1 1783333500
071017  0:50:37 [Note] Recovering after a crash using /srv/db/mysql-bin
071017  0:50:37 [Note] Starting crash recovery...
071017  0:50:37 [Note] Crash recovery finished.
071017  0:50:38 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.32-Debian_7etch1-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Debian etch distribution


mysqld got signal 8;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=58
max_connections=750
threads_connected=35
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 1894138 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=0x7c083980
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0x7bbfd6b8, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x81c0659
0x817a812
0x817b767
0x8143be8
0x814d400
0x8208784
0x821440c
0x8209e85
0x820fc37
0x8209e85
0x820fc37
0x8209e85
0x820fc37
0x8209e85
0x820fc37
0x8213997
0x821fefd
0x8221c20
0x822248e
0x81d74a5
0x81dba17
0x81dbed0
0x81dd198
0x81ddba4
0xb7f77240
0xb7db24ae
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do 
resolve it
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x9af3d90 = SELECT  `cms_pages`.`id` as `id` ,  `cms_pages`.`node_id` as `node_id` ,  `cms_pages`.`article_date` as `article_date` ,  `cms_pages`.`name` as `name` ,  `cms_pages`.`form_id` as `form_id` ,  `cms_pages`.`article_order` as `article_order` ,  `cms_pages`.`published` as `published` ,  `cms_pages`.`gallery_id` as `gallery_id` ,  `cms_pages`.`poll_id` as `poll_id` ,  `cms_pages`.`date_added` as `date_added` ,  `cms_pages`.`user_added` as `user_added` ,  `cms_pages`.`date_modified` as `date_modified` ,  `cms_pages`.`user_modified` as `user_modified` 
,  concat(repeat('   ', cms_nodes.level-1),cms_nodes_to_lang.name, ' - ', cms_pages_to_lang.title) as title  
 FROM `cms_pages` 

 INNER JOIN `db005831_socof000`.`cms_nodes`   ON `db005831_socof000`.`cms_nodes`.`id`=`cms_pages`.`node_id` 
 INNER JOIN `db005831_socof000`.`cms_nodes_to_lang`   ON `db005831_socof000`.`cms_nodes_to_lang`.`node_id`=`cms_nodes`.`id`   
 INNER JOIN `db005831_socof000`.`cms_pages_to_lang`   ON `db005831_socof000`.`cms_pages_to_
thd->thread_id=608147
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
pure virtual method called
pure virtual method called
terminate called without an active exception
terminate called without an active exception
pure virtual method called
Fatal signal 6 while backtracing
terminate called recursively
pure virtual method called
pure virtual method called
pure virtual method called
pure virtual method called
pure virtual method called
terminate called recursively
pure virtual method called
terminate called recursively
pure virtual method called
terminate called recursively
071017 12:37:08  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
071017 12:38:22  InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 1 1795496028.
InnoDB: Doing recovery: scanned up to log sequence number 1 1795496028
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 324, file name mysql-bin.000025
InnoDB: Last MySQL binlog file position 0 85316845, file name /srv/db/mysql-bin.000420
071017 12:38:23  InnoDB: Started; log sequence number 1 1795496028
071017 12:38:23 [Note] Recovering after a crash using /srv/db/mysql-bin
071017 12:38:24 [Note] Starting crash recovery...
071017 12:38:24 [Note] Crash recovery finished.
071017 12:38:24 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.0.32-Debian_7etch1-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Debian etch distribution
-------------- next part --------------
0x81c0659 handle_segfault + 681
0x817a812 _ZN16Item_func_repeat7val_strEP6String + 162
0x817b767 _ZN16Item_func_concat7val_strEP6String + 39
0x8143be8 _ZN4Item13save_in_fieldEP5Fieldb + 72
0x814d400 _ZN17Item_result_field20save_in_result_fieldEb + 32
0x8208784 _Z10copy_funcsPP4Item + 52
0x821440c _Z9end_writeP4JOINP13st_join_tableb + 124
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8213997 _Z9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure + 247
0x821fefd _ZN4JOIN4execEv + 2157
0x8221c20 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_sel + 304
0x822248e _Z13handle_selectP3THDP6st_lexP13select_resultm + 318
0x81d74a5 _Z21mysql_execute_commandP3THD + 9557
0x81dba17 _Z11mysql_parseP3THDPcj + 471
0x81dbed0 _Z16dispatch_command19enum_server_commandP3THDPcj + 1120
0x81dd198 _Z10do_commandP3THD + 136
0x81ddba4 handle_one_connection + 2308
0xb7f55240 _end + -1350121744
0xb7d903de _end + -1351976818


0x81c0659 handle_segfault + 681
0x817a812 _ZN16Item_func_repeat7val_strEP6String + 162
0x817b767 _ZN16Item_func_concat7val_strEP6String + 39
0x8143be8 _ZN4Item13save_in_fieldEP5Fieldb + 72
0x814d400 _ZN17Item_result_field20save_in_result_fieldEb + 32
0x8208784 _Z10copy_funcsPP4Item + 52
0x821440c _Z9end_writeP4JOINP13st_join_tableb + 124
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8213997 _Z9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure + 247
0x821fefd _ZN4JOIN4execEv + 2157
0x8221c20 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_sel + 304
0x822248e _Z13handle_selectP3THDP6st_lexP13select_resultm + 318
0x81d74a5 _Z21mysql_execute_commandP3THD + 9557
0x81dba17 _Z11mysql_parseP3THDPcj + 471
0x81dbed0 _Z16dispatch_command19enum_server_commandP3THDPcj + 1120
0x81dd198 _Z10do_commandP3THD + 136
0x81ddba4 handle_one_connection + 2308
0xb7f10240 _end + -1350404368
0xb7d4b4ae _end + -1352259234


0x81c0659 handle_segfault + 681
0x817a812 _ZN16Item_func_repeat7val_strEP6String + 162
0x817b767 _ZN16Item_func_concat7val_strEP6String + 39
0x8143be8 _ZN4Item13save_in_fieldEP5Fieldb + 72
0x814d400 _ZN17Item_result_field20save_in_result_fieldEb + 32
0x8208784 _Z10copy_funcsPP4Item + 52
0x821440c _Z9end_writeP4JOINP13st_join_tableb + 124
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8209e85 _Z20evaluate_join_recordP4JOINP13st_join_tableiPc + 405
0x820fc37 _Z10sub_selectP4JOINP13st_join_tableb + 119
0x8213997 _Z9do_selectP4JOINP4ListI4ItemEP8st_tableP9Procedure + 247
0x821fefd _ZN4JOIN4execEv + 2157
0x8221c20 _Z12mysql_selectP3THDPPP4ItemP13st_table_listjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_sel + 304
0x822248e _Z13handle_selectP3THDP6st_lexP13select_resultm + 318
0x81d74a5 _Z21mysql_execute_commandP3THD + 9557
0x81dba17 _Z11mysql_parseP3THDPcj + 471
0x81dbed0 _Z16dispatch_command19enum_server_commandP3THDPcj + 1120
0x81dd198 _Z10do_commandP3THD + 136
0x81ddba4 handle_one_connection + 2308
0xb7f77240 _end + -1349982480
0xb7db24ae _end + -1351837346


More information about the pkg-mysql-maint mailing list