[Debian-ha-maintainers] Bug#612915: Random GFS2 partition locking problem

Fri Feb 11 14:27:35 UTC 2011

Package: gfs2-tools
Version: 3.0.12-2
Severity: critical
Tags: squeeze

My mail server (cid02) uses NFS shares mounted from a NFS server (sane02) which is exporting GFS2 cluster partition.

On random times my mail server get IO blocked process over NFS shares, when I start to investigate the problem I found
it is NOT problem with NFS server, network or Mail server (postfix, imap, dovecot).

It appears to be a strange GFS2 filesystem block out.

Any help is appreciated.

Events happens as follow and messages begin to throtle...

On server cid02 file /var/log/kern.log

Feb 11 04:57:49 ccsdtfdemcid02 kernel: [93435.428896] nfs: server 10.0.18.35 not responding, still trying

On server sane02 file /var/log/kern.log

Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168102] INFO: task nfsd:5340 blocked for more than 120 seconds.
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168159] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168214] nfsd          D 0000000000000002     0  5340      2 0x00000000
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168221]  ffff88012c19f810 0000000000000046 0000000000000000 ffffffffa036293b
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168228]  0000000000000000 ffffffffa0362495 000000000000f9e0 ffff880121ec1fd8
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168233]  0000000000015780 0000000000015780 ffff88012c092a60 ffff88012c092d58
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168239] Call Trace:
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168261]  [<ffffffffa036293b>] ? dlm_put_lockspace+0x10/0x1c [dlm]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168271]  [<ffffffffa0362495>] ? dlm_lock+0x135/0x149 [dlm]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168291]  [<ffffffffa04b87c0>] ? gdlm_ast+0x0/0xc2 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168301]  [<ffffffffa04b8731>] ? gdlm_bast+0x0/0x3d [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168309]  [<ffffffffa04a1dcc>] ? gfs2_glock_holder_wait+0x0/0xd [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168317]  [<ffffffffa04a1dd5>] ? gfs2_glock_holder_wait+0x9/0xd [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168327]  [<ffffffff812fae26>] ? __wait_on_bit+0x41/0x70
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168335]  [<ffffffffa04a1dcc>] ? gfs2_glock_holder_wait+0x0/0xd [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168340]  [<ffffffff812faec0>] ? out_of_line_wait_on_bit+0x6b/0x77
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168346]  [<ffffffff81064d58>] ? wake_bit_function+0x0/0x23
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168355]  [<ffffffffa04a1dba>] ? gfs2_glock_wait+0x23/0x28 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168364]  [<ffffffffa04a4703>] ? gfs2_glock_nq_init+0x16/0x2a [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168373]  [<ffffffffa04a505d>] ? gfs2_inode_lookup+0x16c/0x1d1 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168382]  [<ffffffffa04a46fb>] ? gfs2_glock_nq_init+0xe/0x2a [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168392]  [<ffffffffa04b4670>] ? try_rgrp_unlink+0xa3/0xcc [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168401]  [<ffffffffa04b5105>] ? gfs2_inplace_reserve_i+0x285/0x640 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168409]  [<ffffffffa049a243>] ? gfs2_dirent_find+0x0/0x7 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168416]  [<ffffffffa0499b50>] ? gfs2_dirent_search+0x13b/0x167 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168426]  [<ffffffffa04a521d>] ? gfs2_createi+0x15b/0x912 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168430]  [<ffffffff81064c4b>] ? bit_waitqueue+0x10/0xa0
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168438]  [<ffffffffa04a1c5f>] ? do_promote+0x15b/0x1f3 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168447]  [<ffffffffa04a2ed4>] ? run_queue+0xa5/0x127 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168456]  [<ffffffffa04ae156>] ? gfs2_create+0x67/0x131 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168465]  [<ffffffffa04a513e>] ? gfs2_createi+0x7c/0x912 [gfs2]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168472]  [<ffffffff810f78b4>] ? vfs_create+0x6d/0x89
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168484]  [<ffffffffa044eb27>] ? nfsd_create_v3+0x20a/0x358 [nfsd]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168493]  [<ffffffffa0457a14>] ? do_open_lookup+0x91/0x240 [nfsd]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168503]  [<ffffffffa0457d7a>] ? nfsd4_open+0x1b7/0x339 [nfsd]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168533]  [<ffffffffa0458db1>] ? nfsd4_encode_operation+0x5b/0x135 [nfsd]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168548]  [<ffffffffa0457716>] ? nfsd4_proc_compound+0x1fd/0x3b3 [nfsd]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168559]  [<ffffffffa0449329>] ? nfsd_dispatch+0xdd/0x1b9 [nfsd]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168577]  [<ffffffffa03894ff>] ? svc_process+0x403/0x627 [sunrpc]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168588]  [<ffffffffa0449772>] ? nfsd+0x0/0x12e [nfsd]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168598]  [<ffffffffa0449857>] ? nfsd+0xe5/0x12e [nfsd]
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168605]  [<ffffffff81064a5d>] ? kthread+0x79/0x81
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168615]  [<ffffffff81011baa>] ? child_rip+0xa/0x20
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168622]  [<ffffffff810649e4>] ? kthread+0x0/0x81
Feb 11 04:56:12 ccsdtfdemsane02 kernel: [63721.168628]  [<ffffffff81011ba0>] ? child_rip+0x0/0x20

-- System Information:
Debian Release: 6.0
  APT prefers oldstable
  APT policy: (500, 'oldstable'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages gfs2-tools depends on:
ii  cman                      3.0.12-2       Red Hat cluster suite - cluster ma
ii  libc6                     2.11.2-10      Embedded GNU C Library: Shared lib
ii  libldap-2.4-2             2.4.23-7       OpenLDAP libraries
ii  libncurses5               5.7+20100313-5 shared libraries for terminal hand
ii  libxml2                   2.7.8.dfsg-2   GNOME XML library
ii  psmisc                    22.11-1        utilities that use the proc file s

gfs2-tools recommends no packages.

gfs2-tools suggests no packages.

-- no debconf information