Bug#267066: marked as done (libsvn0: repository becomes wedged,
subversion deadlocks)
Debian Bug Tracking System
owner at bugs.debian.org
Thu Jan 12 16:18:44 UTC 2006
Your message dated Thu, 12 Jan 2006 09:13:43 -0700
with message-id <20060112161343.GA13376 at me.troyhebe>
and subject line closing bug
has caused the attached Bug report to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what I am
talking about this indicates a serious mail system misconfiguration
somewhere. Please contact me immediately.)
Debian bug tracking system administrator
(administrator, Debian Bugs database)
--------------------------------------
Received: (at submit) by bugs.debian.org; 20 Aug 2004 14:23:33 +0000
>From marius at pov.lt Fri Aug 20 07:23:33 2004
Return-path: <marius at pov.lt>
Received: from office.pov.lt (mail.pov.lt) [213.197.143.150]
by spohr.debian.org with esmtp (Exim 3.35 1 (Debian))
id 1ByAIm-0005Ew-00; Fri, 20 Aug 2004 07:23:32 -0700
Received: from localhost (localhost [127.0.0.1])
by mail.pov.lt (Postfix) with ESMTP
id DE536C45B; Fri, 20 Aug 2004 17:23:29 +0300 (EEST)
Received: from mail.pov.lt ([127.0.0.1])
by localhost (fridge [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id 30765-03; Fri, 20 Aug 2004 17:23:28 +0300 (EEST)
Received: by mail.pov.lt (Postfix, from userid 1001)
id 4E01DC309; Fri, 20 Aug 2004 17:23:28 +0300 (EEST)
Date: Fri, 20 Aug 2004 17:23:28 +0300
From: Marius Gedminas <marius at pov.lt>
To: Debian Bug Tracking System <submit at bugs.debian.org>
Subject: libsvn0: repository becomes wedged, subversion deadlocks
Message-ID: <20040820142328.GA30559 at fridge.pov.lt>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
X-Reportbug-Version: 2.63
X-GPG-Fingerprint: 8121 AD32 F00A 8094 748A 6CD0 9157 445D E7A6 D78F
X-GPG-Key: http://mg.b4net.lt/mg-pgp-key.txt
User-Agent: Mutt/1.5.6+20040722i
Delivered-To: submit at bugs.debian.org
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2004_03_25
(1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Status: No, hits=-8.0 required=4.0 tests=BAYES_00,HAS_PACKAGE
autolearn=no version=2.60-bugs.debian.org_2004_03_25
X-Spam-Level:
Package: libsvn0
Version: 1.0.6-1.1
Severity: important
Recently (since last weekend) a recurring problem started plaguing the
SchoolTool subversion repository at http://source.schooltool.org/. The
symptoms are quite different from the ones described in bugs #266314 and
#252974: all processes that try to access the repository (including svn,
svnadmin, svnserve, python for viewcvs.cgi, and apache2 with mod_svn)
just hang in an infinite loop. Running strace shows that they all
execute the same loop, repeatedly calling
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
ltrace on those pids reports nothing.
fuser shows that all of those processes are accessing db.lock from the
repository. lsof shows that they all hold read locks on that file.
Looking at the stack trace with gdb reveals that all of them have the
same topmost 9 frames:
#0 0x4036bdd2 in select () from /lib/libc.so.6
#1 0x401bc448 in db_xa_switch_4002 () from /usr/lib/libdb-4.2.so
#2 0x4019d5c9 in __memp_sync_int_4002 () from /usr/lib/libdb-4.2.so
#3 0x4019cd24 in __memp_sync_4002 () from /usr/lib/libdb-4.2.so
#4 0x401a46ee in __txn_checkpoint_4002 () from /usr/lib/libdb-4.2.so
#5 0x401a43cd in __txn_checkpoint_pp_4002 () from /usr/lib/libdb-4.2.so
#6 0x40048e60 in svn_fs_list_transactions () from /usr/lib/libsvn_fs-1.so.0
#7 0x40048f1c in svn_fs_list_transactions () from /usr/lib/libsvn_fs-1.so.0
#8 0x4004904f in svn_fs__retry_txn () from /usr/lib/libsvn_fs-1.so.0
Killing all processes that are accessing the repository does not help --
any newly started processes also hang. Stracing svnadmin verify shows
that the last syscalls performed before the select loop are:
...
stat64("/svn/schooltool/db/uuids", {st_mode=S_IFREG|0664, st_size=8192, ...}) = 0
open("/svn/schooltool/db/uuids", O_RDWR|O_LARGEFILE) = 11
fcntl64(11, F_SETFD, FD_CLOEXEC) = 0
fstat64(11, {st_mode=S_IFREG|0664, st_size=8192, ...}) = 0
time(NULL) = 1093011345
time([1093011345]) = 1093011345
stat64("/svn/schooltool/db/log.0000000275", {st_mode=S_IFREG|0664, st_size=590372, ...}) = 0
open("/svn/schooltool/db/log.0000000275", O_RDWR|O_CREAT|O_LARGEFILE, 0666) = 12
fcntl64(12, F_SETFD, FD_CLOEXEC) = 0
read(12, "\202\372\17\0\34\0\0\0$(\21\324\210\t\4\0\10\0\0\0\0\0"..., 28) = 28
_llseek(12, 590372, [590372], SEEK_SET) = 0
write(12, "\322\1\t\0R\0\0\0\275\23\255\336\2\0\0\0\265\201\7\200"..., 1002) = 1002
fsync(12) = 0
pwrite(7, "\370\0\0\0!\24\10\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t"..., 4096, 0) = 4096
pwrite(6, "\20\1\0\0c\344\n\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t\0"..., 4096, 0) = 4096
pwrite(5, "\371\0\0\0\266c\3\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t"..., 4096, 0) = 4096
pwrite(8, "\23\1\0\0\23B\4\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t\0"..., 4096, 0) = 4096
pwrite(10, "\23\1\0\0\220)\5\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t\0"..., 4096, 0) = 4096
pwrite(9, "\22\1\0\0\362w\t\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t\0"..., 4096, 0) = 4096
pwrite(11, "\0\0\0\0\1\0\0\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t\0\0"..., 4096, 0) = 4096
pwrite(4, "\23\1\0\0\362Q\3\0\0\0\0\0b1\5\0\t\0\0\0\0\20\0\0\0\t\0"..., 4096, 0) = 4096
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
The file descriptors at that time are (from ls -l /proc/$pid/fd):
0 -> /dev/pts/4
1 -> /dev/pts/4
2 -> /dev/pts/4
3 -> /var/lib/svn/schooltool/locks/db.lock
4 -> /var/lib/svn/schooltool/db/nodes
5 -> /var/lib/svn/schooltool/db/revisions
6 -> /var/lib/svn/schooltool/db/transactions
7 -> /var/lib/svn/schooltool/db/copies
8 -> /var/lib/svn/schooltool/db/changes
9 -> /var/lib/svn/schooltool/db/representations
10 -> /var/lib/svn/schooltool/db/strings
11 -> /var/lib/svn/schooltool/db/uuids
12 -> /var/lib/svn/schooltool/db/log.0000000275
Killing all processes that are accessing the repository and then running
svnadmin recover helps.
It is possible, but not certain, that these wedges are triggered by
msnbot indexing the schooltool repository, as indicated by apache's
access.log. There are problems registered in error.log at the time of
the wedge, although there are a number of error messages about problems
closing the Berkeley DB execution environment from earlier days when I
had resorted to killall and killall -9.
-- System Information:
Debian Release: 3.0
APT prefers testing
APT policy: (300, 'testing'), (200, 'unstable')
Architecture: i386 (i686)
Kernel: Linux 2.4.26-1-686
Locale: LANG=C, LC_CTYPE=lt_LT.UTF-8
Versions of packages libsvn0 depends on:
ii libapr0 2.0.50-9 The Apache Portable Runtime
ii libc6 2.3.2.ds1-13 GNU C Library: Shared libraries an
ii libdb4.2 4.2.52-16 Berkeley v4.2 Database Libraries [
ii libexpat1 1.95.6-8 XML parsing C library - runtime li
ii libldap2 2.1.30-2 OpenLDAP libraries
ii libneon24 0.24.7.dfsg-0.1 An HTTP and WebDAV client library
ii libperl5.8 5.8.4-2 Shared Perl library.
ii libssl0.9.7 0.9.7d-4 SSL shared libraries
ii libswig1.3.21 1.3.21-5 Runtime support libraries for swig
ii libxml2 2.6.11-3 GNOME XML library
ii python2.3 2.3.4-5 An interactive high-level object-o
ii zlib1g 1:1.2.1.1-5 compression library - runtime
-- no debconf information
Marius Gedminas
--
We have an advanced scalable groupware communication environment (email)
-- Alan Cox
---------------------------------------
Received: (at 267066-done) by bugs.debian.org; 12 Jan 2006 16:14:17 +0000
>From troyhebe at ldl.fc.hp.com Thu Jan 12 08:14:17 2006
Return-path: <troyhebe at ldl.fc.hp.com>
Received: from atlrel7.hp.com ([156.153.255.213])
by spohr.debian.org with esmtp (Exim 4.50)
id 1Ex55d-0006eO-I7; Thu, 12 Jan 2006 08:14:17 -0800
Received: from smtp2.fc.hp.com (unknown [15.15.136.253])
by atlrel7.hp.com (Postfix) with ESMTP id 6677E36F67;
Thu, 12 Jan 2006 11:13:44 -0500 (EST)
Received: from ldl (lart.fc.hp.com [15.11.146.31])
by smtp2.fc.hp.com (Postfix) with ESMTP
id 139E441E422; Thu, 12 Jan 2006 16:13:44 +0000 (UTC)
Received: by ldl (Postfix, from userid 1000)
id 0B38D7C8034; Thu, 12 Jan 2006 09:13:44 -0700 (MST)
Date: Thu, 12 Jan 2006 09:13:43 -0700
From: Troy Heber <troyh at debian.org>
To: 267066-done at bugs.debian.org
Cc: 267066-submitter at bugs.debian.org
Subject: closing bug
Message-ID: <20060112161343.GA13376 at me.troyhebe>
Reply-To: Troy Heber <troyh at debian.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: mutt-ng devel-20050619 (Debian)
X-Spam-Checker-Version: SpamAssassin 2.60-bugs.debian.org_2005_01_02
(1.212-2003-09-23-exp) on spohr.debian.org
X-Spam-Level:
X-Spam-Status: No, hits=-3.0 required=4.0 tests=BAYES_00 autolearn=no
version=2.60-bugs.debian.org_2005_01_02
Closing bug per Marius's comments above.
Troy
More information about the pkg-subversion-maintainers
mailing list