[Pkg-mailman-hackers] Bug#412142: Upgrade from sarge to etch makes pipermail archives stop working

[EspeonEefi]

Package: mailman
Version: 1:2.1.9-5
Severity: normal

(This report was originally erroneously filed as a followup for bug
#391700.)

We recently moved our mailing lists from an old server running sarge to
a new server running etch. We migrated the lists by simply copying all
the data in /var/lib/mailman/ over.

Most of our lists are internal, but we noticed that a few of our lists
stopped getting their pipermail archives updated at various times after
the migration. These lists were specifically those which often receive
foreign-language spam.

Some extra searching about the problem turned up what seems to be a very
relevant bug filed upstream [1]. In the discussion are pointers to a
couple discussions on the mailman mailing lists about this bug. This
message [2] is particularly interesting, because it speculates that "if
you upgrade in the middle of a month, you get this problem in the
database."

[1] http://sourceforge.net/tracker/index.php?func=detail&aid=1246004&group_id=103&atid=100103
[2] http://mail.python.org/pipermail/mailman-developers/2006-February/018588.html

We ended up regenerating the archives of the broken lists with a
/var/lib/mailman/bin/arch --wipe <listname>. However, this was possible
only because we had both an mbox archive and the HTML archives. After
that, things were fixed.

During the time that the pipermail archives were stuck, we got a large
number of errors of the following form in /var/log/mailman/error:

Feb 17 10:43:12 2007 (23077) Uncaught runner exception: 'ascii' codec can't decode byte 0xae in position 6: ordinal not in range(128)
Feb 17 10:43:12 2007 (23077) Traceback (most recent call last):
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 112, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 170, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/ArchRunner.py", line 73, in _dispose
    mlist.ArchiveMail(msg)
  File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 214, in ArchiveMail
    h.processUnixMailbox(f)
  File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 580, in processUnixMailbox
    self.add_article(a)
  File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 632, in add_article
    article.parentID = parentID = self.get_parent_info(arch, article)
  File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 664, in get_parent_info
    article.subject)
  File "/usr/lib/mailman/Mailman/Archiver/HyperDatabase.py", line 311, in getOldestArticle
    self.__openIndices(archive)
  File "/usr/lib/mailman/Mailman/Archiver/HyperDatabase.py", line 251, in __openIndices
    t = DumbBTree(os.path.join(arcdir, archive + '-' + i))
  File "/usr/lib/mailman/Mailman/Archiver/HyperDatabase.py", line 65, in __init__
    self.load()
  File "/usr/lib/mailman/Mailman/Archiver/HyperDatabase.py", line 179, in load
    self.__sort(dirty=1)
  File "/usr/lib/mailman/Mailman/Archiver/HyperDatabase.py", line 73, in __sort
    self.sorted.sort()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xae in position 6: ordinal not in range(128)

Feb 17 10:43:12 2007 (23077) SHUNTING: 1171726987.576879+a0a8107f50431a926b61f9878aa958559fff8bd5

Anyway, these things pose a very interesting problem for sarge -> etch
upgrades, and it should be addressed somehow...

-- System Information:
Debian Release: 4.0
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.18-3-amd64
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)

Versions of packages mailman depends on:
ii  adduser                     3.102        Add and remove users and groups
ii  apache2                     2.2.3-3.2    Next generation, scalable, extenda
ii  apache2-mpm-prefork [httpd] 2.2.3-3.2    Traditional model for Apache HTTPD
ii  cron                        3.0pl1-100   management of regular background p
ii  debconf [debconf-2.0]       1.5.11       Debian configuration management sy
ii  exim4                       4.63-17      metapackage to ease exim MTA (v4) 
ii  exim4-daemon-light [mail-tr 4.63-17      lightweight exim MTA (v4) daemon
ii  libc6                       2.3.6.ds1-11 GNU C Library: Shared libraries
ii  logrotate                   3.7.1-3      Log rotation utility
ii  lsb-base                    3.1-23       Linux Standard Base 3.1 init scrip
ii  pwgen                       2.05-1       Automatic Password generation
ii  python                      2.4.4-2      An interactive high-level object-o
ii  python-support              0.5.6        automated rebuilding support for p
ii  ucf                         2.0018.1     Update Configuration File: preserv

mailman recommends no packages.

-- debconf information:
* mailman/queue_files_present:
* mailman/default_server_language: en (English)
* mailman/gate_news: false
* mailman/site_languages: en (English)
* mailman/used_languages: en
* mailman/create_site_list: