[Pkg-mailman-hackers] Pkg-mailman commit - rev 479 - in
trunk/debian: . patches
Lionel Elie Mamane
lmamane at alioth.debian.org
Tue Feb 27 21:34:19 CET 2007
Author: lmamane
Date: 2007-02-27 21:34:18 +0100 (Tue, 27 Feb 2007)
New Revision: 479
Added:
trunk/debian/NEWS
trunk/debian/mailman.install
trunk/debian/patches/02_HyperDatabase_mapkeys.patch
trunk/debian/unicodify_archives.py
Modified:
trunk/debian/changelog
trunk/debian/patches/series
trunk/debian/postinst
Log:
A fix for bug: #412142
Convert indexes of current archiving volume to Unicode on upgrade
Added: trunk/debian/NEWS
===================================================================
--- trunk/debian/NEWS 2007-02-26 14:11:06 UTC (rev 478)
+++ trunk/debian/NEWS 2007-02-27 20:34:18 UTC (rev 479)
@@ -0,0 +1,38 @@
+mailman (1:2.1.9-6) unstable; urgency=medium
+
+ * This version will automatically upgrade indexes of the current
+ archiving volume to the new Unicode format, which can lead to a slight
+ corruption / data loss in the form of non-ASCII characters not being
+ transcoded correctly.
+
+ If you kept single-mbox archives of your mailing list additionally to
+ the HTML ones (even if this mbox archive was not downloadable from the
+ web), you are probably better off regenerating the archives from
+ scratch with:
+ cd /var/lib/mailman/lists
+ for list in *; do /var/lib/mailman/bin/arch --wipe "${list}" ; done
+ This will use information in the original email to make a better guess
+ at the encoding of the subject and author of a post to the list.
+
+
+ Details:
+
+ Seome previous versions tried to use Unicode for new entries, mixing
+ them with old 8 bit character string entries left over from even older
+ versions, leading to a cascade of events that broke archiving for
+ mailing lists completely for the rest of the archiving volume period
+ (often the calendar month).
+
+ As the old entries were stored as strings without encoding
+ information, there is no way to automatically reliably convert those
+ to Unicode. The upgrade code will try, in order:
+ - the default Python encoding (usually ASCII)
+ - UTF-8
+ - windows-1252, a superset of iso-8859-1
+ You can change the default Python encoding by creating a file called
+ sitecustomize.py in python's loadpath with e.g.:
+ import sys
+ sys.setdefaultencoding('iso-8859-8')
+
+ -- Lionel Elie Mamane <lmamane at debian.org> Tue, 27 Feb 2007 21:30:49 +0100
+
Modified: trunk/debian/changelog
===================================================================
--- trunk/debian/changelog 2007-02-26 14:11:06 UTC (rev 478)
+++ trunk/debian/changelog 2007-02-27 20:34:18 UTC (rev 479)
@@ -1,14 +1,17 @@
-mailman (1:2.1.9-6) UNRELEASED; urgency=low
+mailman (1:2.1.9-6) unstable; urgency=medium
[ Lionel Elie Mamane ]
* Allow people that have list as a supplementary group to create new
mailing lists. (see bug#309339)
+ * Upgrade subject and author indexes of current archiving volume to
+ Unicode strings; possible slight data loss (non-ASCII characters
+ transcoded wrongly) (closes: #412142)
[ Thijs Kinkhorst ]
* Fix path-typo in apache.conf dedicated virtual host example,
thanks Alberto Furia (Closes: #409180).
- -- Thijs Kinkhorst <thijs at debian.org> Thu, 1 Feb 2007 00:29:46 +0100
+ -- Lionel Elie Mamane <lmamane at debian.org> Tue, 27 Feb 2007 21:02:45 +0100
mailman (1:2.1.9-5) unstable; urgency=medium
Added: trunk/debian/mailman.install
===================================================================
--- trunk/debian/mailman.install 2007-02-26 14:11:06 UTC (rev 478)
+++ trunk/debian/mailman.install 2007-02-27 20:34:18 UTC (rev 479)
@@ -0,0 +1 @@
+debian/unicodify_archives.py usr/lib/mailman/bin/
Added: trunk/debian/patches/02_HyperDatabase_mapkeys.patch
===================================================================
--- trunk/debian/patches/02_HyperDatabase_mapkeys.patch 2007-02-26 14:11:06 UTC (rev 478)
+++ trunk/debian/patches/02_HyperDatabase_mapkeys.patch 2007-02-27 20:34:18 UTC (rev 479)
@@ -0,0 +1,25 @@
+--- Mailman/Archiver/HyperDatabase.py 2005-08-27 03:40:17.000000000 +0200
++++ Mailman/Archiver/HyperDatabase.py 2007-02-27 20:33:41.103527160 +0100
+@@ -324,15 +343,22 @@
+
+ def clearIndex(self, archive, index):
+ self.__openIndices(archive)
+ if hasattr(self.threadIndex, 'clear'):
+ self.threadIndex.clear()
+ return
+ finished=0
+ try:
+ key, msgid=self.threadIndex.first()
+ except KeyError: finished=1
+ while not finished:
+ del self.threadIndex[key]
+ try:
+ key, msgid=self.threadIndex.next()
+ except KeyError: finished=1
++
++ def mapKeys(self, f, archive, index):
++ self.__openIndices(archive)
++ index = getattr(self, index + 'Index')
++ d = index.dict
++ index.dict = dict(zip(map(f, d.keys()), d.values()))
++ index.__dirty = 1
Modified: trunk/debian/patches/series
===================================================================
--- trunk/debian/patches/series 2007-02-26 14:11:06 UTC (rev 478)
+++ trunk/debian/patches/series 2007-02-27 20:34:18 UTC (rev 479)
@@ -1,5 +1,6 @@
00_stolen_from_HEAD.patch -p0
01_defaults.debian.patch -p0
+02_HyperDatabase_mapkeys.patch -p0
07_snooze.patch -p0
10_wrapper_uid.patch -p0
11_handle_propfind.patch -p0
Modified: trunk/debian/postinst
===================================================================
--- trunk/debian/postinst 2007-02-26 14:11:06 UTC (rev 478)
+++ trunk/debian/postinst 2007-02-27 20:34:18 UTC (rev 479)
@@ -4,6 +4,8 @@
# Written 1998 by Gregor Hoffleit <flight at debian.org>.
# Updated 2001 by Tollef Fog Heen <tfheen at debian.org>
# mailman modifications 2004 by Bernd S. Brentrup <bsb at debian.org>
+# Other modifications 2004-2007 by other maintainers of the Debian package:
+# Lionel Elie Mamane, Thijs Kinkhorst, Riccardo Setti, Matej Vela, Hector Garcia, László Böszörményi, Bernd S. Brentrup, ...
#
# $URL$
# $Id$
@@ -56,6 +58,7 @@
# At present ALL directories in /etc/mailman are language directories
# but this may change in the future, better check.
+ # Furthermore, the administrator may have have added other directories
for dir in $(find ${mm_etc} -maxdepth 1 -mindepth 1 -type d -printf '%P\n'); do
if [ -f ${mm_dist}/${dir}/options.html ]; then
old_languages="${old_languages} ${dir}"
@@ -216,4 +219,9 @@
;;
esac
+if [ "$1" = "configure" ] && dpkg --compare-versions "$2" lt "1:2.1.9-6"; then
+ /var/lib/mailman/bin/withlist --all -q -r unicodify_archives
+fi
+
+
#DEBHELPER#
Added: trunk/debian/unicodify_archives.py
===================================================================
--- trunk/debian/unicodify_archives.py 2007-02-26 14:11:06 UTC (rev 478)
+++ trunk/debian/unicodify_archives.py 2007-02-27 20:34:18 UTC (rev 479)
@@ -0,0 +1,66 @@
+#! @PYTHON@
+#
+# Copyright (C) 2007 Lionel Elie Mamane <lmamane at debian.org>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 2
+# of the License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+
+"""Convert a list's archive databases to unicode where appropriate
+
+This script is intended to be run as a bin/withlist script, i.e.
+
+% bin/withlist -l -r unicodify_archives <mylist>
+"""
+
+import paths
+import time
+from Mailman.i18n import _
+from Mailman import mm_cfg
+
+def unicodify_string(s):
+ if isinstance(s,unicode):
+ return s
+ elif isinstance(s,str):
+ try:
+ return s.decode()
+ except UnicodeDecodeError:
+ pass
+ try:
+ return s.decode('utf-8')
+ except UnicodeDecodeError:
+ pass
+ return s.decode('windows-1252', 'replace')
+
+def unicodify_fst(t):
+ l = list(t[1:])
+ l.insert(0, unicodify_string(t[0]))
+ return tuple(l)
+
+def unicodify_archives(mlist):
+ # Only act if we are using the internal archiver
+ if mm_cfg.PUBLIC_EXTERNAL_ARCHIVER:
+ return
+ else:
+ from Mailman.Archiver import HyperArch
+ h = HyperArch.HyperArchive(mlist)
+ currentVolume = h.dateToVolName(time.time())
+ if currentVolume in h.archives:
+ for hdr in ('subject', 'author'):
+ h.database.mapKeys(unicodify_fst, currentVolume, hdr)
+ h.close()
+
+
+
+if __name__ == '__main__':
+ print _(__doc__.replace('%', '%%'))
More information about the Pkg-mailman-hackers
mailing list