Upgrading cyrus

[Patrick Goetz]

On 09/14/2010 04:16 PM, Jeroen van Meeuwen (Kolab Systems) wrote:
>
> Indeed it [cyrus-imapd.cvt_cyrusdb_all] exports the databases on service shutdown, and then imports them
> back on service startup; noted that a condrestart happens on the RPM update,
> this about covers every thinkable scenario (or at least those I've had in the
> past 7 years).
>

This seems like it would considerably slow down startup/shutdown on 
systems with large databases....


>
> If there is no information in /etc/imapd.conf explicitly, the defaults are
> being used -just like the service itself would. The defaults for RPM (for the
> purpose of the script) is to use skiplist for all databases.
>

This is another question that I have:  is the set/type of database files 
determined at run time (i.e. in imapd.conf) or at compile time?  In 
particular, the set of databases I have on 2.1.16 only has a very small 
intersection with the set described in the forwarded email below:
   ------------------------------------------------
   www:cyrus# pwd
   /var/lib/cyrus
   www:cyrus# ls
   db	    db.backup2	log	      msg   quota   tls_sessions.db
   db.backup1  deliver.db	mailboxes.db  proc  socket  user
   ------------------------------------------------

Again, for me this doesn't matter so much, as I'm planning to set up a 
new mail server and imapsync mail from the old server to the new one, 
but if we're talking about a postinst script which smoothly upgrades 
users from one version of cyrus to another, then the database stuff has 
to be solidly nailed down.  Right now I'm confused about what determines 
the number, type, and names of db files in various versions of cyrus.


=========================================================================
-------- Original Message --------
Subject: Re: Wiki, information on database back ends.
Date: Tue, 14 Sep 2010 16:30:55 -0500
From: Dan White <dwhite at olp.net>
To: Patrick Goetz <pgoetz at mail.utexas.edu>
CC: cyrus-devel at lists.andrew.cmu.edu

On 14/09/10 14:24 -0500, Patrick Goetz wrote:
 >There's been some discussion on the Debian cyrus list about how to
 >automate upgrades from cyrus 2.n.k to cyrus 2.m.j.  Jeroen van
 >Meeuwen (on both lists) suggested that the cyrus RPM package features
 >a utility called cyrus-imapd.cvt_cyrusdb_all which might be useful
 >for this.
 >
 >I've been looking at this script, and it mostly appears to be using
 >cvt_cyrusdb to convert particular db files to Cyrus skiplists and
 >then back again to the original db backend format.  I can't follow
 >the script completely as it seems to rely on DB configuration details
 >found in the imapd.conf file I don't have in my Debian 2.1.16 imap
 >server, and it's also not clear how the script is run.
 >
 >This raises a number of questions, though:
 >
 >1.
 >Cyrus skiplists?  I thought all the DB files were in Berkeley DB
 >format.  I tried to find some documentation on skiplists, but only
 >found an old message to the developer list from Bron Gondwana
 >discussing skiplist bugs
 >(http://markmail.org/message/zbaq765brbg2acfj).

The current imapd.conf hints at the available data types for each cyrus
database. Depending on the option, the following database types are
available (in 2.3.16):

annotation_db: skiplist
   Allowed values: berkeley, berkeley-hash, skiplist

duplicate_db: berkeley-nosync
   Allowed values: berkeley, berkeley-nosync, berkeley-hash,
berkeley-hash-nosync, skiplist, sql

mboxkey_db: skiplist
   Allowed values: berkeley, skiplist

mboxlist_db: skiplist
   Allowed values: flat, berkeley, berkeley-hash, skiplist

ptscache_db: berkeley
   Allowed values: berkeley, berkeley-hash, skiplist

quota_db: quotalegacy
   Allowed values: flat, berkeley, berkeley-hash, skiplist, sql, quotalegacy

seenstate_db: skiplist
   Allowed values: flat, berkeley, berkeley-hash, skiplist

subscription_db: flat
   Allowed values: flat, berkeley, berkeley-hash, skiplist

statuscache_db: berkeley-nosync
   Allowed values: berkeley, berkeley-nosync, berkeley-hash,
berkeley-hash-nosync, skiplist

tlscache_db: berkeley-nosync
   Allowed values: berkeley, berkeley-nosync, berkeley-hash,
berkeley-hash-nosync, skiplist, sql

userdeny_db: flat
   Allowed values: flat, berkeley, berkeley-hash, skiplist, sql

http://www.cyrusimap.org/docs/cyrus-imapd/2.3.16/internal/database-formats.php
has some discussion of the databases.

I don't know why the (rpm) upgrade script chose skiplist as an
intermediate format, rather than flat. Perhaps skiplist hasn't changed 
much between major versions, and offers a speed benefit.

The script seems to use the file command to determine the current 
database types, since they can be changed by an admin.

 >2.
 >The Redhat cvt_cyrusdb_all script seems to assume a specific set of
 >database files.  Is the set of cyrus imap DB files fixed, and if so
 >what are they?  Is there any documentation on what each database file
 >contains? This would be very useful to people trying to convert older
 >cyrus IMAP installations to new ones.

As far as skiplist, that seems to be a proprietary format.

As far as I can tell, all databases use the same underlying structure
(except for quotalegacy), which is defined in the database-formats above.

flat: ASCII text file

berkeley: berkeley db (with safer values).

berkeley-nosync: berkeley db with the DB_TXN_NOSYNC flag set.

berkeley-hash: berkeley db with the DB_HASH flag set.

berkeley-hash-nosync: berkeley db with both DB_TXN_NOSYNC and DB_HASH
flags?

sql: A mysql, pgsql, or sqlite store.

quotalegacy: See database-formats

skiplist: ?

With cyrus version 2.3.13 and above, the cyr_dbtool can be used to 
insert and remove records manually for each database.

-- 
Dan White