[PATCH] Re: Syncing gmail labels for gmail accounts
Nicolas Sebrecht
nicolas.s-dev at laposte.net
Sat Jan 26 17:01:40 GMT 2013
Topic with a huge patch with interesting discussion:
http://article.gmane.org/gmane.mail.imap.offlineimap.general/5943
Resending it.
On Thu, Oct 25, 2012 at 08:33:33PM +0200, Abdó Roig-Maranges wrote:
>
> Hi,
>
> I've been trying to find a good way to sync gmail labels (and use them
> in the email client) for some time now. My system up until now involved
> syncing labels into an external file, and using a lot of glue scripts to
> keep them in sync, update notmuch tags, etc. Very messy.
>
> I've now decided to try a different approach, which seems pretty close
> to the right one. I've added support in offlineimap to sync gmail
> labels, and store them inside the messages, under the header
> X-Keywords. This way, the labels "move with the messages" (even if I
> copy the maildir to an other machine, etc etc).
>
> There are some MUA that even recognize to some degree the X-Keywords
> header or a close relative, X-Label [2]. It seems mutt supports X-Label
> [3] and mu supports both [4].
>
> I've been happily using this approach, with mu/mu4e as my MUA / indexer,
> for a week now. So here I attach a couple of patches (against current
> master) which do the following (see the individual commits for more
> details).
>
> bugfixes.patch :
>
> 1. fixes a couple of easy bugs I found. It must be applied first, and
> is independent of the rest.
>
>
> gmail-labels.patch:
>
> 2. When a message goes out of gmail, adds a header X-Keywords with a
> comma separated list of labels.
>
> 3. Updates the LocalStatus Sqlite table to include columns for labels
> and local mtimes. For non-gmail repositories these columns are
> ignored.
>
> 4. When labels change on the gmail side, syncs them the same way as
> flags get synced (comparing with LocalStatus etc)
>
> 5. Adds a GmailMaildir folder type, which keeps track of individual
> message modification times (the POSIX mtime), and uses it to spot
> messages which have been modified locally. Then, only for those
> modified messages (typically very few), reads the labels and syncs
> them back to gmail, the same way as flags.
>
> 6. Adds an option to filter out certain headers when uploading
> messages to gmail. One may want to remove X-Keywords before sending
> a message back to gmail.
>
> 7. Adds an option to ignore certain labels, like \Draft, for which
> flags serve the same purpose. Gmail internally keeps the D flag in
> sync with the \Draft label, or the F flag with the \Starred label.
>
>
> Some comments:
>
> 1. These changes (which are quite a few) should interfere minimally
> with non-gmail users. The only exception is the update on
> LocalStatus sqlite table, which I hope will cause little trouble.
>
> 2. There are some issues with the SQLite backend. Right now, it
> commits to the database too frequently IMHO (after every message
> copy). This produces a lot of disk activity. I may look into
> it... my approach would be storing the status in memory during the
> message copying and commit to database once, at the end. Any
> thoughts? I don't think there is danger of losing data, on
> crashes. The LocalStatus will be updated correctly on the next run.
>
> 3. The slower part is folder.Gmail.cachemessagelist, that downloads
> the uids and labels for all messages. It takes about 17 seconds
> with about 25k moderately labelled messages. I have done some
> experiments with multiple threads without improvement. My guess is
> that on the gmail side there is some sort of bandwith
> throttling. Not even compressing the connection improves matters.
>
> Well, that's it. I send this patch hoping some developer here may take
> the time to look at it. Being able to use and sync labels with the web
> interface or mobile would be a very nice addition for which, as far as I
> know, there is no alternative solution out there.
>
>
> [1] http://comments.gmane.org/gmane.mail.imap.offlineimap.general/5916
> [2] http://does-not-exist.org/mail-archives/mutt-dev/msg08249.html
> [3] http://blitiri.com.ar/p/other/mutt-labels/
> [4] https://github.com/djcb/mu/issues/40
>
> Abdó.
>
> From 76d8ac5343cb5c5f5110c14caf0a30fc7e945cc3 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Abd=C3=B3=20Roig-Maranges?= <abdo.roig at gmail.com>
> Date: Tue, 23 Oct 2012 20:05:59 +0200
> Subject: [PATCH 1/2] change_message_uid did not update messagelist's filename
>
> This broke code that relied on the filename being up to date in memory after
> messages are copied.
> ---
> offlineimap/folder/Maildir.py | 13 +++++++------
> 1 file changed, 7 insertions(+), 6 deletions(-)
>
> diff --git a/offlineimap/folder/Maildir.py b/offlineimap/folder/Maildir.py
> index 24d943c..d080eee 100644
> --- a/offlineimap/folder/Maildir.py
> +++ b/offlineimap/folder/Maildir.py
> @@ -196,7 +196,7 @@ class MaildirFolder(BaseFolder):
> if sorted(self.getmessageuidlist()) != \
> sorted(statusfolder.getmessageuidlist()):
> return True
> - # Also check for flag changes, it's quick on a Maildir
> + # Also check for flag changes, it's quick on a Maildir
> for (uid, message) in self.getmessagelist().iteritems():
> if message['flags'] != statusfolder.getmessageflags(uid):
> return True
> @@ -235,7 +235,7 @@ class MaildirFolder(BaseFolder):
> return '%d_%d.%d.%s,U=%d,FMD5=%s%s2,%s' % \
> (timeval, timeseq, os.getpid(), socket.gethostname(),
> uid, self._foldermd5, self.infosep, ''.join(sorted(flags)))
> -
> +
> def savemessage(self, uid, content, flags, rtime):
> """Writes a new message, with the specified uid.
>
> @@ -263,7 +263,7 @@ class MaildirFolder(BaseFolder):
> fd = os.open(os.path.join(tmpdir, messagename),
> os.O_EXCL|os.O_CREAT|os.O_WRONLY, 0o666)
> except OSError as e:
> - if e.errno == 17:
> + if e.errno == 17:
> #FILE EXISTS ALREADY
> severity = OfflineImapError.ERROR.MESSAGE
> raise OfflineImapError("Unique filename %s already existing." %\
> @@ -344,11 +344,13 @@ class MaildirFolder(BaseFolder):
> dir_prefix, filename = os.path.split(oldfilename)
> flags = self.getmessageflags(uid)
> filename = self.new_message_filename(new_uid, flags)
> + newfilename = os.path.join(dir_prefix, filename)
> os.rename(os.path.join(self.getfullname(), oldfilename),
> - os.path.join(self.getfullname(), dir_prefix, filename))
> + os.path.join(self.getfullname(), newfilename))
> self.messagelist[new_uid] = self.messagelist[uid]
> + self.messagelist[new_uid]['filename'] = newfilename
> del self.messagelist[uid]
> -
> +
> def deletemessage(self, uid):
> """Unlinks a message file from the Maildir.
>
> @@ -373,4 +375,3 @@ class MaildirFolder(BaseFolder):
> os.unlink(filepath)
> # Yep -- return.
> del(self.messagelist[uid])
> -
> --
> 1.8.0
>
>
> From 7fd79f89b684af22f19bc460025e2ed12afaa9ee Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Abd=C3=B3=20Roig-Maranges?= <abdo.roig at gmail.com>
> Date: Thu, 18 Oct 2012 19:23:34 +0200
> Subject: [PATCH 2/2] Changed NotImplementedException to NotImplementedError
>
> It seems NotImplementedException does not exist. It must be a relic from old
> python...
> ---
> offlineimap/folder/Base.py | 28 ++++++++++++++--------------
> 1 file changed, 14 insertions(+), 14 deletions(-)
>
> diff --git a/offlineimap/folder/Base.py b/offlineimap/folder/Base.py
> index c132713..0c8be9e 100644
> --- a/offlineimap/folder/Base.py
> +++ b/offlineimap/folder/Base.py
> @@ -73,7 +73,7 @@ class BaseFolder(object):
> def getcopyinstancelimit(self):
> """For threading folders, returns the instancelimitname for
> InstanceLimitedThreads."""
> - raise NotImplementedException
> + raise NotImplementedError
>
> def storesmessages(self):
> """Should be true for any backend that actually saves message bodies.
> @@ -169,18 +169,18 @@ class BaseFolder(object):
>
> This function needs to be implemented by each Backend
> :returns: UIDVALIDITY as a (long) number"""
> - raise NotImplementedException
> + raise NotImplementedError
>
> def cachemessagelist(self):
> """Reads the message list from disk or network and stores it in
> memory for later use. This list will not be re-read from disk or
> memory unless this function is called again."""
> - raise NotImplementedException
> + raise NotImplementedError
>
> def getmessagelist(self):
> """Gets the current message list.
> You must call cachemessagelist() before calling this function!"""
> - raise NotImplementedException
> + raise NotImplementedError
>
> def uidexists(self, uid):
> """Returns True if uid exists"""
> @@ -197,7 +197,7 @@ class BaseFolder(object):
>
> def getmessage(self, uid):
> """Returns the content of the specified message."""
> - raise NotImplementedException
> + raise NotImplementedError
>
> def savemessage(self, uid, content, flags, rtime):
> """Writes a new message, with the specified uid.
> @@ -209,7 +209,7 @@ class BaseFolder(object):
> If the backend CAN assign a new uid, but cannot find out what
> this UID is (as is the case with some IMAP servers), it
> returns 0 but DOES save the message.
> -
> +
> IMAP backend should be the only one that can assign a new
> uid.
>
> @@ -221,15 +221,15 @@ class BaseFolder(object):
> so you need to ensure that savemessage is never called in a
> dryrun mode.
> """
> - raise NotImplementedException
> + raise NotImplementedError
>
> def getmessagetime(self, uid):
> """Return the received time for the specified message."""
> - raise NotImplementedException
> + raise NotImplementedError
>
> def getmessageflags(self, uid):
> """Returns the flags for the specified message."""
> - raise NotImplementedException
> + raise NotImplementedError
>
> def savemessageflags(self, uid, flags):
> """Sets the specified message's flags to the given set.
> @@ -237,7 +237,7 @@ class BaseFolder(object):
> Note that this function does not check against dryrun settings,
> so you need to ensure that it is never called in a
> dryrun mode."""
> - raise NotImplementedException
> + raise NotImplementedError
>
> def addmessageflags(self, uid, flags):
> """Adds the specified flags to the message's flag set. If a given
> @@ -285,14 +285,14 @@ class BaseFolder(object):
> :param new_uid: (optional) If given, the old UID will be changed
> to a new UID. This allows backends efficient renaming of
> messages if the UID has changed."""
> - raise NotImplementedException
> + raise NotImplementedError
>
> def deletemessage(self, uid):
> """
> Note that this function does not check against dryrun settings,
> so you need to ensure that it is never called in a
> dryrun mode."""
> - raise NotImplementedException
> + raise NotImplementedError
>
> def deletemessages(self, uidlist):
> """
> @@ -492,7 +492,7 @@ class BaseFolder(object):
> continue #don't actually remove in a dryrun
> dstfolder.deletemessagesflags(uids, set(flag))
> statusfolder.deletemessagesflags(uids, set(flag))
> -
> +
> def syncmessagesto(self, dstfolder, statusfolder):
> """Syncs messages in this folder to the destination dstfolder.
>
> @@ -513,7 +513,7 @@ class BaseFolder(object):
> uids present (except for potential negative uids that couldn't
> be placed anywhere).
>
> - Pass3: Synchronize flag changes
> + Pass3: Synchronize flag changes
> Compare flag mismatches in self with those in statusfolder. If
> msg has a valid UID and exists on dstfolder (has not e.g. been
> deleted there), sync the flag change to both dstfolder and
> --
> 1.8.0
>
> From 2233ed9e01647bd41311ef29f221c5a5dbb14d52 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Abd=C3=B3=20Roig-Maranges?= <abdo.roig at gmail.com>
> Date: Wed, 17 Oct 2012 21:45:19 +0200
> Subject: [PATCH 1/6] Restructured folder.IMAP code
>
> Preparing for a sequence of commits implementing gmail label sync, I have split
> some functions so I will be able to change some functionality in folder.Gmail,
> with less code repetition.
>
> Also added some functions to folder.base and imaputil I will later need.
> ---
> offlineimap/folder/Base.py | 116 ++++++++++++++++++++-
> offlineimap/folder/IMAP.py | 244 ++++++++++++++++++++++++---------------------
> offlineimap/imaputil.py | 10 ++
> 3 files changed, 250 insertions(+), 120 deletions(-)
>
> diff --git a/offlineimap/folder/Base.py b/offlineimap/folder/Base.py
> index 0c8be9e..7346965 100644
> --- a/offlineimap/folder/Base.py
> +++ b/offlineimap/folder/Base.py
> @@ -48,6 +48,11 @@ class BaseFolder(object):
> self.visiblename = ''
> self.config = repository.getconfig()
>
> + # Passes for syncmessagesto
> + self.syncmessagesto_passes = [('copying messages' , self.syncmessagesto_copy),
> + ('deleting messages' , self.syncmessagesto_delete),
> + ('syncing flags' , self.syncmessagesto_flags)]
> +
> def getname(self):
> """Returns name"""
> return self.name
> @@ -227,6 +232,10 @@ class BaseFolder(object):
> """Return the received time for the specified message."""
> raise NotImplementedError
>
> + def getmessagemtime(self, uid):
> + """Returns the message modification time of the specified message."""
> + raise NotImplementedError
> +
> def getmessageflags(self, uid):
> """Returns the flags for the specified message."""
> raise NotImplementedError
> @@ -277,6 +286,58 @@ class BaseFolder(object):
> for uid in uidlist:
> self.deletemessageflags(uid, flags)
>
> +
> + def getmessagelabels(self, uid):
> + """Returns the labels for the specified message."""
> + raise NotImplementedError
> +
> + def savemessagelabels(self, uid, labels, ignorelabels=set(), mtime=0):
> + """Sets the specified message's labels to the given set.
> +
> + Note that this function does not check against dryrun settings,
> + so you need to ensure that it is never called in a
> + dryrun mode."""
> + raise NotImplementedError
> +
> + def addmessagelabels(self, uid, labels):
> + """Adds the specified labels to the message's labels set. If a given
> + label is already present, it will not be duplicated.
> +
> + Note that this function does not check against dryrun settings,
> + so you need to ensure that it is never called in a
> + dryrun mode.
> +
> + :param labels: A set() of labels"""
> + newlabels = self.getmessagelabels(uid) | labels
> + self.savemessagelabels(uid, newlabels)
> +
> + def addmessageslabels(self, uidlist, labels):
> + """
> + Note that this function does not check against dryrun settings,
> + so you need to ensure that it is never called in a
> + dryrun mode."""
> + for uid in uidlist:
> + self.addmessagelabels(uid, labels)
> +
> + def deletemessagelabels(self, uid, labels):
> + """Removes each label given from the message's label set. If a given
> + label is already removed, no action will be taken for that label.
> +
> + Note that this function does not check against dryrun settings,
> + so you need to ensure that it is never called in a
> + dryrun mode."""
> + newlabels = self.getmessagelabels(uid) - labels
> + self.savemessagelabels(uid, newlabels)
> +
> + def deletemessageslabels(self, uidlist, labels):
> + """
> + Note that this function does not check against dryrun settings,
> + so you need to ensure that it is never called in a
> + dryrun mode."""
> + for uid in uidlist:
> + self.deletemessagelabels(uid, labels)
> +
> +
> def change_message_uid(self, uid, new_uid):
> """Change the message from existing uid to new_uid
>
> @@ -302,6 +363,51 @@ class BaseFolder(object):
> for uid in uidlist:
> self.deletemessage(uid)
>
> + def message_addheader(self, content, headername, headervalue):
> + """Changes the value of headername to headervalue if the header exists,
> + or adds it if it does not exist"""
> +
> + self.ui.debug('',
> + 'message_addheader: called to add %s: %s' % (headername,
> + headervalue))
> + insertionpoint = content.find("\n\n")
> + self.ui.debug('', 'message_addheader: insertionpoint = %d' % insertionpoint)
> + leader = content[0:insertionpoint]
> + self.ui.debug('', 'message_addheader: leader = %s' % repr(leader))
> + if insertionpoint == 0 or insertionpoint == -1:
> + newline = ''
> + insertionpoint = 0
> + else:
> + newline = "\n"
> +
> + if re.search('^%s:(.*)$' % headername, leader, flags = re.MULTILINE):
> + leader = re.sub('^%s:(.*)$' % headername, '%s: %s' % (headername, headervalue),
> + leader, flags = re.MULTILINE)
> + else:
> + leader = leader + newline + "%s: %s" % (headername, headervalue)
> +
> + self.ui.debug('', 'message_addheader: newline = ' + repr(newline))
> + trailer = content[insertionpoint:]
> + self.ui.debug('', 'message_addheader: trailer = ' + repr(trailer))
> + return leader + trailer
> +
> + def message_getheader(self, content, headername):
> + """Gets the value of the header 'headername' in 'content'. Returns None
> + if can't find the header."""
> +
> + self.ui.debug('',
> + 'message_getheader: called to get %s' % headername)
> + insertionpoint = content.find("\n\n")
> + self.ui.debug('', 'message_getheader: insertionpoint = %d' % insertionpoint)
> + leader = content[0:insertionpoint]
> + self.ui.debug('', 'message_getheader: leader = %s' % repr(leader))
> +
> + m = re.search('^%s:(.*)$' % headername, leader, flags = re.MULTILINE)
> + if m:
> + return m.group(1).strip()
> + else:
> + return None
> +
> def copymessageto(self, uid, dstfolder, statusfolder, register = 1):
> """Copies a message from self to dst if needed, updating the status
>
> @@ -519,14 +625,16 @@ class BaseFolder(object):
> deleted there), sync the flag change to both dstfolder and
> statusfolder.
>
> + Pass4: Synchronize label changes (Gmail only)
> + Compares label mismatches in self with those in statusfolder.
> + If msg has a valid UID and exists on dstfolder, syncs the labels
> + to both dstfolder and statusfolder.
> +
> :param dstfolder: Folderinstance to sync the msgs to.
> :param statusfolder: LocalStatus instance to sync against.
> """
> - passes = [('copying messages' , self.syncmessagesto_copy),
> - ('deleting messages' , self.syncmessagesto_delete),
> - ('syncing flags' , self.syncmessagesto_flags)]
>
> - for (passdesc, action) in passes:
> + for (passdesc, action) in self.syncmessagesto_passes:
> # bail out on CTRL-C or SIGTERM
> if offlineimap.accounts.Account.abort_NOW_signal.is_set():
> break
> diff --git a/offlineimap/folder/IMAP.py b/offlineimap/folder/IMAP.py
> index 298f9fd..9fa88ee 100644
> --- a/offlineimap/folder/IMAP.py
> +++ b/offlineimap/folder/IMAP.py
> @@ -43,7 +43,7 @@ class IMAPFolder(BaseFolder):
>
> Prefer SELECT to EXAMINE if we can, since some servers
> (Courier) do not stabilize UID validity until the folder is
> - selected.
> + selected.
> .. todo: Still valid? Needs verification
> :param: Enforce new SELECT even if we are on that folder already.
> :returns: raises :exc:`OfflineImapError` severity FOLDER on error"""
> @@ -115,59 +115,66 @@ class IMAPFolder(BaseFolder):
> maxmsgid = max(long(msgid), maxmsgid)
> # Different number of messages than last time?
> if maxmsgid != statusfolder.getmessagecount():
> - return True
> + return True
> return False
>
> - def cachemessagelist(self):
> +
> + def _msgs_to_fetch(self, imapobj):
> maxage = self.config.getdefaultint("Account %s" % self.accountname,
> "maxage", -1)
> maxsize = self.config.getdefaultint("Account %s" % self.accountname,
> "maxsize", -1)
> - self.messagelist = {}
>
> + res_type, imapdata = imapobj.select(self.getfullname(), True, True)
> + if imapdata == [None] or imapdata[0] == '0':
> + # Empty folder, no need to populate message list
> + return
> +
> + # By default examine all UIDs in this folder
> + msgsToFetch = '1:*'
> +
> + # Build search condition
> + if (maxage != -1) | (maxsize != -1):
> + search_cond = "(";
> +
> + if(maxage != -1):
> + #find out what the oldest message is that we should look at
> + oldest_struct = time.gmtime(time.time() - (60*60*24*maxage))
> + if oldest_struct[0] < 1900:
> + raise OfflineImapError("maxage setting led to year %d. "
> + "Abort syncing." % oldest_struct[0],
> + OfflineImapError.ERROR.REPO)
> + search_cond += "SINCE %02d-%s-%d" % (
> + oldest_struct[2],
> + MonthNames[oldest_struct[1]],
> + oldest_struct[0])
> +
> + if(maxsize != -1):
> + if(maxage != -1): # There are two conditions, add space
> + search_cond += " "
> + search_cond += "SMALLER %d" % maxsize
> +
> + search_cond += ")"
> +
> + res_type, res_data = imapobj.search(None, search_cond)
> + if res_type != 'OK':
> + raise OfflineImapError("SEARCH in folder [%s]%s failed. "
> + "Search string was '%s'. Server responded '[%s] %s'" % (
> + self.getrepository(), self, search_cond, res_type, res_data),
> + OfflineImapError.ERROR.FOLDER)
> +
> + # Result UIDs are seperated by space, coalesce into ranges
> + msgsToFetch = imaputil.uid_sequence(res_data[0].split())
> +
> + return msgsToFetch
> +
> +
> + def cachemessagelist(self):
> imapobj = self.imapserver.acquireconnection()
> try:
> - res_type, imapdata = imapobj.select(self.getfullname(), True, True)
> - if imapdata == [None] or imapdata[0] == '0':
> - # Empty folder, no need to populate message list
> - return
> - # By default examine all UIDs in this folder
> - msgsToFetch = '1:*'
> -
> - if (maxage != -1) | (maxsize != -1):
> - search_cond = "(";
> -
> - if(maxage != -1):
> - #find out what the oldest message is that we should look at
> - oldest_struct = time.gmtime(time.time() - (60*60*24*maxage))
> - if oldest_struct[0] < 1900:
> - raise OfflineImapError("maxage setting led to year %d. "
> - "Abort syncing." % oldest_struct[0],
> - OfflineImapError.ERROR.REPO)
> - search_cond += "SINCE %02d-%s-%d" % (
> - oldest_struct[2],
> - MonthNames[oldest_struct[1]],
> - oldest_struct[0])
> -
> - if(maxsize != -1):
> - if(maxage != -1): # There are two conditions, add space
> - search_cond += " "
> - search_cond += "SMALLER %d" % maxsize
> -
> - search_cond += ")"
> -
> - res_type, res_data = imapobj.search(None, search_cond)
> - if res_type != 'OK':
> - raise OfflineImapError("SEARCH in folder [%s]%s failed. "
> - "Search string was '%s'. Server responded '[%s] %s'" % (
> - self.getrepository(), self,
> - search_cond, res_type, res_data),
> - OfflineImapError.ERROR.FOLDER)
> -
> - # Result UIDs are seperated by space, coalesce into ranges
> - msgsToFetch = imaputil.uid_sequence(res_data[0].split())
> - if not msgsToFetch:
> - return # No messages to sync
> + msgsToFetch = self._msgs_to_fetch(imapobj)
> + if not msgsToFetch:
> + return # No messages to sync
>
> # Get the flags and UIDs for these. single-quotes prevent
> # imaplib2 from quoting the sequence.
> @@ -182,6 +189,7 @@ class IMAPFolder(BaseFolder):
> finally:
> self.imapserver.releaseconnection(imapobj)
>
> + self.messagelist = {}
> for messagestr in response:
> # looks like: '1 (FLAGS (\\Seen Old) UID 4807)' or None if no msg
> # Discard initial message number.
> @@ -199,9 +207,41 @@ class IMAPFolder(BaseFolder):
> rtime = imaplibutil.Internaldate2epoch(messagestr)
> self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime}
>
> +
> def getmessagelist(self):
> return self.messagelist
>
> +
> + def _fetch_from_imap(self, imapobj, uids, query, retry_num=1):
> + fails_left = retry_num # retry on dropped connection
> + while fails_left:
> + try:
> + imapobj.select(self.getfullname(), readonly = True)
> + res_type, data = imapobj.uid('fetch', uids, query)
> + fails_left = 0
> + except imapobj.abort as e:
> + # Release dropped connection, and get a new one
> + self.imapserver.releaseconnection(imapobj, True)
> + imapobj = self.imapserver.acquireconnection()
> + self.ui.error(e, exc_info()[2])
> + fails_left -= 1
> + if not fails_left:
> + raise e
> + if data == [None] or res_type != 'OK':
> + #IMAP server says bad request or UID does not exist
> + severity = OfflineImapError.ERROR.MESSAGE
> + reason = "IMAP server '%s' failed to fetch messages UID '%s'."\
> + "Server responded: %s %s" % (self.getrepository(), uids,
> + res_type, data)
> + if data == [None]:
> + #IMAP server did not find a message with this UID
> + reason = "IMAP server '%s' does not have a message "\
> + "with UID '%s'" % (self.getrepository(), uids)
> + raise OfflineImapError(reason, severity)
> +
> + return data
> +
> +
> def getmessage(self, uid):
> """Retrieve message with UID from the IMAP server (incl body)
>
> @@ -211,47 +251,24 @@ class IMAPFolder(BaseFolder):
> """
> imapobj = self.imapserver.acquireconnection()
> try:
> - fails_left = 2 # retry on dropped connection
> - while fails_left:
> - try:
> - imapobj.select(self.getfullname(), readonly = True)
> - res_type, data = imapobj.uid('fetch', str(uid),
> - '(BODY.PEEK[])')
> - fails_left = 0
> - except imapobj.abort as e:
> - # Release dropped connection, and get a new one
> - self.imapserver.releaseconnection(imapobj, True)
> - imapobj = self.imapserver.acquireconnection()
> - self.ui.error(e, exc_info()[2])
> - fails_left -= 1
> - if not fails_left:
> - raise e
> - if data == [None] or res_type != 'OK':
> - #IMAP server says bad request or UID does not exist
> - severity = OfflineImapError.ERROR.MESSAGE
> - reason = "IMAP server '%s' failed to fetch message UID '%d'."\
> - "Server responded: %s %s" % (self.getrepository(), uid,
> - res_type, data)
> - if data == [None]:
> - #IMAP server did not find a message with this UID
> - reason = "IMAP server '%s' does not have a message "\
> - "with UID '%s'" % (self.getrepository(), uid)
> - raise OfflineImapError(reason, severity)
> - # data looks now e.g. [('320 (UID 17061 BODY[]
> - # {2565}','msgbody....')] we only asked for one message,
> - # and that msg is in data[0]. msbody is in [0][1]
> - data = data[0][1].replace("\r\n", "\n")
> -
> - if len(data)>200:
> - dbg_output = "%s...%s" % (str(data)[:150],
> - str(data)[-50:])
> - else:
> - dbg_output = data
> - self.ui.debug('imap', "Returned object from fetching %d: '%s'" %
> - (uid, dbg_output))
> + data = self._fetch_from_imap(imapobj, str(uid), '(X-GM-LABELS BODY.PEEK[])', 2)
> finally:
> self.imapserver.releaseconnection(imapobj)
> - return data
> +
> + # data looks now e.g. [('320 (UID 17061 BODY[]
> + # {2565}','msgbody....')] we only asked for one message,
> + # and that msg is in data[0]. msbody is in [0][1]
> + body = data[0][1].replace("\r\n", "\n")
> +
> + if len(body)>200:
> + dbg_output = "%s...%s" % (str(body)[:150], str(body)[-50:])
> + else:
> + dbg_output = body
> +
> + self.ui.debug('imap', "Returned object from fetching %d: '%s'" %
> + (uid, dbg_output))
> +
> + return body
>
> def getmessagetime(self, uid):
> return self.messagelist[uid]['time']
> @@ -287,23 +304,9 @@ class IMAPFolder(BaseFolder):
>
>
> def savemessage_addheader(self, content, headername, headervalue):
> - self.ui.debug('imap',
> - 'savemessage_addheader: called to add %s: %s' % (headername,
> - headervalue))
> - insertionpoint = content.find("\r\n\r\n")
> - self.ui.debug('imap', 'savemessage_addheader: insertionpoint = %d' % insertionpoint)
> - leader = content[0:insertionpoint]
> - self.ui.debug('imap', 'savemessage_addheader: leader = %s' % repr(leader))
> - if insertionpoint == 0 or insertionpoint == -1:
> - newline = ''
> - insertionpoint = 0
> - else:
> - newline = "\r\n"
> - newline += "%s: %s" % (headername, headervalue)
> - self.ui.debug('imap', 'savemessage_addheader: newline = ' + repr(newline))
> - trailer = content[insertionpoint:]
> - self.ui.debug('imap', 'savemessage_addheader: trailer = ' + repr(trailer))
> - return leader + newline + trailer
> + """Adds the header 'headername' with value 'headervalue' to content and
> + returns it"""
> + return self.message_addheader(content, headername, headervalue)
>
>
> def savemessage_searchforheader(self, imapobj, headername, headervalue):
> @@ -516,7 +519,6 @@ class IMAPFolder(BaseFolder):
>
> # get the date of the message, so we can pass it to the server.
> date = self.getmessageinternaldate(content, rtime)
> - content = re.sub("(?<!\r)\n", "\r\n", content)
>
> if not use_uidplus:
> # insert a random unique header that we can fetch later
> @@ -525,7 +527,10 @@ class IMAPFolder(BaseFolder):
> self.ui.debug('imap', 'savemessage: header is: %s: %s' %\
> (headername, headervalue))
> content = self.savemessage_addheader(content, headername,
> - headervalue)
> + headervalue)
> +
> + content = re.sub("(?<!\r)\n", "\r\n", content)
> +
> if len(content)>200:
> dbg_output = "%s...%s" % (content[:150], content[-50:])
> else:
> @@ -615,6 +620,17 @@ class IMAPFolder(BaseFolder):
> self.ui.debug('imap', 'savemessage: returning new UID %d' % uid)
> return uid
>
> + def _store_to_imap(self, imapobj, uid, field, data):
> + imapobj.select(self.getfullname())
> + res_type, retdata = imapobj.uid('store', uid, field, data)
> + if res_type != 'OK':
> + severity = OfflineImapError.ERROR.MESSAGE
> + reason = "IMAP server '%s' failed to store %s for message UID '%d'."\
> + "Server responded: %s %s" % (self.getrepository(), field, uid,
> + res_type, retdata)
> + raise OfflineImapError(reason, severity)
> + return retdata[0]
> +
> def savemessageflags(self, uid, flags):
> """Change a message's flags to `flags`.
>
> @@ -623,17 +639,15 @@ class IMAPFolder(BaseFolder):
> dryrun mode."""
> imapobj = self.imapserver.acquireconnection()
> try:
> - try:
> - imapobj.select(self.getfullname())
> - except imapobj.readonly:
> - self.ui.flagstoreadonly(self, [uid], flags)
> - return
> - result = imapobj.uid('store', '%d' % uid, 'FLAGS',
> - imaputil.flagsmaildir2imap(flags))
> - assert result[0] == 'OK', 'Error with store: ' + '. '.join(result[1])
> + result = self._store_to_imap(imapobj, str(uid), 'FLAGS', imaputil.flagsmaildir2imap(flags))
> +
> + except imapobj.readonly:
> + self.ui.flagstoreadonly(self, [uid], data)
> + return
> +
> finally:
> self.imapserver.releaseconnection(imapobj)
> - result = result[1][0]
> +
> if not result:
> self.messagelist[uid]['flags'] = flags
> else:
> @@ -709,11 +723,11 @@ class IMAPFolder(BaseFolder):
> def change_message_uid(self, uid, new_uid):
> """Change the message from existing uid to new_uid
>
> - If the backend supports it. IMAP does not and will throw errors."""
> + If the backend supports it. IMAP does not and will throw errors."""
> raise OfflineImapError('IMAP backend cannot change a messages UID from '
> '%d to %d' % (uid, new_uid),
> OfflineImapError.ERROR.MESSAGE)
> -
> +
> def deletemessage(self, uid):
> self.deletemessages_noconvert([uid])
>
> @@ -740,5 +754,3 @@ class IMAPFolder(BaseFolder):
> self.imapserver.releaseconnection(imapobj)
> for uid in uidlist:
> del self.messagelist[uid]
> -
> -
> diff --git a/offlineimap/imaputil.py b/offlineimap/imaputil.py
> index fe69b7a..88ae37a 100644
> --- a/offlineimap/imaputil.py
> +++ b/offlineimap/imaputil.py
> @@ -46,6 +46,16 @@ def dequote(string):
> string = string.replace('\\\\', '\\')
> return string
>
> +def quote(string):
> + """Takes an unquoted string and quotes it.
> +
> + It only adds double quotes. This function does NOT consider
> + parenthised lists to be quoted.
> + """
> + string = string.replace('"', '\\"')
> + string = string.replace('\\', '\\\\')
> + return '"%s"' % string
> +
> def flagsplit(string):
> """Converts a string of IMAP flags to a list
>
> --
> 1.8.0
>
>
> From 31c13de625a32f016d8184bc661463bbb2c19f73 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Abd=C3=B3=20Roig-Maranges?= <abdo.roig at gmail.com>
> Date: Wed, 17 Oct 2012 22:07:07 +0200
> Subject: [PATCH 2/6] Adds labels and mtime to LocalStatus Sqlite table
>
> Adds two columns to the LocalStatus Sqlite table:
> * labels: A comma separated list of labels, to be used by Gmail folder.
> * mtime: The POSIX modification time for the message in a local Maildir.
>
> The interface for the class LocalStatusSQLite remains compatible with what it
> was (i.e. new arguments to functions are optional with default values).
> ---
> offlineimap/folder/LocalStatusSQLite.py | 102 +++++++++++++++++++++++++++-----
> 1 file changed, 87 insertions(+), 15 deletions(-)
>
> diff --git a/offlineimap/folder/LocalStatusSQLite.py b/offlineimap/folder/LocalStatusSQLite.py
> index ac67c2f..e4da5da 100644
> --- a/offlineimap/folder/LocalStatusSQLite.py
> +++ b/offlineimap/folder/LocalStatusSQLite.py
> @@ -35,15 +35,15 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
> #though. According to sqlite docs, you need to commit() before
> #the connection is closed or your changes will be lost!"""
> #get db connection which autocommits
> - #connection = sqlite.connect(self.filename, isolation_level=None)
> + #connection = sqlite.connect(self.filename, isolation_level=None)
> #cursor = connection.cursor()
> #return connection, cursor
>
> #current version of our db format
> - cur_version = 1
> + cur_version = 2
>
> def __init__(self, name, repository):
> - super(LocalStatusSQLiteFolder, self).__init__(name, repository)
> + super(LocalStatusSQLiteFolder, self).__init__(name, repository)
> # dblock protects against concurrent writes in same connection
> self._dblock = Lock()
> #Try to establish connection, no need for threadsafety in __init__
> @@ -69,6 +69,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
> if version < LocalStatusSQLiteFolder.cur_version:
> self.upgrade_db(version)
>
> +
> def sql_write(self, sql, vars=None, executemany=False):
> """Execute some SQL, retrying if the db was locked.
>
> @@ -114,9 +115,14 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
> self.connection = sqlite.connect(self.filename,
> check_same_thread = False)
>
> + # Upgrades from plain text to database version 1
> if from_ver == 0:
> - # from_ver==0: no db existent: plain text migration?
> - self.create_db()
> + self.connection.executescript("""
> + CREATE TABLE metadata (key VARCHAR(50) PRIMARY KEY, value VARCHAR(128));
> + INSERT INTO metadata VALUES('db_version', '1');
> + CREATE TABLE status (id INTEGER PRIMARY KEY, flags VARCHAR(50));
> + """)
> +
> # below was derived from repository.getfolderfilename() logic
> plaintextfilename = os.path.join(
> self.repository.account.getaccountmeta(),
> @@ -140,9 +146,22 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
> self.connection.commit()
> file.close()
> os.rename(plaintextfilename, plaintextfilename + ".old")
> +
> + # Upgrade from database version 1 to version 2
> + # This change adds labels and mtime columns, to be used by Gmail IMAP and Maildir folders.
> + if from_ver <= 1:
> + self.ui._msg('Upgrading LocalStatus cache from version 1 to version 2 for %s:%s' %\
> + (self.repository, self))
> + self.connection.executescript("""ALTER TABLE status ADD mtime INTEGER DEFAULT 0;
> + ALTER TABLE status ADD labels VARCHAR(256) DEFAULT '';
> + UPDATE metadata SET value='2' WHERE key='db_version';
> + """)
> + self.connection.commit()
> +
> # Future version upgrades come here...
> - # if from_ver <= 1: ... #upgrade from 1 to 2
> # if from_ver <= 2: ... #upgrade from 2 to 3
> + # if from_ver <= 3: ... #upgrade from 3 to 4
> +
>
> def create_db(self):
> """Create a new db file"""
> @@ -154,7 +173,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
> self.connection.executescript("""
> CREATE TABLE metadata (key VARCHAR(50) PRIMARY KEY, value VARCHAR(128));
> INSERT INTO metadata VALUES('db_version', '1');
> - CREATE TABLE status (id INTEGER PRIMARY KEY, flags VARCHAR(50));
> + CREATE TABLE status (id INTEGER PRIMARY KEY, flags VARCHAR(50), mtime INTEGER, labels VARCHAR(256));
> """)
> self.connection.commit()
>
> @@ -170,10 +189,11 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
>
> def cachemessagelist(self):
> self.messagelist = {}
> - cursor = self.connection.execute('SELECT id,flags from status')
> + cursor = self.connection.execute('SELECT id,flags,mtime,labels from status')
> for row in cursor:
> - flags = set(row[1])
> - self.messagelist[row[0]] = {'uid': row[0], 'flags': flags}
> + flags = set(row[1])
> + labels = set([lb.strip() for lb in row[3].split(',') if len(lb.strip()) > 0])
> + self.messagelist[row[0]] = {'uid': row[0], 'flags': flags, 'mtime': row[2], 'labels': labels}
>
> def save(self):
> #Noop in this backend
> @@ -215,7 +235,7 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
> # return flags
> # assert False,"getmessageflags() called on non-existing message"
>
> - def savemessage(self, uid, content, flags, rtime):
> + def savemessage(self, uid, content, flags, rtime, mtime=0, labels=set()):
> """Writes a new message, with the specified uid.
>
> See folder/Base for detail. Note that savemessage() does not
> @@ -229,17 +249,69 @@ class LocalStatusSQLiteFolder(LocalStatusFolder):
> self.savemessageflags(uid, flags)
> return uid
>
> - self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime}
> + self.messagelist[uid] = {'uid': uid, 'flags': flags, 'time': rtime, 'mtime': mtime, 'labels': labels}
> flags = ''.join(sorted(flags))
> - self.sql_write('INSERT INTO status (id,flags) VALUES (?,?)',
> - (uid,flags))
> + labels = ', '.join(sorted(labels))
> + self.sql_write('INSERT INTO status (id,flags,mtime,labels) VALUES (?,?,?,?)',
> + (uid,flags,mtime,labels))
> return uid
>
> def savemessageflags(self, uid, flags):
> - self.messagelist[uid] = {'uid': uid, 'flags': flags}
> + self.messagelist[uid]['flags'] = flags
> flags = ''.join(sorted(flags))
> self.sql_write('UPDATE status SET flags=? WHERE id=?',(flags,uid))
>
> + def getmessageflags(self, uid):
> + return self.messagelist[uid]['flags']
> +
> + def savemessagelabels(self, uid, labels, mtime=None):
> + self.messagelist[uid]['labels'] = labels
> + if mtime: self.messagelist[uid]['mtime'] = mtime
> +
> + labels = ', '.join(sorted(labels))
> + if mtime:
> + self.sql_write('UPDATE status SET labels=?, mtime=? WHERE id=?',(labels,mtime,uid))
> + else:
> + self.sql_write('UPDATE status SET labels=? WHERE id=?',(labels,uid))
> +
> + def savemessageslabelsbulk(self, labels):
> + """Saves labels from a dictionary in a single database operation."""
> + data = [(', '.join(sorted(lb)), uid) for uid, lb in labels.items()]
> + self.sql_write('UPDATE status SET labels=? WHERE id=?', data, executemany=True)
> + for uid, lb in labels.items():
> + self.messagelist[uid]['labels'] = lb
> +
> + def addmessageslabels(self, uids, labels):
> + data = []
> + for uid in uids:
> + newlabels = self.messagelist[uid]['labels'] | labels
> + data.append((', '.join(sorted(newlabels)), uid))
> + self.sql_write('UPDATE status SET labels=? WHERE id=?', data, executemany=True)
> + for uid in uids:
> + self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] | labels
> +
> + def deletemessageslabels(self, uids, labels):
> + data = []
> + for uid in uids:
> + newlabels = self.messagelist[uid]['labels'] - labels
> + data.append((', '.join(sorted(newlabels)), uid))
> + self.sql_write('UPDATE status SET labels=? WHERE id=?', data, executemany=True)
> + for uid in uids:
> + self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] - labels
> +
> + def getmessagelabels(self, uid):
> + return self.messagelist[uid]['labels']
> +
> + def savemessagesmtimebulk(self, mtimes):
> + """Saves mtimes from the mtimes dictionary in a single database operation."""
> + data = [(mt, uid) for uid, mt in mtimes.items()]
> + self.sql_write('UPDATE status SET mtime=? WHERE id=?', data, executemany=True)
> + for uid, mt in mtimes.items():
> + self.messagelist[uid]['mtime'] = mt
> +
> + def getmessagemtime(self, uid):
> + return self.messagelist[uid]['mtime']
> +
> def deletemessage(self, uid):
> if not uid in self.messagelist:
> return
> --
> 1.8.0
>
>
> From a9885eebd1eee692e6ea69d0d423412a88dd2df7 Mon Sep 17 00:00:00 2001
> From: =?UTF-8?q?Abd=C3=B3=20Roig-Maranges?= <abdo.roig at gmail.com>
> Date: Tue, 16 Oct 2012 20:20:35 +0200
> Subject: [PATCH 3/6] Make GmailFolder sync gmail labels
>
> When synclabels config flag is set to "yes" for the gmail repo, offlineimap
> fetches the message labels along with the messages, and embeds them into the
> body under the header X-Keywords, as a comma separated list.
>
> The configuration option labelsheader allows to change that header under which
> labels are stored. X-Keywords is a useful choice as some mail programs may
> recognize it.
>
> It also adds an extra pass to savemessageto, that performs label synchronization
> on existing messages from gmail to local, the same way it is done with flags.
>
> The ignorelabels configuration seting contains is a list of comma separated
> labels that will be left alone. They will not be added nor removed from any
> message.
> ---
> offlineimap/folder/Gmail.py | 297 ++++++++++++++++++++++++++++++++++++++++
> offlineimap/repository/Gmail.py | 10 +-
> offlineimap/ui/UIBase.py | 28 ++++
> 3 files changed, 333 insertions(+), 2 deletions(-)
>
> diff --git a/offlineimap/folder/Gmail.py b/offlineimap/folder/Gmail.py
> index e3433c0..ed17c08 100644
> --- a/offlineimap/folder/Gmail.py
> +++ b/offlineimap/folder/Gmail.py
> @@ -16,6 +16,12 @@
> # along with this program; if not, write to the Free Software
> # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
>
> +import re
> +
> +from offlineimap import imaputil
> +from offlineimap import imaplibutil
> +import offlineimap.accounts
> +
> """Folder implementation to support features of the Gmail IMAP server.
> """
> from .IMAP import IMAPFolder
> @@ -33,6 +39,7 @@ class GmailFolder(IMAPFolder):
>
> For more information on the Gmail IMAP server:
> http://mail.google.com/support/bin/answer.py?answer=77657&topic=12815
> + https://developers.google.com/google-apps/gmail/imap_extensions
> """
>
> def __init__(self, imapserver, name, repository):
> @@ -40,3 +47,293 @@ class GmailFolder(IMAPFolder):
> self.trash_folder = repository.gettrashfolder(name)
> # Gmail will really delete messages upon EXPUNGE in these folders
> self.real_delete_folders = [ self.trash_folder, repository.getspamfolder() ]
> +
> + # The header under which labels are stored
> + self.labelsheader = self.repository.account.getconf('labelsheader', 'X-Keywords')
> +
> + # enables / disables label sync
> + self.synclabels = self.repository.account.getconfboolean('synclabels', 0)
> +
> + # if synclabels is enabled, add a 4th pass to sync labels
> + if self.synclabels:
> + self.syncmessagesto_passes.append(('syncing labels', self.syncmessagesto_labels))
> +
> + # Labels to be left alone
> + ignorelabels = self.repository.account.getconf('ignorelabels', '')
> + self.ignorelabels = set([lb.strip() for lb in ignorelabels.split(',') if len(lb.strip()) > 0])
> +
> + def getmessage(self, uid):
> + """Retrieve message with UID from the IMAP server (incl body). Also
> + gets Gmail labels and embeds them into the message.
> +
> + :returns: the message body or throws and OfflineImapError
> + (probably severity MESSAGE) if e.g. no message with
> + this UID could be found.
> + """
> + imapobj = self.imapserver.acquireconnection()
> + try:
> + data = self._fetch_from_imap(imapobj, str(uid), '(X-GM-LABELS BODY.PEEK[])', 2)
> + finally:
> + self.imapserver.releaseconnection(imapobj)
> +
> + # data looks now e.g.
> + #[('320 (X-GM-LABELS (...) UID 17061 BODY[] {2565}','msgbody....')]
> + # we only asked for one message, and that msg is in data[0].
> + # msbody is in [0][1].
> + body = data[0][1].replace("\r\n", "\n")
> +
> + # Embed the labels into the message headers
> + if self.synclabels:
> + m = re.search('X-GM-LABELS\s*\(([^\)]*)\)', data[0][0])
> + if m:
> + labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
> + else:
> + labels = set()
> + labels = labels - self.ignorelabels
> + labels = ', '.join(sorted(labels))
> + body = self.savemessage_addheader(body, self.labelsheader, labels)
> +
> + if len(body)>200:
> + dbg_output = "%s...%s" % (str(body)[:150], str(body)[-50:])
> + else:
> + dbg_output = body
> +
> + self.ui.debug('imap', "Returned object from fetching %d: '%s'" %
> + (uid, dbg_output))
> + return body
> +
> + def getmessagelabels(self, uid):
> + if 'labels' in self.messagelist[uid]:
> + return self.messagelist[uid]['labels']
> + else:
> + return set()
> +
> + def cachemessagelist(self):
> + if not self.synclabels:
> + return super(GmailFolder, self).cachemessagelist()
> +
> + self.ui.collectingdata(None, self)
> + self.messagelist = {}
> + imapobj = self.imapserver.acquireconnection()
> + try:
> + msgsToFetch = self._msgs_to_fetch(imapobj)
> + if not msgsToFetch:
> + return # No messages to sync
> +
> + # Get the flags and UIDs for these. single-quotes prevent
> + # imaplib2 from quoting the sequence.
> + data = self._fetch_from_imap(imapobj, "'%s'" % msgsToFetch,
> + '(FLAGS X-GM-LABELS UID)')
> + finally:
> + self.imapserver.releaseconnection(imapobj)
> +
> + for messagestr in data:
> + # looks like: '1 (FLAGS (\\Seen Old) UID 4807)' or None if no msg
> + # Discard initial message number.
> + if messagestr == None:
> + continue
> + messagestr = messagestr.split(' ', 1)[1]
> + options = imaputil.flags2hash(messagestr)
> + if not 'UID' in options:
> + self.ui.warn('No UID in message with options %s' %\
> + str(options),
> + minor = 1)
> + else:
> + uid = long(options['UID'])
> + flags = imaputil.flagsimap2maildir(options['FLAGS'])
> + m = re.search('\(([^\)]*)\)', options['X-GM-LABELS'])
> + if m:
> + labels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(m.group(1))])
> + else:
> + labels = set()
> + labels = labels - self.ignorelabels
> + rtime = imaplibutil.Internaldate2epoch(messagestr)
> + self.messagelist[uid] = {'uid': uid, 'flags': flags, 'labels': labels, 'time': rtime}
> +
> + def savemessage(self, uid, content, flags, rtime):
> + """Save the message on the Server
> +
> + This backend always assigns a new uid, so the uid arg is ignored.
> +
> + This function will update the self.messagelist dict to contain
> + the new message after sucessfully saving it, including labels.
> +
> + See folder/Base for details. Note that savemessage() does not
> + check against dryrun settings, so you need to ensure that
> + savemessage is never called in a dryrun mode.
> +
> + :param rtime: A timestamp to be used as the mail date
> + :returns: the UID of the new message as assigned by the server. If the
> + message is saved, but it's UID can not be found, it will
> + return 0. If the message can't be written (folder is
> + read-only for example) it will return -1."""
> +
> + if not self.synclabels:
> + return super(GmailFolder, self).savemessage(uid, content, flags, rtime)
> +
> + labels = self.message_getheader(content, self.labelsheader)
> + if labels:
> + labels = set([lb.strip() for lb in labels.split(',') if len(lb.strip()) > 0])
> + else:
> + labels = set()
> +
> + ret = super(GmailFolder, self).savemessage(uid, content, flags, rtime)
> + self.savemessagelabels(uid, labels)
> + return ret
> +
> + def _messagelabels_aux(self, arg, uidlist, labels):
> + """Common code to savemessagelabels and addmessagelabels"""
> + labels = labels - self.ignorelabels
> + uidlist = [uid for uid in uidlist if uid > 0]
> + if len(uidlist) > 0:
> + imapobj = self.imapserver.acquireconnection()
> + try:
> + labels_str = '(' + ' '.join([imaputil.quote(lb) for lb in labels]) + ')'
> + # Coalesce uid's into ranges
> + uid_str = imaputil.uid_sequence(uidlist)
> + result = self._store_to_imap(imapobj, uid_str, arg, labels_str)
> +
> + except imapobj.readonly:
> + self.ui.labelstoreadonly(self, uidlist, data)
> + return None
> +
> + finally:
> + self.imapserver.releaseconnection(imapobj)
> +
> + if result:
> + retlabels = imaputil.flags2hash(imaputil.imapsplit(result)[1])['X-GM-LABELS']
> + retlabels = set([imaputil.dequote(lb) for lb in imaputil.imapsplit(retlabels)])
> + return retlabels
> + return None
> +
> + def savemessagelabels(self, uid, labels):
> + """Change a message's labels to `labels`.
> +
> + Note that this function does not check against dryrun settings,
> + so you need to ensure that it is never called in a dryrun mode."""
> + if uid in self.messagelist and 'labels' in self.messagelist[uid]:
> + oldlabels = self.messagelist[uid]['labels']
> + else:
> + oldlabels = set()
> + labels = labels - self.ignorelabels
> + newlabels = labels | (oldlabels & self.ignorelabels)
> + if oldlabels != newlabels:
> + result = self._messagelabels_aux('X-GM-LABELS', [uid], newlabels)
> + if result:
> + self.messagelist[uid]['labels'] = newlabels
> +
> + def addmessageslabels(self, uidlist, labels):
> + """Add `labels` to all messages in uidlist.
> +
> + Note that this function does not check against dryrun settings,
> + so you need to ensure that it is never called in a dryrun mode."""
> +
> + labels = labels - self.ignorelabels
> + result = self._messagelabels_aux('+X-GM-LABELS', uidlist, labels)
> + if result:
> + for uid in uidlist:
> + self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] | labels
> +
> + def deletemessageslabels(self, uidlist, labels):
> + """Delete `labels` from all messages in uidlist.
> +
> + Note that this function does not check against dryrun settings,
> + so you need to ensure that it is never called in a dryrun mode."""
> +
> + labels = labels - self.ignorelabels
> + result = self._messagelabels_aux('-X-GM-LABELS', uidlist, labels)
> + if result:
> + for uid in uidlist:
> + self.messagelist[uid]['labels'] = self.messagelist[uid]['labels'] - labels
> +
> + def copymessageto(self, uid, dstfolder, statusfolder, register = 1):
> + """Copies a message from self to dst if needed, updating the status
> +
> + Note that this function does not check against dryrun settings,
> + so you need to ensure that it is never called in a
> + dryrun mode.
> +
> + :param uid: uid of the message to be copied.
> + :param dstfolder: A BaseFolder-derived instance
> + :param statusfolder: A LocalStatusFolder instance
> + :param register: whether we should register a new thread."
> + :returns: Nothing on success, or raises an Exception."""
> +
> + # Check if we are really copying
> + realcopy = uid > 0 and not dstfolder.uidexists(uid)
> +
> + # first copy the message
> + super(GmailFolder, self).copymessageto(uid, dstfolder, statusfolder, register)
> +
> + # sync labels and mtime now when the message is new (the embedded labels are up to date)
> + # otherwise we may be spending time for nothing, as they will get updated on a later pass.
> + if realcopy and self.synclabels:
> + try:
> + mtime = dstfolder.getmessagemtime(uid)
> + labels = dstfolder.getmessagelabels(uid)
> + statusfolder.savemessagelabels(uid, labels, mtime=mtime)
> +
> + # either statusfolder is not sqlite or dstfolder is not GmailMaildir.
> + except NotImplementedError:
> + return
> +
> + def syncmessagesto_labels(self, dstfolder, statusfolder):
> + """Pass 4: Label Synchronization (Gmail only)
> +
> + Compare label mismatches in self with those in statusfolder. If
> + msg has a valid UID and exists on dstfolder (has not e.g. been
> + deleted there), sync the labels change to both dstfolder and
> + statusfolder.
> +
> + This function checks and protects us from action in dryrun mode.
> + """
> + # This applies the labels message by message, as this makes more sense for a
> + # Maildir target. If applied with an other Gmail IMAP target it would not be
> + # the fastest thing in the world though...
> + uidlist = []
> +
> + # filter the uids (fast)
> + try:
> + for uid in self.getmessageuidlist():
> + # bail out on CTRL-C or SIGTERM
> + if offlineimap.accounts.Account.abort_NOW_signal.is_set():
> + break
> +
> + # Ignore messages with negative UIDs missed by pass 1 and
> + # don't do anything if the message has been deleted remotely
> + if uid < 0 or not dstfolder.uidexists(uid):
> + continue
> +
> + selflabels = self.getmessagelabels(uid) - self.ignorelabels
> + statuslabels = statusfolder.getmessagelabels(uid) - self.ignorelabels
> +
> + if selflabels != statuslabels:
> + uidlist.append(uid)
> +
> + # now sync labels (slow)
> + mtimes = {}
> + labels = {}
> + for i, uid in enumerate(uidlist):
> + # bail out on CTRL-C or SIGTERM
> + if offlineimap.accounts.Account.abort_NOW_signal.is_set():
> + break
> +
> + selflabels = self.getmessagelabels(uid) - self.ignorelabels
> + statuslabels = statusfolder.getmessagelabels(uid) - self.ignorelabels
> +
> + if selflabels != statuslabels:
> + self.ui.settinglabels(uid, i+1, len(uidlist), sorted(selflabels), dstfolder)
> + if self.repository.account.dryrun:
> + continue #don't actually add in a dryrun
> + dstfolder.savemessagelabels(uid, selflabels, ignorelabels = self.ignorelabels)
> + mtime = dstfolder.getmessagemtime(uid)
> + mtimes[uid] = mtime
> + labels[uid] = selflabels
> +
> + # Update statusfolder in a single DB transaction. It is safe, as if something fails,
> + # statusfolder will be updated on the next run.
> + statusfolder.savemessageslabelsbulk(labels)
> + statusfolder.savemessagesmtimebulk(mtimes)
> +
> + except NotImplementedError:
> + self.ui.warn("Can't sync labels. You need to configure a local repository of type GmailMaildir")
> diff --git a/offlineimap/repository/Gmail.py b/offlineimap/repository/Gmail.py
> index f4260c0..61d4486 100644
> --- a/offlineimap/repository/Gmail.py
> +++ b/offlineimap/repository/Gmail.py
> @@ -28,7 +28,7 @@ class GmailRepository(IMAPRepository):
> HOSTNAME = "imap.gmail.com"
> # Gmail IMAP server port
> PORT = 993
> -
> +
> def __init__(self, reposname, account):
> """Initialize a GmailRepository object."""
> # Enforce SSL usage
> @@ -36,6 +36,13 @@ class GmailRepository(IMAPRepository):
> 'ssl', 'yes')
> IMAPRepository.__init__(self, reposname, account)
>
> + if self.account.getconfboolean('synclabels', 0) and \
> + self.account.getconf('status_backend', 'plain') != 'sqlite':
> + raise OfflineImapError("The Gmail repository needs the sqlite backend to sync labels.\n"
> + "To enable it add 'status_backend = sqlite' in the account section",
> + OfflineImapError.ERROR.REPO)
> +
> +
> def gethost(self):
> """Return the server name to connect to.
>
> @@ -71,4 +78,3 @@ class GmailRepository(IMAPRepository):
> def getspamfolder(self):
> #: Gmail also deletes messages upon EXPUNGE in the Spam folder
> return self.getconf('spamfolder','[Gmail]/Spam')
> -
> diff --git a/offlineimap/ui/UIBase.py b/offlineimap/ui/UIBase.py
> index 2c30b94..af77edc 100644
> --- a/offlineimap/ui/UIBase.py
> +++ b/offlineimap/ui/UIBase.py
> @@ -248,6 +248,15 @@ class UIBase(object):
> "for that message." % (
> str(uidlist), self.getnicename(destfolder), destfolder))
>
> + def labelstoreadonly(self, destfolder, uidlist, labels):
> + if self.config.has_option('general', 'ignore-readonly') and \
> + self.config.getboolean('general', 'ignore-readonly'):
> + return
> + self.warn("Attempted to modify labels for messages %s in folder %s[%s], "
> + "but that folder is read-only. No labels have been modified "
> + "for that message." % (
> + str(uidlist), self.getnicename(destfolder), destfolder))
> +
> def deletereadonly(self, destfolder, uidlist):
> if self.config.has_option('general', 'ignore-readonly') and \
> self.config.getboolean('general', 'ignore-readonly'):
> @@ -355,6 +364,25 @@ class UIBase(object):
> self.logger.info("Deleting flag %s from %d messages on %s" % (
> ", ".join(flags), len(uidlist), dest))
>
> + def addinglabels(self, uidlist, label, dest):
> + self.logger.info("Adding label %s to %d messages on %s" % (
> + label, len(uidlist), dest))
> +
> + def deletinglabels(self, uidlist, label, dest):
> + self.logger.info("Deleting label %s from %d messages on %s" % (
> + label, len(uidlist), dest))
> +
> + def settinglabels(self, uid, num, num_to_set, labels, dest):
> + self.logger.info("Setting labels to message %d on %s (%d of %d): %s" % (
> + uid, dest, num, num_to_set, ", ".join(labels)))
> +
> + def collectingdata(self, uidlist, source):
> + if uidlist:
> + self.logger.info("Collecting data from %d messages on %s" % (
> + len(uidlist), source))
> + else:
> + self.logger.info("Collecting data from messages on %s" % source)
> +
> def serverdiagnostics(self, repository, type):
> """Connect to repository and output useful information for debugging"""
> conn = None
--
Nicolas Sebrecht
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bugfixing.patch
Type: text/x-diff
Size: 7966 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20130126/08907d07/attachment-0004.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gmail-labels.patch
Type: text/x-diff
Size: 79518 bytes
Desc: not available
URL: <http://alioth-lists.debian.net/pipermail/offlineimap-project/attachments/20130126/08907d07/attachment-0005.patch>
More information about the OfflineIMAP-project
mailing list