<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div><span>1) IMO imaplib2 should be updated to handle the case where Gmail is sending a new CAPABILITIES response after login, that corresponds to the particular imap server the user's Gmail account is hosted on. Because Brandon is right, IMPA4rev1 protocol says the client must be able to take any response at any time. It's strange to see a second CAPABILITIES response a moment later, with different capabilities, but it's legit, and should be easy to support.<br></span></div><div><br><span></span></div><div><span>The same kind of bug happened recently with an off-topic (but still totally legitimate) response from the IMAP server coming in the middle of a message list operation. imaplib2 incorrectly interpreted this off-topic response as the end of the message list and stopped processing the list right there (but
should've continued). So imaplib2 threw an exception, and caused offlineimap to lose all the rest of the messages in the list !! Very bad critical data loss bug.<br></span></div><div><br></div><div>2) If Google's IMAP implementation is not returning the UID with the APPEND command's OK response, then it's not implementing RFC 4315 ("IMAP - UIDPLUS Extension," December 2005).</div><div><br></div><div>Brandon, I feel Google should implement UIDPLUS according to the RFC4315 on the Gmail servers that support UIDPLUS. Then, offlineimap will be able to communicate with Gmail and get expected UID responses while doing APPENDs, etc.<br></div><div><br></div><div>http://tools.ietf.org/html/rfc4315</div><div><br></div><div>RFC 4315 Abstract:<br></div><div>"The UIDPLUS extension of the Internet Message Access Protocol (IMAP)
provides a set of features intended to reduce the amount of time and
resources used by some client operations."</div><div><br></div><div>RFC 4315 Section 1:<br></div><div>"...this document recommends new status response codes in IMAP that
SHOULD be returned by all server implementations, regardless of
whether or not the UIDPLUS extension is implemented.
The added facilities of the features in UIDPLUS are optimizations;
clients can provide equivalent functionality, albeit less
efficiently, by using facilities in the base protocol."</div><div><br></div><div>RFC 4315 Section 3:<br></div><div>"In this example, A003 and A004 demonstrate successful appending and
copying to a mailbox that returns the UIDs assigned to the messages."</div><div><br></div><div>"Example: <br></div><div> C: A003 APPEND saved-messages (\Seen) {297} <br></div><div>C: Date: Mon, 7 Feb 1994 21:52:25 -0800 (PST) <br></div><div>C: From: Fred Foobar <foobar@example.com> <br></div><div>C: Subject: afternoon meeting <br></div><div> C: To: mooch@example.com <br></div><div>C: Message-Id: <B27397-0100000@example.com> <br></div><div>C: MIME-Version: 1.0 <br></div><div>C: Content-Type: TEXT/PLAIN; CHARSET=US-ASCII <br></div><div> C: <br></div><div>C: Hello Joe, do you think we can meet at 3:30 tomorrow? <br></div><div>C: <br></div><div>S: A003 OK [APPENDUID 38505 3955] APPEND completed"<br></div><div><br></div><div>Best,</div><div>Chris Coleman<br></div><pre class="newpage"><br></pre><div><br></div><div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"><div style="font-family:
times new roman, new york, times, serif; font-size: 12pt;"><font face="Arial" size="2"><hr size="1"><b><span style="font-weight:bold;">From:</span></b> Brandon Long <blong@google.com><br><b><span style="font-weight: bold;">To:</span></b> offlineimap-project@lists.alioth.debian.org<br><b><span style="font-weight: bold;">Cc:</span></b> Sebastian@sspaeth.de; Marc MERLIN <merlin@google.com><br><b><span style="font-weight: bold;">Sent:</span></b> Saturday, June 25, 2011 6:06 PM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: Gmail to "gmail apps" sync: ValueError: Backend could not find uid for message<br></font><br>
Another Googler pointed me to this thread, I think I can clear some things up.<br><br>Sebastian wrote:<br>> Hi,<br>><br>> Alexander has sent me the log files and I analyzed them (the 'backend<br>> could not assign uid' problem). I am a bit puzzled and at the moment, I<br>> blame it entirely on the gmail imap implementation:<br>><br>> Alexander has 2 problems:<br>><br>> 1) His gmail server is not advertising the UIDPLUS extension all the<br>> time which directly allows to get the UID of an APPENDed message. His<br>> gmail imap capabilities reply looked like this:<br>><br>> 'IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA XLIST CHILDREN XYZZY SASL-IR AUTH=XOAUTH'<br><br>This is the pre-auth CAPABILITY response. There is a different<br>response after login, and we also automatically send an untagged<br>CAPABILITY response after login, ie:<br><br>* OK Gimap ready for requests from 71.202.122.239 k6if6834428icw.34<br>a
CAPABILITY<br>* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN<br>X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH<br>a OK Thats all she wrote! k6if6834428icw.34<br>a login <a ymailto="mailto:login@gmail.com" href="mailto:login@gmail.com">login@gmail.com</a> password<br>* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN<br>X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE<br>a OK <a ymailto="mailto:login@gmail.com" href="mailto:login@gmail.com">login@gmail.com</a> User authenticated (Success)<br><br>The reason for this is that we roll out new features on a per-user<br>basis, so we don't know before login what features are available to<br>the user. When we're certain the feature is stable and doesn't need<br>to be rolled back, we can start advertising it pre-login.<br><br>> while mine look like this:<br>><br>> 'IMAP4REV1', 'UIDPLUS', 'CHILDREN', 'NAMESPACE',<br>> 'THREAD=ORDEREDSUBJECT', 'THREAD=REFERENCES', 'SORT',
'QUOTA', 'IDLE',<br>> 'STARTTLS'<br><br>This isn't a Gmail IMAP response, though maybe you aren't implying it<br>is. We don't currently implement any THREAD= or SORT, nor do we<br>advertise STARTTLS because we only provide TLS-wrapped IMAP.<br><br>> both are from <a target="_blank" href="http://imap.gmail.com">imap.gmail.com</a>, so their servers seem very different under<br>> whatever mysterious conditions.<br>><br>> 2)<br>> Without UIDPLUS, we munge our mail header to insert a custom header<br>> like this: X-OfflineIMAP: 3132605382-7478425151<br>> and APPEND the message.<br>><br>> (despite not announcing UIDPLUS, gmail DOES return the new UID ('APPENDUID<br>> 648448589') which we could have used in the first place, so that is<br>> definitely a gmail bug on their side.)<br>><br>> We then try to search gmail for the newly appended message:<br>> UID SEARCH HEADER X-OfflineIMAP
"3132605382-7478425151"<br>><br>> which leads to "OK SEARCH completed (Success)" returning an empty set of<br>> messages.<br><br>Hmm, I don't have access to the source depot at the moment to see when<br>we added support for substring header search, but I think it was last<br>year, so this should definitely work. I don't recommend it, however,<br>as it requires us to load the headers for every message in the folder<br>in order to find the matching messages.<br><br>> The message we just uploaded was not found, although it must be there<br>> (we just uploaded it), so we fail to identify the new UID. Which leads<br>> to the exception as we cannot return the UID of the message in question.<br><br>This probably depends on the exact order of operations. Ie, if you do<br>a SELECT, APPEND, UID SEARCH, I'm not sure if it should be visible,<br>you may need to do a NOOP or CHECK to make sure that the new message<br>is
visible. I guess we could instead generate an untagged EXISTS<br>response after the APPEND to tell the client about the new message,<br>but we currently don't do this, nor do we currently update the folder<br>state when appending to a selected folder.<br><br>From my reading of RFC 3501 5.2 & 5.5, I'm still not sure if this is<br>invalid or not. Its possible we should just update the mailbox state<br>after the APPEND and issue the relevant EXISTS response. Certainly<br>easy enough for us to make that change, though it will definitely make<br>APPENDs more expensive.<br><br>> So 2 things going on: 1) Gmail fails to advertize UIDPLUS (in some cases)<br>> although they definitely support it, which is bad.<br>> 2) Gmail fails to find the message header in a message we have just<br>> uploaded. This is doubly bad on the gmail side.<br>><br>> I know that gmail is popular and often unavoidable. But it is not proper<br>>
IMAP :-(. The one workaround that we could do is to hardcode UIDPLUS<br>> support in the gmail case, since we know that gmail supports it (but who<br>> knows if they really support it in all cases?). I would be rather<br>> hesitant to do that hard coding.<br><br>I don't believe that either of these things is invalid. We are<br>automatically issuing the new CAPABILITY response after login, and<br>clients are required to accept any response at any time, so we handle<br>#1 just fine. #2 may be an edge case in the protocol, or maybe its<br>not and we should fix it, but I can't tell which from the RFC.<br><br>Brandon<br>-- <br> Brandon Long <<a ymailto="mailto:blong@google.com" href="mailto:blong@google.com">blong@google.com</a>><br> Staff Engineer<br> Gmail Delivery TLM<br><br></div></div></div></body></html>