[Bug 104956] dimap: sudden mail loss

[Martin Steigerwald]

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
         
http://bugs.kde.org/show_bug.cgi?id=104956         




------- Additional Comments From Martin Lichtvoll de  2006-06-29 10:06 -------
Hello,

Hmmmm, this is getting more and more complex. I think we are dealing is lots of different bugs here. At least those:

1) Some operations do not return a success or failure indication
2) On some operations the success or failure indication is not checked
3) KMail seems to trust its index file more than it should. If KMail also deletes locally stored mails that are not in the index file the implications are far beyond IMAP related mail losses!

All of this indicate that KMail desperately needs application-internal data journalling and atomicity. What are about mails that have been downloaded and not added to the index? What are about mails that have been marked as downloaded, added to the index, but not saved (due to an error)?

So KMail needs:

1) Journalled index file writing: KMail needs to be absolutely sure, that the index file represents the actual folder contents. So it has to write any changes to a folder to a journal before executing them. So it can check whether all changes have been executed and update the index file accordingly in case it has been stopped abruptly. 

For this to work reliably it needs to be able to sync write the journal, i.e. tell filesystem and block layer to write it immediately and only return when it really has been written to disk (and not only sitting in a write cache). (Not unlike write barrier functionately in kernel 2.6.16+).

This way kmail can tell whether a mail has been written to disk or not, even if it has been aborted abruptly. On next start it will go through the journal if it is not empty. It will add all already commited transactions to the index file. Also it will complete all not yet completed transactions if possible. In case of a downloaded e-mail that was about to be written to disk, but has not been written (completely) it should tell the user to synchronize the IMAP account again in order to re-download that mail - or to redownload mails from a POP3 acocunt.

Or it needs a way to rebuild the index file of a folder if its last access time is newer than the date of the index file. I know that the Amiga mail program YAM (http://www.yam.ch - GPL) rebuilds the index file of a folder in that case. This already is quite reliabled, but is not complete: What if a mail has been added to the index file but not yet been flushed to disk? So preferably journalled index file writing is used! Or each mail has to be sync-written, which probably would slow things down more than journalled index file writing.

2) Complete error handling during the IMAP sync process including reporting to the user. KMail should know whether an IMAP action was successful or not. It should add a new mail to its local mail storage only if downloading it via IMAP has been successful!

Actually I believe this needs some design work before implementing code. It should be documented how mail and index file saving as well as IMAP or POP3 should be done and how errors should be handled.

It would probably be best to rewrite mail syncing, mail download and mail and index file code from scratch. This is a lot of work which likely cannot be done before KDE 4.

For KDE 3.5.x fixing at least the most important cases of non complete error handling might help dramatically as well.

I am willing to help with design work and documentation. I currently don't have enough C++ skills to code the stuff.

Regards, Martin