Disconnected IMAP synchronization method

Copyright Notice

   Copyright (C) Filip Navara (2006).  All Rights Reserved.

Abstract

   This document describes the proposed disconnected IMAP client
   synchronization mechanism as to be implemented in eM Client.
   It builds upon information from RFC 3501 and RFC 4549, but
   goes into detail of specific implementation of the described
   techniques.

Local storage structure

   A local storage structure is expected to be a hierarchy of mailboxes
   where each mailbox holds an arbitrary number of message objects.

   Mailbox cache is defined as a local mailbox which is supposed to be
   a mirror of a single remote mailbox. A mailbox cache differs from
   ordinary local mailbox by a set of associated attributes:

   #RemotePath

       A path that uniquely identifies a remote mailbox. It can be
       specified by an IMAP URL for example.

   #Flags

      \CreatePending
         Folder was created during disconnected operation and is
         scheduled for creation during the synchronization process.

      \DeletePending
         Folder was scheduled for deletion while working in
         disconnected mode.

      \OfflinePending
         Informative flag to prevent complex synchronization on a
         mailbox when not needed. It's set when we add message, remove
         message or change message flags on contained message objects
         during disconnected operation.

      \Interesting
         Folder which is considered "interesting" for disconnected
         usage and whose messages are to be downloaded during
         synchronization process, so they can be viewed in disconnected
         state. Note that there can be further attributes implemented
         which specify the type of messages that are considered to
         be interesting (depending on various criteria like message
         size).

   #UidValidity

      Last known UIDVALIDITY value for given folder. This value is used
      to ensure that the local cache view of the messages matches the
      remote mailbox. Details are available in [IMAP] and [IMAP-DISC]

   #UidNext

      Last known UIDNEXT value for given folder.

   #LastSeenUid

      The highest UID of message in the mailbox that we have seen so far
      on the server (see [IMAP-DISC]).

   #HighestModeSeq
  
      << TODO: Add CONDSTORE support >>

   Message cache object represents a local copy of information for
   specific message contained in a remote mailbox. The following special
   attributes are defined for a message cache object:

   #RemoteId
  
      An unique ID of the message on IMAP server as of last known online
      state. This value has to contain the mailbox path, UIDVALITITY
      for given mailbox and the UID of the message. As specified by
      [IMAP] such an ID must represent the same message on the server
      forever.

   #RemoteFlags

      Last known state of flags for the message on the server.

   #ModSeq

      << TODO: Add CONDSTORE support >>

   It's important to describe the following rules for the local storage
   modification:

   - If any message is moved, whether it is local message or message
     cache, all attributes MUST be preserved.
   - A message which doesn't meet the message cache object criteria
     and is located inside a local mailbox which meets the "mailbox
     cache" criteria is considered a message to be uploaded.
   - A message which meets the "message cache" object criteria, but
     is located inside a mailbox cache with path doesn't match the
     one stored in the #RemoteId attribute is considered a message
     to be copied within the remote mailbox hierarchy.
   - Deleting a "mailbox cache" is accomplished by setting the
     \DeletePending flag. No message objects from inside the mailbox
     cache should be deleted, it will be done during the delayed
     synchronization process.
   - When creating a mailbox cache object in disconnected operation
     the \CreatePending flag MUST be set on the new object and the
     #RemotePath attribute must be filled. If the #UidValidity,
     #UidNext, #LastSeenUid and #HighestModeSeq attributes aren't set
     already (see below for a case when would that happen) they
     should be set to zero.
   - When a creating a mailbox cache object and there is already a
     mailbox cache object for the same remote folder that has the
     \DeletePending flag set, the old object should be reused. Smart
     client should ask the user if he wants to keep the old mailbox
     contents and optionally clear the \DeletePending flag instead
     of setting the \DeletePending flag.
   - Each unique remote folder should have at most one mailbox cache
     object associated. It's particularly important to pay attention
     to it when disconnected mailbox move operation is implemented.
 
Synchronization process

   I. The constructive stage

      In this stage new mailboxes are created, all message copies are
      processed and any new messages are uploaded.
 
      For each local mailbox cache which has one of the \CreatePending,
      \DeletePending or \OfflinePending flags set do the following:

      a) If the mailbox cache is marked as \DeletePending then delete the
         mailbox on the remote server. This has to be done with great
         care, so we don't accidentally delete something we don't want.
         At least the UIDVALIDITY MUST be checked before performing
         the delete operation.

      b) If the mailbox cache is marked as \CreatePending then create the
         mailbox on remote server and clear the \CreatePending flag. In
         case the mailbox already exists an error can be silently ignored.

      c) If the mailbox cache is marked as \OfflinePending then do the
         following:

         - For each message with #RemoteUrl attribute not pointing to a
           folder with the same path as the mailbox cache we're currently
           processing add the message to COPY queue.
         - For each message with no #RemoteUrl attribute set append the
           contents to the server now using the IMAP APPEND command and
           update the references to it. Smart clients SHOULD update all
           references that address the same content, but it is REQUIRED
           to at least update the #RemoteUrl and #RemoteFlags attribute
           of the message cache object just processed.

      d) Sort the operations in the COPY queue by mailbox (actually
         this can be done during the previous step when building the
         queue) and process it. For each unique folder use SELECT to
         activate it, check the UIDVALIDITY and if it matches then
         perform the COPY operation.

   II. The single mailbox synchronization stage

      For each local folder with either the \OfflinePending flag or the
      \Interesting flag set do the following:

      a) Select the folder and check it's UIDVALIDITY. If the value doesn't
         match the locally cached value throw away the local cache of the
         folder and skip steps b), c) and d).

      b) Build an ID to UID map for old (and possibly new) messages.

         For each UID present in remote UID set and not present in local
         message cache add the message to EXPUNGE queue.

         For each UID present in local message cache and not present in
         remote UID set delete the local cache for particular message.

      c) Flush EXPUNGE queue.

      d) Build UID sets for flag STORE operations and process them. The
         +FLAGS.SILENT and -FLAGS.SILENT commands should be used.

      e) Clear the \OfflinePending flag.

      f) Fetch new message descriptors (see [IMAP-DISC]) and for mailboxes
         marked as \Interesting fetch the message contents too.

<< TODO: Add folder move/rename support >>