Internet Draft: IMAP Extension for CHECKPOINT M. Wener Document: draft-wener-lemonade-checkpoint-00.txt Nokia, Inc. Expires: March 2005 October 2004 IMAP Extension for CHECKPOINT Synchronization Status of this Memo By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, or will be disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This document describes an IMAP extension that aids in allowing a client to maintain mailbox synchronization without performing a deep sync after each connection. The goal is to minimize, but not eliminate, the need for deep synchronizations. The extension defines a way for a client to receive and acknowledge state change responses from the server. 1. Introduction and Overview The CHECKPOINT extension is present in any IMAP implementation that returns "CHECKPOINT" as a supported extension in the CAPABILITY response. Clients that are connected over long thin connections may have a need to reduce the overhead of performing mailbox synchronizations. Ideally some approach that allows the cost of the synchronization to be distributed over time and reduces the redundant retrieval of information would reduce the burden of synchronization on the client. This extension provides a mechanism to allow the client to acknowledge to the server that unsolicited responses have been received and processed. Prior to the server sending mailbox state change information via unsolicited responses it will send a new CHECKPOINT response code to demarcate a checkpoint boundary. The client will then process unsolicited responses until at some point it decides to acknowledge to the server that all unsolicited responses since some message response number are committed. This acknowledgment is communicated to the server via an extended CHECK command or through an IDLE CHECKPOINT termination. Each unsolicited message sent from the server to the client over an idle checkpoint session will contain a response number identifying the message. There are two distinct mechanisms for acknowledging a checkpoint to allow clients to take advantage of either a multiple session design or a single session design. A CHECK command could be used in a multiple session design, and an IDLE CHECKPOINT termination could be used in a single session design. The mechanism works within the context of the IDLE command. There is no strict requirement to be part of the IDLE sequence, but doing so provides a context in which the server can restrict the need for the enhanced response queuing that will be needed to implement the extension. It also restricts the scope of when messages are identified by a response number. 1.1 Definitions Full Synchronization Client queries server for all information pertaining to a set of folders. The assumption here is that the client has zero valid information about a mailbox. This would occur at the very first connection or maybe after a UIDVALIDITY change. Deep Synchronization Client queries server for all changes that have occurred since last synchronization. This is typically less costly then a full synchronization because the client retains some information from a previous synchronization. Response Number A non-zero unsigned 32-bit integer. The response number is chosen by the server and MUST be monotonically increasing value. The value is used by the client to keep track of unsolicited responses that are currently queued on the server. Section 2.2 has details of the BNF. 2. IMAP Protocol Changes 2.1 CHECKPOINT Response Code This document adds a new CHECKPOINT response code. The CHECKPOINT response code MUST only occur in an unsolicited untagged response during an IDLE session or as part of the response to the CHECK command. A server supporting the checkpointing mechanism during an IDLE command MUST send a CHECKPOINT response code as part of the first response sent on an IDLE connection. Additional CHECKPOINT response codes will be sent after this as determined by the server. The response code is followed by a checkpoint response number. The response number is chosen by the server and MUST be a monotonically increasing integer value. This value is used by the client to keep track of the unsolicited responses that are currently queued on the server. The value plus one is used by the server as the initial message response number appended to the next unsolicited message. If the response number is followed by the "CONTINUED" keyword then the client should treat this response as an indication there are pending response lines queued up and waiting to be sent. If the CONTINUED keyword is used then the response number MUST be the response number of the original OK CHECKPOINT that started the sequence. The server MUST send these to this idle connection before any new responses are sent. These pending responses may or may not have already been sent. The client MUST be able to handle duplicate response strings, i.e responses with already received response numbers since the last checkpoint. If the response code is part of a NO response then the client MUST terminate all IDLE CHECKPOINT sessions and SHOULD re-issue the IDLE CHECKPOINT command to re-establish queuing. The client SHOULD perform a deep mailbox synchronization due to this NO response. Once a server has sent a NO response containing the CHECKPOINT response code all queuing for the current authentication session MUST stop until a new IDLE CHECKPOINT session becomes active. Example 1: S: * OK [CHECKPOINT 10540] Initial Idle Checkpoint In this example the next unsolicited message will contain the response number 10541. Example 2: S: * OK [CHECKPOINT 10540 CONTINUED] Continuing After Termination In this example the following unsolicited message set to the client will contain the response numbers 10541..10541+N, but some subset of this will have already been sent to the client. Example 3: S: * NO [CHECKPOINT 10540] Checkpoint Failed, Queue Terminated In this example the client SHOULD perform a deep synchronization. 2.2 Message Response Number Addition Each unsolicited message sent from the server to the client over an idle checkpoint session MUST contain a response number identifying the message. For responses containing CHECKPOINT response codes the response number of the message is the CHECKPOINT response number. The response number is only valid until the queue is terminated by the server. The server MAY reuse response numbers after it sends a NO response with a CHECKPOINT response code. The server MUST maintain continuity of response numbers after it sends an OK response with a CHECKPOINT response code. The response number MUST be appended as the last parsable token in the response. The BNF is: response-data-nocrlf = "*" SP (resp-cond-state / resp-cond-bye / mailbox-data / message-data / capability-data) seqnumpart = "(" number ")" ; See [IMAP4] for definition of "number" iresponse = response-data-nocrlf SP seqnumpart response = iresponse CRLF Example 1: S: * 4 FETCH (FLAGS (\Recent) UID 789) (10541) If the client detects gaps in the response number sequence then the client MUST perform a checkpoint of the last number of the last contiguous sequence. This will cause the server to reset the response number and resend the messages if necessary. 2.3 IDLE Command Extension Arguments: CHECKPOINT Responses: A continuation response which is an open prompt indicating the client can send "DONE" to terminate the command. Immediately after the continuation response an untagged response containing a CHECKPOINT response code MUST be sent. The client MAY also send a "PING" continuation line to indicate to the server that it may resets it's timeout timer. A continuation response is then sent by the server. Result: OK - IDLE completed after client sent "DONE" or "DONE CHECKPOINT" NO - Failure: the server will not allow the IDLE command at this time. BAD - Command unknown, arguments invalid or an invalid response number. An IDLE command with the CHECKPOINT argument indicates to the server that it is to use the checkpoint facility to track the unsolicited responses being sent to the client. The tracking is done via a queuing mechanism that is outside the scope of this document. Upon reconnection any queued responses associated with this authentication session that are in the PENDING state MUST be sent on the first IDLE CHECKPOINT connection. Just prior to these messages being sent a CHECKPOINT CONTINUED response code must be sent on this idle connection. If a client sends a "DONE" continuation string with no additional arguments then the server MUST drop any queued responses that have not been acknowledged either via the CHECK command or an IDLE command checkpoint termination. In this case the server MUST send an untagged NO response containing a CHECKPOINT response code indicating the current queued response set has been dropped. The NO response MUST be sent to all IDLE CHECKPOINT sessions. If the client sends a "DONE CHECKPOINT " continuation string the IDLE session is terminated and the server will behave as if a "CHECK CHECKPOINT " command was sent. All queued responses MUST be dropped and a new untagged unsolicited CHECKPOINT response code MUST be issued when the current set of queued responses are acknowledged. 2.4 CHECK Command Extension Arguments: CHECKPOINT keyword followed by a response number. Responses: If the checkpoint is successfully committed an OK response containing a CHECKPOINT response code. If the checkpoint was not committed then a NO response is returned. Result: OK - check command completed BAD - Command unknown, arguments invalid or an invalid response number. This command extension is used by a client to communicate to the server that it has successfully processed all responses upto and including the specified response number. When the CHECK command is given a CHECKPOINT argument the response number identifies the last message response number the client is acknowledging. The only legal response number is a response number in the set N..M where N is the response number on the last OK response containing a CHECKPOINT response code, and M is the response number on the last unsolicited response sent on the idle session. The server MAY drop the currently queued set of responses once they are acknowledged by the client. The command also has the side effect of causing an untagged unsolicited CHECKPOINT response code to be sent on all IDLE CHECKPOINT sessions. This signals the start of a new queued response code set. Example 1: C: 1 CHECK CHECKPOINT 20897 S: * OK [CHECKPOINT 20897] Checkpoint Acknowledged S: 1 OK CHECK Completed Example 2: C: 2 CHECK CHECKPOINT 20897 S: * NO [CHECKPOINT 20897] Checkpoint Failed S: 1 OK Check Complete 3 Queuing Mechanism The response queue maintained by the server is associated with an authentication session. An authentication session is identified when a connection enters an authentication state and terminates when a successful LOGOUT command occurs. In the case of a PREAUTH connection the authentication session is implied in the same external way that the pre-authentication was. The response queue maintained by the server MUST maintain each queued entry in one of two states - PENDING or ACKNOWLEDGED. PENDING The response has been sent on at least one checkpoint idle connection, and has not been acknowledged. ACKNOWLEDGED The response has been acknowledged by the client. A client acknowledges a response via a IDLE CHECKPOINT termination or the CHECK CHECKPOINT command. The server is under no obligation to retain an acknowledged response after this point and MAY drop the queued response at this time. A queue is terminated or dropped when any of the following events occur on the server: 1) An IDLE command with no CHECKPOINT argument is received on any authentication session associated with this queue. An untagged NO CHECKPOINT response is generated on all IDLE connections. 2) A DONE continuation line with no CHECKPOINT argument is received on any authentication session associated with this queue. An untagged NO CHECKPOINT response is generated on all IDLE connections. 3) A DONE continuation line with a CHECKPOINT argument is received on any authentication session associated with this queue. An untagged OK CHECKPOINT response is generated on all IDLE connections. 4) A CHECK command with a CHECKPOINT argument, and a response number that falls within the legal range. An untagged OK CHECKPOINT response is generated on all IDLE connections. 5) A LOGOUT command is received for the last active connection associated with the queue. 6) After some time out period during which there are no active authentication sessions. The time out period MAY be configurable. How the period is set is outside the scope of this document. A NO CHECKPOINT response MUST be sent on the first IDLE CHECKPOINT connection following a timeout. 4 Client Scenarios This section is provided to describe expected behavior on the part of clients in interesting situations. 4.1 Multiple Client Instance Access Multiple clients connecting and using the CHECKPOINT facility concurrently has implications beyond just the simple multiple connection situation. When multiple connections are used by a single client instance the knowledge the client has that it is using multiple connections allows it to coordinate behavior between the two connections. in fact, this latter situation is the expected behavior of clients. While multiple distinct client instances accessing the CHECKPOINT facility is discouraged predictable behavior should be defined. This section describes the behavior that SHOULD occur given the possible cases. Case 1: N Client Concurrent Access to IDLE CHECKPOINT connection First client has an outstanding IDLE CHECKPOINT connection active. When the subsequent client connects and issues the IDLE CHECKPOINT command it will receive an OK CHECKPOINT response communicating the currently active checkpoint. It will then receive all queued responses from the server. If any client issues a successful CHECK CHECKPOINT command or an IDLE CHECKPOINT termination then all outstanding IDLE CHECKPOINT sessions will receive an OK CHECKPOINT response code indicating the start of a new checkpoint. The clients then have the option to either accept this as a valid start of a new checkpoint sequence or perform a deep synchronization. The former choice is a reasonable one if the client has already committed or is able to commit all messages up to the the new checkpoint. Only the client knows it's situation. Case 2: Client "Steals" a Continued Checkpoint Response The server has detected the loss of a client TCP connection and is waiting to send a CHECKPOINT CONTINUED response code. Now the next IDLE CHECKPOINT session is established by a client that is not expecting a continued session. If a client receives an unexpected continued response it SHOULD terminate the IDLE session with a simple "DONE", and perform normal connection synchronization. There is a possibility that a client expecting a continued response may steal another clients continued response. If this is the case then the client should consider this an unexpected response. Case 3: Racing Checkpoint Terminations If two clients have active IDLE CHECKPOINT sessions and both send a CHECK CHECKPOINT then both will receive two OK CHECKPOINT responses. Each client MAY then either accept each as a valid start of a new checkpoint sequence or perform a deep synchronization depending on it's internal knowledge of how it processed any unsolicited responses in between the OK CHECKPOINT responses. 4.2 Client Information Windows While It is expected that the client will use a separate connection to run the IDLE command, the concurrent mailbox issue is mitigated by issuing the IDLE command in the Authenticated state and not in the Selected state. This will allow a server to avoid having to enforce concurrent access limitations for certain mailbox formats. 4.2.1 Initial Connection Synchronization In order to avoid windows of missed information a client SHOULD use the following scheme to monitor mailbox state. The following scenario assumes a single client instance. Connection 1: Client makes first connection to server and authenticates proceeding to the Authenticated state. Client issues an IDLE CHECKPOINT and begins immediately tracking changes received from the server. In order to avoid the loss of the connection a PING continuation line should be sent by the client. Connection 2: Client makes second connection to server and authenticates proceeding to the Authenticated state. Client then performs a deep synchronization of information as required. This second connection can then be used to perform perfunctory IMAP operations as needed by the client while connection 1 remains to monitor the state changes. 4.2.2 Continued Connection Synchronization When a client connects after an abnormal TCP connection termination it MAY reconnect expecting a CHECKPOINT CONTINUED response code. If the expected response is received the client SHOULD process all unsolicited responses flooded to it and update it's mailbox state. If the client receives a NO or OK CHECKPOINT response then it has received an unexpected response and SHOULD proceed with normal initial connection synchronization. 5 Examples 5.1 Smooth Acknowledgment Using Idle Connection C: 1 IDLE CHECKPOINT S: + continuation, PING, DONE or DONE CHECKPOINT S: * OK [CHECKPOINT 1000] New Checkpoint Delimited ... New Mail Arrives S: * 4 EXISTS (1001) S: * 4 FETCH (FLAGS (\Recent) UID 789) (1002) C: PING S: + Still Here, PING, DONE or DONE CHECKPOINT ... Some Client Reads Message 789 S: * 4 FETCH (FLAGS (\Seen) UID 789) (1003) C: DONE CHECKPOINT 1003 S: 1 OK Idle Complete, Checkpoint Acknowledged 5.2 Smooth Acknowledgment Using Check Command The first example in this section demonstrates a normal non-race condition. This is the case where the messages being sent from the server to the client and the client to the server are received in their entirety, which is purely by chance. Client connection one. C: 1 IDLE CHECKPOINT S: + continuation, PING, DONE or DONE CHECKPOINT S: * OK [CHECKPOINT 1000] New Checkpoint Delimited ... New Mail Arrives S: * 4 EXISTS (1001) S: * 4 FETCH (FLAGS (\Recent) UID 789) (1002) ... CHECK Command issued on second connection S: * OK [CHECKPOINT 1003] New Checkpoint Delimited ... Some Client Reads Message 789 S: * 4 FETCH (FLAGS (\Seen) UID 789) (1004) ... Client connection two. C: 2 CHECK CHECKPOINT 1002 S: * OK [CHECKPOINT 1002] Checkpoint Complete S: 1 OK Check Complete, Checkpoint Acknowledged The second example in this section demonstrates a race condition. This is the case where an acknowledgment and an unsolicited response pass each other in the full duplex channel of TCP. Client connection one. C: 1 IDLE CHECKPOINT S: + continuation, PING, DONE or DONE CHECKPOINT S: * OK [CHECKPOINT 1000] New Checkpoint Delimited ... New Mail Arrives S: * 4 EXISTS (1001) S: * 4 FETCH (FLAGS (\Recent) UID 789) (1002) ... Some Client Reads Message 789 ... CHECK Command issued on second connection ... Server sent this response before checkpointing S: * 4 FETCH (FLAGS (\Seen) UID 789) (1003) ... Server sends a CHECKPOINT response code, but it arrives after message 1003, so the server starts a new checkpoint at the last message boundary S: * OK [CHECKPOINT 1002] New Checkpoint Delimited ... Client connection two. C: 2 CHECK CHECKPOINT 1002 S: * OK [CHECKPOINT 1002] Checkpoint Complete S: 1 OK Check Complete, Checkpoint Acknowledged 5.3 Termination With Forced Queue Flushing C: 1 IDLE CHECKPOINT S: + continuation, PING, DONE or DONE CHECKPOINT S: * OK [CHECKPOINT 1000] New Checkpoint Delimited ... New Mail Arrives S: * 4 EXISTS (1001) S: * 4 FETCH (FLAGS (\Recent) UID 789) (1002) C: DONE S: * NO [CHECKPOINT 1000] Queue Terminated S: 1 OK Idle Complete 5.4 Abrupt Termination With Pending Queue C: 1 IDLE CHECKPOINT S: + continuation, PING, DONE or DONE CHECKPOINT S: * OK [CHECKPOINT 1000] New Checkpoint Delimited ... New Mail Arrives S: * 4 EXISTS (1001) S: * 4 FETCH (FLAGS (\Recent) UID 789) (1002) ... Connection dropped via TCP Reset ... Some client reads message 789 ... Client reconnects and reauthenticates C: 1 IDLE CHECKPOINT S: + continuation, PING, DONE or DONE CHECKPOINT S: * OK [CHECKPOINT 1000 CONTINUED] Checkpoint Continued S: * 4 EXISTS (1001) S: * 4 FETCH (FLAGS (\Recent) UID 789) (1002) S: * 4 FETCH (FLAGS (\Seen) UID 789) (1003) C: DONE CHECKPOINT 1003 S: 1 OK Idle Complete, Checkpoint Acknowledged 5.5 Abrupt Termination Due To Timeout C: 1 IDLE CHECKPOINT S: + continuation, PING, DONE or DONE CHECKPOINT S: * OK [CHECKPOINT 1000] New Checkpoint Delimited ... New Mail Arrives S: * 4 EXISTS (1001) S: * 4 FETCH (FLAGS (\Recent) UID 789) (1002) ... Connection dropped via TCP Reset ... Some client reads message 789 ... Client is disconnected due to some timeout period ... Client reconnects and reauthenticates C: 1 IDLE CHECKPOINT S: + continuation, PING, DONE or DONE CHECKPOINT S: * NO [CHECKPOINT 1000] Queue Terminated, Time out C: DONE S: 1 OK Idle Complete C: 1 IDLE CHECKPOINT S: + continuation, PING, DONE or DONE CHECKPOINT S: * OK [CHECKPOINT 1001] New Checkpoint Delimited ... Start listening for fresh events ... Client performs deep synchronization 6 Normative References [IMAP4] Crispin, M., "Internet Message Access Protocol Version 4rev1", RFC 3501, March 2003. [IDLE] Leiba, B., "IMAP4 IDLE command", RFC 2177, June 1997. 7 Author Michael J. Wener Enterprise Mobility Systems Nokia, Inc. Phone: (412) 551-9860 EMail: Michael.Wener@Nokia.com 8 Aknowledgments Rich Newpol Tony DeWitt David DeCamp 9 Full Copyright Statement Copyright (C) The Internet Society (2004). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. 10 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.