Network Working Group D. Cridland Internet-Draft C. King Expires: April 14, 2007 Isode Limited October 11, 2006 Contexts for IMAP4 draft-cridland-imap-context-00 Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on April 14, 2007. Copyright Notice Copyright (C) The Internet Society (2006). Abstract The IMAP4rev1 protocol has powerful search facilities as part of the core protocol, and a similarly powerful SORT extension, but lacks the ability to create live, updated results which can be easily handled. This memo provides such an extension, and shows how it can be used to provide a facility similar to virtual mailboxes. Cridland & King Expires April 14, 2007 [Page 1] Internet-Draft IMAP CONTEXT October 2006 Table of Contents 1. Conventions used in this document . . . . . . . . . . . . . . 3 2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Protocol Changes . . . . . . . . . . . . . . . . . . . . . . . 3 3.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.2. Extended SORT . . . . . . . . . . . . . . . . . . . . . . 4 3.3. Context Hint . . . . . . . . . . . . . . . . . . . . . . . 4 3.4. Notifications of changes . . . . . . . . . . . . . . . . . 4 3.4.1. ADDTO Return Data Item . . . . . . . . . . . . . . . . 5 3.4.2. REMOVEFROM Return Data Item . . . . . . . . . . . . . 5 3.5. Partial results . . . . . . . . . . . . . . . . . . . . . 6 3.6. Caching results . . . . . . . . . . . . . . . . . . . . . 6 4. Formal Syntax . . . . . . . . . . . . . . . . . . . . . . . . 7 5. Security Considerations . . . . . . . . . . . . . . . . . . . 7 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 8 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 8 8.1. Normative References . . . . . . . . . . . . . . . . . . . 8 8.2. Informative References . . . . . . . . . . . . . . . . . . 8 Appendix A. Cookbook . . . . . . . . . . . . . . . . . . . . . . 9 A.1. Virtual Mailboxes . . . . . . . . . . . . . . . . . . . . 9 A.2. Other uses . . . . . . . . . . . . . . . . . . . . . . . . 9 A.3. Resynchronizing Contexts . . . . . . . . . . . . . . . . . 9 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 10 Intellectual Property and Copyright Statements . . . . . . . . . . 11 Cridland & King Expires April 14, 2007 [Page 2] Internet-Draft IMAP CONTEXT October 2006 1. Conventions used in this document In examples, "C:" and "S:" indicate lines sent by the client messaging user agent and IMAP4rev1 ([IMAP]) server respectively. The examples show a server which supports [ESEARCH] and [IDLE], neither extension is required for this specification. The IDLE command is used to denote an extended period of time during which any response may be sent to the client. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [KEYWORDS]. Other capitalised words are typically names of IMAP extensions or commands - these are uppercased for clarity only, and are case- insensitive. [[ Editorial comments are like this. XML2RFC working source is held at http://svn.dave.cridland.net/svn/ietf-drafts/ draft-cridland-imap-contexts.xml ]] 2. Introduction Although the basic SEARCH command defined in [IMAP], as enhanced by [ESEARCH], is relatively compact in its representation, this reduction only saves a certain amount of data, and huge mailboxes can overwhelm the storage available for results on even relatively high- end desktop machines. For SORT, as defined in [SORT], use of set-syntax saves considerably less, and so the problem is more acute. This memo borrows concepts from [ACAP], providing a windowed view onto search and sort results, as well as low-bandwidth updates. 3. Protocol Changes 3.1. Overview This extension is present in any IMAP4rev1 server which includes the string "CONTEXTS", or any string beginning "CONTEXTS=", within its advertised capabilities. These servers handle an extended form of the SORT command accepting return options similar to SEARCH, and an extended sorted results response ESORT, described in Section 3.2, which mirrors the extended Cridland & King Expires April 14, 2007 [Page 3] Internet-Draft IMAP CONTEXT October 2006 SEARCH described in [IMAP-ABNF]. [[ Of course, if this were moved to SORT, we'd be happy. ]] Such servers also accept three additional return options, and provide three new result data items, and no new responses. The first search return option is CONTEXT, an optional hint that the criteria will be used repeatedly, and is defined in Section 3.3. The second is NOTIFY, which causes the server to provide efficient notifications of changes to the results. This is defined in Section 3.4. Finally, the PARTIAL return specifier causes the server to return a subset of the results in set-syntax. This allows for "virtual scrollbars" and other UI conveniences to be achieved without having to preload the entire result set, and is described in Section 3.5. 3.2. Extended SORT [[ This needs to describe extended SORT in the same terms as RFC4466. For now, just close your eyes and imagine something largely similar to ESEARCH and SEARCH return options. ]] 3.3. Context Hint The return option CONTEXT SHOULD be used by a client to indicate that subsequent use of the criteria (and sort order, for SORT) are likely. Servers MAY ignore this return option, or use it as a hint to maintain a full result set, or index. A client might choose to obtain a count of matching messages prior to obtaining actual results. Here, the client signifies its intention to fetch the results themselves: C: A01 SEARCH RETURN (CONTEXT COUNT) UNDELETED UNKEYWORD $Junk S: * ESEARCH (TAG A01) COUNT 23765 S: A01 OK Search completed. 3.4. Notifications of changes The search return option NOTIFY, if used by a client, causes the server to issue unsolicited notifications containing updates to the SEARCH or SORT results. These results are carried in ADDTO and REMOVEFROM data items in ESEARCH/ESORT responses. Both ADDTO and REMOVEFROM data items SHOULD be delivered to clients in a timely manner, as and when results changes, whether by new Cridland & King Expires April 14, 2007 [Page 4] Internet-Draft IMAP CONTEXT October 2006 messages arriving in the mailbox, metadata such as flags being changed, or messages being expunged. Typically, this would occur at the same time as the FETCH, EXISTS or EXPUNGE responses carrying the source of the change. Unlike [ACAP], there is no requirement that a context need be created with CONTEXT to use NOTIFY, and in addition, the lack of NOTIFY with a CONTEXT does not affect the results caused by later SEARCH commands - there is no snapshot facility. This time, the client will require notifications of updates, and chooses to obtain a count: C: B01 UID SEARCH RETURN (NOTIFY COUNT) DELETED KEYWORD $Junk S: * ESEARCH (TAG B01) COUNT 74 S: B01 OK Search completed, will notify. 3.4.1. ADDTO Return Data Item The ADDTO return data item contains, as payload, a list containing a position and a set of results to be inserted at the position. For ESEARCH responses, the position MAY be zero, and MAY be ignored by clients. For ESORT responses, the position MUST be the position of the first message referred to in the sequence-set of results, counting 1 as the first position. C: B02 IDLE S: + Idle [...] S: * 23762 FETCH (FLAGS (\Deleted \Seen)) S: * ESEARCH (TAG B01) UID ADDTO (0 32768) C: DONE S: B02 OK Not Idle. Note that this example assumes message 23762 with UID 32768 previously had neither \Deleted nor $Junk set. Also note that only the ADDTO is included, and not the COUNT. 3.4.2. REMOVEFROM Return Data Item The REMOVEFROM return data item contains a set of results to be removed. The results to be removed are referenced by message number or UID, as appropriate, and need not be in the same order as the results. Servers SHOULD sort the results in order to use the sequence-set syntax as efficiently as possible. Cridland & King Expires April 14, 2007 [Page 5] Internet-Draft IMAP CONTEXT October 2006 There is no requirement on servers to avoid issuing REMOVEFROM return data at any particular moment, in particular this is distinct from EXPUNGE responses. Command B03 here is purely an example of a command which prohibits EXPUNGE messages. The REMOVEFROM could have been sent without any command in progress. C: B03 SEARCH RETURN () 1:* ALL S: * ESEARCH (TAG B03) ALL 1:49152 S: * ESEARCH (TAG B01) UID REMOVEFROM 32768 S: B03 OK Search completed. C: B04 IDLE S: + Idle S: * EXPUNGE 23762 [...] C: DONE S: B04 OK Not Idle. 3.5. Partial results The PARTIAL search return option causes the server to provide in an ESEARCH or ESORT response the range from the results denoted by the sequence range given as the mandatory argument. The first result is 1, thus the first 500 results would be obtained by a return option of "PARTIAL 1:500", and the second 500 by "PARTIAL 501:1000". This intentionally mirrors message sequence numbers. The subset of results are returned in sequence-set syntax, and servers SHOULD order results from a SEARCH for maximum efficiency. Clients need not request PARTIAL results in any particular order. C: A02 UID SEARCH RETURN (PARTIAL 23500:24000) UNDELETED UNKEYWORD $Junk S: * ESEARCH (TAG A02) PARTIAL (23500:24000 ...) // Elided results include 264 results in this sequence. S: A02 OK Completed. 3.6. Caching results Server implementations MAY cache results from a search or sort, whether or not hinted to by CONTEXT, in order to make subsequent searches more efficient, perhaps by recommencing a subsequent PARTIAL search where a previous search left off. However servers MUST behave identically whether or not internal caching is taking place, therefore any such cache is required to be updated as changes to the Cridland & King Expires April 14, 2007 [Page 6] Internet-Draft IMAP CONTEXT October 2006 mailbox occur. An alternate strategy would be to discard results when any change occurs to the mailbox. 4. Formal Syntax The collected formal syntax. This includes definitions from [IMAP] and [IMAP-ABNF], and uses ABNF as defined in [ABNF]. addto-position = number ;; Number can be 0 for SEARCH result additions. ;; number from RFC3501 modifier-context = "CONTEXT" modifier-partial = "PARTIAL" SP seq-range ;; seq-range from RFC3501 modifier-notify = "NOTIFY" search-return-opt =/ modifier-context / modifier-partial / modifier-notify ;; All conform to search-return-opt, from RFC4466 ret-data-addto = "ADDTO" SP "(" addto-position SP sequence-set ")" ;; sequence-set from RFC3501 ret-data-partial = "PARTIAL" SP "(" seq-range SP sequence-set ")" ;; sequence-set from RFC3501 ret-data-removefrom = "REMOVEFROM" SP sequence-set ;; sequence-set from RFC3501 search-return-data =/ ret-data-partial / ret-data-addto / ret-data-removefrom ;; All conform to search-return-data, from RFC4466. 5. Security Considerations The authors cannot think of any. 6. IANA Considerations [[ The extension needs registering - TODO: find correct text for this Cridland & King Expires April 14, 2007 [Page 7] Internet-Draft IMAP CONTEXT October 2006 bit. ]] 7. Acknowledgements Much of the design of this extension can be found in ACAP. Valuable comments, both in agreement and in dissent, were received from Alexey Melnikov, Arnt Gulbrandsen, Randall Gellens, Cyrus Daboo, and others, and many of these comments have had significant influence on the design or the text. The authors are grateful to all those involved, including those not mentioned here. 8. References 8.1. Normative References [ABNF] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 4234, October 2005. [IMAP] Crispin, M., "INTERNET MESSAGE ACCESS PROTOCOL - VERSION 4rev1", RFC 3501, March 2003. [IMAP-ABNF] Melnikov, A. and C. Daboo, "Collected Extensions to IMAP4 ABNF", RFC 4466, April 2006. [KEYWORDS] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [SORT] Crispin, M. and K. Murchison, "INTERNET MESSAGE ACCESS PROTOCOL - SORT AND THREAD EXTENSION", draft-ietf-imapext-sort-17 (work in progress), May 2004. 8.2. Informative References [ACAP] Newman, C. and J. Myers, "ACAP -- Application Configuration Access Protocol", RFC 2244, November 1997. [ESEARCH] Melnikov, A. and D. Cridland, "IMAP4 extension to SEARCH command for controlling what kind of information is returned", draft-melnikov-imap-search-ret-03 (work in progress), June 2006. [IDLE] Leiba, B., "IMAP4 IDLE command", RFC 2177, June 1997. Cridland & King Expires April 14, 2007 [Page 8] Internet-Draft IMAP CONTEXT October 2006 Appendix A. Cookbook A.1. Virtual Mailboxes It is possible to use the facilities described within this memo to create a facility largely similar to a virtual mailbox, but handled on the client side. Because the messages are still referred to by their native UIDs, and there is no constraint to fit the IMAP mailbox interface, this is a simpler method without the potential pitfalls of stretching metaphors. Initially, the client SELECTs the real "backing" mailbox. Next, it can switch to a filtered view at any time by issuing a SEARCH RETURN (COUNT NOTIFY CONTEXT), and using SEARCH RETURN (PARTIAL x:y) as the user scrolls, feeding the results into a FETCH to populate summary views as required. Similarly, a sorted display can be obtained by using SORT instead of SEARCH. A.2. Other uses It is possible to simultaneously have two or more NOTIFY searches or sorts in operation. This can be used to build a grouped message display. A.3. Resynchronizing Contexts Although not specifically provided by the protocol as defined, it is possible for clients to resynchronize a notify unsorted context without having to refetch the entire result set, if the server supports CONDSTORE. The following steps are believed to be correct: Recreate the context as soon as possible, returning the COUNT. Check the HIGHESTMODSEQ value for changes. Resynchronize UID mapping, removing any messages from the contexts which have been expunged. Find all changed messages (using, perhaps, UID FETCH 1:* FLAGS (CHANGEDSINCE)). Either calculate locally, or research across those messages, to discover the changes to the new context. Client authors should note that for contexts which contain only a small percentage of the entire mailbox - or have a very fluid membership - it will be cheaper in terms of bandwidth to simply recreate the context from scratch. Cridland & King Expires April 14, 2007 [Page 9] Internet-Draft IMAP CONTEXT October 2006 Authors' Addresses Dave Cridland Isode Limited 5 Castle Business Village 36, Station Road Hampton, Middlesex TW12 2BX GB Email: dave.cridland@isode.com Curtis King Isode Limited 5 Castle Business Village 36, Station Road Hampton, Middlesex TW12 2BX GB Email: cking@mumbo.ca Cridland & King Expires April 14, 2007 [Page 10] Internet-Draft IMAP CONTEXT October 2006 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2006). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Cridland & King Expires April 14, 2007 [Page 11]