Internet DRAFT - draft-fanf-smtp-rfc1845bis

draft-fanf-smtp-rfc1845bis






SMTP extensions                                                 T. Finch
Internet-Draft                                   University of Cambridge
Obsoletes: 1845 (if approved)                           February 6, 2007
Intended status: Standards Track
Expires: August 10, 2007


         SMTP service extensions for transaction checkpointing
                     draft-fanf-smtp-rfc1845bis-01

Status of this Memo

   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on August 10, 2007.

Copyright Notice

   Copyright (C) The IETF Trust (2007).













Finch                    Expires August 10, 2007                [Page 1]

Internet-Draft       SMTP transaction checkpointing        February 2007


Abstract

   This memo describes the SMTP service extension for checkpoint/resume,
   which allows a client to recover from a lost connection to the server
   without having to repeat all of the commands and message content sent
   prior to the interruption, and with less risk of duplicated messages.
   It also includes an updated specification of the predecessor SMTP
   service extension for checkpoint/restart.











































Finch                    Expires August 10, 2007                [Page 2]

Internet-Draft       SMTP transaction checkpointing        February 2007


Document revision

   $Cambridge: hermes/doc/qsmtp/draft-fanf-smtp-rfc1845bis.xml,v 1.75
   2007/02/06 00:56:44 fanf2 Exp $


Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  4
     1.1.  Overview . . . . . . . . . . . . . . . . . . . . . . . . .  4
     1.2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . .  5
     1.3.  IANA Considerations  . . . . . . . . . . . . . . . . . . .  5
   2.  SMTP service extension for checkpoint/resume . . . . . . . . .  6
     2.1.  Framework  . . . . . . . . . . . . . . . . . . . . . . . .  6
     2.2.  Overview . . . . . . . . . . . . . . . . . . . . . . . . .  7
     2.3.  Transaction IDs  . . . . . . . . . . . . . . . . . . . . .  7
     2.4.  Octet offsets  . . . . . . . . . . . . . . . . . . . . . .  8
     2.5.  Server-side resume state . . . . . . . . . . . . . . . . .  9
     2.6.  Retry versus resume  . . . . . . . . . . . . . . . . . . . 10
     2.7.  Re-establishing a connection . . . . . . . . . . . . . . . 10
     2.8.  The RESUME command . . . . . . . . . . . . . . . . . . . . 11
     2.9.  The MAIL command TRANSID and TRANSOFF parameters . . . . . 12
     2.10. The QUIT command . . . . . . . . . . . . . . . . . . . . . 13
   3.  SMTP service extension for checkpoint/restart  . . . . . . . . 14
     3.1.  Framework  . . . . . . . . . . . . . . . . . . . . . . . . 14
     3.2.  Description  . . . . . . . . . . . . . . . . . . . . . . . 14
     3.3.  Changes from RFC 1845  . . . . . . . . . . . . . . . . . . 16
     3.4.  Prefer checkpoint/resume to checkpoint/restart . . . . . . 17
   4.  Security considerations  . . . . . . . . . . . . . . . . . . . 18
     4.1.  Server storage . . . . . . . . . . . . . . . . . . . . . . 18
     4.2.  Transaction IDs  . . . . . . . . . . . . . . . . . . . . . 19
     4.3.  Unnecessary bounces  . . . . . . . . . . . . . . . . . . . 19
   5.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 21
     5.1.  Normative references . . . . . . . . . . . . . . . . . . . 21
     5.2.  Informative references . . . . . . . . . . . . . . . . . . 21
   Appendix A.  Acknowledgments . . . . . . . . . . . . . . . . . . . 23
   Appendix B.  Changes since version -00 . . . . . . . . . . . . . . 24
   Appendix C.  Draft discussion venue  . . . . . . . . . . . . . . . 25
   Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 26
   Intellectual Property and Copyright Statements . . . . . . . . . . 27











Finch                    Expires August 10, 2007                [Page 3]

Internet-Draft       SMTP transaction checkpointing        February 2007


1.  Introduction

   Although SMTP is widely and robustly deployed, it does not handle the
   loss of its underlying connection particularly gracefully.  There are
   two problems, and they are becoming more of a concern because of the
   way the Internet is changing.

   Firstly, if the connection is lost part-way through the transmission
   of a message, the client must retry the transmission starting from
   the beginning.  When dealing with very large messages over less
   reliable connections it is possible for substantial resources to be
   consumed by repeated unsuccessful attempts to transmit the message in
   its entirety.  Messages are getting larger on average.  Wireless
   connections, which are much more likely to be lost in normal use, are
   becoming more common.

   Secondly, a connection that is lost after the client has transmitted
   the message but before it receives the server's final reply can
   result in a duplicated message.  This problem is described in
   [RFC1047], which recommends that servers should minimize the time
   between receiving the last of the message data and sending their
   final reply.  However it is often preferable to analyse the message
   data for spam and viruses at this point so that the server can avoid
   taking responsibility for an undesirable message, and this can be
   time consuming.

   Furthermore, the problem is worse for clients that try to make the
   most of high-latency connections.  The minimum time for an SMTP
   transaction is normally one round trip, since the client has to wait
   for all outstanding server replies after sending the DATA command
   [RFC2920].  If the client's messages are smaller than the amount of
   data it can transmit in a round trip, i.e. the bandwidth*delay
   product, then the connection is sitting idle for some of the time.  A
   client can work around this problem by using multiple concurrent SMTP
   connections, or by using the SMTP service extension for large
   messages [RFC3030].  The latter eliminates pipeline stalls so the
   client can stream multiple messages to the server while waiting for
   replies.  However, with both work-arounds, one loss of network
   connectivity means multiple messages will need retransmission and may
   be duplicated.

1.1.  Overview

   This memo provides a facility by which a client can uniquely identify
   a particular SMTP transaction.  The server stores this identifying
   information along with all the information it receives and sends as
   the transaction proceeds.  If the connection is lost during the
   transaction the SMTP client may establish a new connection and ask



Finch                    Expires August 10, 2007                [Page 4]

Internet-Draft       SMTP transaction checkpointing        February 2007


   the server to resume the transaction.  The server tells the client
   how much message data it saved from the interrupted transaction.  The
   client can then perform an abbreviated transaction, repeating only
   those commands to which it did not receive replies, and transmitting
   only message data that the server does not yet have.

   These problems were originally tackled by the SMTP service extension
   for checkpoint/restart [RFC1845].  However it has a number of
   limitations (Section 3.3) so this memo describes the similar but
   improved SMTP service extension for checkpoint/resume in Section 2.
   We address backwards compatibility in Section 3 by re-defining
   checkpoint/restart as a modification of checkpoint/resume.  Finally,
   Section 4 gives proper consideration to the security implications of
   these extensions.

1.2.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119.

   The "message envelope" comprises the SMTP MAIL and RCPT commands.
   The "message data" is transmitted using the SMTP DATA command, or
   alternatives such as the BDAT command [RFC3030].

   An SMTP "transaction" is the sequence of commands necessary to
   transmit a message, i.e. a message envelope followed by message data.
   It starts with a MAIL command and ends with the server's "final
   reply" to the last of the message data, or with a reset.  The
   server's final reply in an LMTP (local mail transport protocol)
   transaction can comprise more than one per-recipient sub-reply
   [RFC2033].  A "reset" is caused by the RSET, HELO, or EHLO commands.

   The metalinguistic notation used in this memo corresponds to the
   "Augmented Backus-Naur Form" defined in [RFC4234].  Rules not defined
   here are either defined in the ABNF core rules or in [RFC2821] or
   [RFC2822].  Metalanguage terms used in running text are surrounded by
   pointed brackets (e.g., <transid-spec>).

1.3.  IANA Considerations

   This memo defines the SMTP service extension for checkpoint/resume,
   with the EHLO keyword value "RESUME", in Section 2.

   This memo replaces RFC 1845 as the definition of the SMTP service
   extension for checkpoint/restart, with the EHLO keyword value
   "CHECKPOINT", in Section 3.




Finch                    Expires August 10, 2007                [Page 5]

Internet-Draft       SMTP transaction checkpointing        February 2007


2.  SMTP service extension for checkpoint/resume

   This section uses the SMTP extension model specified in [RFC2821].

2.1.  Framework

   The SMTP service extension for checkpoint/resume is defined as
   follows:

   o  The name of the service extension is "checkpoint/resume".

   o  The EHLO keyword associated with the extension is "RESUME".

   o  The RESUME EHLO keyword has no parameters.

   o  This extension defines one additional command, RESUME, which MAY
      appear anywhere in a pipelined group [RFC2920].

      Syntax:

            resume-command = "RESUME" SP transid-value CRLF

   o  This extension defines the 355 reply to the RESUME command.

      Syntax:

            resume-reply = "355" SP octet-offset SP text CRLF

   o  This extension defines two additional parameters to the MAIL
      command.

      The TRANSID parameter has the following syntax:

            esmtp-param     =/  transid-param
            transid-param   =   "TRANSID=" transid-value
            transid-value   =   "<" transid-spec ">"
            transid-spec    =   dot-string "@" domain

      The TRANSOFF parameter has the following syntax:

            esmtp-param     =/  transoff-param
            transoff-param  =   "TRANSOFF=" octet-offset
            octet-offset    =   1*20DIGIT

   o  The maximum length of the MAIL command is increased by 297
      characters.  The maximum length of a <transid-spec> is 256
      characters.




Finch                    Expires August 10, 2007                [Page 6]

Internet-Draft       SMTP transaction checkpointing        February 2007


   o  There are no additional parameters to the RCPT command defined by
      this extension and its maximum length is not increased.

   o  This extension is suitable for use with message submission
      [RFC4409] and LMTP [RFC2033].

2.2.  Overview

   A server that supports the SMTP service extension for checkpoint/
   resume SHALL include the RESUME keyword in its reply to the client's
   EHLO command.  A client that wishes to use this extension MUST first
   check that this EHLO keyword is present.

   The client then proceeds as usual, except that it issues MAIL
   commands with TRANSID and TRANSOFF parameters (Section 2.9).  The
   TRANSID parameter's value is described in Section 2.3 and the
   TRANSOFF parameter's value is "0" (zero).  The server periodically
   checkpoints the transaction, retaining state so that it can later be
   resumed (Section 2.5).  If all goes well, the client will close the
   connection normally and the server can discard the resume state
   (Section 2.10).

   If the connection is lost (Section 2.6) then the client can reconnect
   (Section 2.7) and issue a RESUME command (Section 2.8) for each
   transaction in the lost connection.  It thereby discovers the <octet-
   offset> at which it must resume transmitting each transaction's data
   (Section 2.4).  The client then issues MAIL commands with TRANSID and
   non-zero TRANSOFF parameters to resume any transactions that are
   missing data on the server or for which the client lost any of the
   server's replies.

2.3.  Transaction IDs

   A <transid-spec> serves to uniquely identify a particular SMTP
   transaction started by a particular client.  The <transid-spec> and
   client identity together form the "transaction ID".

   The <transid-spec> is structured to ensure global uniqueness without
   any additional registry.  Its domain part SHOULD be a valid domain
   name that uniquely identifies the SMTP client.  This is usually the
   same as the domain name given in the EHLO command, but not always.
   The EHLO domain name identifies the specific host the SMTP connection
   originated from, whereas the <transid-spec> domain can refer to a
   group of hosts that collectively host a multi-homed SMTP client, or
   it can refer to a mobile client's home domain name, etc.

   The <transid-spec> local part MUST be an identifier that
   distinguishes this SMTP transaction from any other originating from



Finch                    Expires August 10, 2007                [Page 7]

Internet-Draft       SMTP transaction checkpointing        February 2007


   this SMTP client.  Care must be used in constructing a <transid-spec>
   to simultaneously ensure both uniqueness, unguessability, and the
   ability to reidentify interrupted transactions.  It MUST be
   unguessable for security reasons; see Section 4.

   Despite the structured nature of a <transid-spec> the server MUST
   treat the value as an opaque, case-sensitive string.

   Note that the contents of the [RFC2822] Message-ID: header field, or
   the MAIL FROM: ENVID parameter [RFC3461], typically are NOT
   appropriate for use as a <transid-spec>, since such identifiers may
   be associated with multiple copies of the same message - e.g., as it
   is split during transmission to different recipients - and hence with
   multiple distinct SMTP transactions.

   For security reasons, servers MUST treat the same <transid-spec> from
   different clients as different transaction IDs.  Servers MUST use a
   secure identifier to distinguish clients, such as credentials from
   the SMTP AUTH command [RFC2554] or from TLS negotiation [RFC3207].
   Note that these identifiers are independent of the IP address a
   client connects from: servers MUST allow authenticated mobile clients
   to reconnect and resume transactions from different IP addresses.  If
   a client is not authenticated the server SHOULD use its IP address to
   identify it.

2.4.  Octet offsets

   The <octet-offset> represents an offset, counting from zero, to the
   particular octet in the actual message data the server expects to see
   next.  (This is also a count of how many octets the server has
   received and stored successfully.)  This offset SHALL NOT account for
   the message envelope.  A value of 0 would indicate that the client
   needs to start sending the message from the beginning, a value of 1
   would indicate that the client should skip one octet, and so on, up
   to a maximum equal to the message size.  Additional requirements
   apply depending on the command(s) used by the client to transmit the
   message data:

   DATA  [RFC2821] section 4.5.2: The <octet-offset> SHALL NOT count any
      octets added by the dot-stuffing algorithm.  The offset MUST also
      correspond to the start of a line, i.e. equal to zero or a point
      immediately after a <CRLF>.

   BDAT  [RFC3030]: The <octet-offset> SHALL count octets in the same
      way as the <chunk-size> parameter to the BDAT command.  It MAY
      point anywhere within a chunk.  It SHALL NOT count any octets for
      the BDAT command itself.




Finch                    Expires August 10, 2007                [Page 8]

Internet-Draft       SMTP transaction checkpointing        February 2007


   BURL  [RFC4468]: The <octet-offset> SHALL count the size of the
      content retrieved from the URL given in the BURL command.  The
      offset MUST NOT correspond to a point within the BURL data; that
      is, the server stores the whole URL content or none of it.  The
      <octet-offset> SHALL NOT count any octets for the BURL command
      itself.

   These requirements are consistent with the semantics of the SIZE
   parameter to the MAIL command [RFC1870].

2.5.  Server-side resume state

   The server's "resume state" is the additional information it retains
   in order to implement checkpoint/resume.

   The server SHOULD keep track of the transaction IDs associated with
   each connection, that is the <transid-spec>s used by the client in
   RESUME commands or TRANSID parameters.  This so that it can prevent
   multiple connections from trying to modify the same transaction, as
   described in Section 2.7, and so that the server can discard the
   resume state of the connection's transactions when the connection is
   closed cleanly, as described in Section 2.10.

   What is included in a transaction's resume state varies depending on
   whether or not the server has received any message data, and whether
   or not it has committed the transaction.  The server commits the
   transaction when it decides what its final reply to the last of the
   message data is.

   Servers retain no resume state before they receive any message data.
   This implies that clients must resume the transaction from the
   beginning if the connection is lost while the client is transmitting
   the envelope.

   Before the server commits a transaction, its resume state comprises
   the message envelope (including all client commands and server
   replies) and any message data that the server has received so far.
   It is OPTIONAL for the server to retain this state.  That is, a
   server MAY be configured to require some or all clients to resume
   interrupted transactions from the beginning.

   When the server has received the last of the message data it SHOULD
   proceed to commit the transaction regardless of whether the client
   connection is lost while it does so.  Since committing can be a
   multi-stage process (especially with LMTP [RFC2033]) this ensures
   that the client sees a consistent result even after a connection loss
   and resume.  If the server commits to a 2yz final reply then it can
   continue with onward delivery of the message independent of client



Finch                    Expires August 10, 2007                [Page 9]

Internet-Draft       SMTP transaction checkpointing        February 2007


   activity.

   After the transaction is committed, its resume state comprises the
   message envelope (as before), the size of the message data, and the
   server's final reply.  If the client loses a connection and takes a
   while to reconnect and resume the transactions, it can be necessary
   for the server to retain this resume state for longer than it takes
   to transmit the message on to its next destination.

   If the client terminates a transaction with a reset, the server SHALL
   discard any resume state for the transaction ID and remove the
   transaction ID from the list of those associated with the current
   connection.  If the client issues a reset between transactions, the
   server SHALL retain any resume state for the preceding
   transaction(s).

2.6.  Retry versus resume

   A client SHOULD NOT attempt to resume a transaction unless the SMTP
   connection was lost after the start of the transaction.  A connection
   is lost if it is terminated without the client sending a QUIT command
   to the server, or receiving a 421 reply to any command.  This
   includes SMTP-level timeouts as well as loss or premature closure of
   the underlying connection.  If the connection is lost before the
   first MAIL command, or message transmission fails for another reason,
   the client MUST use the retry algorithm specified by [RFC2821].
   These non-resume situations include, but are not limited to, the
   following:

   o  failure to establish a connection;

   o  failure to establish a security layer;

   o  failure to authenticate;

   o  a temporary failure indicated by a 4yz reply from the server;

   o  a connection forcibly closed by the server with a 421 reply.

2.7.  Re-establishing a connection

   When a connection has been lost as described in the previous section,
   the client MAY reconnect immediately.  It SHALL first re-check the
   DNS if necessary, as determined by the DNS records' time-to-live.  If
   the IP address (target of A or AAAA record) used to connect to the
   original server is still on this list it SHOULD be tried first, since
   this server is most likely to be capable of resuming the transaction.
   If that IP address is no longer on the list or if the connection



Finch                    Expires August 10, 2007               [Page 10]

Internet-Draft       SMTP transaction checkpointing        February 2007


   fails, then the client SHOULD next try any other IP addresses
   associated with the host name (target of MX record) used to connect
   to the original server.  If these also fail then the client SHOULD
   fall back to the ranking algorithm described in [RFC2821] section 5.

   A client SHOULD NOT wait for a server with an interrupted
   transaction's resume state to come back online.  Multi-homed and
   clustered SMTP servers do exist, which means that it is entirely
   possible for a transaction to be resumed on a different server host.

   Note that connection loss can appear different from the client and
   the server ends, so a client might detect the loss and reconnect
   before the server detects the loss.  This can lead to legitimate
   circumstances where there are multiple connections to the server that
   appear to be trying to use the same transaction ID.  Therefore, if a
   client issues a command containing a <transid-spec> that the client
   has used in another connection that is currently active from the
   server's point of view, then the server SHOULD prevent the older
   connection from performing further actions that affect the same
   transaction.  For example, the server MAY drop all but the connection
   with the most recent activity.  Even if the server is still
   committing a transaction when the client resumes it, the server MUST
   issue the final reply that it commits to.

2.8.  The RESUME command

   A client uses the RESUME command to discover how much message data
   the server has stored for a given <transid-spec>, which is specific
   to that client.  If the server has no resume state associated with
   the transaction ID, it SHALL reply with a 355 code and an <octet-
   offset> of "0" (zero).  If the server has resume state associated
   with the transaction ID the server SHALL reply with a 355 code and an
   <octet-offset> equal to the amount of message data received by the
   server so far.

   The <octet-offset> field in a 355 reply is REQUIRED to be the first
   thing on the first line of the reply.  It MUST be separated from any
   following <text> by linear whitespace.  The <text> can vary and MUST
   be ignored by the client.

   For example,

          355 64735 is the transaction offset

   The client MUST NOT issue a RESUME command during a MAIL transaction.
   The server SHOULD reject a RESUME command issued during a transaction
   with a "503 Bad sequence of commands" reply.




Finch                    Expires August 10, 2007               [Page 11]

Internet-Draft       SMTP transaction checkpointing        February 2007


   After reconnecting, clients SHOULD issue RESUME commands for all the
   transactions in a lost connection, not just for the transactions that
   were interrupted.  This is so that when the client QUITs the server
   can discard all their resume state.  See Section 2.10.

   The possible failure replies to the RESUME command are 500, 501, and
   421, as described in [RFC2821] section 4.3.2.

2.9.  The MAIL command TRANSID and TRANSOFF parameters

   A client SHALL start or resume a resumable transaction by issuing a
   MAIL command with one TRANSID parameter and one TRANSOFF parameter.
   It MUST NOT include more than one of either parameter, or omit either
   parameter.  (However see Section 3 for the meaning of a TRANSID
   parameter without a TRANSOFF parameter.)

   If this is a new transaction, the TRANSOFF MUST be "0" (zero).  The
   server SHALL discard any resume state that it may have retained for
   the given <transid-spec>, which is specific to that client.  The
   transaction then proceeds as normal, with the server retaining
   additional resume state as described in Section 2.5.

   If the client is resuming a transaction, then it MUST have previously
   issued a RESUME command in this connection with the same <transid-
   spec> as in the TRANSID parameter.  The value of the TRANSOFF
   parameter MUST be the same as the <octet-offset> given in the
   server's reply to the RESUME command.  Apart from the non-zero
   <octet-offset> the MAIL command MUST be the same as the one issued by
   the client to start the transaction originally.  If the client issues
   a MAIL command with a non-zero TRANSOFF that does not meet these
   requirements then the server SHALL reply with "503 Bad sequence of
   commands".  Otherwise the server SHALL give the same reply to the
   MAIL command that it retained in the transaction's resume state.

   After the client has issued a MAIL command with a non-zero TRANSOFF,
   it MAY re-issue the transaction's RCPT commands.  The client MAY omit
   some or all of the RCPT commands but it MUST NOT re-order them.  The
   server SHALL give the same reply to each RCPT command that it
   retained in the transaction's resume state.  If the client issues a
   RCPT command that was not retained in the resume state then the
   server SHALL reject it with a "553 Requested action not taken:
   mailbox name not allowed" reply.

   After the client has issued the envelope commands, it SHALL issue
   another data transfer command (e.g.  DATA or BDAT) and send the
   remaining message data starting from the <octet-offset>.  The
   requirement in [RFC3030] against mixing DATA and BDAT in the same
   transaction still applies if the transaction is interrupted and later



Finch                    Expires August 10, 2007               [Page 12]

Internet-Draft       SMTP transaction checkpointing        February 2007


   resumed.  If the <octet-offset> is equal to the message size, then
   the client SHALL issue a data transfer command with no data (e.g.
   DATA<CRLF>.<CRLF> or BDAT 0 LAST<CRLF>) and the server SHALL give the
   final reply it retained in the transaction's resume state.

2.10.  The QUIT command

   The client MUST issue a QUIT command before closing the connection.
   It MUST NOT pipeline the QUIT command; that is, it SHALL wait to
   receive the replies to all outstanding commands before issuing the
   QUIT command.  Note that this is stricter than [RFC2920] which allows
   the QUIT command to appear as the last command in a pipelined group.

   This requirement means that when the server receives a QUIT command,
   it can be sure that the client has received all the replies to all
   the transactions in the connection, so the server MAY then discard
   any resume state associated with these transactions.  The server MUST
   NOT rely on this for resource reclamation and MUST still time out old
   resume state.  This protects against malicious clients and some
   legitimate failure modes.  For example, if the connection is lost
   after the client sends QUIT but before the server receives it, the
   client will not reconnect and the server will not immediately free
   its resume state.




























Finch                    Expires August 10, 2007               [Page 13]

Internet-Draft       SMTP transaction checkpointing        February 2007


3.  SMTP service extension for checkpoint/restart

   This section re-defines checkpoint/restart in terms of checkpoint/
   resume, and lists the differences between this specification and
   [RFC1845].  We also explain why checkpoint/resume is better than
   checkpoint/restart.

3.1.  Framework

   The SMTP service extension for checkpoint/restart is defined as
   follows:

   o  The name of the service extension is "checkpoint/restart".

   o  The EHLO keyword associated with the extension is "CHECKPOINT".

   o  The CHECKPOINT EHLO keyword has no parameters.

   o  This extension defines one additional parameter to the MAIL
      command.  The TRANSID parameter has the syntax defined in
      Section 2.

   o  The maximum length of the MAIL command is increased by 88
      characters.  The maximum length of a <transid-spec> is 77
      characters.  If the server also supports the checkpoint/resume
      extension, then its larger limits apply instead of (not in
      addition to) these limits.

   o  This extension defines the 355 reply to the MAIL command which has
      the syntax defined in Section 2.

   o  There are no additional parameters to the RCPT command defined by
      this extension and its maximum length is not increased.

   o  There are no additional commands defined by this extension.

   o  This extension is suitable for use with message submission
      [RFC4409].

3.2.  Description

   A server that supports the SMTP service extension for checkpoint/
   restart SHALL include the CHECKPOINT keyword in its reply to the
   client's EHLO command.  A client that wishes to use this extension
   MUST first check that this EHLO keyword is present.  It SHALL then
   issue a MAIL command with one TRANSID parameter and without a
   TRANSOFF parameter.  Such a MAIL command MUST appear as the last
   command in a pipelined group; note that this is stricter than the



Finch                    Expires August 10, 2007               [Page 14]

Internet-Draft       SMTP transaction checkpointing        February 2007


   usual pipelining requirements for the MAIL command specified in
   [RFC2920].

   Issuing a MAIL FROM:<...> TRANSID=<transid-spec> command to a server
   that supports checkpoint/restart is equivalent to issuing the
   following hypothetical pair of commands to a server that supports
   checkpoint/resume:

         RESUME <transid-spec>
         MAIL FROM:<...> TRANSID=<transid-spec> TRANSOFF=<octet-offset>

   The <octet-offset> value of the hypothetical TRANSOFF parameter is
   the same as the <octet-offset> given in the server's hypothetical 355
   reply to the RESUME command.

   One of the two hypothetical replies to this pair of checkpoint/resume
   commands is given in response to the equivalent real checkpoint/
   restart MAIL...TRANSID command.  The reply is chosen according to the
   following rules:

   o  If the hypothetical reply to the RESUME command would have
      indicated failure (i.e. not 355) then that is used as the real
      reply;

   o  Otherwise, if the hypothetical reply to the MAIL FROM command
      would have indicated failure (i.e. not 250) then that is used as
      the real reply;

   o  Otherwise, if the hypothetical <octet-offset> given by the
      server's 355 reply and used in the MAIL FROM command would have
      been zero, then the hypothetical 250 reply to the MAIL FROM
      command is used as the real reply; (This is the case for new
      transactions.)

   o  Otherwise, the hypothetical 355 reply to the RESUME command is
      used as the real reply.  (This is the case for resumed
      transactions.)

   When a client wishes to start a new resumable transaction, it SHALL
   issue a MAIL...TRANSID command.  If it does not get the expected 250
   reply (which would indicate an accidental or malicious transaction ID
   collision) it SHALL issue a RSET command to reset the transaction,
   and re-issue the MAIL...TRANSID command.

   When a client wishes to resume a transaction after a lost connection,
   it SHALL issue the same MAIL...TRANSID command again.  If it receives
   a 250 reply, it MUST repeat the transaction in full.  If it receives
   a 355 reply, it MAY re-issue the rest of the transaction's envelope



Finch                    Expires August 10, 2007               [Page 15]

Internet-Draft       SMTP transaction checkpointing        February 2007


   commands, then it SHALL issue a data transfer command and resume
   transmission of the message data at the exact <octet-offset>
   indicated in the 355 reply.

   In all other respects this extension works in the same way as
   checkpoint/resume (Section 2).

3.3.  Changes from RFC 1845

   This section lists differences between checkpoint/restart as
   specified in this memo and as specified in [RFC1845].  We also give
   the reasons for each change.

   o  Interworking with pipelining [RFC2920] is improved in a number of
      ways:

      *  We explicitly state that a checkpoint/restart MAIL command has
         stricter pipelining requirements than specified in [RFC2920].

      *  A client that pipelines its envelope commands and message data
         (using BDAT [RFC3030] or BURL [RFC4468]) can lose the server's
         envelope replies when a connection is lost.  Clients may now
         re-issue envelope commands when resuming a transaction.

      *  A client can pipeline the end of one transaction with the start
         of the next, so a new transaction does not indicate that the
         previous transaction is complete from the client's point of
         view.  Servers are no longer permitted to discard a
         transaction's resume state at this point.

   o  We now specify the semantics of the <octet-offset> when the
      message data is transmitted using the BDAT [RFC3030] or BURL
      [RFC4468] commands.

   o  We now specify how to use this extension with LMTP [RFC2033].

   o  The server's data retention requirements have been loosened.  This
      allows a server to advertise support for checkpointing to less
      trustworthy clients, with less security exposure.  See Section 4.

   o  In [RFC1845] the <transid-spec> is coupled to the client's EHLO
      host name.  This memo specifies the use of higher-level
      authenticated client identities where possible, so that mobile
      clients can re-connect from a different IP address (which implies
      a different EHLO host name) and resume interrupted transactions.

   o  This specification allows a client to re-connect to the server
      sooner when resuming a transaction than is usual for normal SMTP



Finch                    Expires August 10, 2007               [Page 16]

Internet-Draft       SMTP transaction checkpointing        February 2007


      retries, as specified in [RFC2821] section 4.5.4.  This is
      friendlier to message submission clients that involve a human in
      the retry strategy.

   o  We have identified some security concerns that are discussed in
      Section 4.

   o  A couple of trivial matters:

      *  [RFC2119] keywords for normative requirements.

      *  Corrected octet counts in the MAIL command length limit
         increase.

3.4.  Prefer checkpoint/resume to checkpoint/restart

   Even after updating and clarifying [RFC1845], there remains a
   significant problem with checkpoint/restart, which is why this memo
   defines and prefers the not-quite-backwards-compatible variant
   checkpoint/resume.

   The problem is that checkpoint/restart adds a pipeline stall to each
   transaction.  This increases the time taken to send a message, which
   is particularly undesirable for message submission clients on high-
   latency connections.  It also prevents clients from using the BDAT
   command [RFC3030] to stream messages to the server without pipeline
   stalls.  This is particularly unfortunate because losing a connection
   during pipelined streaming can affect multiple messages, so it is
   desirable to have some way of recovering the state of transactions
   after a lost connection.

   The pipeline stall in checkpoint/restart has two functions, which
   checkpoint/resume handles in ways that avoid the stall.

   o  In checkpoint/restart, the server tells the client whether the
      transaction is new or is being resumed.  In checkpoint/resume,
      this information goes from the client to the server, which removes
      the need for an extra round trip in normal transactions.

   o  When recovering from a lost connection, the client must find out
      the <octet-offset> at which it will resume transmitting message
      data.  This implies a client-server round trip before resuming the
      transaction.  In checkpoint/resume this round trip is detached
      from the transaction using the RESUME command.  The RESUME command
      can be pipelined, so if multiple transactions are to be resumed
      only one extra round trip per connection is needed, not one per
      message.




Finch                    Expires August 10, 2007               [Page 17]

Internet-Draft       SMTP transaction checkpointing        February 2007


4.  Security considerations

   The difficulties fall into three areas: additional storage
   requirements on the server; vulnerabilities associated with spoofed
   transaction IDs; and undesirable bounce messages.  All are
   significantly mitigated if checkpoint/resume is only permitted for
   trustworthy clients (which generally implies secure authentication).
   This section also applies to [RFC1845], which does not discuss
   security.

4.1.  Server storage

   A significant difference between partial transactions and complete
   transactions is that the server can recover the storage used by
   complete transactions by delivering their messages.  Thus if a server
   gives partial transactions a long lifetime, it can be easier for an
   attacker to exhaust the server's disk space than with un-extended
   SMTP.  The attacker does not have to outrun the server's ability to
   process messages, nor search for a destination address to which the
   server cannot deliver immediately (such as an over-quota user).

   The extensions in this memo permit a server to choose whether or not
   to store partial messages according to operational needs.  For
   example, partial transaction storage might be permitted for
   authorized message submission clients, but MX clients might be
   required to recover from failed transactions by retrying the
   transaction from the start.

   When partial messages are stored, their lifetime should be scaled
   according to the typical time it takes for a client to recover from a
   lost connection, which will depend on the operational environment but
   will often be on the order of several minutes.

   Once a transaction is complete, the server still keeps some resume
   state so that clients can recover from connection loss during the
   [RFC1047] synchronization gap.  Abusive clients can waste this space
   by failing to close connections cleanly with the QUIT command.  The
   server has to restrict the lifetime of this resume state in a similar
   way to partial transactions.  This resume state is relatively small,
   and is important for correctness (avoidance of duplicate messages) so
   its lifetime can be longer than that of partial messages.

   This memo does not provide a mechanism for clients to discover the
   lifetime of partial transactions and resume state on the server.







Finch                    Expires August 10, 2007               [Page 18]

Internet-Draft       SMTP transaction checkpointing        February 2007


4.2.  Transaction IDs

   If an attacker can guess a client's transaction IDs, it can perform a
   variety of attacks based on confusing a client about the state of a
   transaction or inserting unwanted data into a message.  These attacks
   are made much easier by the extensions defined in this memo than in
   un-extended SMTP.  We reduce the opportunity for attack by making
   transaction IDs unguessable and by tying them to the client's
   authenticated identity.

   Resumable transactions can be used to turn a truncation attack into a
   message modification attack.  If an attacker can cause the client's
   connection to break in the middle of a message, the attacker can
   resume the transaction and append any data to the end of the message.
   TCP truncation attacks are much easier than TCP modification attacks.
   Un-guessable transaction IDs prevent attackers that cannot sniff the
   client-server connection (e.g. because they are off the client-server
   path or because the connection has a security layer) from doing this.

   Unguessability can be achieved by including enough random data.
   [RFC4086] provides some guidelines on how to do this securely.

   An attacker that guesses a transaction ID before the client uses it
   could potentially append data to the start of the message.  In
   checkpoint/resume this is prevented by the client asserting that a
   transaction is new.  In checkpoint/restart this is prevented by
   requiring clients to detect this attack and reset the transaction.

   As well as these client-side protections, the server has to tie
   transaction IDs to the client's authenticated identity.  Even if an
   attacker can guess a transaction ID, it also has to break the
   client's authentication credentials in order to succeed.  If it
   doesn't manage to do both then the server will consider the client's
   and attacker's transactions to have different IDs.

   These extensions can be used with unauthenticated SMTP, in which case
   the server uses the client's IP address to identify it.  However this
   is usually not secure, especially with wireless or dial-up
   connections where it is relatively easy for an attacker to steal the
   client's IP address.

4.3.  Unnecessary bounces

   It is generally preferable for SMTP servers to reject messages during
   the SMTP transaction instead of accepting them and later generating a
   bounce message.  This allows message submission clients to present
   the error to the user immediately in a friendly manner.  It also
   reduces the problem of backscatter from spam with bogus return paths,



Finch                    Expires August 10, 2007               [Page 19]

Internet-Draft       SMTP transaction checkpointing        February 2007


   assuming that spam sending software has a partial SMTP implementation
   that doesn't emit bounces when its spam is rejected.

   The RESUME specification requires a server to honor its original
   replies to the RCPT commands.  If the status of a recipient address
   changes from good to bad between the start of a transaction and its
   completion, it would seem that the server is forced to accept the
   message then generate a bounce.  This should be a rare situation if
   we assume clients will prefer to resume transactions promptly.
   However it is possible for servers to reduce the problem by giving a
   negative reply to the end of the message data: in general a 450 reply
   if not all the recipients' statuses have changed, so that the client
   will retry the delivery later.






































Finch                    Expires August 10, 2007               [Page 20]

Internet-Draft       SMTP transaction checkpointing        February 2007


5.  References

5.1.  Normative references

   [RFC2033]  Myers, J., "Local Mail Transfer Protocol", RFC 2033,
              October 1996.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC2821]  Klensin, J., "Simple Mail Transfer Protocol", RFC 2821,
              April 2001.

   [RFC2822]  Resnick, P., "Internet Message Format", RFC 2822,
              April 2001.

   [RFC2920]  Freed, N., "SMTP Service Extension for Command
              Pipelining", STD 60, RFC 2920, September 2000.

   [RFC3030]  Vaudreuil, G., "SMTP Service Extensions for Transmission
              of Large and Binary MIME Messages", RFC 3030,
              December 2000.

   [RFC4234]  Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax
              Specifications: ABNF", RFC 4234, October 2005.

   [RFC4468]  Newman, C., "Message Submission BURL Extension", RFC 4468,
              May 2006.

5.2.  Informative references

   [RFC1047]  Partridge, C., "Duplicate messages and SMTP", RFC 1047,
              February 1988.

   [RFC1845]  Crocker, D. and N. Freed, "SMTP Service Extension for
              Checkpoint/Restart", RFC 1845, September 1995.

   [RFC1870]  Klensin, J., Freed, N., and K. Moore, "SMTP Service
              Extension for Message Size Declaration", STD 10, RFC 1870,
              November 1995.

   [RFC2554]  Myers, J., "SMTP Service Extension for Authentication",
              RFC 2554, March 1999.

   [RFC3207]  Hoffman, P., "SMTP Service Extension for Secure SMTP over
              Transport Layer Security", RFC 3207, February 2002.

   [RFC3461]  Moore, K., "Simple Mail Transfer Protocol (SMTP) Service



Finch                    Expires August 10, 2007               [Page 21]

Internet-Draft       SMTP transaction checkpointing        February 2007


              Extension for Delivery Status Notifications (DSNs)",
              RFC 3461, January 2003.

   [RFC4086]  Eastlake, D., Schiller, J., and S. Crocker, "Randomness
              Requirements for Security", BCP 106, RFC 4086, June 2005.

   [RFC4409]  Gellens, R. and J. Klensin, "Message Submission for Mail",
              RFC 4409, April 2006.











































Finch                    Expires August 10, 2007               [Page 22]

Internet-Draft       SMTP transaction checkpointing        February 2007


Appendix A.  Acknowledgments

   Thanks to Dave Crocker and Ned Freed for [RFC1845], from which this
   document borrows freely.  Thanks are also due to Dave Cridland,
   Randall Gellens, and Alexey Melnikov for encouragement, review, and
   comments.













































Finch                    Expires August 10, 2007               [Page 23]

Internet-Draft       SMTP transaction checkpointing        February 2007


Appendix B.  Changes since version -00

   o  LMTP support

   o  Clarified maximum value of <octet-offset>

   o  <octet-offset> doesn't count BURL commands

   o  Do not completely forbid clients from attempting to resume when
      there hasn't been a prior connection loss

   o  Clarify normal connection closure with QUIT from the client's
      point of view

   o  Resuming MAIL commands must be substantially the same as the
      transaction's original MAIL command

   o  Clarify ordering of RCPT commands in a resumed transaction

   o  Forbid mixing of DATA and BDAT

   o  Clarify failure replies to RESUME.





























Finch                    Expires August 10, 2007               [Page 24]

Internet-Draft       SMTP transaction checkpointing        February 2007


Appendix C.  Draft discussion venue

   Feedback about this draft should be emailed to the author, or to the
   IETF SMTP discussion mailing list, <ietf-smtp@imc.org>, or to the
   Lemonade working group mailing list, <lemonade@ietf.org>.














































Finch                    Expires August 10, 2007               [Page 25]

Internet-Draft       SMTP transaction checkpointing        February 2007


Author's Address

   Tony Finch
   University of Cambridge Computing Service
   New Museums Site
   Pembroke Street
   Cambridge  CB2 3QH
   ENGLAND

   Phone: +44 797 040 1426
   Email: dot@dotat.at
   URI:   http://dotat.at/







































Finch                    Expires August 10, 2007               [Page 26]

Internet-Draft       SMTP transaction checkpointing        February 2007


Full Copyright Statement

   Copyright (C) The IETF Trust (2007).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
   THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
   OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
   THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.


Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at
   ietf-ipr@ietf.org.


Acknowledgment

   Funding for the RFC Editor function is provided by the IETF
   Administrative Support Activity (IASA).





Finch                    Expires August 10, 2007               [Page 27]