INTERNET-DRAFT S. Varshavchik Expires Jan 26, 2000 Double Precision, Inc. Jul 26, 1999 Variable Envelope Return Path SMTP Extension draft-varshavchik-verp-smtpext-01.txt Status Of This Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. 0. Revision history 01 - added additional comments on DSNs not being very successful in automatically handling mailing list functions. Additional section added with comments regarding vacation autoresponders. 1. Abstract This document describes an extension to the SMTP service [1], called Variable Envelope Return Path (VERP). The VERP extension implements a way of automatically identifying undeliverable mail recipients, even when non-delivery reports originate from mail systems that do not implement delivery status notifications, as specified in [2] and [3]. 2. Introduction All E-mail software can expect to deal with undeliverable mail. [2] and [3] specify a machine-readable format for delivery status S. Varshavchik Expires Jan 26, 2000 [Page 1] VERP SMTP Extension S. Varshavchik Jul 26, 2000 notifications (DSNs, or non-delivery reports). DSNs allow undeliverable mail to be handled in a totally automatic fashion, without requiring manual intervention. For example, mailing list managers can automatically identify addresses that are no longer deliverable, and remove them from the mailing list. Although [2] and [3] are now widely implemented, there are still many systems that do not use them. This makes it impractical to completely rely on DSNs for automatic mailing list management. Undeliverable addresses accumulate quickly even from a very small percentage of non-DSN systems. This results in a non-trivial amount of manual work to identify undeliverable addresses and purge them from the mailing list. Mailing list software began to use VERPs (the acronym stands for Variable Envelope Return Path) after DSNs were found to be impractical for totally automatic mailing list management. VERPs are an alternative way to handle non-delivery notices. The advantage of VERPs is that they can be made to work automatically, even when non- delivery notices are not in the format specified by [2]. Unfortunately, VERPs require much more bandwidth and network resources than DSNs because VERPs cannot be used to send one copy of a mailing list message addressed to all the recipients in the same E- mail domain. This SMTP service extension allows E-mail software to send a single VERP message to all addresses in the same mail domain, for as long as mail servers, which relay the message, support the VERP SMTP extension. The VERP message may be eventually relayed to a mail server that does not support this extension. Separate messages - with variable envelope return paths - will be sent when this happens. So the worst case scenario results in the same situation where traditional VERPs are used right from the start. The best case scenario results in significant savings of network resources and bandwidth, from eliminating hundreds (or more) copies of the same message. Essentially, the VERP extension postpones the generation of multiple messages with different return paths as much as possible, until it is absolutely required. 2.1 VERP overview The traditional VERP message encodes the recipient address as a S. Varshavchik Expires Jan 26, 2000 [Page 2] VERP SMTP Extension S. Varshavchik Jul 26, 2000 portion of the return address. When undeliverable mail comes back, the mail software decodes the return address (now the recipient address) and obtains the address responsible for the non-delivery notice. For example: mail sent by a mailing list manager to the address carries a return address of . The mailing list software at domain.com handles all mail with the local portion of the address starting with "mlist-return-". If a non-delivery notice is generated because the address is not deliverable, the mailing list software takes the address where the non-delivery report was sent, retrieves the remaining portion of the local address, "john=example.org", and determines that the undeliverable address was . This does not rely on RFC 1894, and will work for all non-delivery notices. 3. Framework for the VERP SMTP transport extension This SMTP transport extension [1] is laid out as follows. (1) The name of the SMTP transport extension defined here is Variable Envelope Return Path. (2) The EHLO keyword associated with this extension is VERP. (3) The VERP EHLO keyword takes no parameters. (4) One optional ESMTP keyword VERP is associated with the MAIL FROM command. This parameter takes no values. (5) No additional ESMTP verbs are defined by this extension. (6) The next section specifies how support for this extension affects the behavior of a server and client SMTP. 4. The VERP SMTP extension When a VERP keyword is present in the MAIL FROM command, [4], some additional restrictions are imposed on the RFC 822 address [5], specified by that MAIL FROM command, and on all RFC 822 addresses in the subsequent RCPT TO commands that refer to the same message (that is, until the next DATA, RSET, or QUIT command). The term "VERP message" refers to any E-mail message whose MAIL FROM command includes the VERP keyword. The term "VERP-compliant server" refers to any E-mail server that supports the Variable Envelope Return Path SMTP extension. When a VERP keyword is present in the MAIL FROM S. Varshavchik Expires Jan 26, 2000 [Page 3] VERP SMTP Extension S. Varshavchik Jul 26, 2000 command: (1) The address specified by the MAIL FROM verb MUST contain at least one @ character. (2) The address in every RCPT TO verb referring to the same message MUST contain at least one @ character. (3) The domain portion of the address in the MAIL FROM and RCPT TO verbs MUST be compliant with the definition of in [6]. That is, it MUST contain only letters, digits, hyphens, and periods. The domain portion of the address is the one that follows the last @ character, 4.1 Delivery failures When a VERP-compliant server is unable to deliver a VERP message to one or more recipients, the VERP server MUST do one of the following: 1) Return an RFC 1891 delivery status notification to the return address, or: 2) Transmit a separate non-delivery notice for each failed recipient. The return address for each non-delivery notice MUST be the address that's formed by applying the procedure described in section 7 of this document to the return address of the message and the failed recipient's address. If more than one recipient was undeliverable a separate notice MUST be sent for each undeliverable address. 5. Final delivery Section 4.3.1 of [5] specifies that the mail server performing final delivery of a message will generate a Return-Path: header containing the return address of the message. This return address MUST be formed by applying the procedure described in section 7 of this document to the return address and the recipient's address. This also applies if the mail server invokes some other external process to handle final delivery, instead of placing the message into the recipient's mailbox. In all cases, the return address specified by the mail server to any external environment or process MUST be derived by applying the procedure in section 7 to the return address and the recipient's address. 6. Relaying S. Varshavchik Expires Jan 26, 2000 [Page 4] VERP SMTP Extension S. Varshavchik Jul 26, 2000 When a VERP-compliant server determines that a recipient of a VERP message is not a local mailbox, and the message must be relayed to another server, the VERP-compliant server MUST: (1) If the VERP-compliant server's local policies require the return and/or recipient addresses MUST comply with the restrictions specified in section 4 of this document. (2) If the VERP-compliant server determines that the remote server is also a VERP compliant server, the VERP keyword MUST be included in the MAIL FROM command used to relay the VERP message to the remote server. (3) If the remote server is not a VERP compliant server, The VERP compliant server SHOULD send a separate copy of the message for every recipient. The return address of each copy of the message MUST be formed by applying the procedure described in section 7 of this document to the original return address, and the address of each individual recipient. Although the message SHOULD NOT be returned as undeliverable, if it is then the rules defined in section 4.1 MUST be applied. These rules also apply if the SMTP-compliant server determines that the VERP message must be forwarded via some other protocol to a non-SMTP gateway, unless the non-SMTP protocol has equivalent features that are completely identical in function to Variable Envelope Return Path SMTP service extension (including any translations of E-mail addresses to and from the non-RFC822 format). 7. Variable envelope return path encoding This encoding method starts with a return address and one recipient address. As mentioned previously, both addresses MUST be valid RFC822 addresses, [5], and MUST contain at least one @ character. The portion of each address following the last @ character MUST be compliant with [6]. Let "sdomain" represent the portion of the return address that follows the last @ character. Let "slocal" represent the portion of the return address that precedes the last @ character. Let "rdomain" represent the portion of the recipient address that follows the last @ character. Let "rlocal" represent the portion of the recipient address that S. Varshavchik Expires Jan 26, 2000 [Page 5] VERP SMTP Extension S. Varshavchik Jul 26, 2000 precedes the last @ character. To encode the recipient address within the envelope sender address, create an address of the following form: slocal-encodedrlocal=rdomain@sdomain Where "encodedrlocal" is formed by taking rlocal and encoding it as follows: 1) Each @, :, %, !, and + character in rlocal is replaced by a single '+' character followed by two uppercase hexadecimal characters whose value is the ASCII code of the replaced character. 2) All other characters are unchanged. Other characters MAY, but SHOULD NOT be also encoded in the same fashion. This can be represented using BNF as follows: encodedverp: slocal "-" encodedrlocal "=" rdomain "@" sdomain encodedrlocal: * (char-literal / char-encoded ) char-literal: any character valid in an RFC821 address [4], except @, :, %, !, and + char-encoded: "+" hexdigit hexdigit hexdigit: ("0" / "1" / "2" / "3" / "4" / "5" / "6" / "7" / "8" / "9" / "A" / "B" / "C" / "D" / "E" / "F" ) 8. Variable envelope return path decoding Non-delivery notices for VERP messages will be sent to either the original address, , or to the VERP-encoded address, . Messages sent to will be RFC 1891-compliant delivery status notifications. These messages will be machine-readable, and the mail software will be able to identify failed addresses from the RFC 1891 delivery report. Non-delivery notices will also be sent to the VERP-encoded address, and the mail software will be able to reconstruct the failed address from the VERP-encoded address by simply reversing the steps used in encoding: S. Varshavchik Expires Jan 26, 2000 [Page 6] VERP SMTP Extension S. Varshavchik Jul 26, 2000 1) Extracting encodedrlocal and rdomain from the recipient address. There will be at least one = character in the encoded portion of the return address. encodedrlocal is everything up to the last = character. Everything following the last = character is rdomain. 2) Replacing all occurrences of "+" followed by two hexadecimal digits in encodedrlocal with the equivalent ASCII character. 3) Using the decoded rlocal, @, then rdomain. 9. Examples Suppose that a VERP-compliant server named "example.com" receives a message via the following SMTP conversation (for brevity, non- relevant headers have been omitted): 250 example.com ESMTP EHLO domain.com 250-example.com ESMTP 250-SIZE 250-DSN 250-VERP 250 HELP MAIL FROM: VERP SIZE=100 250 Ok RCPT TO: 250 Ok RCPT TO: 250 Ok RCPT TO: 250 Ok RCPT TO: 250 Ok RCPT TO: 250 Ok DATA 250 Ok From: "John" Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST) Subject: Meeting canceled. Today's 2pm meeting has been rescheduled for tomorrow, 9am, due to a scheduling conflict. . The message is delivered to the local mailbox for . The message looks like this: S. Varshavchik Expires Jan 26, 2000 [Page 7] VERP SMTP Extension S. Varshavchik Jul 26, 2000 Return-Path: From: "John" Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST) Subject: Meeting canceled. Today's 2pm meeting has been rescheduled for tomorrow, 9am, due to a scheduling conflict. The VERP-compliant server at example.com connects to the mail server for old.example.com. old.example.com does not support the Variable Envelope Return Path extension. Therefore, old.example.com receives two messages. The SMTP conversation for the first message is as follows: 250 old.example.com ESMTP EHLO example.com 250-old.example.com ESMTP 250-SIZE 250-DSN 250 HELP MAIL FROM: 250 Ok RCPT TO: 250 Ok DATA 250 Ok From: "John" Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST) Subject: Meeting canceled. Today's 2pm meeting has been rescheduled for tomorrow, 9am, due to a scheduling conflict. . The SMTP conversation for the second message is as follows: MAIL FROM: 250 Ok RCPT TO: 250 Ok DATA 250 Ok From: "John" Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST) Subject: Meeting canceled. Today's 2pm meeting has been rescheduled for tomorrow, 9am, due to a scheduling conflict. S. Varshavchik Expires Jan 26, 2000 [Page 8] VERP SMTP Extension S. Varshavchik Jul 26, 2000 . example.com connects to new.example.com and determines that new.example.com runs a modern ESMTP server that supports the VERP keyword. The SMTP conversation then goes like this: 250 new.example.com ESMTP EHLO example.com 250-new.example.com ESMTP 250-SIZE 250-DSN 250-VERP 250 HELP MAIL FROM: VERP SIZE=100 250 Ok RCPT TO: 250 Ok RCPT TO: 250 Ok DATA 250 Ok From: "John" Date: Thu, 16 Jan 1997 14:49:31 -0500 (EST) Subject: Meeting canceled. Today's 2pm meeting has been rescheduled for tomorrow, 9am, due to a scheduling conflict. . 10. Security concerns All the usual security considerations applicable to SMTP are also applicable to this extension. Relay of VERP messages to non-VERP servers requires a single message with many recipients to be exploded into many messages with one recipient. In all cases, however, there will never be any additional overhead beyond the resources that are required when VERPs are manually implemented by the mail sender, instead of the VERP SMTP extension. Mail systems which support the VERP extension SHOULD have adequate security measures, including blocks against unauthorized access and relaying. 10.1 Vacation programs, and other autoresponders "Vacation" type autoresponders are often used in practice. A vacation autoresponder is a program that automatically replies to every message, informing the sender that the recipient is on S. Varshavchik Expires Jan 26, 2000 [Page 9] VERP SMTP Extension S. Varshavchik Jul 26, 2000 vacation, or is generally unavailable at this time. Vacation autoresponders MUST NOT generate autoresponses to mailing list messages, but people often forget to do set them up to do so. Because autoresponses are sent to the same address that's used to receive non-delivery reports, malfunctioning autoresponders result in the recipient being removed from mailing lists. Advanced autoresponders send automatic replies in the format specified by [2], as a "delayed" notification. DSN-aware software will not remove addresses from mailing lists due to delayed notifications. Section 5 of this document specifies that the mail server MUST replace the original return address with a VERP-modified address when delivering the message to a mailbox or an external process. Therefore it is possible that RFC 1891 reports may also be sent to a VERP-encoded address, as specified by sections 5 and 7 of this document. Mail software SHOULD ignore any RFC 1891 "delayed" or "success" reports that sent to a VERP-encoded address. If it is a "failed" report, note that the VERP address will be more reliable than the address specified in the report itself. S. Varshavchik Expires Jan 26, 2000 [Page 10] VERP SMTP Extension S. Varshavchik Jul 26, 2000 11. References [1] Klensin, J., Freed, N., Rose, M., Stefferud, E., Crocker, D. "SMTP Service Extensions", RFC 1425, United Nations University, Innosoft International, Inc., Dover Beach Consulting, Inc., Network Management Associates, Inc., The Branch Office, February 1993 [2] Moore, K., and G. Vaudreuil, "An Extensible Message Format for Delivery Status Notifications", RFC 1894, University of Tennessee, Octel Network Services, January 1996. [3] Moore, K. "SMTP Service Extension for Delivery Status Notifications", RFC 1891, University of Tennessee, January 1996. [4] Postel, J., "Simple Mail Transfer Protocol", STD 10, RFC 821, USC/Information Sciences Institute, August 1982. [5] Crocker, D., "Standard for the Format of ARPA Internet Text Messages", STD 11, RFC 822, UDEL, August 1982. [6] Mockapetris, P., "Domain Names - Implementation and Specification", RFC 1035, ISI, November 1987 12. Author's address Sam Varshavchik Double Precision, Inc. PO Box 668 Greenwood Lake, NY 10925 S. Varshavchik Expires Jan 26, 2000 [Page 11]