Internet-Draft Matt Mathis John Heffner PSC Kevin Lahey Freelance 14 Feb, 2004 Path MTU Discovery draft-ietf-pmtud-method-01.txt Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Abstract This document describes a robust new method for Path MTU Discovery that relies on TCP or other Packetization Layer to probe an Internet path with progressively larger packets. This method is described as an extension to RFC 1191 and RFC 1981, which specify ICMP based Path MTU Discovery for IP versions 4 and 6. This document does not define a protocol, but rather a method to use features of existing protocols to discover the path MTU. The general strategy of the new algorithm is to start with a small MTU and probe upward, testing successively larger MTUs by probing Mathis, et al [Page 1] Internet-Draft Expires Sept 2004 14 Feb, 2004 with single packets. If the probe is successfully delivered, then the MTU is raised. If the probe is lost, it is treated as an MTU limitation and not as a congestion signal. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . 3 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . 3 3. Overview . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.1. General Method . . . . . . . . . . . . . . . . . . . . . 8 3.2. Generating Probes . . . . . . . . . . . . . . . . . . . . 9 3.3. Normal sequence of events to raise the MTU . . . . . . . 10 3.4. Processing MTU Indications . . . . . . . . . . . . . . . 11 3.4.1. Processing Packet Too Big Messages . . . . . . . . . . 11 3.4.2. Packetization Layer retransmits lost packets . . . . . 11 3.4.3. Packetization Layer Retransmission Timeout . . . . . . 13 3.5. Probing Intervals . . . . . . . . . . . . . . . . . . . . 14 3.6. Interoperation with prior algorithms . . . . . . . . . . 15 4. Requirements . . . . . . . . . . . . . . . . . . . . . . . 15 5. Implementation Issues . . . . . . . . . . . . . . . . . . . 16 5.1. Layering and Accounting for Header Sizes. . . . . . . . . 17 5.2. Storing PMTU information . . . . . . . . . . . . . . . . 18 5.3. Host fragmentation . . . . . . . . . . . . . . . . . . . 19 5.4. Multicast . . . . . . . . . . . . . . . . . . . . . . . . 19 5.5. Path MTU Search Strategy . . . . . . . . . . . . . . . . 20 5.5.1. Search . . . . . . . . . . . . . . . . . . . . . . . . 20 5.5.2. Monitor . . . . . . . . . . . . . . . . . . . . . . . . 21 5.5.3. Suspend . . . . . . . . . . . . . . . . . . . . . . . . 21 5.6. Implementation issues for specific Packetization Layers . 21 5.6.1. Probing method using TCP . . . . . . . . . . . . . . . 21 5.6.2. Probing method using SCTP . . . . . . . . . . . . . . . 22 5.6.3. Issues for tunnels . . . . . . . . . . . . . . . . . . 23 5.6.4. Issues for other transport protocols . . . . . . . . . 23 5.7. Diagnostic tools . . . . . . . . . . . . . . . . . . . . 23 5.8. Management interface . . . . . . . . . . . . . . . . . . 23 6. Normative references . . . . . . . . . . . . . . . . . . . 24 7. Informative references . . . . . . . . . . . . . . . . . . 24 8. Security considerations . . . . . . . . . . . . . . . . . . 24 9. IANA considerations . . . . . . . . . . . . . . . . . . . . 25 10. Contributors . . . . . . . . . . . . . . . . . . . . . . . 25 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . 25 12. Authors' addresses . . . . . . . . . . . . . . . . . . . . 25 13. Intellectual Property . . . . . . . . . . . . . . . . . . 25 14. Full copyright statement . . . . . . . . . . . . . . . . . 26 Mathis, et al [Page 2] Internet-Draft Expires Sept 2004 14 Feb, 2004 1. Introduction This document describes a method for Packetization Layer Path MTU Discovery (PLPMTUD) which is an extension to existing Path MTU discovery methods. The proper MTU is determined by starting with small packets and probing with successively larger packets. The bulk of the algorithm is implemented above IP, in the transport layer (e.g. TCP) or other "Packetization Protocol" that is responsible for determining packet boundaries. This document draws heavily RFC1191 and RFC1981 for terminology, ideas and some of the text. The methods described in this document apply both IPv4 and IPv6, and to many transport protocols such as TCP. This document does not define a protocol, but rather a method to use features of existing protocols to discover the path MTU. It does not require cooperation from the lower layers (except that they are consistent about what packet sizes are acceptable) or the far node. Variants in implementations will not cause interoperability problems. The methods described in this document are carefully designed to maximize robustness in the presence of less than ideal implementations of other protocols or Internet components. For sake of clarity we uniformly prefer TCP and IPv6 terminology. In the terminology section we also present the analogous IPv4 terms and concepts for the IPv6 terminology. In a few situations we describe specific details that are different between IPv4 and IPv6. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119]. 2. Terminology IP - Either IPv4 [IPv4-SPEC] or IPv6 [IPv6-SPEC]. node - A device that implements IP. router - A node that forwards IP packets not explicitly addressed to itself. host - Any node that is not a router. Mathis, et al [Page 3] Internet-Draft Expires Sept 2004 14 Feb, 2004 upper layer - A protocol layer immediately above IP. Examples are transport protocols such as TCP and UDP, control protocols such as ICMP, routing protocols such as OSPF, and Internet or lower-layer protocols being "tunneled" over (i.e., encapsulated in) IP such as IPX, AppleTalk, IP itself. link - A communication facility or medium over which nodes can communicate at the link layer, i.e., the layer immediately below IP. Examples are Ethernets (simple or bridged); PPP links; X.25, Frame Relay, or ATM networks; and Internet (or higher) layer "tunnels", such as tunnels over IPv4 or IPv6. In some earlier documents the term "lower layer" was used for this concept. interface - A node's attachment to a link. address - An IP-layer identifier for an interface or a set of interfaces. packet - An IP header plus payload. MTU - Maximum Transmission Unit, the size in bytes of the largest IP packet, including the IP header and payload, that can be transmitted on a link or path. Note that this could more properly be called the IP MTU, to be consistent with how other standards organizations use the acronym MTU. link MTU - The Maximum Transmission Unit, i.e., maximum IP packet size in octets, that can be conveyed in one piece over a link. Beware that this definition differers from the definition used by other standards organizations. For IETF documents, link MTU is uniformly defined as the IP MTU over the link. This includes the IP header, but excludes link layer headers and other framing which is not part of IP or the IP payload. Other standards organizations generally define link MTU to include the link layer headers. path - The set of links traversed by a packet between a source node and a destination node path MTU - The minimum link MTU of all the links in a path between a source node and a destination node. Mathis, et al [Page 4] Internet-Draft Expires Sept 2004 14 Feb, 2004 PMTU - Path MTU classical PMTU discovery, - Process described in RFC 1191 and RFC 1981, in which nodes rely on ICMP messages to learn the MTU of a path. PL, packetization layer - The layer of the network stack which segments data into packets. PLPMTUD - Packetization Layer Path MTU Discovers, the method described in this document, which is an extension to classical PMTU discovery. Packet Too Big message - An ICMP message reporting that an IP packet is too large to forward. This is the IPv6 term that corresponds to the IPv4 "ICMP Can't fragment" message. flow - A context in which MTU discovery is applied. This is naturally an instance of the packetization protocol, e.g. one side of a TCP connection. MPS - The maximum IP payload size available over a specific path. This is typically the path MTU minus the IP header As an example, this is the maximum TCP packet size, including TCP payload and headers but not including IP headers. This has also been called the "L3 MTU". MSS - The TCP Maximum Segment Size, the maximum payload size available to the TCP layer. This is typically the path MPS minus the size of the TCP headers. probe packet- A packet which is being used to test for a larger MTU. probe size - The size of a packet being used to probe for a larger MTU. successful probe - The probe packet was delivered through the network. inconclusive probe - The probe packet was not delivered, but there were other lost packets close enough to the probe where can not presume that the probe was lost due to MTU. By implication the probe might have been lost due to something other than MTU (such congestion), so the results are inconclusive. Inconclusive probes are generally repeated at the same probe size, after a suitable delay. Mathis, et al [Page 5] Internet-Draft Expires Sept 2004 14 Feb, 2004 failed probe - The probe packet was not delivered and there were no other lost packets close to the probe. This is taken as an indication that the probe was larger than the path MTU, and future probes should generally be for at smaller sizes. errored probe - There were losses or timeouts during the verification phase which suggest a potentially disruptive failure or network condition. These are generally retried only after substantially longer intervals. @@@ not used probe gap - The expected missing payload data that will need to be retransmitted if the probe is not delivered. probe phase - The interval (time or protocol events) between when a probe is sent, and when it is determined that the the probe succeeded, failed or was inconclusive verification phase - An additional interval during which the new path MTU is considered provisional. Packet losses or timeouts are treated as an indication that there may be a problem with the provisional MTU. Transition phase - The interval between the probe phase and the verification phase, during which packets using the new MTU propagate to the far node and the acknowledgment propagates back. full stop timeout - a timeout where the none of the packets transmitted after some specific event at the sender (e.g. entering the probe or verification phase) is acknowledged by the receiver. This is taken as an indication that the MTU change caused some failure in the network. search strategy - the heuristics used to choose successive probe sizes to converge to the proper path MTU, as described in section 5.5. 3. Overview This document describes a method for TCP or other packetization Mathis, et al [Page 6] Internet-Draft Expires Sept 2004 14 Feb, 2004 protocols to dynamically discover the MTU of a path without relying on explicit signals from the network. These procedures are applicable to TCP and other transport- or application-level packetization protocols in which the receiver always reports to the sender complete information about which packets were lost in the network. The general strategy of the new procedure is for the packetization layer to find the proper MTU by probing with progressively larger packets, without disrupting its normal protocol operation. If a probe packet is successfully delivered, then the path MTU is provisionally raised. If there are no additional losses during the subsequent verification phase, then the path MTU is confirmed (verified) to be at least as large as the provisional MTU. PLPMTUD can then probe again with an even larger MTU, according to MTU search strategy described in section 5.5. The verification phase is used to detect situations where raising the MTU greatly raises the packet loss rate. For example this might happen if some link is striped across multiple physical channels and the stripes have inconsistent MTUs. A conservative implementation of PLPMTUD would use a full round trip time for the verification phase. In this case each time PLPMTUD raises the MTU it takes three full round trip times to do so. It takes one round trip for the probe phase, during which the probe propagates to the far node and an acknowledgment is returned. The second round trip is the transitional phase, during which data packets using the provisional MTU propagate to the far node and are acknowledged. During he third and final round trip time, it is verified that raising the MTU does not cause excessive loss. The isolated loss of a probe packet (with or without a Packet Too Big message) is treated as an indication of an MTU limit, and not as a congestion indicator. In this case alone, the packetization protocol is permitted to retransmit the probe gap without adjusting the congestion window. If there is a timeout or additional lost packets during any of the three phases, the loss is treated as a congestion indication as well as a indication of some sort of failure of the PLPMTUD process. The congestion indication is treated like any other congestion indication: window or rate adjustments are mandatory per the relevant congestions control standards [Congestion]. Probing can resume with some new probe size after a delay which is determined by the nature of the indicated failure. The most likely (and least serious) PLPMTUD failure is the link Mathis, et al [Page 7] Internet-Draft Expires Sept 2004 14 Feb, 2004 experiencing legitimate congestion related losses at about the same time as the probe. In this case, it is appropriate to retry the probe (with the same probe size) as soon as the packetization layer has fully adapted to the congestion and recovered from the losses. In other cases, additional losses or timeouts indicate problems with the link or packetization layer, and that probes may be disruptive. In these situations it is desirable to use progressively longer delays depending on the severity of the failure and if it is repeated. PLPMTUD can optionally process Packet Too Big messages to select the provisional MTU for faster convergence in exchange for a slight decrease in robustness. Processing malicious or erroneous Packet Too Big messages can cause PLPMTUD to arrive at the incorrect MTU for a path, which is likely to reduce protocol performance. The document describes three options for processing Packet Too Big messages: completely ignore them, only accept them in response to probes or accept all Packet Too Big messages (fully implementing classic PMTUD within PLPMTUD). Theses are further described in section 3.8. Relatively few details of this procedure affect interoperability with other standards or Internet protocols. These details are specified in RFC2119 standards language in the requirements section. The vast majority of the implementation details are recommendations based on experiences with earlier versions of path MTU discovery. These are motivated by a desire to maximize robustness of PLPMTUD in the presence of less than ideal implementations as they exist in the field. 3.1. General Method Most of the difficulty in implementing PLPMTUD arises because it needs to be implemented in several different places within a single node. In general each packetization protocol needs to have it's own implementation of PLPMTUD. Furthermore, the natural mechanism to share path MTU information between concurrent or subsequent connections over the same path is a path information cache in the IP layer. The various packetization protocols need to have the means to access and update the shared cache in the IP layer. Rather than prescribing implementation details this memo describes PLPMTUD in terms of its primary subsystems, without fully describing how they are integrated into a complete implementation. The subsystems are: generating probes, processing probe responses, the search strategy and, the supporting infrastructure (including the Mathis, et al [Page 8] Internet-Draft Expires Sept 2004 14 Feb, 2004 path cache). The first two are introduced in this section and are subject to the requirements specified in the following section. The probe strategy and issues related to the support infrastructure and cache are covered in the implementation section. 3.2. Generating Probes A new candidate MTU is tested by sending one "probe packet", which is larger than the current MTU. In this section we present a couple of possible ways to alter packetization layers to generate probe packets. The different techniques incur different overheads in three areas: difficulty in generating the probe packet (in terms of packetization layer implementation complexity and computational overhead) possible additional network capacity consumed by the probes and the overhead of recovering from failed probes (both network and protocol overhead). For example a protocol such as SCTP might be extended to allow padding with dummy data inside the SCTP packets. This greatly simplify the implementation because the probing can be performed without participation from the application and if the probe fails, the missing data (the "probe gap") is assured to fit within the current MTU when it is retransmitted. However, the padding does consume network capacity without carrying any useful payload. This technique does not work for TCP, because there is not a separate length field or other mechanism to differentiate between padding and real payload data. With TCP the natural approach is to send additional payload data in an over-sized segment. There are several variants which have different tradeoffs. In one method, after a TCP probe segment has been sent the subsequent segment(s) may be sent as though the probe segment was not over- sized. Thus if the probe segment is lost, it will leave a gap in the sequence space that is exactly the correct size to be filled by one segment at the current MTU. Since this method generates overlapping data, it will cause duplicate acknowledgments if the probe is successfully delivered. The sender must be capable of ignoring these expected duplicate acknowledgments in a manner which will not cause unnecessary retransmission or congestion window reduction. In the second method, after a TCP probe segment has been sent, subsequent TCP segments are sent in a non-overlapping manner. If the probe segment is lost, it will leave a gap which will require retransmission of multiple segments to fill. This method has lower overhead for successful probes, but it requires more complexity in the retransmit logic to correctly retransmit the missing data with multiple segments that fit into the old MTU, while properly Mathis, et al [Page 9] Internet-Draft Expires Sept 2004 14 Feb, 2004 suppressing the congestion adjustments for this one situation and no others. Under all conditions it is important that the packetization layer sends additional data (packets) after the probe, such that Fast Retransmit or equivalent algorithms in the packetization layer will trigger the retransmission of the probe if it is lost in the network. 3.3. Normal sequence of events to raise the MTU If the probe size is smaller than the path MTU and there are no other losses, the normal sequence of events will be: Step 1) The probe is sent, followed by more packets at the current MTU. By definition PLPMTUD enters the probe phase. The probe propagates through the network and the far node acknowledges it (or possibly latter data, if ACKs are cumulative and delayed ACK is in effect). Step 2) The ACK for the probe reaches the data sender. By definition, this ends the probe phase. Step 3) The packetization layer provisionally raises the MTU to the probe size. PLPMTUD enters the transitional phase when it starts sending data using the provisional MTU. Note that implementations that use packet counts for congestion accounting (e.g. keep cwnd in units of packets) must rescale their congestion accounting such that raising the MTU does not raise the total congestion window or data rate. If the implementation packetizes the data at the application API, it may transmit already queued data at the current MTU before raising the MTU. In this case this data is not part of either the probing or transition phases, because all of the packets in flight fit within the current MTU. Step 4) Once the first packet of the transitional phase is acknowledged, PLPMTUD enters the verification phase to determine if raising the MTU causes packet loss. In principle the verification phase can be of arbitrary duration, however at this time we are recommending one full window of data (i.e one full round trip time). Step 5) Once there has been sufficient data delivered and acknowledged in the provisional MTU is considered verified and the path MTU is updated. PLPMTUD can then probe for an even larger MTU, as described in the searching strategy in section 5.5. Mathis, et al [Page 10] Internet-Draft Expires Sept 2004 14 Feb, 2004 3.4. Processing MTU Indications Other events described below are treated as exceptions and alter or cancel some of the steps above. 3.4.1. Processing Packet Too Big Messages Classical PMTU discovery specifies the generation of Packet Too Big Messages if an over-sized packet (e.g. a probe) encounters a link that has a smaller MTU. Since these messages can not be authenticated they introduce a number of well documented denial of service attacks against classical PMTUD [DOS]. In PLPMTUD these messages are not required for correct operation, so in principle they can be summarily ignored at the expense of slower convergence to the proper MTU. However we believe that a slightly better compromise is to process Packet too big messages in two specific contexts: in conjunction with a PLPMTUD probe or a retransmission timeout in the packetization layer (indication a re- route to a link with a smaller MTU). Every Packet Too Big Message should be subjected to the following checks: o If globally forbidden then discard the message. o If forbidden by the application then discard the message. o If this path has been tagged "bogus ICMP messages" then discard the message. o If the reported MTU fails consistency checks then set "bogus ICMP messages" flag for this path and discards the message. These consistency checks include: unrecognized or unparseable enclosed header, reported MTU is larger than the size indicated by the enclosed header or larger than the current MTU, provisional MTU or probe size as appropriate. o If the Packet Too Big Message is acceptable under all of these checks, save the "ICMP MTU", pending another packetization layer protocol event. 3.4.2. Packetization Layer retransmits lost packets Mathis, et al [Page 11] Internet-Draft Expires Sept 2004 14 Feb, 2004 Each packetization protocol has it's own mechanism to detect lost packets and request the retransmission of missing data. The primary signals used by the packetization layer are these protocol specific loss indications. In all cases the packetization layer is responsible for retransmitting the lost data and notifying PLPMTUD that there was a loss. o If the probe itself was lost, and there were no other losses during the probe phase (The RTT between when the probe was sent and the loss detected) than it is taken as an indication that the path MTU is smaller than the probe size. In this situation alone the Packetization Layer is permitted to retransmit the missing data (the "probe gap") without adjusting its congestion window or data transmission rate. If an accepted Packet Too Big Message was received after the probe was sent, and it passes the additional checks that the ICMP MTU is greater than the current MTU, then set the provisional MTU to the ICMP MTU and proceed from step 3 in section 3.3 above. If there was not a accepted Packet Too Big Message, then the indicated event is a "probe failure", which can be retried with a smaller probe size after a suitable delay for a probe_failure_event. See section 3.7 for more complete descriptions of failure events. o If there are losses during the probe phase and the probe was not lost, then the probe was successful. However, since additional loses have the potential to spoil the verification phase, it is important that PLPMTUD not progress into the transition phase (step 3 above) until after the Packetization Layer has fully recovered from the losses and completed the congestion window (or rate) adjustment. o If there are losses during the probe phase and the probe was also lost the outcome depends on the presence an ICMP MTU set by an acceptable Packet Too Big Message. If there was an accepted Packet Too Big Message received since the probe was sent, and it passes the additional checks that the ICMP MTU is greater than the current MTU, then set the provisional MTU to the ICMP MTU, and once the Packetization Layer has fully recovered from the losses and completed the congestion window (or rate) adjustment then proceed to step 3 in section 3.3 above. If there was not an accepted Packet Too big Message, then the probe is inconclusive because the lost probe might have been caused by congestion. The probe can be retried after a suitable delay for a inconclusive_probe_event. Mathis, et al [Page 12] Internet-Draft Expires Sept 2004 14 Feb, 2004 o Losses during the transition phase do not receive special treatment. o Losses during the verification phase are taken as a indication that the path may have a non-uniform MTU or some other problems such that raising the MTU substantially raises the loss rate. If so, this is potentially a very serious problem, so the provisional MTU is considered to have failed the verification phase and the path MTU is set back to the previously verified MTU (the previously the current MTU). Packet loss during the verification phase might also be due to coincidental congestion on the path, unrelated to the probe, so it would seem to be desirable that PLPMTUD re-probes the path. The risk is that this effectively raises the tolerated loss threshold because even though raising the MTU causes additional loss, there is a statistical chance that repeated attempts to verify a the new MTU may yield as false pass. The compromise is to re-probe once with the same probe size (after delay inconclusive_probe_event), and if this also fails, then the probe may not be retried until after a suitable delay for a verification_fail_event, which exponentially increases on each successive failure. Losses during the verification phase may indicate that a Packet Too Big Message reported the incorrect ICMP MTU, so if the provisional MTU was updated from the ICMP MTU (which was from an earlier Packet Too Big Message), set the "bogus ICMP message" flag for this path. This will prevent PLPMTUD from processing further "Packet Too Big Messages" for this path. If the provisional MTU was correct, the re-probe above will correctly use it. If it was not correct, then by definition the path reported at least one incorrect "Packet Too Big Message", and should not process any additional messages. 3.4.3. Packetization Layer Retransmission Timeout If there is a retransmission timeout during the probe or verification phase it may be an indication of a serious problem with the path or the Packetization Layer. We first define the notion of a "full stop timeout" to be a timeout where the none of the packets transmitted after some specific event at the sender (e.g. entering the probe or verification phase) is acknowledged by the receiver. If a retransmission timeout is not full stop it is processed above as loss, except using longer delays before re-probing. (probe_timeout_event, verification_timeout_event) If there is a full stop timeout following a probe then it is taken as an indication that probing may be disruptive to either the network or the far node (e.g. it triggered a bug halt due to a buffer overrun, Mathis, et al [Page 13] Internet-Draft Expires Sept 2004 14 Feb, 2004 etc). The probe should not be retried until after a long delay, for probe_stop_event. Not that this makes it particularly important that probes are only sent when the sender can send sufficient additional data to assure the correct operation of Fast Retransmit and similar algorithms in the Packetization Layer. If there is a full stop timeout when the path MTU is raised to the provisional MTU and the provisional MTU was updated from the ICMP MTU, then it is assumed that the MTU reported in the Packet Too Big Message was incorrect. Set the "bogus ICMP message" flag for this path and re-probe with a smaller probe size after a suitable delay for an ICMP_fail_event. If there is a full stop timeout when the path MTU is raised to the provisional MTU and the provisional MTU is the same as the probe size (because the probe packet was not lost), then something truly unexpected happened. It is possible that the timeout is unrelated to the probe, so in this situation re-probe with a smaller probe size after a suitable delay for an verification_stop_event. 3.5. Probing Intervals Section 3.4 above describes a number of probe failure events. In all cases the basic response is the same: to wait some time interval (dependent on the specific event and possibly the history) and then to probe again. For events that are "inconclusive", is is generally appropriate to re-probe with the same probe size. For events that are identified as "failed probes" is is generally appropriate to re- probe with a smaller probe size. The search strategy described in section 5.5 is used to select probe sizes. Many of the intervals below are specified in terms of elapsed round trips relative to the current congestion window. This is because TCP and other Packetization Layer protocols tend to exhibit periodic loses which cause periodic variations of the congestion window and possibly the data rate. It is preferable that the PLPMTUD probes are scheduled near the low point of these cycles. In order from least to most serious: inconclusive_probe_event - Packet loss near the lost probe marked the result ambiguous. Since the loss of non-probe causes a window (or data rate) reduction, it is desirable to schedule the re-probe (of the same probe size) a few round trips after the end of the recovery. ICMP_fail_event - Since this is detected by a timeout, it is first desirable for the packetization protocol to come back into Mathis, et al [Page 14] Internet-Draft Expires Sept 2004 14 Feb, 2004 equilibrium with the network (for TCP, this generally means recover it's self clock by completing slowstart up to one half of the old congestion window) before probing again with a smaller probe segment. @@@ TODO finish probe_failure_event - verification_fail_event - verification_timeout_event - probe_timeout_event - verification_stop_event - 3.6. Interoperation with prior algorithms To cache or not. To ICMP or not - Three choices for processing packet too big: ignore all, accept all or only on probes. Three choices for starting size: cached, small, or interface 4. Requirements All Internet nodes SHOULD implement PLPMTUD in order to discover and take advantage of the largest MTU supported along the Internet path. Links MUST not deliver packets that are larger than their MTU. Links that have parametric limitations (e.g. MTU bounds due to limited clock stability) MUST include explicit mechanisms to consistently reject packets that might otherwise be nondeterministically delivered. The requirements below only apply to those implementations that include PLPMTUD. If the IP protocol is IPv4 the DF bit must be set. A packetization protocol MUST use a loss reporting mechanism mechanism (e.g. SACK) which avoids spurious retransmission of any other data when a probe packet is lost. Mathis, et al [Page 15] Internet-Draft Expires Sept 2004 14 Feb, 2004 A Packetization Layer SHOULD NOT send a probe packet unless the flow is expected to have at least the 3 round trips worth of data needed to successfully complete the probing and verification process. A Packetization Layer MUST NOT send a probe unless it has sufficient data available to send such that a lost packet will trigger Fast Retransmit or similar algorithm. Failed and inconclusive probes MUST NOT be sent more frequently than the normal congestion interval for the current average window size. @@@@ too TCP specific During the probe, the normal congestion control machinery MUST remain in effect except when only the probe gap is detected as lost. In this case the normal multiplicative congestion window reduction is suppressed. If any other lost data is detected, all normal congestion control MUST take place. If the probe is successful, the current MPS is updated to the candidate MPS. If window and other congestion state variables are kept in units of packets, they MUST be rescaled to preserve the current window size in bytes. @@ move 5. Implementation Issues This section discusses a number of issues related to the implementation of Path MTU Discovery. This is not a specification, but rather a set of notes provided as an aid for implementers. The issues include: - What layer or layers implement Path MTU Discovery? - Accounting for headers - How is the PMTU information cached? - How are ICMP messages processed - How to implement PMTUD with TCP? - What should other transport and higher layers do? Mathis, et al [Page 16] Internet-Draft Expires Sept 2004 14 Feb, 2004 - What should tunnels above IP do? 5.1. Layering and Accounting for Header Sizes. Packetization Layer Path MTU Discovery is most easily implemented by splitting its functions between layers. The IP layer is in the best place to keep shared state, collect the ICMP messages, track IP headers sizes and manage MTU information from the link layer interfaces. However the procedures that PLPMTUD uses for probing, verifications and scanning for the path MTU are very tightly coupled to the data recovery and congestion control state machines in the Packetization Layer. The most difficult part of implementing PLPMTUD is properly splitting the implementation between the layers. Note that this layering is constant with the advice in the current PMTUD specifications [RFC1191, RFC1981]. Today, many implementations of classical PMTU Discovery are already split along these same layers. Early implementation of PLPMTUD revealed that it is critically important to have a good clean mechanism for accounting header sizes at all layers. This is because the Packetization Layer does its calculations in its own natural data unit, which are almost always a reflection of the service that the Packetization Layer provides to the application or other upper layers. For example, TCP naturally performs all of its calculations in terms of sequence numbers and segment sizes, and the probe gap is the segment that was carried by the probe packet. However, the probe size, ICMP MTU, etc are measures of full packets, which not only include the TCP data and fixed IP and TCP headers, but may also include IP extension headers or options, TCP options and even IPsec AH or ESP headers. PLPMTUD requires frequent bidirectional translation between these two domains: the Packetization Layer's natural data unit and full IP packet sizes. While there are a number of possible ways to accurately implement this duality of size measures, our experience has been that it is best if the boundary between the IP layer and the Packetization layer communicate in terms of the IP Maximum Payload Size or MPS. The MPS is the only size measure that is common to both the IP and Packetization Layers, because it exactly matches the boundary between the layers. The IP Layer is responsible for adding or deducting it's own headers when translating between MTU and MPS. Likewise the Packetization Layer is responsible for adding or deducting its own headers when calculations in it's own natural data units. This document does not take a stance on the placement of IPsec, which Mathis, et al [Page 17] Internet-Draft Expires Sept 2004 14 Feb, 2004 logically sits between IP and the Packetization Layer. IPsec can be treated either as part of IP or as part of the Packetization Layer, as long as the accounting is consistent within any given implementation. If IPsec is treated as part of the IP layer, then each security association to a remote node may need to be treated as a separate flow if they have different length security headers. If IPsec is treated as part of the packetization layer, the IPsec header size has to be included in the Packetization Layer's header size calculations. 5.2. Storing PMTU information This memo uses the concept of a "flow" to define the scope in which path MTU information is used. Each flow locally stores its maximum payload size (MPS), which is used for packetizing data. Flows may communicate with the IP layer to store or access cached PMTU values, providing a means by which similar flows may share information. To do so, the flow must convert between these two values by adding or subtracting the size of the IP header plus any additional intermediate headers. The IP layer also stores PMTU information from the ICMP layer when it receives Packet Too Big messages. Ideally, a PMTU value should be associated with a specific path traversed by packets exchanged between the source and destination nodes. However, in most cases a node will not have enough information to completely and accurately identify such a path. Rather, a node must associate a PMTU value with some local representation of a path. It is left to the implementation to select the local representation of a path. An implementation could use the destination address as the local representation of a path. The PMTU value associated with a destination would be the minimum PMTU learned across the set of all paths in use to that destination. The set of paths in use to a particular destination is expected to be small, in many cases consisting of a single path. This approach will result in the use of optimally sized packets on a per-destination basis. This approach integrates nicely with the conceptual model of a host as described in [ND]: a PMTU value could be stored with the corresponding entry in the destination cache. However, NAT and other forms of middle boxes may exhibit differing MTUs at as single IP address. If IPv6 flows [IPv6-SPEC] are in use, an implementation could use the IPv6 flow id as the local representation of a path. Packets sent to a particular destination but belonging to different flows may use different paths, with the choice of path depending on the flow id. This approach will result in the use of optimally sized packets on a per-flow basis, providing finer granularity than PMTU values Mathis, et al [Page 18] Internet-Draft Expires Sept 2004 14 Feb, 2004 maintained on a per-destination basis. For source routed packets (i.e. packets containing an IPv6 Routing header, or IPv4 LSRR or SSRR options), the source route may further qualify the local representation of a path. An implementation could use source route information in the local representation of a path. If IPsec is in use, the security association can also be used to represent paths. 5.3. Host fragmentation Packetization layers are encouraged to avoid sending messages that will require fragmentation (for the case against fragmentation, see [FRAG]). However this is not always possible. Some packetization layers, such as a UDP application outside the kernel, may be unable to change the size of messages it sends. This may result in packet sizes that exceeds the Path MTU. IPv4 permitted such applications to send packets without DF set. These packets would be fragmented in the IP layer in the host or fragmented by the network. This approach is no longer recommended. We recommend that IPv4 implementation use a new strategy to mimic IPv6 functionality. When the application sends packets that are too large for the path they should be fragmented in the host IP layer. However, the DF bit should be set on the fragments, so they will not be fragmented again in the network. Note that in principle the IP fragmentation layer is an example of a Packetization Layers, it could implement full PLPMTUD in the fragmentation process. At lease one major operating system already uses this strategy. 5.4. Multicast In the case of a multicast destination address, copies of a packet may traverse many different paths to reach many different nodes. The local representation of the "path" to a multicast destination must in fact represent a potentially large set of paths. Minimally, an implementation could maintain a single MPS value to be used for all packets originated from the node. This MPS value would be the minimum MPS learned across the set of all paths in use by the node. This approach is likely to result in the use of smaller packets than is necessary for many paths. Alternatively, if the application using multicast gets complete Mathis, et al [Page 19] Internet-Draft Expires Sept 2004 14 Feb, 2004 delivery reports (unlikely because this requirement has poor scaling properties), PLPMTUD could be implemented in multicast protocols. 5.5. Path MTU Search Strategy The probe strategy described here is a recommended baseline algorithm. It is not presented in formal standards language because the probe strategy can include heuristics to help select an optimal MSS for a given path. As a consequence there is opportunity for future improvements to this algorithms. The probing strategy has three major states: Search, Monitor and Suspend. In the Search state, it sequentially searches for the largest MSS that the path can support. Once the appropriate MPS has been discovered, the probing algorithm enters the Monitor state where it probes infrequently to detect if the path MPS has become larger. If the MPS probing persistently fails it may be desirable to suspend MPS probing and heuristically select one of the common default MSSs: 576, 1240, or 1460 Bytes. 5.5.1. Search @@@ this entire section still needs to be rewritten @@@ The recommended search strategy is a multi-phase scan: First, a coarse scan for the approximate MTU using factor of 2 steps starting at 1024 Bytes until a probe fails, followed by successively finer scans between the largest previously successful and unsuccessful probes. The TCP should use its best knowledge of the lower@@ layer header sizes to appropriately determine the MPS from the MTUs listed in the table below. Table 1: Recommended MTU scanning sequence (Coarse scan down column 1, fine scan across each row) 512, [Use only after repeated timeouts] 1024, 1492, 1500, 2002 2048 4096, 4352 8192, 9000 16384, 17914 32768 64512 ((Additional values needed)) During the scan it is recommended that the MPS not be raised if cwnd Mathis, et al [Page 20] Internet-Draft Expires Sept 2004 14 Feb, 2004 is too small as determined by a heuristic. The recommended heuristic is that the MPS is only raised when the cwnd is larger than 20 segments. @@@ This may be too high. 5.5.2. Monitor Once the scan has found an appropriate MPS, the probe strategy enters the Monitor state, where it re-probes the most recent failed MTU, once every MONITOR_INTERVAL seconds. If the probe fails, it remains in the Monitor state. If it succeeds, it enters the scanning state. If the network becomes too congested during either the Search or the Monitor states, it is recommended that the MPS be reduced to a smaller size as determined by a heuristic. The recommended heuristic is to reduce the MSS if ssthresh is reduced to 5 segments or smaller. The recommended reduction is to the next smaller coarse step in Table 1. When there are repeated timeouts (MAX_TIMO or more retransmissions, without any received ACKs), it is presumed that the connection was re-routed onto a link with a smaller MSS, and that ICMP messages are not being delivered. The MSS probing algorithms is reset by pulling back the MSS to 1024 Bytes, rescaling the congestion control variables and reentering the Search state. 5.5.3. Suspend If there is a timeout, and cwnd prior to the timeout was smaller than 6 packets, then the probe strategy can enter the Suspend state and set the MSS to 512 or 1240 Bytes. This has the effect of reducing the minimum data rate that TCP can stably manage. 5.6. Implementation issues for specific Packetization Layers Different protocols introduce specific problems. 5.6.1. Probing method using TCP TCP has no mechanism that could be used to distinguish between real application data and some other form of padding that might be used to fill out probe packets. Therefor, TCP must generate probes by sending oversized segments that are carrying real application date. As previously mentioned there are two approaches that TCP might use to minimize the overheads associated with the probing process. Mathis, et al [Page 21] Internet-Draft Expires Sept 2004 14 Feb, 2004 A TCP implementation of PLPMTUD can elect to send subsequent segments as though the probe segment was not oversized. This has the advantage that TCP only need to retransmit a segment at the current MTU to recover from failed probes. However the duplicate data in the probe does consume network resources and will cause duplicate acknowledgments. It is important that these extra duplicate acknowledgments not trigger Fast Retransmit. This can be guaranteed by limiting the largest probe segment size to twice the current segment size (causing at most 1 duplicate acknowledgment) three times the current segment size (causing at most 2 duplicate acknowledgments. The other approach is to send non-overlapping segments following the probe. Although this is cleaner from a protocol architecture standpoint it clashes with many of the optimizations used improve the efficiency of data motion withing many operating systems. In particular many implementations divide the data into segments and precompute checksums as the data is copied out of user space. In these implementation it can be very expensive to adjust segment boundaries after the data is already queued. If TCP is using SACK or any other variable length headers, the headers on the probe and verification packets should be padded to the maximum possible length. Otherwise, large headers may cause delivery problems for future segments. Note that the header size and overhead calculations described in section @@@ apply here. TCP's natural data accounting units are sequence space and Maximum Segment Size. However the the PLPMTUD process is described in terms of total packet size, which is larger than the MSS by all fixed and optional headers. At the point when TCP is ready to start verification, it is permitted to not re-packetize already queued data. This postpones the verification process by the time required to send the queued data. If the verification phase experiences any segment losses, TCP is required to pull back to the prior MSS. Since failing the verification phase should be an infrequent error condition it is less important that this be as efficient as probing. 5.6.2. Probing method using SCTP In the SCTP protocol packetization is the responsibility of the application or protocol above SCTP. By implication SCTP can not easily generate probes sending additional application data. Mathis, et al [Page 22] Internet-Draft Expires Sept 2004 14 Feb, 2004 For SCTP the recommended method for generating probes is to pad messages by @@@@@@ or by sending a message that consists entirely of padding and no application data. The verification phase ...... 5.6.3. Issues for tunnels @@@ to be written 5.6.4. Issues for other transport protocols Some transport protocols (such as ISO TP4 [ISOTP]) are not allowed to repacketize when doing a retransmission. That is, once an attempt is made to transmit a segment of a certain size, the transport cannot split the contents of the segment into smaller segments for retransmission. In such a case, the original segment can be fragmented by the IP layer during retransmission. Subsequent segments, when transmitted for the first time, should be no larger than allowed by the Path MTU. 5.7. Diagnostic tools All implementations MUST include a mechanism to implement diagnostic tools that do not rely on the operating systems implementation of path MTU discovery. This requires an mechanism where an application can send oversized packets that are not subjected to the operating systems notion of the current path MTU, up to the physical MTU limit as supported by the network interface, as well as a mechanism to collect any Packet Too Big Messages. 5.8. Management interface It is suggested that an implementation provide a way for a system utility program to: - Specify that Path MTU Discovery not be done on a given path. - Change the PMTU value associated with a given path. - Global controls on ICMP processing - Per connection or per application controls on ICMP processing The former can be accomplished by associating a flag with the path; when a packet is sent on a path with this flag set, the IP layer does not send packets larger than the IPv6 minimum link MTU. Mathis, et al [Page 23] Internet-Draft Expires Sept 2004 14 Feb, 2004 These features might be used to work around an anomalous situation, or by a routing protocol implementation that is able to obtain Path MTU values. The implementation should also provide a way to change the timeout period for aging stale PMTU information. 6. Normative references [RFC1191] Path MTU discovery. J.C. Mogul, S.E. Deering. Nov-01-1990. (Format: TXT=47936 bytes) (Obsoletes RFC1063) (Status: DRAFT STANDARD) [RFC1981] Path MTU Discovery for IP version 6. J. McCann, S. Deering, J. Mogul. August 1996. (Status: PROPOSED STANDARD) [RFC2119] Key words for use in RFCs to Indicate Requirement Levels. S. Bradner. March 1997. (Status: BEST CURRENT PRACTICE) 7. Informative references [RFC1063] IP MTU discovery options. J.C. Mogul, C.A. Kent, C. Par- tridge, K. McCloghrie. Jul-01-1988. (Obsoleted by RFC1191) [RFC1435] IESG Advice from Experience with Path MTU Discovery. S. Knowles. March 1993. (Format: TXT=2708 bytes) (Status: INFORMATIONAL) [RFC1626] Default IP MTU for use over ATM AAL5. R. Atkinson. May 1994. (Status: PROPOSED STANDARD) [RFC1791] TCP And UDP Over IPX Networks With Fixed Path MTU. T. Sung. April 1995. (Status: EXPERIMENTAL) [RFC2923] TCP Problems with Path MTU Discovery. K. Lahey. September 2000. (Status: INFORMATIONAL) 8. Security considerations Since the MTU reported in the ICMP messages is constrained to be Mathis, et al [Page 24] Internet-Draft Expires Sept 2004 14 Feb, 2004 between the old MTU and the candidate MTU, this algorithm is more difficult to attack through fraudulent ICMP messages. Furthermore, since this algorithm can function properly without ICMP messages that part of the algorithm can be disabled for additional robustness in hostile environments. 9. IANA considerations 10. Contributors 11. Acknowledgements Matt Mathis and John Heffner are supported by a grant from Cisco Sys- tems, Inc. 12. Authors' addresses Please send comments and suggestions to pmtud@ietf.org. Matt Mathis and John Heffner Pittsburgh Supercomputing Center 4400 Fifth Ave. Pittsburgh, PA 15213 mathis@psc.edu jheffner@psc.edu Kevin Lahey Freelance kml@patheticgeek.net 13. Intellectual Property The IETF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to per- tain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on the IETF's procedures with respect to rights in standards-track and standards- related documentation can be found in BCP-11. Copies of claims of rights made available for publication and any assurances of licenses Mathis, et al [Page 25] Internet-Draft Expires Sept 2004 14 Feb, 2004 to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF Secretariat. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this standard. Please address the information to the IETF Executive Director. 14. Full copyright statement Copyright (C) The Internet Society 14 Feb, 2004. All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this doc- ument itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of develop- ing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. Mathis, et al [Page 26]