HTTP/1.1 200 OK Date: Tue, 09 Apr 2002 08:39:24 GMT Server: Apache/1.3.20 (Unix) Last-Modified: Mon, 24 Nov 1997 18:49:00 GMT ETag: "323c6f-fe40-3479cc1c" Accept-Ranges: bytes Content-Length: 65088 Connection: close Content-Type: text/plain Network Working Group V. Paxson, Editor Internet Draft M. Allman S. Dawson I. Heavens B. Volz Expiration Date: May 1998 November 1997 Known TCP Implementation Problems 1. Status of this Memo This document is an Internet Draft. Internet Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet Drafts. Internet Drafts are draft documents valid for a maximum of six months, and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet Drafts as reference material or to cite them other than as ``work in progress''. To learn the current status of any Internet Draft, please check the ``1id-abstracts.txt'' listing contained in the Internet Drafts shadow directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). This memo provides information for the Internet community. This memo does not specify an Internet standard of any kind. Distribution of this memo is unlimited. 2. Introduction This memo catalogs a number of known TCP implementation problems. The goal in doing so is to improve conditions in the existing Internet by enhancing the quality of current TCP/IP implementations. It is hoped that both performance and correctness issues can be resolved by making implementors aware of the problems and their solutions. In the long term, it is hoped that this will provide a reduction in unnecessary traffic on the network, the rate of connection failures due to protocol errors, and load on network servers due to time spent processing both unsuccessful connections and retransmitted data. This will help to ensure the stability of the global Internet. Paxson, Editor [Page 1] ID Known TCP Implementation Problems November 1997 Each problem is defined as follows: Name The name associated with the problem. In this memo, the name is given as a subsection heading. Classification One or more problem categories for which the problem is classi- fied. Categories used so far: "congestion control", "perfor- mance", "reliability", "resource management". Others antici- pated: "security", "interoperability", "configuration". Description A definition of the problem, succinct but including necessary background material. Significance A brief summary of the sorts of environments for which the prob- lem is significant. Implications Why the problem is viewed as a problem. Relevant RFCs Brief discussion of the RFCs with respect to which the problem is viewed as an implementation error. Trace file demonstrating the problem One or more ASCII trace files demonstrating the problem, if applicable. These may in the future be replaced with URLs to on-line traces. Trace file demonstrating correct behavior One or more examples of how correct behavior appears in a trace, if applicable. These may in the future be replaced with URLs to on-line traces. References References that further discuss the problem. How to detect How to test an implementation to see if it exhibits the problem. This discussion may include difficulties and subtleties associ- ated with causing the problem to manifest itself, and with interpreting traces to detect the presence of the problem (if applicable). In the future, this may include URLs for diagnos- tic tools. Paxson, Editor [Page 2] ID Known TCP Implementation Problems November 1997 How to fix For known causes of the problem, how to correct the implementa- tion. Implementation specifics If it is viewed as beneficial to document particular implementa- tions exhibiting the problem, and if the corresponding implemen- tors approve, then this section gives the specifics of those implementations, along with a contact address for the implemen- tors. 3. Known implementation problems 3.1. No initial slow start Classification Congestion control Description When a TCP begins transmitting data, it is required by RFC 1122, 4.2.2.15, to engage in a "slow start" by initializing its conges- tion window, cwnd, to one packet (one segment of the maximum size). It subsequently increases cwnd by one packet for each ACK it receives for new data. The minimum of cwnd and the receiver's advertised window bounds the highest sequence number the TCP can transmit. A TCP that fails to initialize and increment cwnd in this fashion exhibits "No initial slow start". Significance In congested environments, detrimental to the performance of other connections, and possibly to the connection itself. Implications A TCP failing to slow start when beginning a connection results in traffic bursts that can stress the network, leading to excessive queueing delays and packet loss. Implementations exhibiting this problem might do so because they suffer from the general problem of not including the required congestion window. These implementations will also suffer from "No slow start after retransmission timeout". There are different shades of "No initial slow start". From the perspective of stressing the network, the worst is a connection that simply always sends based on the receiver's advertised window, with no notion of a separate congestion window. Some other forms Paxson, Editor [Page 3] ID Known TCP Implementation Problems November 1997 are described in "Uninitialized CWND" and "Initial CWND of 2 pack- ets". Relevant RFCs RFC 1122 requires use of slow start. RFC 2001 gives the specifics of slow start. Trace file demonstrating it Made using tcpdump/BPF recording at the connection responder. No losses reported. 10:40:42.244503 B > A: S 1168512000:1168512000(0) win 32768 (DF) [tos 0x8] 10:40:42.259908 A > B: S 3688169472:3688169472(0) ack 1168512001 win 32768 10:40:42.389992 B > A: . ack 1 win 33580 (DF) [tos 0x8] 10:40:42.664975 A > B: P 1:513(512) ack 1 win 32768 10:40:42.700185 A > B: . 513:1973(1460) ack 1 win 32768 10:40:42.718017 A > B: . 1973:3433(1460) ack 1 win 32768 10:40:42.762945 A > B: . 3433:4893(1460) ack 1 win 32768 10:40:42.811273 A > B: . 4893:6353(1460) ack 1 win 32768 10:40:42.829149 A > B: . 6353:7813(1460) ack 1 win 32768 10:40:42.853687 B > A: . ack 1973 win 33580 (DF) [tos 0x8] 10:40:42.864031 B > A: . ack 3433 win 33580 (DF) [tos 0x8] After the third packet, the connection is established. A, the con- nection responder, begins transmitting to B, the connection initia- tor. Host A quickly sends 6 packets comprising 7812 bytes, even though the SYN exchange agreed upon an MSS of 1460 bytes (implying an initial congestion window of 1 segment corresponds to 1460 bytes), and so A should have sent at most 1460 bytes. The ACKs sent by B to A in the last two lines indicate that this trace is not a measurement error (slow start really occurring but the corresponding ACKs having been dropped by the packet filter). A second trace confirmed that the problem is repeatable. Trace file demonstrating correct behavior Made using tcpdump/BPF recording at the connection originator. No losses reported. 12:35:31.914050 C > D: S 1448571845:1448571845(0) win 4380 12:35:32.068819 D > C: S 1755712000:1755712000(0) ack 1448571846 win 4096 12:35:32.069341 C > D: . ack 1 win 4608 Paxson, Editor [Page 4] ID Known TCP Implementation Problems November 1997 12:35:32.075213 C > D: P 1:513(512) ack 1 win 4608 12:35:32.286073 D > C: . ack 513 win 4096 12:35:32.287032 C > D: . 513:1025(512) ack 1 win 4608 12:35:32.287506 C > D: . 1025:1537(512) ack 1 win 4608 12:35:32.432712 D > C: . ack 1537 win 4096 12:35:32.433690 C > D: . 1537:2049(512) ack 1 win 4608 12:35:32.434481 C > D: . 2049:2561(512) ack 1 win 4608 12:35:32.435032 C > D: . 2561:3073(512) ack 1 win 4608 12:35:32.594526 D > C: . ack 3073 win 4096 12:35:32.595465 C > D: . 3073:3585(512) ack 1 win 4608 12:35:32.595947 C > D: . 3585:4097(512) ack 1 win 4608 12:35:32.596414 C > D: . 4097:4609(512) ack 1 win 4608 12:35:32.596888 C > D: . 4609:5121(512) ack 1 win 4608 12:35:32.733453 D > C: . ack 4097 win 4096 References This problem is documented in [Paxson97]. How to detect For implementations always manifesting this problem, it shows up immediately in a packet trace or a sequence plot, as illustrated above. How to fix If the root problem is that the implementation lacks a notion of a congestion window, then unfortunately this requires significant work to fix. However, doing so is important, as such implementa- tions also exhibit "No slow start after retransmission timeout". 3.2. No slow start after retransmission timeout Classification Congestion control Description When a TCP experiences a retransmission timeout, it is required by RFC 1122, 4.2.2.15, to engage in "slow start" by initializing its congestion window, cwnd, to one packet (one segment of the maximum size). It subsequently increases cwnd by one packet for each ACK it receives for new data until it reaches the "congestion avoidance" threshold, ssthresh, at which point the congestion avoidance algorithm for updating the window takes over. A TCP that fails to enter slow start upon a timeout exhibits "No slow start after retransmission timeout". Significance Paxson, Editor [Page 5] ID Known TCP Implementation Problems November 1997 In congested environments, severely detrimental to the performance of other connections, and also the connection itself. Implications Entering slow start upon timeout forms one of the cornerstones of Internet congestion stability, as outlined in [Jacobson88]. If TCPs fail to do so, the network becomes at risk of suffering "congestion collapse" [RFC896]. Relevant RFCs RFC 1122 requires use of slow start after loss. RFC 2001 gives the specifics of how to implement slow start. RFC 896 describes congestion collapse. The retransmission timeout discussed here should not be confused with the separate "fast recovery" retransmission mechanism dis- cussed in RFC 2001. Trace file demonstrating it Made using tcpdump/BPF recording at the sending TCP (A). No losses reported. 10:40:59.090612 B > A: . ack 357125 win 33580 (DF) [tos 0x8] 10:40:59.222025 A > B: . 357125:358585(1460) ack 1 win 32768 10:40:59.868871 A > B: . 357125:358585(1460) ack 1 win 32768 10:41:00.016641 B > A: . ack 364425 win 33580 (DF) [tos 0x8] 10:41:00.036709 A > B: . 364425:365885(1460) ack 1 win 32768 10:41:00.045231 A > B: . 365885:367345(1460) ack 1 win 32768 10:41:00.053785 A > B: . 367345:368805(1460) ack 1 win 32768 10:41:00.062426 A > B: . 368805:370265(1460) ack 1 win 32768 10:41:00.071074 A > B: . 370265:371725(1460) ack 1 win 32768 10:41:00.079794 A > B: . 371725:373185(1460) ack 1 win 32768 10:41:00.089304 A > B: . 373185:374645(1460) ack 1 win 32768 10:41:00.097738 A > B: . 374645:376105(1460) ack 1 win 32768 10:41:00.106409 A > B: . 376105:377565(1460) ack 1 win 32768 10:41:00.115024 A > B: . 377565:379025(1460) ack 1 win 32768 10:41:00.123576 A > B: . 379025:380485(1460) ack 1 win 32768 10:41:00.132016 A > B: . 380485:381945(1460) ack 1 win 32768 10:41:00.141635 A > B: . 381945:383405(1460) ack 1 win 32768 10:41:00.150094 A > B: . 383405:384865(1460) ack 1 win 32768 10:41:00.158552 A > B: . 384865:386325(1460) ack 1 win 32768 10:41:00.167053 A > B: . 386325:387785(1460) ack 1 win 32768 10:41:00.175518 A > B: . 387785:389245(1460) ack 1 win 32768 10:41:00.210835 A > B: . 389245:390705(1460) ack 1 win 32768 10:41:00.226108 A > B: . 390705:392165(1460) ack 1 win 32768 10:41:00.241524 B > A: . ack 389245 win 8760 (DF) [tos 0x8] Paxson, Editor [Page 6] ID Known TCP Implementation Problems November 1997 The first packet indicates the ack point is 357125. 130 msec after receiving the ACK, A transmits the packet after the ACK point, 357125:358585. 640 msec after this transmission, it retransmits 357125:358585, in an apparent retransmission timeout. At this point, A's cwnd should be one MSS, or 1460 bytes, as A enters slow start. The trace is consistent with this possibility. B replies with an ACK of 364425, indicating that A has filled a sequence hole. At this point, A's cwnd should be 1460*2 = 2920 bytes, since in slow start receiving an ACK advances cwnd by MSS. However, A then launches 19 consecutive packets, which is incon- sistent with slow start. A second trace confirmed that the problem is repeatable. Trace file demonstrating correct behavior Made using tcpdump/BPF recording at the sending TCP (C). No losses reported. 12:35:48.442538 C > D: P 465409:465921(512) ack 1 win 4608 12:35:48.544483 D > C: . ack 461825 win 4096 12:35:48.703496 D > C: . ack 461825 win 4096 12:35:49.044613 C > D: . 461825:462337(512) ack 1 win 4608 12:35:49.192282 D > C: . ack 465921 win 2048 12:35:49.192538 D > C: . ack 465921 win 4096 12:35:49.193392 C > D: P 465921:466433(512) ack 1 win 4608 12:35:49.194726 C > D: P 466433:466945(512) ack 1 win 4608 12:35:49.350665 D > C: . ack 466945 win 4096 12:35:49.351694 C > D: . 466945:467457(512) ack 1 win 4608 12:35:49.352168 C > D: . 467457:467969(512) ack 1 win 4608 12:35:49.352643 C > D: . 467969:468481(512) ack 1 win 4608 12:35:49.506000 D > C: . ack 467969 win 3584 After C transmits the first packet shown to D, it takes no action in response to D's ACKs for 461825, because the first packet already reached the advertised window limit of 4096 bytes above 461825. 600 msec after transmitting the first packet, C retransmits 461825:462337, presumably due to a timeout. Its congestion window is now MSS (512 bytes). D acks 465921, indicating that C's retransmission filled a sequence hole. This ACK advances C's cwnd from 512 to 1024. Very shortly after, D acks 465921 again in order to update the offered window from 2048 to 4096. This ACK does not advance cwnd since it is not for new data. Very shortly after, C responds to the newly enlarged window by transmitting two packets. D acks both, advancing cwnd from 1024 to 1536. C in turn transmits three packets. Paxson, Editor [Page 7] ID Known TCP Implementation Problems November 1997 References This problem is documented in [Paxson97]. How to detect Packet loss is common enough in the Internet that generally it is not difficult to find an Internet path that will force retransmis- sion due to packet loss. If the effective window prior to loss is large enough, however, then the TCP may retransmit using the "fast recovery" mechanism described in RFC 2001. In a packet trace, the signature of fast recovery is that the packet retransmission occurs in response to the receipt of three duplicate ACKs, and subsequent duplicate ACKs may lead to the transmission of new data, above both the ack point and the highest sequence transmitted so far. An absence of three duplicate ACKs prior to retransmission suffices to distinguish between timeout and fast recovery retransmissions. In the face of only observing fast recovery retransmissions, generally it is not difficult to repeat the data transfer until observing a timeout retransmission. Once armed with a trace exhibiting a timeout retransmission, deter- mining whether the TCP follows slow start is done by computing the correct progression of cwnd and comparing it to the amount of data transmited by the TCP subsequent to the timeout rtransmission. How to fix If the root problem is that the implementation lacks a notion of a congestion window, then unfortunately this requires significant work to fix. However, doing so is critical, for reasons outlined above. 3.3. Inconsistent retransmission Classification Reliability Description If, for a given sequence number, a sending TCP retransmits dif- ferent data than previously sent for that sequence number, then a strong possibility arises that the receiving TCP will reconstruct a different byte stream than that sent by the sending application, depending on which instance of the sequence number it accepts. Such a sending TCP exhibits "Inconsistent retransmission". Paxson, Editor [Page 8] ID Known TCP Implementation Problems November 1997 Significance Critical for all environments. Implications Reliable delivery of data is a fundamental property of TCP. Relevant RFCs RFC 793, section 1.5, discusses the central role of reliability in TCP operation. Trace file demonstrating it Made using tcpdump/BPF recording at the receiving TCP (B). No losses reported. 12:35:53.145503 A > B: FP 90048435:90048461(26) ack 393464682 win 4096 4500 0042 9644 0000 3006 e4c2 86b1 0401 83f3 010a b2a4 0015 055e 07b3 1773 cb6a 5019 1000 68a9 0000 data starts here>504f 5254 2031 3334 2c31 3737*2c34 2c31 2c31 3738 2c31 3635 0d0a 12:35:53.146479 B > A: R 393464682:393464682(0) win 8192 12:35:53.851714 A > B: FP 90048429:90048463(34) ack 393464682 win 4096 4500 004a 965b 0000 3006 e4a3 86b1 0401 83f3 010a b2a4 0015 055e 07ad 1773 cb6a 5019 1000 8bd3 0000 data starts here>5041 5356 0d0a 504f 5254 2031 3334 2c31 3737*2c31 3035 2c31 3431 2c34 2c31 3539 0d0a The sequence numbers shown in this trace are absolute and not adjusted to reflect the ISN. The 4-digit hex values show a dump of the packet's IP and TCP headers, as well as payload. A first sends to B data for 90048435:90048461. The corresponding data begins with hex words 504f, 5254, etc. B responds with a RST. Since the recording location was local to B, it is unknown whether A received the RST. A then sends 90048429:90048463, which includes six sequence posi- tions below the earlier transmission, all 26 positions of the ear- lier transmission, and two additional sequence positions. The retransmission disagrees starting just after sequence 90048447, annotated above with a leading '*'. These two bytes were origi- nally transmitted as hex 2c34 but retransmitted as hex 2c31. Sub- sequent positions disagree as well. This behavior has been observed in other traces involving different Paxson, Editor [Page 9] ID Known TCP Implementation Problems November 1997 hosts. It is unknown how to repeat it. In this instance, no corruption would occur, since B has already indicated it will not accept further packets from A. A second example illustrates a slightly different instance of the problem. The tracing again was made with tcpdump/BPF at the receiving TCP (D). 22:23:58.645829 C > D: P 185:212(27) ack 565 win 4096 4500 0043 90a3 0000 3306 0734 cbf1 9eef 83f3 010a 0525 0015 a3a2 faba 578c 70a4 5018 1000 9a53 0000 data starts here>504f 5254 2032 3033 2c32 3431 2c31 3538 2c32 3339 2c35 2c34 330d 0a 22:23:58.646805 D > C: . ack 184 win 8192 4500 0028 beeb 0000 3e06 ce06 83f3 010a cbf1 9eef 0015 0525 578c 70a4 a3a2 fab9 5010 2000 342f 0000 22:31:36.532244 C > D: FP 186:213(27) ack 565 win 4096 4500 0043 9435 0000 3306 03a2 cbf1 9eef 83f3 010a 0525 0015 a3a2 fabb 578c 70a4 5019 1000 9a51 0000 data starts here>504f 5254 2032 3033 2c32 3431 2c31 3538 2c32 3339 2c35 2c34 330d 0a In this trace, sequence numbers are relative. C sends 185:212, but D only sends an ACK for 184 (so sequence number 184 is missing). C then sends 186:213. The packet payload is identical to the previ- ous payload, but the base sequence number is one higher, resulting in an inconsistent retransmission. Neither trace exhibits checksum errors. Trace file demonstrating correct behavior (Omitted, as presumably correct behavior is obvious.) References None known. How to detect This problem unfortunately can be very difficult to detect, since available experience indicates it is quite rare that it is mani- fested. No "trigger" has been identified that can be used to reproduce the problem. How to fix Paxson, Editor [Page 10] ID Known TCP Implementation Problems November 1997 In the absence of a known "trigger", we cannot always assess how to fix the problem. In one implementation (not the one illustrated above), the problem manifested itself when (1) the sender received a zero window and stalled; (2) eventually an ACK arrived that offered a window larger than that in effect at the time of the stall; (3) the sender transmitted out of the buffer of data it held at the time of the stall, but (4) failed to limit this transfer to the buffer length, instead using the newly advertised (and larger) offered window. Consequently, in addition to the valid buffer contents, it sent whatever garbage values followed the end of the buffer. If it then retransmitted the corresponding sequence numbers, at that point it sent the correct data, resulting in an inconsistent retransmission. Note that this instance of the problem reflects a more general problem, that of initially transmitting incorrect data. 3.4. Failure to retain above-sequence data Classification Congestion control, performance Description When a TCP receives an "above sequence" segment, meaning one with a sequence number exceeding RCV.NXT but below RCV.NXT+RCV.WND, it SHOULD queue the segment for later delivery (RFC 1122, 4.2.2.20). A TCP that fails to do so is said to exhibit "Failure to retain above-sequence data". It may sometimes be appropriate for a TCP to discard above-sequence data to reclaim memory. If they do so only rarely, then we would not consider them to exhibit this problem. Instead, the particular concern is with TCPs that always discard above-sequence data. Significance In environments prone to packet loss, detrimental to the perfor- mance of both other connections and the connection itself. Implications In times of congestion, a failure to retain above-sequence data will lead to numerous otherwise-unnecessary retransmissions, aggra- vating the congestion and potentially reducing performance by a large factor. Relevant RFCs RFC 1122 revises RFC 793 by upgrading the latter's MAY to a SHOULD Paxson, Editor [Page 11] ID Known TCP Implementation Problems November 1997 on this issue. Trace file demonstrating it Made using tcpdump/BPF recording at the receiving TCP. No losses reported. B is the TCP sender, A the receiver. A exhibits failure to retain above sequence data: 10:38:10.164860 B > A: . 221078:221614(536) ack 1 win 33232 [tos 0x8] 10:38:10.170809 B > A: . 221614:222150(536) ack 1 win 33232 [tos 0x8] 10:38:10.177183 B > A: . 222150:222686(536) ack 1 win 33232 [tos 0x8] 10:38:10.225039 A > B: . ack 222686 win 25800 Here B has sent up to (relative) sequence 222686 in-sequence, and A accordingly acknowledges. 10:38:10.268131 B > A: . 223222:223758(536) ack 1 win 33232 [tos 0x8] 10:38:10.337995 B > A: . 223758:224294(536) ack 1 win 33232 [tos 0x8] 10:38:10.344065 B > A: . 224294:224830(536) ack 1 win 33232 [tos 0x8] 10:38:10.350169 B > A: . 224830:225366(536) ack 1 win 33232 [tos 0x8] 10:38:10.356362 B > A: . 225366:225902(536) ack 1 win 33232 [tos 0x8] 10:38:10.362445 B > A: . 225902:226438(536) ack 1 win 33232 [tos 0x8] 10:38:10.368579 B > A: . 226438:226974(536) ack 1 win 33232 [tos 0x8] 10:38:10.374732 B > A: . 226974:227510(536) ack 1 win 33232 [tos 0x8] 10:38:10.380825 B > A: . 227510:228046(536) ack 1 win 33232 [tos 0x8] 10:38:10.387027 B > A: . 228046:228582(536) ack 1 win 33232 [tos 0x8] 10:38:10.393053 B > A: . 228582:229118(536) ack 1 win 33232 [tos 0x8] 10:38:10.399193 B > A: . 229118:229654(536) ack 1 win 33232 [tos 0x8] 10:38:10.405356 B > A: . 229654:230190(536) ack 1 win 33232 [tos 0x8] A now receives 13 additional packets from B. These are above- sequence because 222686:223222 was dropped. The packets do however fit within the offered window of 25800. A does not generate any duplicate ACKs for them. The trace contributor (V. Paxson) verified that these 13 packets had valid IP and TCP checksums. 10:38:11.917728 B > A: . 222686:223222(536) ack 1 win 33232 [tos 0x8] 10:38:11.930925 A > B: . ack 223222 win 32232 B times out for 222686:223222 and retransmits it. Upon receiving it, A only acknowledges 223222. Had it retained the valid above- sequence packets, it would instead have ack'd 230190. 10:38:12.048438 B > A: . 223222:223758(536) ack 1 win 33232 [tos 0x8] 10:38:12.054397 B > A: . 223758:224294(536) ack 1 win 33232 [tos 0x8] Paxson, Editor [Page 12] ID Known TCP Implementation Problems November 1997 10:38:12.068029 A > B: . ack 224294 win 31696 B retransmits two more packets, and A only acknowledges them. This pattern continues as B retransmits the entire set of previously- received packets. A second trace confirmed that the problem is repeatable. Trace file demonstrating correct behavior Made using tcpdump/BPF recording at the receiving TCP (C). No losses reported. 09:11:25.790417 D > C: . 33793:34305(512) ack 1 win 61440 09:11:25.791393 D > C: . 34305:34817(512) ack 1 win 61440 09:11:25.792369 D > C: . 34817:35329(512) ack 1 win 61440 09:11:25.792369 D > C: . 35329:35841(512) ack 1 win 61440 09:11:25.793345 D > C: . 36353:36865(512) ack 1 win 61440 09:11:25.794321 C > D: . ack 35841 win 59904 A sequence hole occurs because 35841:36353 has been dropped. 09:11:25.794321 D > C: . 36865:37377(512) ack 1 win 61440 09:11:25.794321 C > D: . ack 35841 win 59904 09:11:25.795297 D > C: . 37377:37889(512) ack 1 win 61440 09:11:25.795297 C > D: . ack 35841 win 59904 09:11:25.796273 C > D: . ack 35841 win 61440 09:11:25.798225 D > C: . 37889:38401(512) ack 1 win 61440 09:11:25.799201 C > D: . ack 35841 win 61440 09:11:25.807009 D > C: . 38401:38913(512) ack 1 win 61440 09:11:25.807009 C > D: . ack 35841 win 61440 (many additional lines omitted) 09:11:25.884113 D > C: . 52737:53249(512) ack 1 win 61440 09:11:25.884113 C > D: . ack 35841 win 61440 Each additional, above-sequence packet C receives from D elicits a duplicate ACK for 35841. 09:11:25.887041 D > C: . 35841:36353(512) ack 1 win 61440 09:11:25.887041 C > D: . ack 53249 win 44032 D retransmits 35841:36353 and C acknowledges receipt of data all the way up to 53249. References This problem is documented in [Paxson97]. Paxson, Editor [Page 13] ID Known TCP Implementation Problems November 1997 How to detect Packet loss is common enough in the Internet that generally it is not difficult to find an Internet path that will result in some above-sequence packets arriving. A TCP that exhibits "Failure to retain ..." may not generate duplicate ACKs for these packets. However, some TCPs that do retain above-sequence data also do not generate duplicate ACKs, so failure to do so does not definitively identify the problem. Instead, the key observation is whether upon retransmission of the dropped packet, data that was previously above-sequence is acknowledged. Two considerations in detecting this problem using a packet trace are that it is easiest to do so with a trace made at the TCP receiver, in order to unambiguously determine which packets arrived successfully, and that such packets may still be correctly dis- carded if they arrive with checksum errors. The latter can be tested by capturing the entire packet contents and performing the IP and TCP checksum algorithms to verify their integrity; or by confirming that the packets arrive with the same checksum and con- tents as that with which they were sent, with a presumption that the sending TCP correctly calculates checksums for the packets it transmits. It is considerably easier to verify that an implementation does NOT exhibit this problem. This can be done by recording a trace at the data sender, and observing that sometimes after a retransmission the receiver acknowledges a higher sequence number than just that which was retransmitted. How to fix If the root problem is that the implementation lacks buffer, then then unfortunately this requires significant work to fix. However, doing so is important, for reasons outlined above. 3.5. Excessively short keepalive connection timeout Classification Reliability Description Keep-alive is a mechanism for checking whether an idle connection is still alive. According to RFC-1122, keepalive should only be invoked in server applications that might otherwise hang indefin- itely and consume resources unnecessarily if a client crashes or Paxson, Editor [Page 14] ID Known TCP Implementation Problems November 1997 aborts a connection during a network failure. RFC-1122 also specifies that if a keep-alive mechanism is imple- mented it MUST NOT interpret failure to respond to any specific probe as a dead connection. The RFC does not specify a particular mechanism for timing out a connection when no response is received for keepalive probes. However, if the mechanism does not allow ample time for recovery from network congestion or delay, connec- tions may be timed out unnecessarily. Significance In congested networks, can lead to unwarranted termination of con- nections. Implications It is possible for the network connection between two peer machines to become congested or to exhibit packet loss at the time that a keep-alive probe is sent on a connection. If the keep-alive mechanism does not allow sufficient time before dropping connec- tions in the face of unacknowledged probes, connections may be dropped even when both peers of a connection are still alive. Relevant RFCs RFC 1122 specifies that the keep-alive mechanism may be provided. It does not specify a mechanism for determining dead connections when keepalive probes are not acknowledged. Trace file demonstrating it Made using the Orchestra tool at the peer of the machine using keep-alive. After connection establishment, incoming keep-alives were dropped by Orchestra to simulate a dead connection. 22:11:12.040000 A > B: 22666019:0 win 8192 datasz 4 SYN 22:11:12.060000 B > A: 2496001:22666020 win 4096 datasz 4 SYN ACK 22:11:12.130000 A > B: 22666020:2496002 win 8760 datasz 0 ACK (more than two hours elapse) 00:23:00.680000 A > B: 22666019:2496002 win 8760 datasz 1 ACK 00:23:01.770000 A > B: 22666019:2496002 win 8760 datasz 1 ACK 00:23:02.870000 A > B: 22666019:2496002 win 8760 datasz 1 ACK 00:23.03.970000 A > B: 22666019:2496002 win 8760 datasz 1 ACK 00:23.05.070000 A > B: 22666019:2496002 win 8760 datasz 1 ACK The initial three packets are the SYN exchange for connection setup. About two hours later, the keepalive timer fires because Paxson, Editor [Page 15] ID Known TCP Implementation Problems November 1997 the connection has been idle. Keepalive probes are transmitted a total of 5 times, with a 1 second spacing between probes, after which the connection is dropped. This is problematic because a 5 second network outage at the time of the first probe results in the connection being killed. Trace file demonstrating correct behavior Made using the Orchestra tool at the peer of the machine using keep-alive. After connection establishment, incoming keep-alives were dropped by Orchestra to simulate a dead connection. 16:01:52.130000 A > B: 1804412929:0 win 4096 datasz 4 SYN 16:01:52.360000 B > A: 16512001:1804412930 win 4096 datasz 4 SYN ACK 16:01:52.410000 A > B: 1804412930:16512002 win 4096 datasz 0 ACK (two hours elapse) 18:01:57.170000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK 18:03:12.220000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK 18:04:27.270000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK 18:05:42.320000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK 18:06:57.370000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK 18:08:12.420000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK 18:09:27.480000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK 18:10:43.290000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK 18:11:57.580000 A > B: 1804412929:16512002 win 4096 datasz 0 ACK 18:13:12.630000 A > B: 1804412929:16512002 win 4096 datasz 0 RST ACK In this trace, when the keep-alive timer expires, 9 keepalive probes are sent at 75 second intervals. 75 seconds after the last probe is sent, a final RST segment is sent indicating that the con- nection has been closed. This implementation waits about 11 minutes before timing out the connection, while the first implemen- tation shown allows only 5 seconds. References This problem is documented in [Dawson97]. How to detect For implementations manifesting this problem, it shows up on a packet trace after the keepalive timer fires if the peer machine receiving the keepalive does not respond. Usually the keepalive timer will fire at least two hours after keepalive is turned on, but it may be sooner if the timer value has been configured lower, or if the keepalive mechanism violates the specification (see Insufficient interval between keepalives problem). In this exam- ple, suppressing the response of the peer to keepalive probes was Paxson, Editor [Page 16] ID Known TCP Implementation Problems November 1997 accomplished using the Orchestra toolkit, which can be configured to drop packets. It could also have been done by creating a con- nection, turning on keepalive, and disconnecting the network con- nection at the receiver machine. How to fix This problem can be fixed by using a different method for timing out keepalives that allows a longer period of time to elapse before dropping the connection. For example, the algorithm for timing out on dropped data could be used. Another possibility is an algorithm such as the one shown in the trace above, which sends 9 probes at 75 second intervals and then waits an additional 75 seconds for a response before closing the connection. 3.6. Insufficient interval between keepalives Classification Reliability Description Keep-alive is a mechanism for checking whether an idle connection is still alive. According to RFC-1122, keep-alive may be included in an implementation. If it is included, the interval between keep-alive packets MUST be configurable, and MUST default to no less than two hours. Significance In congested networks, can lead to unwarranted termination of con- nections. Implications According to RFC-1122, keep-alive is not required of implementa- tions because it could: (1) cause perfectly good connections to break during transient Internet failures; (2) consume unnecessary bandwidth ("if no one is using the connection, who cares if it is still good?"); and (3) cost money for an Internet path that charges for packets. Regarding this last point, we note that in addition the presence of dial-on-demand links in the route can greatly mag- nify the cost penalty of excess keepalives, potentially forcing a full-time connection on a link that would otherwise only be con- nected a few minutes a day. Paxson, Editor [Page 17] ID Known TCP Implementation Problems November 1997 If keepalive is provided the RFC states that the required inter- keepalive distance MUST default to no less than two hours. If it does not, the probability of connections breaking increases, the bandwidth used due to keepalives increases, and cost increases over paths which charge per packet. Relevant RFCs RFC 1122 specifies that the keep-alive mechanism may be provided. It also specifies the two hour minimum for the default interval between keepalive probes. Trace file demonstrating it Made using the Orchestra tool at the peer of the machine using keep-alive. Machine A was configured to use default settings for the keepalive timer. 11:36:32.910000 A > B: 3288354305:0 win 28672 datasz 4 SYN 11:36:32.930000 B > A: 896001:3288354306 win 4096 datasz 4 SYN ACK 11:36:32.950000 A > B: 3288354306:896002 win 28672 datasz 0 ACK 11:50:01.190000 A > B: 3288354305:896002 win 28672 datasz 0 ACK 11:50:01.210000 B > A: 896002:3288354306 win 4096 datasz 0 ACK 12:03:29.410000 A > B: 3288354305:896002 win 28672 datasz 0 ACK 12:03:29.430000 B > A: 896002:3288354306 win 4096 datasz 0 ACK 12:16:57.630000 A > B: 3288354305:896002 win 28672 datasz 0 ACK 12:16:57.650000 B > A: 896002:3288354306 win 4096 datasz 0 ACK 12:30:25.850000 A > B: 3288354305:896002 win 28672 datasz 0 ACK 12:30:25.870000 B > A: 896002:3288354306 win 4096 datasz 0 ACK 12:43:54.070000 A > B: 3288354305:896002 win 28672 datasz 0 ACK 12:43:54.090000 B > A: 896002:3288354306 win 4096 datasz 0 ACK The initial three packets are the SYN exchange for connection setup. About 13 minutes later, the keepalive timer fires because the connection is idle. The keepalive is acknowledged, and the timer fires again in about 13 more minutes. This behavior contin- ues indefinitely until the connection is closed, and is a violation of the specification. Trace file demonstrating correct behavior Made using the Orchestra tool at the peer of the machine using keep-alive. Machine A was configured to use default settings for Paxson, Editor [Page 18] ID Known TCP Implementation Problems November 1997 the keepalive timer. 17:37:20.500000 A > B: 34155521:0 win 4096 datasz 4 SYN 17:37:20.520000 B > A: 6272001:34155522 win 4096 datasz 4 SYN ACK 17:37:20.540000 A > B: 34155522:6272002 win 4096 datasz 0 ACK 19:37:25.430000 A > B: 34155521:6272002 win 4096 datasz 0 ACK 19:37:25.450000 B > A: 6272002:34155522 win 4096 datasz 0 ACK 21:37:30.560000 A > B: 34155521:6272002 win 4096 datasz 0 ACK 21:37:30.570000 B > A: 6272002:34155522 win 4096 datasz 0 ACK 23:37:35.580000 A > B: 34155521:6272002 win 4096 datasz 0 ACK 23:37:35.600000 B > A: 6272002:34155522 win 4096 datasz 0 ACK 01:37:40.620000 A > B: 34155521:6272002 win 4096 datasz 0 ACK 01:37:40.640000 B > A: 6272002:34155522 win 4096 datasz 0 ACK 03:37:45.590000 A > B: 34155521:6272002 win 4096 datasz 0 ACK 03:37:45.610000 B > A: 6272002:34155522 win 4096 datasz 0 ACK The initial three packets are the SYN exchange for connection setup. Just over two hours later, the keepalive timer fires because the connection is idle. The keepalive is acknowledged, and the timer fires again just over two hours later. This behavior continues indefinitely until the connection is closed. References This problem is documented in [Dawson97]. How to detect For implementations manifesting this problem, it shows up on a packet trace. If the connection is left idle, the keepalive probes will arrive closer together than the two hour minimum. 3.7. Stretch ACK Violation Classification Congestion Control/Performance Description To improve efficiency (both computer and network) a data receiver may refrain from sending an ACK for each incoming segment, Paxson, Editor [Page 19] ID Known TCP Implementation Problems November 1997 according to [RFC1122]. However, an ACK should not be delayed an inordinate amount of time. Specifically, ACKs MUST be sent for every second full-sized segment that arrives. If a second full- sized segment does not arrive within a given timeout (of no more than 0.5 seconds), an ACK must be transmitted, according to [RFC1122]. A TCP receiver which does not generate an ACK for every second full-sized segment exhibits a "Stretch ACK Violation". Significance TCP receivers exhibiting this behavior will cause TCP senders to generate burstier traffic, which can degrade performance in cong- ested environments. In addition, generating fewer ACKs increases the amount of time needed by the slow start algorithm to open the congestion window to an appropriate point, which diminishes perfor- mance in environments with large bandwidth-delay products. Finally, generating fewer ACKs may cause needless retransmission timeouts in lossy environments, as it increases the possibility that an entire window of ACKs is lost, forcing a retransmission timeout. Implications When not in loss recovery, every ACK received by a TCP sender triggers the transmission of new data segments. The burst size is determined by the number of previously unacknowledged segments each ACK covers. Therefore, a TCP receiver ACKing more than 2 segments at a time causes the sending TCP to generate a larger burst of traffic upon receipt of the ACK. This large burst of traffic can overwhelm an intervening gateway, leading to higher drop rates for both the connection and other connections passing through the cong- ested gateway. In addition, the TCP slow start algorithm increases the congestion window by 1 segment for each ACK received. Therefore, increasing the ACK interval (thus decreasing the rate at which ACKs are transmitted) increases the amount of time it takes slow start to increase the congestion window to an appropriate operating point, and the connection consequently suffers from reduced performance. This is especially true for connections using large windows. Relevant RFCs RFC 1122 outlines delayed ACKs as a recommended mechanism. Trace file demonstrating it Trace file taken using tcpdump at host B, the data receiver (and Paxson, Editor [Page 20] ID Known TCP Implementation Problems November 1997 ACK originator). The advertised window (which never changed) and timestamp options have been omitted for clarity, except for the first packet sent by A: 12:09:24.820187 A.1174 > B.3999: . 2049:3497(1448) ack 1 win 33580 [tos 0x8] 12:09:24.824147 A.1174 > B.3999: . 3497:4945(1448) ack 1 12:09:24.832034 A.1174 > B.3999: . 4945:6393(1448) ack 1 12:09:24.832222 B.3999 > A.1174: . ack 6393 12:09:24.934837 A.1174 > B.3999: . 6393:7841(1448) ack 1 12:09:24.942721 A.1174 > B.3999: . 7841:9289(1448) ack 1 12:09:24.950605 A.1174 > B.3999: . 9289:10737(1448) ack 1 12:09:24.950797 B.3999 > A.1174: . ack 10737 12:09:24.958488 A.1174 > B.3999: . 10737:12185(1448) ack 1 12:09:25.052330 A.1174 > B.3999: . 12185:13633(1448) ack 1 12:09:25.060216 A.1174 > B.3999: . 13633:15081(1448) ack 1 12:09:25.060405 B.3999 > A.1174: . ack 15081 This portion of the trace clearly shows that the receiver (host B) sends an ACK for every third full sized packet received. Further investigation of this implementation found that the cause of the increased ACK interval was the TCP options being used. The imple- mentation sent an ACK after it was holding 2*MSS worth of unack- nowledged data. In the above case, the MSS is 1460 bytes so the receiver transmits an ACK after it is holding at least 2920 bytes of unacknowledged data. However, the length of the TCP options being used [RFC1323] took 12 bytes away from the data portion of each packet. This produced packets containing 1448 bytes of data. But the additional bytes used by the options in the header were not taken into account when determining when to trigger an ACK. There- fore, it took 3 data segments before the data receiver was holding enough unacknowledged data (>= 2*MSS, or 2920 bytes in the above example) to transmit an ACK. Trace file demonstrating correct behavior Trace file taken using tcpdump at host B, the data receiver (and ACK originator), again with window and timestamp information omit- ted except for the first packet: 12:06:53.627320 A.1172 > B.3999: . 1449:2897(1448) ack 1 win 33580 [tos 0x8] 12:06:53.634773 A.1172 > B.3999: . 2897:4345(1448) ack 1 12:06:53.634961 B.3999 > A.1172: . ack 4345 12:06:53.737326 A.1172 > B.3999: . 4345:5793(1448) ack 1 12:06:53.744401 A.1172 > B.3999: . 5793:7241(1448) ack 1 12:06:53.744592 B.3999 > A.1172: . ack 7241 Paxson, Editor [Page 21] ID Known TCP Implementation Problems November 1997 12:06:53.752287 A.1172 > B.3999: . 7241:8689(1448) ack 1 12:06:53.847332 A.1172 > B.3999: . 8689:10137(1448) ack 1 12:06:53.847525 B.3999 > A.1172: . ack 10137 This trace shows the TCP receiver (host B) ack'ing every second full-sized packet, according to [RFC1122]. This is the same imple- mentation shown above, with slight modifications that allow the receiver to take the length of the options into account when decid- ing when to transmit an ACK. References This problem is documented in [Allman97] and [Paxson97]. How to detect Stretch ACK violations show up immediately in receiver-side packet traces of bulk transfers, as shown above. However, packet traces made on the sender side of the TCP connection may lead to ambigui- ties when diagnosing this problem due to the possibility of lost ACKs. 3.8. Failure to send FIN notification promptly Classification Performance Description When an application closes a connection, the corresponding TCP should send the FIN notification promptly to its peer (unless prevented by the congestion window). If a TCP implementation delays in sending the FIN notification, for example due to waiting until unacknowledged data has been acknowledged, then it is said to exhibit "Failure to send FIN notification promptly". Also, while not strictly required, FIN segments should include the PSH flag to ensure expedited delivery of any pending data at the receiver. Significance The greatest impact occurs for short-lived connections, since for these the additional time required to close the connection intro- duces the greatest relative delay. Paxson, Editor [Page 22] ID Known TCP Implementation Problems November 1997 The additional time can be significant in the common case of the sender waiting for an ACK that is delayed by the receiver. Implications Can diminish total throughput as seen at the application layer, because connection termination takes longer to complete. Relevant RFCs RFC 793 indicates that a receiver should treat an incoming FIN flag as implying the push function. Trace file demonstrating it Made using tcpdump (no losses reported). 10:04:38.68 A > B: S 1031850376:1031850376(0) win 4096 (DF) 10:04:38.71 B > A: S 596916473:596916473(0) ack 1031850377 win 8760 (DF) 10:04:38.73 A > B: . ack 1 win 4096 (DF) 10:04:41.98 A > B: P 1:4(3) ack 1 win 4096 (DF) 10:04:42.15 B > A: . ack 4 win 8757 (DF) 10:04:42.23 A > B: P 4:7(3) ack 1 win 4096 (DF) 10:04:42.25 B > A: P 1:11(10) ack 7 win 8754 (DF) 10:04:42.32 A > B: . ack 11 win 4096 (DF) 10:04:42.33 B > A: P 11:51(40) ack 7 win 8754 (DF) 10:04:42.51 A > B: . ack 51 win 4096 (DF) 10:04:42.53 B > A: F 51:51(0) ack 7 win 8754 (DF) 10:04:42.56 A > B: FP 7:7(0) ack 52 win 4096 (DF) 10:04:42.58 B > A: . ack 8 win 8754 (DF) Machine B in the trace above does not send out a FIN notification promptly if there is any data outstanding. It instead waits for all unacknowledged data to be acknowledged before sending the FIN segment. The connection was closed at 10:04.42.33 after requesting 40 bytes to be sent. However, the FIN notification isn't sent until 10:04.42.51, after the (delayed) acknowledgement of the 40 bytes of data. Trace file demonstrating correct behavior Made using tcpdump (no losses reported). 10:27:53.85 C > D: S 419744533:419744533(0) win 4096 (DF) 10:27:53.92 D > C: S 10082297:10082297(0) ack 419744534 Paxson, Editor [Page 23] ID Known TCP Implementation Problems November 1997 win 8760 (DF) 10:27:53.95 C > D: . ack 1 win 4096 (DF) 10:27:54.42 C > D: P 1:4(3) ack 1 win 4096 (DF) 10:27:54.62 D > C: . ack 4 win 8757 (DF) 10:27:54.76 C > D: P 4:7(3) ack 1 win 4096 (DF) 10:27:54.89 D > C: P 1:11(10) ack 7 win 8754 (DF) 10:27:54.90 D > C: FP 11:51(40) ack7 win 8754 (DF) 10:27:54.92 C > D: . ack 52 win 4096 (DF) 10:27:55.01 C > D: FP 7:7(0) ack 52 win 4096 (DF) 10:27:55.09 D > C: . ack 8 win 8754 (DF) References This problem is documented in [Dawson97]. How to detect For implementations manifesting this problem, it shows up on a packet trace. If the connection is left idle, the keepalive probes will arrive closer together than the two hour minimum. Here, Machine D sends a FIN with 40 bytes of data even before the original 10 octets have been acknowledged. This is correct behavoir as it provides for the highest performance. 3.9. Failure to send a RST after Half Duplex Close Classification Resource management [[ editor's comment: this is a new classification category ]] Description RFC 1122 4.2.2.13 states that a TCP SHOULD send a RST if data is received after "half duplex close", i.e. if it cannot be delivered to the application. A TCP that fails to do so is said to exhibit "Failure to send a RST after Half Duplex Close". Significance Potentially serious for TCP endpoints that manage large numbers of connections, due to exhaustion of memory available for managing connection state. Paxson, Editor [Page 24] ID Known TCP Implementation Problems November 1997 Implications Failure to send the RST can lead to permanently hung TCP connec- tions. This problem has been demonstrated when HTTP clients abort connections, common when users move on to a new page before the current page has finished downloading. The HTTP client closes by transmitting a FIN while the server is transmitting images, text, etc. The server TCP receives the FIN, but its application does not close the connection until all data has been queued for transmission. Since the server will not transmit a FIN until all the preceding data has been transmitted, deadlock results if the client TCP does not consume the pending data or tear down the con- nection: the window decreases to zero, since the client cannot pass the data to the application, and the server sends probe segments. The client acknowledges the probe segments with a zero window. As mandated in RFC1122 4.2.2.17, the probe segments are transmitted forever. Server connection state remains in CLOSE_WAIT, and even- tually server processes are exhausted. Note that there are two bugs. First, probe segments should be ignored if the window can never subsequently increase. Second, a RST should be sent when data is received after half duplex close. Fixing the first bug, but not the second, results in the probe seg- ments eventually timing out the connection, but the server remains in CLOSE_WAIT for a significant and unnecessary period. Relevant RFCs RFC 1122 sections 4.2.2.13 and 4.2.2.17. Trace file demonstrating it Made using an unknown network analyzer. No drop information avail- able. client.1391 > server.8080: S 0:1(0) ack: 0 win: 2000 server.8080 > client.1391: SA 8c01:8c02(0) ack: 1 win: 8000 client.1391 > server.8080: PA client.1391 > server.8080: PA 1:1c2(1c1) ack: 8c02 win: 2000 server.8080 > client.1391: [DF] PA 8c02:8cde(dc) ack: 1c2 win: 8000 server.8080 > client.1391: [DF] A 8cde:9292(5b4) ack: 1c2 win: 8000 server.8080 > client.1391: [DF] A 9292:9846(5b4) ack: 1c2 win: 8000 server.8080 > client.1391: [DF] A 9846:9dfa(5b4) ack: 1c2 win: 8000 client.1391 > server.8080: PA server.8080 > client.1391: [DF] A 9dfa:a3ae(5b4) ack: 1c2 win: 8000 server.8080 > client.1391: [DF] A a3ae:a962(5b4) ack: 1c2 win: 8000 server.8080 > client.1391: [DF] A a962:af16(5b4) ack: 1c2 win: 8000 server.8080 > client.1391: [DF] A af16:b4ca(5b4) ack: 1c2 win: 8000 client.1391 > server.8080: PA Paxson, Editor [Page 25] ID Known TCP Implementation Problems November 1997 server.8080 > client.1391: [DF] A b4ca:ba7e(5b4) ack: 1c2 win: 8000 server.8080 > client.1391: [DF] A b4ca:ba7e(5b4) ack: 1c2 win: 8000 client.1391 > server.8080: PA server.8080 > client.1391: [DF] A ba7e:bdfa(37c) ack: 1c2 win: 8000 client.1391 > server.8080: PA server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c2 win: 8000 client.1391 > server.8080: PA [ HTTP client aborts and enters FIN_WAIT_1 ] client.1391 > server.8080: FPA [ server ACKs the FIN and enters CLOSE_WAIT ] server.8080 > client.1391: [DF] A [ client enters FIN_WAIT_2 ] server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c3 win: 8000 [ server continues to try to send its data ] client.1391 > server.8080: PA < window = 0 > server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c3 win: 8000 client.1391 > server.8080: PA < window = 0 > server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c3 win: 8000 client.1391 > server.8080: PA < window = 0 > server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c3 win: 8000 client.1391 > server.8080: PA < window = 0 > server.8080 > client.1391: [DF] A bdfa:bdfb(1) ack: 1c3 win: 8000 client.1391 > server.8080: PA < window = 0 > [ ... repeat ad exhaustium ... ] Trace file demonstrating correct behavior Made using an unknown network analyzer. No drop information avail- able. client > server D=80 S=59500 Syn Seq=337 Len=0 Win=8760 server > client D=59500 S=80 Syn Ack=338 Seq=80153 Len=0 Win=8760 client > server D=80 S=59500 Ack=80154 Seq=338 Len=0 Win=8760 [ ... normal data omitted ... ] client > server D=80 S=59500 Ack=14559 Seq=596 Len=0 Win=8760 server > client D=59500 S=80 Ack=596 Seq=114559 Len=1460 Win=8760 Paxson, Editor [Page 26] ID Known TCP Implementation Problems November 1997 [ client closes connection ] client > server D=80 S=59500 Fin Seq=596 Len=0 Win=8760 server > client D=59500 S=80 Ack=597 Seq=116019 Len=1460 Win=8760 [ client sends RST (RFC1122 4.2.2.13) ] client > server D=80 S=59500 Rst Seq=597 Len=0 Win=0 server > client D=59500 S=80 Ack=597 Seq=117479 Len=1460 Win=8760 [[ editor's comment: why doesn't the RST result in the connection being torn down at this point ??]] client > server D=80 S=59500 Rst Seq=597 Len=0 Win=0 server > client D=59500 S=80 Ack=597 Seq=118939 Len=1460 Win=8760 client > server D=80 S=59500 Rst Seq=597 Len=0 Win=0 server > client D=59500 S=80 Ack=597 Seq=120399 Len=892 Win=8760 client > server D=80 S=59500 Rst Seq=597 Len=0 Win=0 server > client D=59500 S=80 Ack=597 Seq=121291 Len=1460 Win=8760 client > server D=80 S=59500 Rst Seq=597 Len=0 Win=0 How to detect The problem can be detected by inspecting packet traces of a large, interrupted bulk transfer. 4. Security Considerations This version of this memo does not discuss any security-related implementation problems. Futures versions most likely will, so secu- rity considerations will require revisiting. 5. Acknowledgements Thanks to numerous correspondents on the tcp-impl mailing list for their input: Steve Alexander, Mark Allman, Larry Backman, Jerry Chu, Alan Cox, Kevin Fall, Richard Fox, Jim Gettys, Rick Jones, Allison Mankin, Neal McBurnett, Perry Metzger, der Mouse, Thomas Narten, Andras Olah, Steve Parker, Francesco Potorti`, Luigi Rizzo, Allyn Romanow, Jeff Semke, Al Smith, Jerry Toporek, Joe Touch, and Curtis Villamizar. Thanks also to Josh Cohen for the traces documenting the "Failure to send a RST after Half Duplex Close" problem. Paxson, Editor [Page 27] ID Known TCP Implementation Problems November 1997 6. References [Allman97] Mark Allman, "Fixing Two BSD TCP Bugs," Technical Report CR-204151, NASA Lewis Research Center, October 1997. http://gigahertz.lerc.nasa.gov/~mallman/papers/bug.ps [RFC1122] R. Braden, Editor, "Requirements for Internet Hosts -- Communica- tion Layers," Oct. 1989. [Dawson97] S. Dawson, F. Jahanian, and T. Mitton, "Experiments on Six Commer- cial TCP Implementations Using a Software Fault Injection Tool," to appear in Software Practice & Experience, 1997. A technical report version of this paper can be obtained at ftp://rtcl.eecs.umich.edu/outgoing/sdawson/CSE-TR-298-96.ps.gz. [Jacobson88] V. Jacobson, "Congestion Avoidance and Control," Proc. SIGCOMM '88. ftp://ftp.ee.lbl.gov/papers/congavoid.ps.Z [RFC896] J. Nagle, "Congestion Control in IP/TCP Internetworks," Jan. 1984. [Paxson97] V. Paxson, "Automated Packet Trace Analysis of TCP Implementa- tions," Proc. SIGCOMM '97, available from ftp://ftp.ee.lbl.gov/papers/vp-tcpanaly-sigcomm97.ps.Z. [RFC793] J. Postel, Editor, "Transmission Control Protocol," Sep. 1981. [RFC2001] W. Stevens, "TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery Algorithms," Jan. 1997. 7. Authors' Addresses Vern Paxson Network Research Group Lawrence Berkeley National Laboratory Berkeley, CA 94720 USA Phone: +1 510/486-7504 Paxson, Editor [Page 28] ID Known TCP Implementation Problems November 1997 Mark Allman NASA Lewis Research Center/Sterling Software 21000 Brookpark Road MS 54-2 Cleveland, OH 44135 USA Phone: +1 216/433-6586 Scott Dawson Real-Time Computing Laboratory EECS Building University of Michigan Ann Arbor, MI 48109-2122 USA Phone: +1 313/763-5363 Ian Heavens Spider Software Ltd. 8 John's Place, Leith Edinburgh EH6 7EL UK Phone: +44 131/475-7015 Bernie Volz Process Software Corporation 959 Concord Street Framingham, MA 01701 USA Phone: +1 508/879-6994 Paxson, Editor [Page 29]