HTTP/1.1 200 OK Date: Tue, 09 Apr 2002 12:10:19 GMT Server: Apache/1.3.20 (Unix) Last-Modified: Tue, 17 Mar 1998 16:34:00 GMT ETag: "3ddc17-f1ae-350ea5f8" Accept-Ranges: bytes Content-Length: 61870 Connection: close Content-Type: text/plain Network Working Group John C. C. White Internet-Draft The MITRE Corporation Obsoletes: RFC998 April 1997 Category: Proposed Standard Expires in six months NETBLT (Network Block Transfer Protocol) draft-white-protocol-stack-00.txt Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as ``work in progress.'' To learn the current status of any Internet-Draft, please check the ``1id-abstracts.txt'' listing contained in the Internet- Drafts Shadow Directories on ftp.is.co.za (Africa), nic.nordu.net (Europe), munnari.oz.au (Pacific Rim), ds.internic.net (US East Coast), or ftp.isi.edu (US West Coast). Abstract The NETBLT protocol [RFC998] was designed as an experimental transport layer protocol, intended for moving large quantities of data across a wide variety of networks. It provides reliable bulk transfer with an end-to-end flow-control mechanism meant to deal with network congestion by throttling the rate at which data is inserted into the network. However, experiments with NETBLT across shared links revealed problems with fairness; traffic from one connection could hog most of a link's bandwidth, and there seems to be no way to prevent this under the current rate-control scheme, so further application of NETBLT was not pursued by its original developers. However, NETBLT has a number of characteristics which make it very attractive for use across noisy, long-delay, slow-turnaround, or asymmetric communications links. Such links are common in military usage, and may become more widespread with the development of mobile computing. NETBLT's attractive characteristics include selective retransmission of lost packets, potentially large transmission windows, and control of transmission from the receiving, rather than the sending side; the latter makes NETBLT relatively insensitive to White [Page 1] INTERNET DRAFT NETBLT April 1997 network delays. NETBLT, with minor modifications, was adopted as the transport layer of the military standard TACO2 (Tactical Communications Protocol 2) [MIL- STD], which is intended for the transmission of images and other bulk data across links that cannot support the usual TCP/IP operation. This document describes NETBLT as it is currently used, and is intended partly to encourage consideration of NETBLT in other difficult communications environments. 1. Document History The military standard definition of NETBLT was developed from RFC998 by expunging most of the tutorial information and translating the remainder into the language required by military standards. This document was then prepared from the military standard; as a result, there may be some unnecessary rearrangements and rewordings. In any case, most of the protocol remains as designed by the original developers. Modifications have been made to simplify protocol operation and to extend its capability. 2. NETBLT Overview The bulk data transfer protocol NETBLT works by opening a connection between a sender and a receiver, transferring a message in one or more buffers, and closing the connection. Each buffer is transferred as a sequence of packets; the interaction between sender and receiver is primarily on a per-buffer basis. This section provides an overview of NETBLT; further explanation and detailed requirements are found in the following sections. The material here assumes the existence of a full- duplex connection between sender and receiver, such that information can be transferred in both directions more or less concurrently. Changes for half-duplex and simplex operation are provided later. Specific packet types are identified in the following sections by upper-case names (e.g., DATA packets), in contrast with packet functions (e.g., keepalive packets) which are accomplished by more than one packet type. 2.1 Single-buffer operation. In its simplest form, a NETBLT transfer works as follows: first, a connection is opened between a sending and a receiving NETBLT. That step includes negotiation of various transmission parameters. The sending client loads a buffer of data and passes it to the NETBLT layer to be transferred. NETBLT breaks the buffer up into packets and sends these packets across the network via datagrams. The receiving NETBLT loads these packets into a matching buffer. When White [Page 2] INTERNET DRAFT NETBLT April 1997 the last packet in the buffer should have arrived at the receiver, the receiving NETBLT checks whether all packets in that buffer have been received correctly. If some packets have not been received correctly, the receiving NETBLT requests that they be resent. When the buffer has been completely received, the receiving NETBLT passes it to the receiving client, and confirms its receipt to the sender. When a new buffer is ready to receive more data, the receiving NETBLT notifies the sender that the new buffer is ready, and the sender prepares and sends the next buffer in the same manner. This continues until all the data has been sent; at that time the sender notifies the receiver that the transmission has been completed. The two sides then close the connection. 2.2 Multiple-buffer operation. As described above the NETBLT protocol is "lock-step". Action halts after a buffer is transmitted, and begins again after confirmation is received from the receiver of data. NETBLT provides for multiple buffering, so that the sending NETBLT can transmit new buffers while waiting for confirmation of earlier buffers from the receiving NETBLT. 2.3 Buffers and packets. The data to be transmitted is broken up into buffers by the sending client. All buffers are the same size, except for the last buffer. During connection setup, the sending and receiving NETBLTs negotiate the buffer size. Buffer sizes are in bytes; data is placed in buffers on byte boundaries. Each buffer is broken down by NETBLT into a sequence of DATA packets terminated by an LDATA packet. DATA packet size is negotiated between the sending and receiving NETBLTs during connection setup. All DATA packets are the same size. DATA and LDATA packets are identical in format except for the packet type. 2.4 Flow control. NETBLT uses two strategies for flow control, one at the client level and one internal. 2.4.1 Client level flow control. The sending and receiving NETBLTs transmit data in buffers; client flow control is therefore by buffer. Before a buffer can be transmitted, NETBLT confirms that both clients have set up matching buffers, that one is ready to send data, and that the other is ready to receive data. Either client can therefore control the flow of data White [Page 3] INTERNET DRAFT NETBLT April 1997 by not providing a new buffer. Clients cannot stop a buffer transfer once it is in progress, except by aborting the entire transfer. 2.4.2 Internal flow control. The internal flow control mechanism for NETBLT is rate control. The transmission rate is negotiated by the sending and receiving NETBLTs during connection setup and after each buffer transmission. The sender uses timers to maintain the negotiated rate, by controlling the time to transmit groups of packets. The sender transmits a burst of packets over the negotiated time interval, and sends another burst in the next interval. NETBLT's rate control therefore has two parts, a burst size and a burst interval, with (burst interval)/(burst size) equal to the average transmission time per packet. A burst interval value of zero means that internal flow control is turned off, so that only client level flow control is in effect. In this case, the sending NETBLT will transmit packets without regard for the rate control mechanism. All NETBLT flow control parameters (packet size, buffer size, number of buffers outstanding, burst size, and burst interval) are negotiated during connection setup. The negotiation process is the same for all parameters. The client initiating the connection (the active side) sends a value for each parameter in its OPEN packet. The other client (the passive side) will compare these values with the highest- performance values it can support. The passive side can modify any of the parameters, but only by making them more restrictive; i. e., smaller packet size, smaller buffer size, fewer buffers, smaller burst size, and larger burst interval. The (possibly modified) parameters are sent back to the active side in the RESPONSE packet. The burst size and burst interval may also be re-negotiated after each buffer transmission to adjust the transfer rate according to the performance observed from transferring the previous buffer. The receiving end sends burst size and burst interval values in its OK messages (which acknowledge successful receipt of a buffer) and in its RESEND messages (which request retransmission of specific packets). The sender will compare these values with the values it can support. Again, it may modify either of these parameters, but only by making them more restrictive. The modified parameters will then be communicated to the receiver in DATA, LDATA, or NULL-ACK packets. 2.5 Checksumming. NETBLT automatically checksums each packet header and, optionally, White [Page 4] INTERNET DRAFT NETBLT April 1997 the data portion of each DATA and LDATA packet. The checksum value is the bitwise negation of the ones-complement sum of the 16-bit words being checksummed. If a packet to be transferred has an odd number of bytes, it is padded with a final null byte (binary 0's) to make the number of bytes even for the purpose of checksum calculation. The extra byte is not transmitted as part of the packet, but its existence is assumed at the receiving end for checksum verification. 3. NETBLT detailed operation Each NETBLT transfer has three stages: connection setup, data transfer, and connection close. The stages are described in detail below, along with methods for insuring that each stage completes reliably. State diagrams are provided at the end of the description for each stage of the transfer. Each transition in the diagrams is labelled with the event that causes the transition, and optionally, in parentheses, actions that occur at the time of the transition. 3.1 Connection setup. A NETBLT connection is set up by an exchange of two packets between the active NETBLT and the passive NETBLT. The active end sends an OPEN packet; the passive end acknowledges the OPEN packet in one of two ways: it either sends a REFUSED packet, indicating that the connection cannot be completed for some reason, or it completes the connection setup by sending a RESPONSE packet. After a successful connection setup, the transfer can begin. Figure 1 illustrates the opening of a connection by the active end, and figure 2 shows the same process for the passive end. Each side of the connection transmits its death-timeout value in seconds in the OPEN or the RESPONSE packet. The death-timeout value is used to determine the frequency with which to send keepalive packets during idle periods of an opened connection (death timers and keepalive packets are discussed later). The sending NETBLT specifies a passive client through a client- specific "well-known" 16 bit logical port number on which the receiving end listens. The sending client identifies itself through a 32 bit Internet address and a unique 16 bit port number. An unstructured, variable-length client message field is provided in OPEN and RESPONSE packets for any client-specific information that may be required. In addition, a "reason for refusal" field is provided in REFUSED packets. Recovery for lost OPEN and RESPONSE packets is provided by the use of timers. The sending end sets a timer when it sends an OPEN packet. White [Page 5] INTERNET DRAFT NETBLT April 1997 When the timer expires, another OPEN packet is sent, until some predetermined maximum number (at least five) of OPEN packets have been sent. The timer is cleared upon receipt of a RESPONSE or REFUSED packet. To prevent duplication of OPEN and RESPONSE packets, the OPEN packet contains a 32 bit connection unique ID (UID) that must be returned in the RESPONSE packet. This unique ID prevents the initiator from confusing the response to the current request with the response to an earlier connection request (there can only be one connection open between any pair of logical ports). Any OPEN or RESPONSE packet with a port pair matching that of an open connection will have its unique ID checked. If the unique ID of the packet matches the unique ID of the connection, then the packet type is checked. If it is a RESPONSE packet, it is treated as a duplicate and ignored. If it is an OPEN packet, the passive NETBLT will send another RESPONSE (on the assumption that a previous RESPONSE packet was sent and lost, causing the initiating NETBLT to retransmit its OPEN packet). A non-matching unique ID is treated as an attempt to open a second connection between the port pair and is rejected by sending a REFUSED message. +------------+ | |--------<-----------------------------+ | Inactive |-->-+ | | | | | +------------+ | | ^ Connect request from client | | (Send OPEN, start Open Timer) | REFUSED received | <=max # OPENs sent | | | +------------+ | | | Opening |-<--+---<---------+ ^ | | | | | |->-+ -------+-------------------+ RESPONSE received (clear Open Timer) | +------------+ | | | +-->----| Connected | | | +------------+ Figure 1. Active side open state diagram White [Page 6] INTERNET DRAFT NETBLT April 1997 +------------+ | |--->----------+ +-<--| Inactive | Unacceptable OPEN received | | | (send REFUSED) | | |---<----------+ | +------------+ Acceptable OPEN received (send RESPONSE) +-------+--------->-----------+ | +------------+ | | | | | |--->--+ Acceptable OPEN Unacceptable OPEN +->--| Connected | received received | | (send RESPONSE) (send REFUSED) | |--<---+ | | +------------+ +-------+---------<-----------+ Figure 2. Passive side open state diagram 3.2 Data transfer The simplest full-duplex mode of data transfer proceeds as follows. The sending client sets up a buffer full of data. The receiving NETBLT sends a GO message inside a CONTROL packet to the sender, signifying that it too has set up a buffer and is ready to receive data. Once the GO message is received, the sender transmits the buffer as a series of DATA packets followed by an LDATA packet. When the last packet in the buffer should have been received (as determined by a timer), if any packets in the buffer have not been received the receiver sends a RESEND message inside a CONTROL packet containing a list of packets that were not received. The sender will resend these packets. This process continues until there are no missing packets. At that time the receiver sends an OK message inside a CONTROL packet, sets up another buffer to receive data, and sends another GO message. The sender, having received the OK message, will set up another buffer, wait for the GO message, and repeat the process. A more efficient full-duplex transfer mode uses multiple buffering, in which the sender and receiver allocate and transfer buffers in a manner that allows error recovery or successful transmission confirmation of previous buffers to be concurrent with transmission of the current buffer. During the connection setup phase, one of the negotiated parameters is the number of concurrent buffers permitted during the transfer. If there is more than one buffer available, transfer of the next buffer will start right after the current buffer finishes, and the receiver is ready to receive the buffer. The receiver signals that it is ready for the next buffer by sending a GO message. This is illustrated in the following example: Assume the sender has available two buffers A and B in a multiple-buffer White [Page 7] INTERNET DRAFT NETBLT April 1997 transfer, with A preceding B. When A has been transferred and the sending NETBLT is waiting for either an OK or a RESEND message for it, the sending NETBLT can start sending B immediately. If the receiver of data sends an OK for A, all is well; if it sends a RESEND, the missing packets specified in the RESEND message are retransmitted. In the multiple-buffer transfer mode, all packets to be sent are ordered by buffer number (lowest number first). Since buffer numbers increase monotonically, packets from an earlier buffer will precede packets from a later buffer. 3.2.1 Control Messages. NETBLT uses a single long-lived control packet; the packet is treated like a FIFO queue, with new control messages added on at the end and acknowledged control messages removed from the front. The implementation places control messages in the control packet and transmits the entire control packet, consisting of any unacknowledged control messages plus new messages just added. Since control packet transmissions are fairly frequent, unacknowledged messages may be transmitted several times before they are finally acknowledged. The receiver may send zero or more control messages (OK, GO, or RESEND) within a single CONTROL packet. In order to limit the size of the control packet, it is permissible to send fewer than the full set of unacknowledged control messages in a control packet; it is however required that the control messages in a control packet be consecutive, starting with the lowest-numbered unacknowledged control message. Each control message includes a sequence number, which starts at one and increases by one for each control message generated. The sending NETBLT checks the sequence number of every incoming control and stores the highest sequence number below which all other sequence numbers have been received (in following paragraphs this is called the high-acknowledged- sequence-number). It returns this number in every packet flowing back to the receiver. The receiver removes control messages with sequence numbers less than or equal to the high-acknowledged-sequence-number from the control packet. Whenever the receiver sends a control packet, it starts a control timer. When the control timer expires, the receiving NETBLT will resend the control packet and reset the timer. The receiving NETBLT will continue to resend control packets in response to control timer expiration until either the control timer is cleared or the receiving NETBLT's death timer (described later) expires (at which time it will shut down the connection). The control timer may have as its initial value an arbitrary number. Subsequent control timer values are based on the network round-trip transit time (the time between sending the control packet and receiving the acknowledgment of all messages in White [Page 8] INTERNET DRAFT NETBLT April 1997 the control packet) plus a variance factor. The timer value is regularly updated, based on a smoothed average of collected round- trip transit times. The control timer is set to the keepalive value when a packet is received from the sender with high-acknowledged- sequence-number equal to the highest sequence number in the control packet most recently sent. The exact algorithm for control timer calculation is not mandated. The suggested algorithm, similar to that [Jacobsen] now used in TCP, is as follows: Initially, the round trip time is set to one-half the keepalive value and the deviation is set to zero. When a control packet is sent, if the round-trip-delay (RTD) timer is not running, its highest sequence number is stored and the RTD timer is started. When acknowledgement of that sequence number is received, the RTD timer value is used to calculate a new RTD estimate. However, if a control packet is retransmitted, the RTD timer is zeroed, its value is not used (to avoid ambiguous RTD measurements), and the estimated RTD is increased by 1/2 of its value (to allow recovery from too low an RTD estimate). To calculate the new RTD estimate, New smoothed round trip time = (1-a) * old smoothed round trip time + a * latest round trip measurement New deviation = (1-b) * old deviation + b * |latest round trip measurement - old smoothed round trip time| where a = 1/8 and b = 1/4, allowing computations to be done with add and shift operations. The control timer is set equal to the new smoothed round trip time plus twice the new deviation, or to the keepalive value, whichever is less, if the control packet is not empty. If the control packet is empty, the control timer is set to the keepalive value. The sending NETBLT, upon receiving a previously unseen control message, will either set up a new buffer (upon receipt of an OK message for a previous buffer), mark data for resending (upon receipt of a RESEND message), or prepare a buffer for sending (upon receipt of a GO message). If the sending NETBLT is not in a position to send data, it sends a NULL-ACK packet, which contains its high- acknowledged-sequence- number (this permits the receiving NETBLT to resend any outstanding control messages or to clear its control timer), and waits until it can send more data. 3.2.2 Send buffer state sequence. The state sequence for a sending buffer is as follows: when a GO White [Page 9] INTERNET DRAFT NETBLT April 1997 message for the buffer is received, the buffer is created, filled with data, and placed in a SENDING state. When an OK for that buffer has been received, it goes into a SENT state and may be released. Figure 3 illustrates this sequence. +-----------+ | | GO for buffer n received --->---| Ready |-->-------+ (create and fill buffer n) | | | +-----------+ | Start sending buffer n (set last-buffer-touched to n) +-----------+ | | | | +------<------| Sending |--<-------+ | | | OK for buffer n received +-----------+ | +-----------+ +------>------| | | Sent |-->- (remove buffer n) | | +-----------+ Figure 3. Sending buffer state diagram 3.2.3 Receive buffer state sequence. The state sequence for a receiving buffer is more complicated. Assume existence of a Buffer A. When a control message for Buffer A is sent, the buffer will move into state ACK-WAIT (it is waiting for acknowledgement of the control message). As soon as the control message has been acknowledged, Buffer A will move from the ACK-WAIT state into the ACKED state (it is now waiting for DATA packets to arrive). At this point, the control message is removed from the control packet. Buffer A will stay in the ACKED state until a DATA, LDATA, or NULL-ACK packet arrives with its "Last Buffer Touched" number greater than or equal to Buffer A's number. At this time, Buffer A's data timer is set to the time expected for the remaining packets in the buffer to be received plus a variance, and Buffer A will move to the RECEIVING state. (Note: This mechanism is different from, and simpler than, the "loose/tight" timer mechanism described in RFC 998). When all DATA packets for A have been received, it will move from the RECEIVING state to the RECEIVED state and may be passed to the receiving client. Had any packets been missing, Buffer A's data timer would have expired; in that case, Buffer A will move into the ACK-WAIT state after sending a RESEND message. The sending of a RESEND message will cause the data timers of all buffers currently in White [Page 10] INTERNET DRAFT NETBLT April 1997 the RECEIVING state to be recalculated, since the presence of re-sent packets will change the expected completion time for later buffers. The state progression would then move as in the above example. Figure 4 illustrates this sequence. < maximum # buffers exist & last buffer not detected --->---------------------+ (create buffer n; send GO n) | +--------------+ | | +--<--ACK for buffer n GO or --<--| ACK-wait | | RESEND message received | | | +--------------+ +------------+ | | | ^ | ACKed | | | |-->---+ RESEND sent +------------+ | (set all receiving | buffer data timers) DATA/LDATA/NULL-ACK with | last-buffer-touched >= n received | (set buffer n data timer) +-------------+ | | | | | Resend-wait | | | | | +-------------+ +-------------+ | | | |---<--+ | | Receiving | | | |-->-- Buffer n data timeout & ->--+ +-------------+ buffer n not complete | (add RESEND to control packet) | Buffer n complete +------------+ | | | +---->--------| Received |--->--- Buffer n flushed | | (remove buffer n) +------------+ Figure 4. Receiving buffer state diagram 3.2.4 Data Timers. NETBLT solves the problem of DATA and LDATA packet loss by using a data timer for each buffer at the receiving end. The simplest data timer model has a data timer set when a buffer is ready to be received; if the data timer expires, the receiving NETBLT will send a White [Page 11] INTERNET DRAFT NETBLT April 1997 RESEND message requesting all missing DATA/LDATA packets in the buffer. When all packets have been received, the timer is cleared. Data timer values are based on the amount of time taken to transfer a buffer plus a variance factor. The exact algorithm for data timer estimation is not mandated. The suggested algorithm is to compute the number of packets expected before the buffer is complete, multiply that by the time required to transmit a packet, and add a variance. The receiver uses both the minimum time per packet established by the burst size/burst interval, and the measured time per packet with mean deviation, to establish two estimates of the expected time per packet. These two estimates are then used to calculate data timer settings for each buffer, and the maximum value is used for that buffer's data timer. This combination allows the demand placed on the net capacity by a given transfer to be limited, while still avoiding unnecessary retransmissions if the available net capacity is less than that requested. The algorithm is as follows: The average time-per-packet A and the mean deviation D are initialized. The first-packet time and packet number are stored when the first packet from a given buffer arrives, and the latest-packet time and packet number are stored or updated as each packet from that same buffer arrives. When a packet from a different buffer or a NULL-ACK arrives, or a RESEND is sent for the given buffer, the difference between the first-packet time and the latest-packet time is divided by the difference between the first-packet number and the latest-packet number, to provide a sample time-per-packet. Then, New smoothed time-per-packet = (1-a) * old smoothed time-per-packet + a * latest time-per-packet measurement New time-per-packet deviation = (1-b) * old time-per-packet deviation + b * |latest time-per-packet measurement - old smoothed time-per- packet | where a = 1/8 and b = 1/4, allowing computations to be done with add and shift operations, as with the control timer. When the data timer for a buffer must be set, two times are calculated using N, the number of packets which must be received before the buffer is completely filled: T1 = 1.25 * N * smoothed time-per-packet + 2 * time-per-packet deviation T2 = 1.5 * N * burst interval / burst size. White [Page 12] INTERNET DRAFT NETBLT April 1997 The data timer is then set to the maximum of (T1, T2). 3.2.5 Death timers. At connection startup, each NETBLT sends its death value to the other end in either the OPEN or the RESPONSE packet. As soon as the connection is opened, each end sets its death timer to its chosen value; this timer is reset every time a packet is received. When a NETBLT's death timer expires, it will close the connection without sending any more packets. 3.2.6 Keepalive packets. NETBLT includes a keepalive function, which sends packets repeatedly at fixed intervals when a NETBLT has no other reason to send packets. The sender uses NULL-ACKs as keepalive packets; the receiver uses empty CONTROL packets. If the sending NETBLT is not ready to send upon receipt of a control packet, it sends a single NULL-ACK packet to clear any outstanding control timers at the receiving end. Each end uses the other end's death-timeout value to compute a frequency with which to send keepalive packets. The keepalive frequency should be high enough that several keepalive packets can be lost before the other end's death timer expires; suitable values are the sender's death timer value divided by seven for the receiver, and the receiver's death timer value divided by eight for the sender (keepalive intervals should be different to avoid repeated collisions in half-duplex operations). 3.3 Connection termination. There are four conditions under which a connection is terminated: a successful transfer, a client quit, a NETBLT abort, and a death timer timeout. 3.3.1 Successful transfer. After a successful data transfer, NETBLT closes the connection. When the sender is transmitting the last buffer of data, it sets a "last-buffer" flag on every DATA packet in the buffer. The receiver will recognize that the transfer has completed successfully when all of the following are true: (1) it has received DATA packets with a "last- buffer" flag set, (2) all its control messages have been acknowledged, and (3) it has no outstanding buffers with missing packets. The DONE packet is transmitted when the receiver recognizes that the transfer has been successfully completed. At that point, the receiver closes its half of the connection. Figure 5 illustrates this sequence. White [Page 13] INTERNET DRAFT NETBLT April 1997 +-------------+ +------------+ | | | | | Connected |-->-- Last buffer received & --->---| Inactive | | | all buffers disposed of & | | +-------------+ all messages acked +------------+ (send DONE) Figure 5. Receiver successful close state diagram The sender will recognize that the transfer has completed when the following are true: (1) it has transmitted DATA packets with a "last- buffer" flag set and (2) it has received OK messages for all its buffers. At that point, it will "dally" for a predetermined period of time before closing its half of the connection. If the NULL-ACK packet acknowledging the receiver's last OK message was lost, the receiver has time to retransmit the OK message, receive a new NULL- ACK, and recognize a successful transfer. The dally timer value is based on the receiver's control timer value; it should be long enough to allow the receiver's control timer to expire so that the OK message can be re-sent. A value of twice the receiver's control timer value is suitable for the dally timer. When the sender receives a DONE packet, it clears its dally timer and close its half of the connection. Figure 6 illustrates this sequence. +-----------+ | | | Connected |--->---+ | | | +-----------+ All buffers flushed (send NULL-ACK; set dally timer) +-----------+ | | |---<---+-------<-----------------+ | Dallying | | | |----->--- OK message received ->-+ +-----------+ (send NULL-ACK; | set dally timer) +----------+ | | | +-->-- DONE received or dally timeout ->--| Inactive | | | +----------+ Figure 6. Sender successful close state diagram 3.3.2 Client QUIT. White [Page 14] INTERNET DRAFT NETBLT April 1997 During a NETBLT transfer, one client may send a QUIT packet to the other, to terminate the transfer prematurely. The NETBLT receiving the QUIT packet will take no action other than immediately notifying its client and transmitting a QUITACK packet. The QUIT sender will time out and retransmit until a QUITACK has been received or its death timer expires. The sender of the QUITACK will dally before quitting, so that it can respond to a retransmitted QUIT. Figure 7 illustrates this sequence. +-----------+ | | +---| Connected |--->--- Quit request from client ----->---+ | | | (send QUIT; set quit timer) | | +-----------+ +-----------+ | +- Quit timer timeout ->-| | +->- QUIT received -->--+ | (send QUIT) | Quit-sent | (send QUIT-ACK; | +--------<---------------| | set dally timer) | +-----------+ +------------+ | | | |--<---+--------<-----------+ +->--+ +---| Ouit-rcvd | | | | | |---->----- QUIT received --+ | | +------------+ (send QUIT-ACK; | | set dally timer) | | +------------+ | | | | | +-->-- Dally timeout -->---| Inactive |-<--QUIT-ACK received -+ | | or death timeout +------------+ Figure 7. Quit state diagram 3.3.3 NETBLT ABORT. An ABORT will take place when an unrecoverable malfunction occurs. Since the ABORT originates in the NETBLT layer, it may be sent at any time. The ABORT implies a malfunction, so no transmit reliability is expected, and the sender will immediately close its connection. Figure 8 illustrates this sequence. White [Page 15] INTERNET DRAFT NETBLT April 1997 +------------+ | | +-<--| Connected |-->---+ | | | | | +------------+ | | | ABORT received Internal malfunction | (send ABORT) | +------------+ | | | | | +->--| Inactive |--<---+ | | +------------+ Figure 8. Abort state diagram 3.3.4 Death timer timeout. When a NETBLT's death timer expires, it closes the connection without sending further packets. 4. Protocol layering structure. NETBLT may be implemented directly on top of the Internet Protocol (IP), in which case it has been assigned an official protocol number of 30 (decimal), which is 0x1e (hexadecimal). In other instances, it has been implemented over UDP, for which an official protocol number will be requested. 5. Packet formats. NETBLT packets are divided into three categories, all of which share a common 12-byte packet header. a. There are three packet types that travel only from data sender to receiver; these include the high-acknowledged-sequence-numbers which the receiver uses for control of message transmission reliability. They are the NULL-ACK, DATA, and LDATA packets. b. There are two packet types that travels only from receiver to sender. One is the CONTROL packet. Each CONTROL packet can contain an arbitrary number of control messages (GO, OK, or RESEND), each with its own sequence number. The other is the unreliably-transmitted DONE packet. c. There are six packet types which can travel in either White [Page 16] INTERNET DRAFT NETBLT April 1997 direction. These packet types either have special ways of insuring reliability, or are not transmitted reliably. They are the OPEN, RESPONSE, REFUSED, QUIT, QUITACK, and ABORT packets. The OPEN packet travels from active side to passive side; the RESPONSE and REFUSED packets travel from passive side to active side; and the QUIT, QUITACK, and ABORT packets can be sent by either side. All packet headers are "longword-aligned," such that all packet headers are a multiple of four bytes in length and all four-byte fields start on a longword boundary. The content of the longword alignment fields is zeros. The Client String field is terminated with at least one null byte, with extra null bytes added at the end to create a field that is a multiple of four bytes long. All numeric values are coded as binary integers. OPEN (type 0) and RESPONSE (type 1) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+---------------+---------------+ | Checksum | Version | Type | +---------------+---------------+---------------+---------------+ | Length | Local Port | +---------------+---------------+---------------+---------------+ | Foreign Port | Longword Alignment Padding | +---------------+---------------+---------------+---------------+ | Connection Unique ID | +---------------+---------------+---------------+---------------+ | Buffer Size | +---------------+---------------+---------------+---------------+ | DATA packet size | Burst Size | +---------------+---------------+---------------+---------------+ | Burst Interval | Death Timer Value | +---------------+---------------+---------------+---------------+ | Reserved (must be zero) |C|M| Maximum # Outstanding Buffers | +---------------+---------------+---------------+---------------+ | Client String ... +---------------+---------------+--------------- Longword Alignment Padding | ---------------+-------------------------------+ a. Checksum: to generate the checksum, the checksum field itself is cleared, the 16-bit ones-complement sum is computed over the packet, and the ones complement of this sum is placed in the checksum field. b. Version: the NETBLT protocol version number. This document describes version 4 of NETBLT. White [Page 17] INTERNET DRAFT NETBLT April 1997 c. Type: the NETBLT packet type number (OPEN = 0, RESPONSE = 1, etc.) d. Length: the total length (NETBLT header plus data, if present) of the NETBLT packet in bytes e. Local Port: the local NETBLT's 16-bit port number f. Foreign Port: the foreign NETBLT's 16-bit port number g. Connection UID: the 32 bit connection unique identifier. Connection UID may be any randomly-selected value, which is unique in that if more than one NETBLT connection is supported by a single host interface, it will not be duplicated. h. Buffer size: the size in bytes of each NETBLT buffer (except the last) i. Data packet size: length of each DATA packet in bytes j. Burst Size: Number of DATA packets in a burst k. Burst Interval: Transmit time in milliseconds of a single burst l. Death timer: Packet sender's death timer value in seconds m. "C": the DATA/LDATA packet data checksum flag (0 = do not checksum DATA and LDATA packet data, 1 = do). n. "M": the transfer mode (0 = READ, 1 = WRITE). o. Maximum Outstanding Buffers: maximum number of buffers that can be transferred before waiting for an OK message from the receiving NETBLT. p. Client string: an arbitrary, null-terminated, longword-aligned string for use by NETBLT clients. QUITACK (type 3), and DONE (type 10) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+---------------+---------------+ | Checksum | Version | Type | +---------------+---------------+---------------+---------------+ | Length | Local Port | +---------------+---------------+---------------+---------------+ | Foreign Port | Longword Alignment Padding | +---------------+---------------+---------------+---------------+ White [Page 18] INTERNET DRAFT NETBLT April 1997 QUIT (type 2), ABORT (type 4), and REFUSED (type 9) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+---------------+---------------+ | Checksum | Version | Type | +---------------+---------------+---------------+---------------+ | Length | Local Port | +---------------+---------------+---------------+---------------+ | Foreign Port | Longword Alignment Padding | +---------------+---------------+---------------+---------------+ | Reason for QUIT/ABORT/REFUSE... +---------------+---------------+--------------- Longword Alignment Padding | ---------------+-------------------------------+ DATA (type 5) and LDATA (type 6) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+---------------+---------------+ | Checksum | Version | Type | +---------------+---------------+---------------+---------------+ | Length | Local Port | +---------------+---------------+---------------+---------------+ | Foreign Port | Longword Alignment Padding | +---------------+---------------+---------------+---------------+ | Buffer Number | +---------------+---------------+---------------+---------------+ | Last Buffer Touched | +---------------+---------------+---------------+---------------+ | High Consecutive Seq Num Rcvd | Packet Number | +---------------+---------------+---------------+---------------+ | Data Area Checksum Value | Reserved (MBZ) |L| +---------------+---------------+---------------+---------------+ | New Burst Size | New Burst Interval | +---------------+---------------+---------------+---------------+ a. Checksum: checksum of the packet header only, including the Data Area Checksum Value. b. Buffer number: a 32 bit unique number assigned to every buffer. Buffers are sequentially numbered, starting with 1. c. Last Buffer Touched: the number of the highest buffer transmitted so far. White [Page 19] INTERNET DRAFT NETBLT April 1997 d. High Consecutive Sequence Number Received: Highest control message sequence number below which all control messages have been received. e. Packet number: sequential, monotonically increasing DATA packet identifier, starting with 0 in each buffer. f. Data Area Checksum Value: Checksum of the DATA packet's data. Algorithm used is the same as that used to compute checksums of other NETBLT packets. g. "L" is a bit that is set to 1 when the buffer that this DATA packet belongs to is the last buffer in the transfer. h. New Burst Size: Burst size as negotiated from value given by receiving NETBLT in OK message. i. New Burst Interval: Burst interval as negotiated from value given by receiving NETBLT in OK message. Value is in milliseconds. NULL-ACK (type 7) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+---------------+---------------+ | Checksum | Version | Type | +---------------+---------------+---------------+---------------+ | Length | Local Port | +---------------+---------------+---------------+---------------+ | Foreign Port | Longword Alignment Padding | +---------------+---------------+---------------+---------------+ | Last Buffer Touched | +---------------+---------------+---------------+---------------+ | High Consecutive Seq Num Rcvd | New Burst Size | +---------------+---------------+---------------+---------------+ | New Burst Interval | Longword Alignment Padding |L| +---------------+---------------+---------------+---------------+ a. Last Buffer Touched: the number of the highest buffer transmitted so far. b. High Consecutive Sequence Number Received: same as in DATA/LDATA packet. c. New Burst Size: Burst size as negotiated (half- and full-duplex only) from value given by receiving NETBLT in OK message. White [Page 20] INTERNET DRAFT NETBLT April 1997 d. New Burst Interval: Burst interval as negotiated (half- and full- duplex only) from value given by receiving NETBLT in OK message. Value is in milliseconds. e. "L" is a bit that is set to 1 when the buffer identified in the Last Buffer Touched field is the last buffer in the transfer. CONTROL (type 8) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+---------------+---------------+ | Checksum | Version | Type | +---------------+---------------+---------------+---------------+ | Length | Local Port | +---------------+---------------+---------------+---------------+ | Foreign Port | Longword Alignment Padding | +---------------+---------------+---------------+---------------+ Followed by any number of messages, each of which is longword aligned, with the following formats: GO message (subtype 0) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+---------------+---------------+ | Subtype | Word Padding | Sequence Number | +---------------+---------------+---------------+---------------+ | Buffer Number | +---------------+---------------+---------------+---------------+ a. Subtype: message type (GO = 0, OK = 1, RESEND = 2) b. Sequence number: A 16 bit unique message number. Sequence numbers must be monotonically increasing, starting with 1. c. Buffer number: as in DATA/LDATA packet White [Page 21] INTERNET DRAFT NETBLT April 1997 OK message (subtype 1). 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+---------------+---------------+ | Subtype | Word Padding | Sequence Number | +---------------+---------------+---------------+---------------+ | Buffer Number | +---------------+---------------+---------------+---------------+ | New Offered Burst Size | New Offered Burst Interval | +---------------+---------------+---------------+---------------+ | Current control timer value | Longword Alignment Padding | +---------------+---------------+---------------+---------------+ a. New offered burst size: burst size for subsequent buffer transfers, possibly based on performance information for previous buffer transfers. b. New offered burst interval: burst rate for subsequent buffer transfers, possibly based on performance information for previous buffer transfers. Rate is in milliseconds. c. Current control timer value: Receiving NETBLT's control timer value in milliseconds. RESEND message (subtype 2) 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +---------------+---------------+---------------+---------------+ | Subtype | Word Padding | Sequence Number | +---------------+---------------+---------------+---------------+ | Buffer Number | +---------------+---------------+---------------+---------------+ | Number of Missing Packets | New Offered Burst Size | +---------------+---------------+---------------+---------------+ | New Offered Burst Interval | Longword Alignment Padding | +---------------+---------------+---------------+---------------+ | Packet Number (2 bytes/packet)| ... +---------------+---------------+---------- | Padding (if necessary) | -----------+---------------+---------------+ a. Packet number: the 16 bit data packet identifier of a DATA packet, from the buffer identified by Buffer Number, whose retransmission is requested. Multiple packet numbers may occur in one RESEND message. 6. NETBLT modes of operation. White [Page 22] INTERNET DRAFT NETBLT April 1997 NETBLT supports three modes of operation; simplex, half-duplex, and full-duplex. This section identifies the required components of NETBLT for simplex and half-duplex modes of operation. Across full- duplex connections the normal NETBLT as described above is used. 6.1 Simplex. The only NETBLT packet types used in the simplex case are the following: a. OPEN b. QUIT c. ABORT d. DATA e. LDATA f. NULL-ACK 6.1.1 Sender simplex operation. Operation of NETBLT in simplex send mode is as follows: the OPEN message is sent; DATA and LDATA packets are sent; and the connection is closed. Any packet may be sent more than once, for redundancy, but for all n, packets from buffer(n - 1) will not be sent after packets from buffer(n). QUIT and ABORT packets may be sent at any time, and will have the same effect. The Maximum Number of Outstanding Buffers (in the OPEN packet) is set to 2. 6.1.2 Receiver simplex operation. Operation of NETBLT in simplex receive mode is as follows: when an OPEN packet is received, a connection is considered to be established. Packets received are stored into NETBLT buffers. The receiving NETBLT will pass a buffer to the client when the buffer is filled with correct packets or when good packets for a higher- numbered buffer are received. A list of packets which are possibly bad, or missing, is passed to the client. When the last buffer (L flag set in packet headers) has been passed to the client, or when the death timeout has expired, the receiving connection is terminated. The receiving NETBLT will discard redundant packets. In the case of errors, the following rules apply at the receiving NETBLT: a. A NETBLT packet with a bad header checksum is discarded. b. A NETBLT DATA or LDATA packet with a good header checksum and a bad data area checksum may optionally be saved but flagged as possibly bad. Reasonableness checks may be used to insure that good White [Page 23] INTERNET DRAFT NETBLT April 1997 data is not affected by the possibly bad packet data. If a good NETBLT packet (redundantly transmitted) is received with the same buffer and packet number as a possibly bad one, the possibly bad packet is replaced with the good one. 6.2 Half-duplex. The normal, full-duplex version of NETBLT operates across half-duplex connections with the following modification: keepalive packets will not be sent by the receiver while it is in the process of receiving a packet. The burst timer and burst size counter are reset at the start of each transmission period. If the Maximum Number of Outstanding Buffers (in the OPEN packet) is set to 1, the sending and receiving NETBLTs will operate in lockstep. If the Maximum Number of Outstanding Buffers is set to a value N greater than 1, the receiving NETBLT will wait until N buffers have been completely received or have had their data timers expire before sending a CONTROL packet. An exception occurs when the last buffer is sent; when all buffers up to and including the last buffer have been completely received or have had their data timers expire, the receiving NETBLT is permitted to send its CONTROL packet. The last buffer is identified by the receiver as the buffer for which the "L" bit is set in a DATA/LDATA packet, or as the Last Buffer Touched in a NULL-ACK packet with its "L" bit set to 1. 7. Security Considerations Security considerations for NETBLT operation have not been addressed in this document. 8. Possible Extensions Two forms of extension to NETBLT are being examined. The first would make it possible for NETBLT to operate as a unidirectional stream protocol, by allowing all buffers, not just the last one, to be of any size less than or equal to the negotiated size. The second would provide a "Transaction NETBLT", using an approach similar to Transaction TCP [RFC1379]. 9. References [Jacobsen] Jacobsen, V. "Congestion Avoidance and Control", ACM SIGCOMM 88 Symposium Proceedings, August 1988 [MIL-STD] MIL-STD-2045-44500 "Tactical Communications Protocol 2 (TACO2) for the National Imagery Transmission Format Standard", June 1993 White [Page 24] INTERNET DRAFT NETBLT April 1997 [RFC998] Clark, D., Lambert, M., and Zhang, L. "NETBLT: A Bulk Data Transfer Protocol", RFC 998, March 1987 [RFC1379] Braden, R., "Extending TCP for Transactions -- Concepts", RFC 1379, November 1992 10. Author's Address John C. C. White The MITRE Corporation 202 Burlington Road Bedford, MA 01730-1420 Phone: 617-271-3284 Fax: 617-271-2721 Email: jccw@mitre.org White [Page 25]