Internet DRAFT - draft-briscoe-tcpm-inspace-mode-tcpbis
draft-briscoe-tcpm-inspace-mode-tcpbis
TCP Maintenance and Minor Extensions (tcpm) B. Briscoe
Internet-Draft BT
Updates: 793 (if approved) March 09, 2015
Intended status: Experimental
Expires: September 10, 2015
Inner Space for all TCP Options (Kitchen Sink Draft - to be Split Up)
draft-briscoe-tcpm-inspace-mode-tcpbis-00
Abstract
This document describes an experimental redesign of TCP's
extensibility mechanism. It aims to traverse most known middleboxes
including connection splitters, by making it possible to tunnel all
TCP options within the TCP Data. It provides a choice between in-
order and out-of-order delivery for TCP options. In-order delivery
is a useful new facility for options that control datastream
processing. Out-of-order delivery has been the norm for TCP options
until now, and is necessary for options involved with acknowledging
data, otherwise flow control can deadlock. TCP's original design
limits TCP option space to 40B. In the new design there is no such
arbitrary limit, other than the maximum size of a segment. The TCP
client can immediately start to use the extra option space
optimistically from the very first SYN segment, by using a dual
handshake. The dual handshake is designed to prevent a legacy server
from getting confused and sending the control options to the
application as user-data. The dual handshake is only one strategy -
a single handshake will usually suffice once deployment is underway.
In summary, the protocol should allow new TCP options to be
introduced i) with minimal middlebox traversal problems; ii) with
incremental deployment from legacy servers; iii) with zero
handshaking delay iv) with a choice of in-order and out-of-order
delivery v) without arbitrary limits on available space.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
Briscoe Expires September 10, 2015 [Page 1]
Internet-Draft Inner Space for all TCP Options March 2015
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 10, 2015.
Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1. Motivation for Adoption Now (to be removed before
publication) . . . . . . . . . . . . . . . . . . . . . . 7
1.2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3. Experiment Goals . . . . . . . . . . . . . . . . . . . . 8
1.4. Wider Implications . . . . . . . . . . . . . . . . . . . 8
1.5. Document Roadmap . . . . . . . . . . . . . . . . . . . . 9
1.6. Terminology . . . . . . . . . . . . . . . . . . . . . . . 10
2. Protocol Specification . . . . . . . . . . . . . . . . . . . 11
2.1. Protocol Interaction Model . . . . . . . . . . . . . . . 11
2.1.1. Dual 3-Way Handshake . . . . . . . . . . . . . . . . 11
2.1.2. Dual Handshake Retransmission Behaviour . . . . . . . 14
2.1.3. Continuing the Upgraded Connection . . . . . . . . . 14
2.2. Upgraded Segment Structure and Format . . . . . . . . . . 15
2.2.1. Structure of an Upgraded Segment . . . . . . . . . . 15
2.2.2. Format of the InSpace Option . . . . . . . . . . . . 16
2.3. Inner TCP Option Processing . . . . . . . . . . . . . . . 18
2.3.1. Writing Inner TCP Options . . . . . . . . . . . . . . 19
2.3.1.1. Constraints on TCP Fast Open . . . . . . . . . . 19
2.3.1.2. Option Alignment . . . . . . . . . . . . . . . . 19
2.3.1.3. Sequence Space Consumption . . . . . . . . . . . 20
2.3.1.4. Flow Control Coverage . . . . . . . . . . . . . . 20
2.3.1.5. Presence or Absence of Flow-Controlled Data . . . 21
2.3.1.6. Construction Order for TCP Data . . . . . . . . . 22
2.3.2. Reading Inner TCP Options . . . . . . . . . . . . . . 22
2.3.2.1. Reading Inner TCP Options (SYN=1) . . . . . . . . 22
Briscoe Expires September 10, 2015 [Page 2]
Internet-Draft Inner Space for all TCP Options March 2015
2.3.2.2. Reading Inner TCP Options (SYN=0) . . . . . . . . 24
2.3.3. Forwarding Inner TCP Options . . . . . . . . . . . . 26
2.4. Exceptions . . . . . . . . . . . . . . . . . . . . . . . 26
2.5. SYN Flood Protection . . . . . . . . . . . . . . . . . . 27
3. Design Rationale . . . . . . . . . . . . . . . . . . . . . . 27
3.1. Dual Handshake and Migration to Single Handshake . . . . 27
3.2. Inner Option Space . . . . . . . . . . . . . . . . . . . 28
3.2.1. Header Extension by Encapsulation . . . . . . . . . . 28
3.2.2. Non-Deterministic Magic Number Approach . . . . . . . 29
3.2.3. Non-Goal: Security Middlebox Evasion . . . . . . . . 31
3.2.4. Avoiding the Start of the First Two Segments . . . . 32
3.2.5. Framing Segments . . . . . . . . . . . . . . . . . . 32
3.2.6. Control Options Within Data Sequence Space . . . . . 33
3.2.6.1. In-Order Flow-Controlled Options . . . . . . . . 33
3.2.6.2. Fire-and-Forget Options . . . . . . . . . . . . . 35
3.3. Deployment Approach . . . . . . . . . . . . . . . . . . . 38
3.3.1. Substrate Protocol: TCP vs. UDP . . . . . . . . . . . 38
3.3.2. Kernel-Space vs. User-Space . . . . . . . . . . . . . 38
3.4. Rationale for the InSpace Option Format . . . . . . . . . 38
4. Protocol Overhead . . . . . . . . . . . . . . . . . . . . . . 40
5. Interaction with Pre-Existing TCP Implementations . . . . . . 42
5.1. Compatibility with Pre-Existing TCP Variants . . . . . . 42
5.2. Interaction with Middleboxes . . . . . . . . . . . . . . 44
5.3. Interaction with the Pre-Existing TCP API . . . . . . . . 45
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 47
7. Security Considerations . . . . . . . . . . . . . . . . . . . 47
8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 49
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 50
9.1. Normative References . . . . . . . . . . . . . . . . . . 50
9.2. Informative Reference . . . . . . . . . . . . . . . . . . 50
Appendix A. Zero Overhead Message Boundary Insertion (ZOMBI) . . 52
Appendix B. Generic Connection Mode Switching . . . . . . . . . 55
Appendix C. Protocol Extension Specifications . . . . . . . . . 57
C.1. Dual Handshake: The Explicit Variant . . . . . . . . . . 57
C.1.1. SYN-O Structure . . . . . . . . . . . . . . . . . . . 59
C.1.2. Retransmission Behaviour - Explicit Variant . . . . . 60
C.1.3. Corner Cases . . . . . . . . . . . . . . . . . . . . 60
C.1.4. Workround if Data in SYN is Blocked . . . . . . . . . 61
C.2. Jumbo InSpace TCP Option (only if SYN=0) . . . . . . . . 62
C.3. Optional Segment Structure to Traverse DPI boxes . . . . 63
Appendix D. Comparison of Alternatives . . . . . . . . . . . . . 66
D.1. Implicit vs Explicit Dual Handshake . . . . . . . . . . . 66
Appendix E. Protocol Design Issues (to be Deleted before
Publication) . . . . . . . . . . . . . . . . . . . . 67
Appendix F. Change Log (to be Deleted before Publication) . . . 68
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 72
Briscoe Expires September 10, 2015 [Page 3]
Internet-Draft Inner Space for all TCP Options March 2015
1. Introduction
TCP has become hard to extend, partly because the option space was
limited to 40B when TCP was first defined [RFC0793] and partly
because many middleboxes only forward TCP headers that conform to the
stereotype they expect.
In 2011, [Honda11] tested a broad but small set of paths and found
that there were few if any middlebox traversal problems over
residential access networks, but the chance of a new option
traversing other types of access was terrible. Cellular was
especially bad (stripping options on 40% of paths for port 80 and 20%
for other ports), but WiFi hotspots, enterprise, and university
networks were close behind (typically, about 18% of paths blocked new
extensions). This specification ensures new TCP capabilities can
traverse most middleboxes by tunnelling TCP options within the TCP
Data as 'Inner Options' (Figure 1). Then the TCP receiver can
reconstruct the Inner Options sent by the sender, even if a middlebox
resegments the datastream and even if it strips 'Outer' options from
the TCP header that it does not recognise.
The two words 'Inner Space' are appropriate as a name for the scheme;
'Inner' because it encapsulates options within the TCP Data and
'Space' because the space for TCP options within the TCP Data is
virtually unlimited--constrained only by the maximum segment size.
,-----. TCP Payload ,-----.
| App |<----------------------------------------->| App |
|-----| |-----|
| | Inner Options within TCP Data | |
| |<----------------------------------------->| |
| | | |
| TCP | TCP Header and TCP header and | TCP |
| | Outer Options ,---------. Outer Options | |
| |<-------------->|Middlebox|<-------------->| |
|-----| |---------| |-----|
| IP | | IP | | IP |
: : : : : :
Figure 1: Encapsulation Approach
Tunnelling options within TCP Data raises two difficult questions: i)
immediate (out-of-order) delivery of certain options and ii)
bootstrapping the inner control channel.
Traditional TCP options [RFC0793] are delivered unreliably and out of
order, because they are within the main header, outside the TCP
sequence space. This document calls these 'Outer Options'. When TCP
Briscoe Expires September 10, 2015 [Page 4]
Internet-Draft Inner Space for all TCP Options March 2015
options are placed within the TCP Data (Inner Options), it is easiest
to include them within TCP's sequence space. Then TCP naturally
delivers them reliably and in order without any extra machinery.
However, in-order delivery is unacceptable for some options.
TCP options fall into three categories:
Segment-related (out-of-order): These have to be delivered to the
receiver's TCP stack as soon as they are received (i.e. not
necessarily in the order sent). They are generally concerned with
transmission of each TCP segment, e.g. Timestamps, Selective
ACKnowledgements (SACK), the Data ACK of Multipath TCP [RFC6824]
and the message authentication code (MAC) of tcpcrypt
[I-D.bittau-tcpinc-tcpcrypt].
Datastream-related (in-order): These would ideally be applied in the
order that the sender inserted them into the datastream. They are
generally concerned with controlling the transmission of the
ordered datastream, e.g. the options of the TCP AO [RFC5925] that
control data authentication or the suboptions of tcpcrypt that
control data encryption [I-D.bittau-tcpinc-tcpcrypt]. At the time
these were designed, TCP only provided Outer Options, so it was
complex to apply TCP-AO options reliably and in order and similar
complexity is being included in tcpcrypt;
Connection-related (order-agnostic): These are typically applied at
the start of a connection which is also inherently the start of
the first segment so the order of segment delivery is not a
concern, e.g. TCP fast option [I-D.ietf-tcpm-fastopen], the sub-
options of MPTCP [RFC6824] (except the Data ACK), and most of the
TCP options that are in common usage;
The simplest ('default') variant of the Inner Space protocol
[I-D.briscoe-tcpm-inner-space] delivers all Inner Options reliably
and in order within the datastream.Therefore the default-mode Inner
Space protocol can only support segment-related options as Outer
Options. This is irritating because even though only a few options
are segment-related, if just one kind of option cannot traverse a
middlebox, it often prevents a whole set of other extensions from
being used even though they would have no problem traversing the
middlebox as Inner Options. For instance, one MPTCP option (the Data
ACK) and one tcpcrypt option (the MAC) have to be delivered
immediately (out of order), even though all the other MPTCP and
tcpcrypt options can be delivered in order.
The present specification extends the default-mode Inner Space
protocol to add out-of-order delivery of Inner Options. It can then
support all TCP options as Inner Options. This offers the prospect
Briscoe Expires September 10, 2015 [Page 5]
Internet-Draft Inner Space for all TCP Options March 2015
of completely circumventing middlebox problems and space problems for
all TCP extensions.
The second difficult question addressed by the present specification
is how to bootstrap the inner control channel--without any visible
difference to the TCP wire protocol that would otherwise be unlikely
to traverse many middleboxes. Given the Inner Space protocol places
control options within TCP Data, it is critical that a legacy TCP
receiver is never confused into passing this mix to an application as
if it were pure data. Naively, both ends could handshake to check
they understand the protocol, but this would introduce a round of
delay.
The Inner Space protocol will have to use whichever bootstrap
approach is least bad, because they all involve compromises. For the
present specification, the dual handshake has been chosen over the
only other candidate currently in the running
[I-D.touch-tcpm-tcp-syn-ext-opt], in which the client complements the
SYN with an out-of-band (OOB) segment. In both approaches the client
starts the connection with two segments. However, with the OOB
approach the two segments will always be necessary, whereas the dual
handshake is only a transition strategy that becomes unnecessary for
each server as it is upgraded. Both approaches will need to be
tested for middlebox traversal. It seems likely that many firewalls
will block the OOB segment and it is also expected that some
middleboxes will block the data in the SYN used for one of the dual
handshakes.
In the dual handshake approach the client sends two SYNs; one for an
upgraded server, and the other for an ordinary server. Then, if the
client discovers that the server does not understand the new
protocol, it can abort the upgraded handshake before the server
corrupts the application by passing it Inner Options. Otherwise, if
the server does understand the new protocol, the client can abort the
ordinary handshake, given it offers no extra option space. Either
way, zero extra delay is added. Interworking of the dual handshake
with TCP Fast Open [I-D.ietf-tcpm-fastopen] is carefully defined so
that either server can pass data to the application as soon as the
initial SYN arrives.
Solving the five problems of i) option-space exhaustion; ii)
middlebox traversal; iii) legacy server confusion; iv) a choice of
in-order and out-of-order frame delivery; and v) handshake latency;
does not come without cost:
o So that the Inner Space protocol is immune to option stripping, it
avoids a conventional TCP option in the header. Instead it
signals its presence using a magic number within the TCP Data of
Briscoe Expires September 10, 2015 [Page 6]
Internet-Draft Inner Space for all TCP Options March 2015
the initial segment in each direction. This introduces a risk
that payload in an ordinary SYN or SYN/ACK might be mistaken for
the Inner Space protocol (an initial worst-case estimate of the
probability is one connection globally every 40 years).
Nonetheless, the risk is zero in the (currently common) case of an
ordinary connection without payload during the handshake. There
is also no risk of a mistake the other way round--an upgraded
connection cannot be mistaken for an ordinary connection.
o Although the dual handshake introduces no extra latency, it
introduces extra connection processing & state, extra traffic and
extra header processing. Initial estimates put the percentage
overhead in single digits for connection processing and state, and
traffic overhead at only a few hundredths of a percent. Once the
most popular TCP servers have upgraded, only a single handshake
will be necessary most of the time and overhead should drop to
vanishingly small proportions.
1.1. Motivation for Adoption Now (to be removed before publication)
A number of extensions to TCP are in the process of definition and
experimentation (TCPINC, MPTCP, etc). If a general-purpose middlebox
traversal solution were available now, each new protocol design would
not need complex machinery to detect and work round the byzantine
range of middlebox behaviours. It would also make these extensions
available to many more users.
It seems inevitable that ultimately more option space will be needed,
particularly given that many of the TCP options introduced recently
consume large numbers of bits in order to provide sufficient
information entropy, which is not amenable to compression.
Extension of TCP option space requires support from both ends. This
means it will take many years before the facility is functional for
most pairs of end-points. Therefore, given the problem is already
becoming pressing, a solution needs to start being deployed now.
1.2. Scope
This experimental specification extends the TCP wire protocol. It is
independent of the dynamic rate control behaviour of TCP and it is
independent of (and thus compatible with) any protocol that
encapsulates TCP, including IPv4 and IPv6.
Briscoe Expires September 10, 2015 [Page 7]
Internet-Draft Inner Space for all TCP Options March 2015
1.3. Experiment Goals
TCP is critical to the robust functioning of the Internet, therefore
any proposed modifications to TCP need to be thoroughly tested.
Success criteria: The experimental protocol will be considered
successful if it satisfies the following requirements in the
consensus opinion of the IETF tcpm working group. The protocol
needs to be sufficiently well specified so that more than one
implementation can be built in order to test its function,
robustness, overhead and interoperability (with itself, with
previous version of TCP, and with various commonly deployed
middleboxes). Non-functional issues such as recommendations on
message timing also need to be tested. Various optional
extensions to the protocol are proposed in Appendix C so
experiments are also needed to determine whether these extensions
ought to remain optional, or perhaps be removed or become
mandatory.
Duration: To be credible, the experiment will need to last at least
12 months from publication of the present specification. If
successful, it would then be appropriate to progress to a
standards track specification, complemented by a report on the
experiments.
1.4. Wider Implications
The implications of this work are more than 'just' a low latency
incrementally deployable way to extend TCP option space:
End-to-middle signalling channel: Once endpoints have an end-to-end
control channel within the TCP Data, they can use authentication
or even encryption to stop middleboxes interfering with it. Then
given middleboxes already interfere with Outer TCP Options, they
can serve a new purpose as a channel for end-system TCP stacks to
interact with middleboxes, but only if they choose to.
Multiplexed streams, compression, encryption (transport services):
The Inner Space protocol has been designed generically, so that
different delivery modes such as in-order and out-of-order
delivery can be applied to different frames within the TCP Data.
An additional mode could be added to extend out-of-order delivery
to user-data, not just TCP control options. Then a single TCP
connection could deliver data in multiple independent streams to
minimise latency while one stream is blocked by a loss without the
overhead of multiple connections. Inner Space is also structured
so that data transformations such as compression or encryption can
Briscoe Expires September 10, 2015 [Page 8]
Internet-Draft Inner Space for all TCP Options March 2015
easily be introduced and controlled by TCP options, as a generic
facility available to any application layer protocol.
All these transport services (multiplexed streams, compression,
encryption) are sought after by Web applications. However
attempts to make them available in new transport protocols (e.g.
SCTP) have proved impossible to deploy over the public Internet
because too many middleboxes block new protocol identifiers. To
work round this impasse, these transport services are being
embedded within the application layer as part of the next
generation of the HTTP protocol [I-D.ietf-httpbis-http2]. Inner
Space has been designed so that these transport services would be
straightforward to add in a structured way at the transport layer,
using a new TCP mode. A separate document is planned to specify
this mode. The present document focuses solely on TCP control
options, which meets specific immediate needs. Nonetheless, the
similarity is close enough to extrapolate that it will be
straightforward to provide the transport services that Web
applications need as well.
1.5. Document Roadmap
The body of the document starts with a full specification of the
Inner Space extension to TCP (Section 2). It is rather terse,
answering 'What?' and 'How?' questions, but deferring 'Why?' to
Section 3. The careful design choices made are not necessarily
apparent from a superficial read of the specification, so the Design
Rationale section is fairly extensive. The body of the document ends
with Section 5 that checks possible interactions between the new
scheme and pre-existing variants of TCP, including interaction with
partial implementations of TCP in known middleboxes.
Appendix A defines the encoding that the Inner Space protocol uses
for TCP Data. Eventually, this appendix is likely to be published
separately because the encoding is more generally applicable.
Appendix B defines an Inner TCP Option that provides a capability to
switch the mode of a TCP connection, where the term 'mode' is a very
general concept that might be used to change the ordering semantics
of a connection, or switch off the Inner Space capability part way
through a connection. Eventually this appendix is likely to be
published separately due to its general applicability. Appendix C
specifies optional extensions to the protocol that will need to be
implemented experimentally to determine whether they are useful. And
Appendix D discusses the merits of the chosen design against some of
the optional extensions.
Briscoe Expires September 10, 2015 [Page 9]
Internet-Draft Inner Space for all TCP Options March 2015
1.6. Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119]. In this
document, these words will appear with that interpretation only when
in ALL CAPS. Lower case uses of these words are not to be
interpreted as carrying RFC-2119 significance.
TCP Header: As defined in [RFC0793]. Even though the present
specification places TCP options beyond the Data Offset, the term
'TCP Header' is still used to mean only those fields at the head
of the segment, delimited by the TCP Data Offset.
Inner TCP Options (or just Inner Options): TCP options placed in the
space that the present specification makes available beyond the
Data Offset.
Outer TCP Options (or just Outer Options): The TCP options in the
traditional location directly after the base TCP Header and before
the TCP Data Offset.
Prefix TCP Options: Inner Options to be processed before the Outer
Options.
Suffix TCP Options: Inner Options to be processed after the Outer
Options, in sequence with the data.
TCP options: Any TCP options, whether inner, outer or both. This
specification makes this term on its own ambiguous so it should be
qualified if it is intended to mean TCP options in a certain
location.
TCP Payload: Data to be passed to the layer above TCP. The present
specification redefines the TCP Payload so that it does not
include the Inner TCP Options, the InSpace Option or any inner
padding, even though they are located beyond the Data Offset.
TCP Data: The information in a TCP segment after the Data Offset,
including the TCP Payload, Inner TCP Options, any inner padding
and the InSpace Option defined in the present specification.
Pure ACK: A TCP acknowledgement with no TCP Data at all.
Impure ACK: A TCP acknowledgement with no TCP Payload or Suffix
Options, but with at least an InSpace Option and possibly padding
and Prefix Options.
Briscoe Expires September 10, 2015 [Page 10]
Internet-Draft Inner Space for all TCP Options March 2015
Flow-Controlled ACK: A TCP acknowledgement containing at least TCP
Payload and/or Suffix Options.
client: The process taking the role of actively opening a TCP
connection.
server: The process taking the role of TCP listener.
Upgraded Segment: A segment that will only be fully understood by a
host complying with the present specification (even though it
might appear valid to a pre-existing TCP receiver). Similarly,
Upgraded SYN, Upgraded SYN/ACK etc.
Ordinary Segment: A segment complying with pre-existing TCP
specifications but not the present specification. Similarly,
Ordinary SYN, Ordinary SYN/ACK etc.
Upgraded Connection: A connection starting with an Upgraded SYN.
Ordinary Connection: A connection starting with an Ordinary SYN.
Upgraded Host: A host complying with the present document as well as
with pre-existing TCP specifications. Similarly Upgraded TCP
Client, Upgraded TCP Server, etc.
Legacy Host: A host complying with pre-existing TCP specifications,
but not with the present document. Similarly Legacy TCP Client,
Legacy TCP Server, etc.
Note that the term 'Ordinary' is used for segments and connections,
but the term 'Legacy' is used for hosts. This is because, if the
Inner Space protocol were widely used in future, a host that could
not open an Upgraded Connection would be considered deficient and
therefore 'Legacy', whereas an Ordinary Connection would not be
considered deficient; because it will always be legitimate to open an
Ordinary Connection if extra option space or middlebox traversal is
not needed.
2. Protocol Specification
2.1. Protocol Interaction Model
2.1.1. Dual 3-Way Handshake
During initial deployment, an Upgraded TCP Client sends two
alternative SYNs: an Ordinary SYN in case the server is legacy and a
SYN-U in case the server is upgraded. The two SYNs MUST have the
same network addresses and the same destination port, but different
Briscoe Expires September 10, 2015 [Page 11]
Internet-Draft Inner Space for all TCP Options March 2015
source ports. Once the client establishes which type of server has
responded, it continues the connection appropriate to that server
type and aborts the other without completing the 3-way handshake.
The format of the SYN-U will be described later (Section 2.2.2). At
this stage it is only necessary to know that the client can put
either TCP options or payload (or both) in a SYN-U, in the space
traditionally intended only for payload. So if the server's response
shows that it does not recognise the Upgraded SYN-U, the client is
responsible for aborting the Upgraded Connection. This ensures that
a Legacy TCP Server will never erroneously confuse the application by
passing it TCP options as if they were user-data.
Section 3.1 explains various strategies the client can use to send
the SYN-U first and defer or avoid sending the Ordinary SYN.
However, such strategies are local optimizations that do not need to
be standardized. The rules below cover the most aggressive case, in
which the client sends the SYN-U then the Ordinary SYN back-to-back
to avoid any extra delay. Nonetheless, the rules are just as
applicable if the client defers or avoids sending the Ordinary SYN.
Table 1 summarises the TCP 3-way handshake exchange for each of the
two SYNs in the two right-hand columns, between an Upgraded TCP
Client (the active opener) and either:
1. a Legacy Server, in the top half of the table (steps 2-4), or
2. an Upgraded Server, in the bottom half of the table (steps 2-4)
Because the two SYNs come from different source ports, the server
will treat them as separate connections, probably using separate
threads (assuming a threaded server). A load balancer might forward
each SYN to separate replicas of the same logical server. Each
replica will deal with each incoming SYN independently - it does not
need to co-ordinate with the other replica.
Briscoe Expires September 10, 2015 [Page 12]
Internet-Draft Inner Space for all TCP Options March 2015
+------+------------------+--------------------+--------------------+
| | | Ordinary | Upgraded |
| | | Connection | Connection |
+------+------------------+--------------------+--------------------+
| 1 | Upgraded Client | >SYN | >SYN-U |
| | | | |
| /\/\ | /\/\/\/\/\/\/\/\ | /\/\/\/\/\/\/\/\/\ | /\/\/\/\/\/\/\/\/\ |
| 2 | Legacy Server | <SYN/ACK | <SYN/ACK |
| | | | |
| 3a | Upgraded Client | Waits for response | |
| | | to both SYNs | |
| | | | |
| 3b | " | >ACK | >RST |
| | | | |
| 4 | | Cont... | |
| | | | |
| /\/\ | /\/\/\/\/\/\/\/\ | /\/\/\/\/\/\/\/\/\ | /\/\/\/\/\/\/\/\/\ |
| 2 | Upgraded Server | <SYN/ACK | <SYN/ACK-U |
| | | | |
| 3a | Upgraded Client | Waits for response | |
| | | to SYN-U | |
| | | | |
| 3b | " | >RST | >ACK |
| | | | |
| 4 | | | Cont... |
+------+------------------+--------------------+--------------------+
Table 1: Dual 3-Way Handshake in Two Server Scenarios
Each column of the table shows the required 3-way handshake exchange
within each connection, using the following symbols:
> means client to server;
< means server to client;
Cont... means the TCP connection continues.
The connection that starts with an Ordinary SYN is called the
'Ordinary Connection' and the one that starts with a SYN-U is called
the 'Upgraded Connection'. An Upgraded Server MUST respond to a
SYN-U with an Upgraded SYN/ACK (termed a SYN/ACK-U and defined in
Section 2.2.2). Then the client recognises that it is talking to an
Upgraded Server. The client's behaviour depends on which response it
receives first, as follows:
Briscoe Expires September 10, 2015 [Page 13]
Internet-Draft Inner Space for all TCP Options March 2015
o If the client first receives a SYN/ACK response on the Ordinary
Connection, it MUST wait for the response on the Upgraded
Connection. It then proceeds as follows:
* If the response on the Upgraded Connection is an Ordinary SYN/
ACK, the client MUST reset (RST) the Upgraded Connection and it
can continue with the Ordinary Connection.
* If the response on the Upgraded Connection is an Upgraded SYN/
ACK-U, the client MUST reset (RST) the Ordinary Connection and
it can continue with the Upgraded Connection.
o If the client first receives an Ordinary SYN/ACK response on the
Upgraded Connection, it MUST reset (RST) the Upgraded Connection
immediately. It can then wait for the response on the Ordinary
Connection and, once it arrives, continue as normal.
o If the client first receives an Upgraded SYN/ACK-U response on the
Upgraded Connection, it MUST reset (RST) the Ordinary Connection
immediately and continue with the Upgraded Connection.
2.1.2. Dual Handshake Retransmission Behaviour
If the client receives a response to the SYN, but a short while after
that {ToDo: duration TBA} the response to the SYN-U has not arrived,
it SHOULD retransmit the SYN-U. If latency is more important than
the extra TCP option space, in parallel to any retransmission, or
instead of any retransmission, the client MAY give up on the Upgraded
(SYN-U) Connection by sending a reset (RST) and completing the 3-way
handshake of the Ordinary Connection.
If the client receives no response at all to either the SYN or the
SYN-U, it SHOULD solely retransmit one or the other, not both. If
latency is more important than the extra TCP option space, it will
retransmit the SYN. Otherwise it will retransmit the SYN-U. It MUST
NOT retransmit both segments, because the lack of response could be
due to severe congestion.
2.1.3. Continuing the Upgraded Connection
Once an Upgraded Connection has been successfully negotiated in the
SYN, SYN/ACK exchange, either host can allocate any amount of the TCP
Data space in any subsequent segment for extra TCP options. In fact,
the sender has to use the upgraded segment structure in every
subsequent segment of the connection that contains non-zero TCP
Payload. The sender can use the upgraded structure in a segment
carrying no TCP Payload, but it does not have to (see
Section 2.3.1.5).
Briscoe Expires September 10, 2015 [Page 14]
Internet-Draft Inner Space for all TCP Options March 2015
As well as extra option space, the facility offers other advantages,
such as reliable ordered delivery of Inner TCP Options on empty
segments and more robust middlebox traversal. If none of these
features is needed, at any point the facility can be disabled for the
rest of the connection, using the ModeSwitch TCP option in
Appendix B. Interestingly, the ModeSwitch options itself can be very
simple because it uses the reliable ordered delivery property of
Inner Options, rather than having to cater for the possibility that a
message to switch modes might be lost or reordered.
2.2. Upgraded Segment Structure and Format
2.2.1. Structure of an Upgraded Segment
An Upgraded Segment is structured as shown in Figure 2. Up to the
TCP Data Offset, the structure is identical to an Ordinary TCP
Segment, with a base TCP Header (BaseHdr) and the usual facility to
set the Data Offset (DO) to allow space for TCP options. These
regular TCP options are renamed by this specification to Outer TCP
Options or just Outer Options, and labelled as OuterOpts in the
figure.
| SDS |
|--------------------------------------------->|
|P| | SOO | |
|a| ,--------->| |
| DO |d| Len+1 | InOO | |
,------------------>| ,------->,-------------------->| |
+--------+----------+-+--------+----------+----------+-------------+
| BaseHdr| OuterOpts| | InSpace|PrefixOpts|SuffixOpts| Payload |
+--------+----------+-+--------+----------+----------+-------------+
| '----------.----------' |
| Inner Options |
`-----------------------.----------------------'
TCP Data
All offsets are specified in 4-octet (32-bit) words, except SDS and
Pad, which are in octets.
Figure 2: The Structure of an Upgraded Segment (not to scale)
Unlike an Ordinary TCP Segment, the Payload of an Upgraded Segment
does not start straight after the TCP Data Offset. Instead, Figure 2
shows that space is provided for additional Inner TCP Options before
the TCP Payload. The size of this space is termed the Inner Options
Offset (InOO). The TCP receiver reads the InOO field from the Inner
Option Space (InSpace) option defined in Section 2.2.2.
Briscoe Expires September 10, 2015 [Page 15]
Internet-Draft Inner Space for all TCP Options March 2015
Padding might have to be included at the start of the TCP Data to
align the InSpace option on a 4-octet boundary from the start of the
datastream (see Section 2.3.1.2).
Because the InSpace Option is only ever located in a standardized
location it does not need to follow the RFC 793 format of a TCP
option. Therefore, although we call InSpace an 'option', we do not
describe it as a 'TCP option'. The Length (Len) of the InSpace
option itself is read from a fixed location within the InSpace
option.
The Sent Data Size (SDS) is also read from within the InSpace Option.
If the datastream has been resegmented, it allows the receiver to
know the size of the segment as it was when it was sent, even if the
InSpace Options are no longer at the start of each segment (see
Section 2.3).
The Suffix Options Offset (SOO) is also read from within the InSpace
Option. It delineates the end of the Prefix TCP Options (PrefixOpts
in the figure) and the start of the Suffix TCP Options (SuffixOpts).
The receiver processes PrefixOpts before OuterOpts, then SuffixOpts
afterwards in order with the datastream. Full details of option
processing are given in Section 2.3.
The first segment in each direction (i.e. the SYN or the SYN/ACK) is
identifiable as upgraded by the presence of 6-octets of magic number
at the start of the TCP Data. The probability that an Upgraded
Server will mistake arbitrary data at the beginning of the payload of
an Ordinary Segment for the Magic Number has to be allowed for, but
it is vanishingly small (see Section 3.2.2). Once an Upgraded
Connection has been negotiated during the SYN - SYN/ACK exchange, a
magic number is not needed to identify Upgraded Segments, because
both ends then know the protocol that determines where subsequent
InSpace options will be located.
2.2.2. Format of the InSpace Option
The internal structure of the InSpace Option for an Upgraded SYN or
SYN/ACK segment (SYN=1) is defined in Figure 3a) and for a segment
with SYN=0 in Figure 3b) or an abbreviated form in Figure 3c).
Briscoe Expires September 10, 2015 [Page 16]
Internet-Draft Inner Space for all TCP Options March 2015
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
a) +---------------------------------------------------------------+
| Magic Number A |
+-------------------------------+---------------------------+---+
| Sent TCP Data Size (SDS) |Inner Options Offset (InOO)|Len|
+-------------------------------+---------------------------+---+
| Magic Number B |Suffix Options Offset (SOO)|CU |
+-------------------------------+---------------------------+---+
b) +-------------------------------+-----------------------------+-+
| Marker | ZOMBI |CU
+-------------------------------+---------------------------+-+-+
| Sent TCP Data Size (SDS) |Inner Options Offset (InOO)|Len|
+-------------------------------+---------------------------+---+
| Currently Unused (CU) |Suffix Options Offset (SOO)|CU |
+-------------------------------+---------------------------+---+
c) +-------------------------------+-----------------------------+-+
| Marker | ZOMBI |P|
+-------------------------------+---------------------------+-+-+
| Sent TCP Data Size (SDS) |Inner Options Offset (InOO)|Len|
+-------------------------------+---------------------------+---+
Figure 3: InSpace Option Format a) SYN=1; b) SYN=0, Len=2; c) SYN=0,
Len=1
The fields are defined as follows (see Section 3.4 for the rationale
behind these format choices):
Option Length (Len): The 2-bit Len field specifies the length of the
InSpace Option in 4-octets words excluding the first 4-octet word.
In other words, the option is (Len + 1) * 4 octets long. For this
experimental specification:
When SYN=1: the sender MUST use Len=2 (12 octets);
When SYN=0: the sender MUST use either Len = 2 (12 octets) or
Len=1 (8 octets). If Len = 1, the fields in the last 4-octet
word (CU and InOO) are omitted.
Sent Data Size (SDS): In this 16-bit field the sender MUST record
the size in octets of the TCP Data when it was sent. This
specification defines the TCP Data as all the octets after the TCP
Data Offset, including Inner TCP options, the InSpace Option and
any padding.
Briscoe Expires September 10, 2015 [Page 17]
Internet-Draft Inner Space for all TCP Options March 2015
Inner Options Offset (InOO): This 14-bit field defines the total
size of the Inner TCP Options in 4-octet words.
Suffix Options Offset (SOO): The 14-bit SOO field defines the offset
in 4-octet words from the start of the Inner Options to the start
of the Suffix Options. It represents the size of the Prefix
Options (see Section 2.3.2).
Prefix (P) flag: The P flag is only defined if Len=1 and SYN=0. In
this case the SOO field is not present. Then If P=1, SOO = InOO
(i.e. there are only Prefix Options), and if P=0, SOO=0 (i.e.
there are only Suffix Options).
Currently Unused (CU): The sender MUST fill the CU fields with zeros
and they MUST be ignored and forwarded unchanged by other nodes,
even if their value is different.
The following field is only defined within a segment with SYN=1 (i.e.
a SYN or SYN/ACK):
Magic Numbers A & B: The sizes of these fields are respectively 32 &
16 bits. The sender MUST fill them with Magic Numbers A & B
{ToDo: Values TBA}.
The following fields are only defined within a segment with SYN=0:
Marker: The sender must fill this 16-bit field with zeros (0x00).
ZOMBI: This 15-bit field is used to start encoding or decoding the
ZOMBI encoding (respectively see Section 2.3.1.6 or
Section 2.3.2.2).
2.3. Inner TCP Option Processing
The objects that Inner Space places within the TCP Data can be
divided into two types:
In-Order Flow-Controlled Objects: The receiver processes Suffix
Options and the TCP Payload in order, so it might have to buffer
them while waiting for a gap in the datastream to be filled by a
retransmission. Buffering requires flow control, therefore these
will be called In-Order Flow-Controlled objects.
Fire-and-Forget Objects: In contrast, when a segment arrives at the
receiver, it never buffers the padding, InSpace Option and any
Prefix Options; it immediately processes and removes them. The
sender does not need to retransmit these objects if they do not
arrive; it creates them on-the-fly to complement each sent
Briscoe Expires September 10, 2015 [Page 18]
Internet-Draft Inner Space for all TCP Options March 2015
segment. If it has to re-send a segment, it will create new ones
relevant to the re-sent segment. Therefore, these will be called
fire-and-forget objects.
The rationale for these choices is given in Section 3.2.6. The
following two subsections lay out the order in which these options
are processed respectively when the sender writes them and when the
receiver reads them.
2.3.1. Writing Inner TCP Options
2.3.1.1. Constraints on TCP Fast Open
If an Upgraded TCP Client uses a TCP Fast Open (TFO) cookie
[I-D.ietf-tcpm-fastopen] in an Upgraded SYN-U, it MUST place the TFO
option within the Inner TCP Options, beyond the Data Offset.
This rule is specific to TFO, but it can be generalised to any
capability similar to TFO as follows: An Upgraded TCP Client MUST NOT
place any TCP option in the Outer TCP Options of a SYN if it might
cause a TCP server to pass user-data directly to the application
before its own 3-way handshake completes.
If a client uses TCP Fast Open cookies on both the parallel
connection attempts of a dual handshake, an Upgraded Server will
deliver the TCP Payload to the application twice before the client
aborts the Ordinary Connection. This is not a problem, because
[I-D.ietf-tcpm-fastopen] requires that TFO is only used for
applications that are robust to duplicate requests.
2.3.1.2. Option Alignment
The sender MUST add ("3 - ((seqno - isn - 1) % 4")) octets of non-
zero padding ("Pad" in Figure 2) to align the start of the InSpace
option on a 4-octet word boundary from the start of the datastream,
where "seqno" is the TCP sequence number of the segment, "isn" is the
initial sequence number and '%' is the modulo operation.
If the end of the last Inner TCP Option does not align on a 4-octet
boundary, the sender MUST append sufficient no-op TCP options. The
end of the Prefix TCP Options MUST be similarly aligned.
If the sending TCP is applying a block-mode transformation to the TCP
Data (e.g. compression or encryption), the sender might have to add
some padding options to align the end of the Inner Options with the
end of a block. Any yet-to-be-written encryption specification will
need to carefully define this padding in order not to weaken the
cipher.
Briscoe Expires September 10, 2015 [Page 19]
Internet-Draft Inner Space for all TCP Options March 2015
2.3.1.3. Sequence Space Consumption
The sender MUST include all the TCP Data in TCP's sequence number and
acknowledgement number space, i.e. any padding, the InSpace Option
and any Inner Options as well as the TCP Payload.
Whenever the sender includes non-zero TCP Payload in a segment, it
MUST also include an InSpace Option, whether or not there are any
Inner Options (to enable reconstruction in case of resegmentation).
On the other hand, if the sender includes no TCP Payload in a segment
(e.g. ACKs, RSTs), it SHOULD NOT include an InSpace Option unless it
is necessary to send an Inner Option. {ToDo: Consider whether there
is any reason to preclude Inner Options on a RST, FIN or FIN-ACK.}
A sender MUST consider the sequence space consumed by InSpace
options, any padding and any Prefix Options as implicitly
acknowledged. Therefore, the sender has no need to hold these items
in its retransmit buffer. A sender MUST hold Suffix Options (and TCP
Payload, of course) in its retransmit buffer until they are
acknowledged.
These rules and those below concerning flow control and pure ACKs
have significant implications, which are discussed alongside their
rationale in Section 3.2.6.
2.3.1.4. Flow Control Coverage
The sender MUST count Suffix Options and the TCP Payload towards
consumption of the receive window advertised by the remote host.
Nonetheless, the sender MUST NOT count any padding, the InSpace
Option and any Prefix Options towards consumption of the advertised
receive window.
There might be a legacy middlebox on the path that discards segments
containing out-of-window data but does not understand the way the
Inner Space protocol modifies flow control. To traverse such a
middlebox, a sending implementation SHOULD use a modified flow
control algorithm that avoids the send window dropping below a
minimum threshold Snd.Wind.Min (instead of zero). Each sender
unilaterally chooses Snd.Wind.Min to allow for Fire-and-Forget
Objects it might need in flight on its half-connection. The
receiving sides of both half-connections play no part in this
allowance. Section Section 3.2.6.2 discusses the rationale for this
approach.
A reasonable value for the sender to choose for "Snd.Wind.Min" would
be twice the size of the fire-and-forget objects currently in flight.
Briscoe Expires September 10, 2015 [Page 20]
Internet-Draft Inner Space for all TCP Options March 2015
This would ensure that a middlebox still considers all the fire-and-
forget objects are in-window, even if a whole window were lost and
retransmitted.
2.3.1.5. Presence or Absence of Flow-Controlled Data
There are three types of acknowledgement segment:
1. An ACK containing no TCP Data is called a Pure ACK;
2. An ACK with no Flow-Controlled Objects (no TCP Payload and no
Suffix Options) but some Fire-and-Forget Objects (i.e. an InSpace
Option and possibly some padding and Prefix Options) is called an
Impure ACK
3. An ACK can be piggy-backed on a segment containing Flow-
Controlled In-Order Objects (either TCP Payload or Suffix
Options).
It is expected that impure ACKs will rarely be necessary. An example
of an Impure ACK is a segment containing no TCP Payload, but still
carrying a message authentication code (MAC) in a Prefix Option in
order to authenticate and protect the integrity of the TCP header of
the ACK.
If an Inner Space TCP implementation currently has no further TCP
Payload or Suffix Options to send, and it receives Impure ACKs, it
MUST NOT itself respond with further impure ACKs, i.e. it MUST NOT
consume further sequence space solely to acknowledge impure ACKs.
Nonetheless, while it has no further TCP Payload or Suffix Options to
send, it MAY cumulatively acknowledge the TCP Data in the impure ACKs
it has received by emitting a pure ACK, but no more often than once
per round trip time (see Section 3.2.6.2 for rationale). If it later
starts sending further Payload Data and/or Suffix Options, it will
cumulatively acknowledge the sequence space of all the TCP Data in
the intervening impure ACKs it has received, as would be expected.
If a sequence of one or more Impure ACKs is dropped, the receiver
will not know whether they were impure. The receiver's normal ACK
feedback will request a retransmission of the missing sequence space.
By definition, the sender does not hold fire-and-forget options in
its retransmit buffer. Therefore, the sender MUST reconstruct a new
impure ACK of at least the same size as the gap in fire-and-forget
options (if SACK has not been negotiated the sender will only know
the size of the gap up to any subsequent in-order objects). The
sender will include whatever Prefix options are relevant at the time
of retransmission (which might be none). If the size of the new
Briscoe Expires September 10, 2015 [Page 21]
Internet-Draft Inner Space for all TCP Options March 2015
Prefix Options is less than the gap to be filled, the sender MUST
make up the shortfall with noop Prefix Options. If the size of the
new Prefix Options is greater than the gap to be filled, no harm will
be done. This is because the receiver discards fire-and-forget
options after processing them, so any overflow will not overwrite
flow-controlled in-order data already in the receive buffer.
2.3.1.6. Construction Order for TCP Data
The sender constructs the TCP Data in the following order:
1. It writes any padding, the Inspace Option, Prefix Options, Suffix
Options and Payload Data into the TCP Data of the segment.
2. It applies any transformation of the data that might be required,
e.g. compression or encryption initiated by a previous control
message applied at the TCP layer.
If SYN=0, and if any such transformation is sensitive to the
delivery order of segments, the padding, InSpace Option and
Prefix Option MUST remain unaltered (because they need to be
processed as soon as they arrive, without waiting to fill gaps
in the sequence space).
3. If SYN=0, the sender MUST apply the zero overhead message
boundary insertion (ZOMBI) encoding to the segment (see
Appendix A).
2.3.2. Reading Inner TCP Options
The rules for reading Inner TCP Options are divided between the
following two subsections, depending on whether SYN=1 or SYN=0.
2.3.2.1. Reading Inner TCP Options (SYN=1)
This subsection applies when TCP receives a segment with SYN=1, e.g.
when the server receives a SYN or the client receives a SYN/ACK.
Before processing any TCP options, unless the size of the TCP Data is
less than 12 octets, an Upgraded Receiver MUST determine whether the
segment is an Upgraded Segment by checking that all the following
conditions apply:
o The first 4 octets of the segment match Magic Number A;
o The value of the Length field of the InSpace Option is 2;
o The value of Magic Number B in the InSpace Option is correct;
Briscoe Expires September 10, 2015 [Page 22]
Internet-Draft Inner Space for all TCP Options March 2015
o The value of the Sent Data Size matches the size of the TCP Data.
If all these conditions pass, the receiver MAY walk the sequence of
Inner TCP Options, using the length of each to check that the sum of
their lengths equals InOO. The receiver then concludes that the
received segment is an Upgraded Segment.
The receiver then processes the TCP Options in the following order:
1. Any Prefix TCP options (PrefixOpts in Figure 2)
2. Any Outer TCP options (OuterOpts in Figure 2);
3. Any Suffix TCP options (SuffixOpts in Figure 2)
The receiver removes the magic number, the InSpace Option and each
TCP Option from the TCP Data as it processes each.
The receiver MUST NOT count the size of Prefix Options against the
receive window. Strictly it ought to subtract the size of Suffix
Options from the receive window on arrival, then add the size back
again as it removes them. However, when SYN=1, the Suffix Options
will never have to be buffered, so these redundant steps can be
skipped.
Once only the TCP Payload (if any) remains, the receiver holds it
ready to pass to the application. It then emits the appropriate
Upgraded Acknowledgement to progress the handshake (see
Section 2.1.1).
If any of the above tests to find the InSpace Option fails:
1. the receiver concludes that the received segment is an Ordinary
Segment. It MUST then proceed by processing any Outer TCP
options in the TCP Header in the normal order (OuterOpts in
Figure 2).
2. If some previous control message causes the TCP receiver to alter
the TCP Data (e.g. decompression, decryption), it reruns the
above tests to check whether the altered TCP Data now looks like
an Upgraded Segment.
3. If it finds an InSpace Option, it suspends processing the Outer
TCP Options and instead processes and removes TCP Options in the
following order:
1. Any Prefix Inner Options;
Briscoe Expires September 10, 2015 [Page 23]
Internet-Draft Inner Space for all TCP Options March 2015
2. Any remaining Outer TCP Options;
3. Any Suffix Inner Options.
4. If it does not find an InSpace Option, it continues processing
the remaining Outer TCP Options as normal.
For the avoidance of doubt the above rules imply that, as long as an
InSpace Option has not been found in the segment, the receiver might
rerun the tests for it multiple times if multiple Outer TCP Options
alter the TCP Data. However, once the receiver has found an InSpace
Option, it MUST NOT rerun the tests for an Upgraded Segment in the
same segment.
If the receiver has not found an InSpace Option after processing all
the Outer Options, it emits the appropriate Ordinary Acknowledgement
to progress the handshake (see Section 2.1.1). As normal, it holds
any TCP Payload ready to pass to the application.
2.3.2.2. Reading Inner TCP Options (SYN=0)
This subsection applies once the TCP connection has successfully
negotiated to use the upgraded InSpace structure.
The receiver processes Prefix Options and Outer Options in the order
they are received. But it processes Suffix Options in the order they
were sent, which is not necessarily the order in which they are
received. The receiver achieves this by processing an arriving
segment with SYN=0 in the following order. (Steps 3 & 6 are included
for completeness even though no current TCP options apply data
transformations):
1. It buffers the TCP Data in sequence space order along with any
previously buffered data. There might be sequence gaps at this
stage.
2. It MUST then ZOMBI decode the buffered data Appendix A. If the
stream has not been resegmented, the process is straightforward,
but the following steps also check for the more general case
where resegmentation might have occurred:
A. When it finished ZOMBI decoding the immediately preceding TCP
Data, the receiver might have run out of data in the middle
of a segment and stored the outstanding segment length to
decode. If so, the receiver simply continues the unfinished.
ZOMBI decoding as long as there is contiguous data to decode.
Briscoe Expires September 10, 2015 [Page 24]
Internet-Draft Inner Space for all TCP Options March 2015
B. Otherwise, the receiver checks for a 0x0000 marker in the new
segment. It starts at the first 4-octet-aligned word in the
segment (counting from the ISN). If not present, it scans
the TCP Data for the first occurence of such a marker. It
classifies any data before the marker as undecoded
(conceivably it could find no marker, then the whole arriving
segment would remain buffered for later decoding).
C. Starting from the first marker found, the receiver reads the
SDS field from the InSpace option and runs the ZOMBI decode
algorithm over the extent of the sent data segment. It
repeats this for any following sent segments (which might be
present due to segment coalescing).
The receiver uses each InSpace Option to calculate the extent of
the associated Inner Options (using SOO and InOO).
3. It applies any order-insensitive transformation of the TCP Data
that might be required, e.g. counter-mode decryption initiated by
a previous control message applied at the TCP layer:
4. It MUST then remove the InSpace Option and it MUST process and
remove TCP options in the following order:
A. It processes and removes any Prefix TCP Options. (During the
decoding process the receiver might find Prefix Options on
multiple sent segments within a single newly arrived segment,
due to prior resegmentation.) Note: it does not subtract the
size of Fire-and-Forget Objects from the receive window.
B. It processes and removes any Outer TCP Options of the newly
arrived segment (note that if an arriving segment contains
multiple sent segments, the receiver processes all the Prefix
Options within it before processing any Outer Options).
C. It buffers Suffix Options and TCP Payload, subtracting from
the receive window ("Rcv.Wind") accordingly.
5. It emits an ACK if appropriate (typically using regular TCP ACK
behaviour, but see Section 2.3.1.5 concerning Impure ACKs).
6. Once gaps (if any) in the datastream have been filled, the
receiver applies any order-sensitive transformation of the TCP
data that might be required, e.g. decompression or decryption
initiated by a previous control message applied at the TCP layer:
A. The TCP receiver MUST apply an order-sensitive transformation
progressively, to one sent segment at a time in sequence
Briscoe Expires September 10, 2015 [Page 25]
Internet-Draft Inner Space for all TCP Options March 2015
order from the start of one Payload up to the end of the next
set of Suffix Options (which might change the way it
transforms the next segment, e.g. a rekey option).
B. Having established the extent of the next sent segment, The
receiver returns to step 6A.
7. It processes and removes any Suffix Options strictly in
datastream order, as illustrated in Figure 4a) in Section 3.2.6.
It adds to "Rcv.Wind" accordingly.
Once only the TCP Payload remains, the TCP receiver passes it to the
application as normal.
2.3.3. Forwarding Inner TCP Options
Middleboxes exist that process some aspects of the TCP Header. The
present specification defines a new location for Inner TCP Options
beyond the Data Offset, this is intended for the exclusive use of the
destination TCP implementation. Therefore:
o A middlebox MUST treat any octets beyond the Data Offset as
immutable user-data. Section 3.2.3 explains how the endpoints
will be able to force middleboxes to comply with this rule once
they can authenticate of even encrypt TCP options within the TCP
Data, whereas if they tried to enforce this rule today they would
only damage their own transmissions. Legacy Middleboxes already
do not expect to find options beyond the Data Offset anyway.
o A middlebox MUST NOT defer data in a segment with SYN=1 to a
subsequent segment.
A TCP implementation is not necessarily aware whether it is deployed
in a middlebox or in a destination, e.g. a split TCP connection might
use a regular off-the-shelf TCP implementation. Therefore, a
general-purpose TCP that implements the present specification will
need a configuration switch to disable any search for options beyond
the Data Offset and to enable immediate forwarding of data in a SYN.
2.4. Exceptions
{ToDo: Define behaviour of forwarding or receiving nodes if the
structure or format of an Upgraded Segment is not as specified.}
If an Upgraded TCP Receiver receives an InSpace Option with a Length
it does not recognise as valid, it MUST drop the packet and
acknowledge the octets up to the start of the unrecognised option.
Briscoe Expires September 10, 2015 [Page 26]
Internet-Draft Inner Space for all TCP Options March 2015
Values of Sent Data Size greater than 2^16 - 21 (=65,515 = 0xFFEB)
octets in a regular (non-jumbo) InSpace Option MUST be treated as the
distance to the next InSpace option, but they MUST NOT be taken as
indicative of the size of the TCP Data when it was sent. This is
because the TCP Data in a regular IPv6 packet cannot be greater than
(2^16 -1 - 20) octets (given the minimum TCP header is 20 octets).
If the size of the TCP Data is greater than 0xFFEB octets, the sender
MUST use a Jumbo InSpace Option (Appendix C.2).
A Sent Data Size of 0xFFFF octets MAY be used to minimise the
occurrence of empty InSpace options without permanently disabling the
Inner Space protocol for the rest of the connection.
2.5. SYN Flood Protection
An implementation of the Inner Space protocol MUST support the
EchoCookie TCP option [I-D.briscoe-tcpm-echo-cookie]. To indicate
its support for EchoCookie, an Ordinary Client would send an empty
EchoCookie TCP option on the SYN. Support for the Inner Space
protocol makes this redundant. Therefore an Inner Space client MUST
NOT send an empty EchoCookie TCP option on a SYN-U.
The EchoCookie TCP option replaces the SYN Cookie mechanism
[RFC4987], which only has sufficient space to hold the result of one
TCP option negotiation (the MSS), and then only a subset of the
possible values (see the discussion under Security Considerations
Section 7).
3. Design Rationale
This section is informative, not normative.
3.1. Dual Handshake and Migration to Single Handshake
In traditional [RFC0793] TCP, the space for options is limited to 40B
by the maximum possible Data Offset. Before a TCP sender places
options beyond that, it has to be sure that the receiver will
understand the upgraded protocol, otherwise it will confuse and
potentially crash the application by passing it TCP options as if
they were payload data.
The Dual Handshake (Section 2.1.1) ensures that a Legacy TCP Server
will never pass on TCP options as if they were user-data. If a SYN
carries TCP Data, a TCP server typically holds it back from the
application until the 3-way handshake completes. This gives the
client the opportunity to abort the Upgraded Connection if the
response from the server shows it does not recognise an Upgraded SYN.
Briscoe Expires September 10, 2015 [Page 27]
Internet-Draft Inner Space for all TCP Options March 2015
The strategy of sending two SYNs in parallel is not essential to the
Alternative SYN approach. It is merely an initial strategy that
minimises latency when the client does not know whether the server
has been upgraded. Evolution to a single SYN with greater option
space could proceed as follows:
o Clients could maintain a white-list of upgraded servers discovered
by experience and send just the Upgraded SYN-U in these cases.
o Then, for white-listed servers, the client could send an Ordinary
SYN only in the rare cases when an attempt to use an Upgraded
Connection had previously failed (perhaps a mobile client
encountering a new blockage on a new path to a server that it had
previously accessed over a good path).
o In the longer term, once it can be assumed that most servers are
upgraded and the risk of having to fall back to legacy has dropped
to near-zero, clients could send just the Upgraded SYN first,
without maintaining a white-list, but still be prepared to send an
Ordinary SYN in the rare cases when that might fail.
There is concern that, although dual handshake approaches might well
eventually migrate to a single handshake, they do not scale when
there are numerous choices to be made simultaneously. For instance:
o trying IPv6 then IPv4 [RFC6555];
o and trying SCTP and TCP in parallel
[I-D.wing-tsvwg-happy-eyeballs-sctp];
o and trying ECN and non-ECN in parallel;
o and so on.
Nonetheless, it is not necessary to try every possible combination of
N choices, which would otherwise require 2^N handshakes (assuming
each choice is between two options). Instead, a selection of the
choices could be attempted together. At the extreme, two handshakes
could be attempted, one with all the new features, and one without
all the new features.
3.2. Inner Option Space
3.2.1. Header Extension by Encapsulation
It has been proposed [Briscoe14] that extension of a header (as
opposed to options) at layer X ought not to be located within the
header at layer X, but instead within the layer encapsulated by that
Briscoe Expires September 10, 2015 [Page 28]
Internet-Draft Inner Space for all TCP Options March 2015
header (layer X+1), for a selection of principled and pragmatic
reasons:
1. Implementations of layer X that have not implemented or are not
interested in an extension to layer X need not be bothered with
walking over a load of extensions they do not know or care about.
2. An extension always requires a new implementation, which can be
coded to know where to look for the extensions it implements;
extensions never need to be located where unmodified code can
find them.
3. Layer-X middleboxes that do not correctly forward layer-X
extensions are common, but they do tend to forward their layer-
X+1 payload correctly. Therefore extending layer-X within an
encapsulation is more likely to traverse badly designed
middleboxes.
4. Extension by encapsulation is not a manifesto for extending layer
X at layer X+1, X+2,... and ever-deeper. Usually a base protocol
design is sound, and an an extension is not permanently necessary
to make it fit for purpose; the extension merely adds something
needed in circumstances not originally conceived. Therefore it
is rare that an extension becomes so ubiquitous that extensions
to the extension become necessary.
5. Extending layer X within a layer-X+1 encapsulation should not be
confused with an attempt to evade security middleboxes. If an
attack on layer X is encapsulated in layer X+1, security
middleboxes will be reprogrammed to block it. Whereas, if a
useful extension to layer X were encapsulated in layer X+1,
security middleboxes would not be reprogrammed to block it.
6. If the endpoints of layer X don't want layer-X middleboxes to
intervene in their layer-X extension, they can encapsulate it
within layer X+1. In contrast, if they want an extension for co-
operation with layer-X middleboxes, they can place it in the
layer-X header. Then everything at layer X+1 can be
authenticated and/or encrypted to structure and enforce the
distinction between the types of extension, without having to
selectively authenticate and/or encrypt parts of the layer X
header.
3.2.2. Non-Deterministic Magic Number Approach
This section justifies the magic number approach by contrasting it
with a more 'conventional' approach. A conventional approach would
Briscoe Expires September 10, 2015 [Page 29]
Internet-Draft Inner Space for all TCP Options March 2015
use a regular (Outer) TCP option to point to the dividing line within
the TCP Data between the extra Inner Options and the TCP Payload.
This 'conventional' approach cannot provide extra option space over a
path on which a middlebox strips TCP options that it does not
recognise. [Honda11] quantifies the prevalence of such paths. It
reports on experiments conducted in 2010-2011 that found unknown
options were stripped from the SYN-SYN/ACK exchange on 14% of paths
to port 80 (HTTP), 6% of paths to port 443 (HTTPS) and 4% of paths to
port 34343 (unassigned). Further analysis found that the option-
stripping middleboxes fell into two main categories:
o about a quarter appeared to actively remove options that they did
not recognise (perhaps assuming they might be indicative of an
attack?);
o the rest were some type of higher layer proxy that split the TCP
connection, unwittingly failing to pass unknown options between
the two connections.
The magic number approach ensures that all the TCP Headers and
options up to the Data Offset are completely indistinguishable from
an Ordinary Segment. Therefore, it will be highly likely (but not
certain--see Appendix C.1.4) that the extra Inner Options will always
be forwarded, while the conventional approach would fall far short of
ths ideal.
The magic number approach also ensures that the Inner Options and the
option that points to them are both tucked away beyond the Data
Offset (see Section 2.2.1). This makes it highly likely that the two
will share the same fate--it would be extremely unusual for a
middlebox to treat different parts of the TCP Data selectively.
Typically, if a TCP option were stripped, the concern would only be
lack of function, not safety. But with option space extension, the
concern is serious application corruption. If control options are
placed beyond the Data Offset, and the option that says they are
there gets stripped, it risks control options being passed to the
application as (corrupt) data. Although option stripping can be
detected during the handshake, this consumes round trips and it is
does not guarantee that option stripping will not start part-way
through a connection (e.g. due to a path change). In contrast the
magic number approach is inherently safe.
The downside of the magic number approach is that it is slightly non-
deterministic, quantified as follows:
Briscoe Expires September 10, 2015 [Page 30]
Internet-Draft Inner Space for all TCP Options March 2015
o The probability that an Upgraded SYN=1 segment will be mistaken
for an Ordinary Segment is precisely zero.
o In the currently common case of a SYN with zero payload, the
probability that it will be mistaken for an Upgraded Segment is
also precisely zero.
o However, there will be a very small probability (roughly 2^{-66}
or 1 in 74 billion billion (74 * 10^18)) that payload data in an
Ordinary SYN=1 segment could be mistaken for an Upgraded SYN or
SYN/ACK, if it happens to contain a pattern in exactly the right
place that matches the correct Sent Data Size, Length and Magic
Numbers of an InSpace Option. {ToDo: Estimate how often a
collision will occur globally. Rough estimate: 1 connection
collision globally every 40 years.}
The above probability is based on the assumptions that:
o the magic numbers will be chosen randomly (in reality they will
not--for instance, a magic number that looked just like the start
of an HTTP connection would be rejected)
o data at the start of Ordinary SYN=1 segments is random (in reality
it is not--the first few bytes of most payloads are very
predictable).
Therefore even though 2^{-66} is a vanishingly small probability, the
actual probability of a collision will be much lower.
If a perfect collision does occur, it will result in TCP removing a
number of 32-bit words of data from the start of a byte-stream before
passing it to the application.
3.2.3. Non-Goal: Security Middlebox Evasion
The purpose of locating control options within the TCP Data is not to
evade security. Security middleboxes can be expected to evolve to
examine control options in the new inner location. Instead, the
purpose is to traverse middleboxes that block new TCP options
unintentionally--as a side effect of their main purpose--merely
because their designers were too careless to consider that TCP might
evolve. This category of middleboxes tends to forward the TCP
Payload unaltered.
By sitting within the TCP Data, the Inner Space protocol should
traverse enough existing middleboxes to reach critical mass and prove
itself useful. In turn, this will open an opportunity to introduce
integrity protection for the TCP Data (which includes Inner Options).
Briscoe Expires September 10, 2015 [Page 31]
Internet-Draft Inner Space for all TCP Options March 2015
Whereas today, no operating system would introduce integrity
protection of Outer TCP options, because in too many cases it would
fail and abort the connection.
Once the integrity of Inner Options is protected, it will raise the
stakes. Any attempt to meddle with control options within the TCP
Data will not just close off the theoretical potential benefit of a
protocol advance that no-one knows they want yet; it will fail
integrity checks and therefore completely break any communication.
It is unlikely that a network operator will buy a middlebox that does
that.
Then middlebox designers will be on the back foot. To completely
block communications they will need a sound justification. If they
block an attack, that will be fine. But if they want to block
everything abnormal, they will have to block the whole communication,
or nothing. So the operator will want to choose middlebox vendors
who take much more care to ensure their policies track the latest
protocol advances--to avoid costly support calls.
3.2.4. Avoiding the Start of the First Two Segments
Some middleboxes discard a segment sent to a well-known port
(particularly port 80) if the TCP Data does not conform to the
expected app-layer protocol (particularly HTTP). Often such
middleboxes only parse the start of the app-layer header (e.g. Web
filters only continue until they find the URL being accessed, or DPI
boxes only continue until they have identified the application-layer
protocol).
The segment structure defined in Section 2.2.1 would not traverse
such middleboxes. An alternative segment structure that avoids the
start of the first two segments in each direction is defined in
Appendix C.3. It is not mandatory to implement in the present
specification. However, it is hoped that it will be included in some
experimental implementations so that it can be decided whether it is
worth making mandatory.
3.2.5. Framing Segments
A middlebox that splits a TCP connection can coalesce and/or divide
the original segments. Segmentation offload hardware is another
common cause of resegmentation. Inclusion of the marker in the
InSpace Option allows the receiver to reconstruct the original
segment boundaries. The ZOMBI encoding Appendix A removes any
occurrences of the marker other than those at the start of each
segment.
Briscoe Expires September 10, 2015 [Page 32]
Internet-Draft Inner Space for all TCP Options March 2015
Superficially, the receiver does not need the sent data size (SDS)
field to find the end of each sent segment; it could scan for the
marker at the start of the next segment instead. However, in the
common case when a stream has _not_ been resegmented, the receiver
will find the marker at the start of the segment, but the next marker
will not have been received yet. The SDS field allows the receiver
to know immediately whether a whole segment has been received as
sent. For the same reason, Minion [I-D.iyengar-minion-protocol] uses
a (different) marker to tag the end of each message. In contrast,
the Inner Space approach uses 2B to declare the original segment
size, which saves having to scan the stream for an end marker.
Equally, one could argue that markers are unnecessary, because the
sequence of sent data size fields from the start of the stream seem
sufficient to find all the segment boundaries. Using markers ensures
that the receiver can pick out segment boundaries immediately on
arrival, which is important for deadlock avoidance (see
Section 3.2.6).
The Sent Data Size is not strictly necessary on a SYN (SYN=1, ACK=0)
because a SYN is never resegmented. However, for simplicity, the
layout for a SYN is made the same as for a SYN/ACK. This future-
proofs the protocol against the possibility that SYNs might be
resegmented in future. And it makes it easy to introduce the
alternative segment structure of Appendix C.3 if it is needed.
3.2.6. Control Options Within Data Sequence Space
Section 2.3 introduced the two types of objects that Inner Space
places within the TCP Data:
In-Order Flow-Controlled Objects: Suffix Options and the TCP
Payload;
Fire-and-Forget Objects: Padding, the InSpace Option and any Prefix
Options.
The following two sections address each in turn: i) explaining why it
is useful to introduce in-order flow-controlled TCP options and ii)
explaining why it is feasible to encapsulate fire-and-forget options
within the TCP datastream, despite its reliable ordered semantics.
3.2.6.1. In-Order Flow-Controlled Options
Including Suffix Options within TCP's sequence space gives the sender
a simple way to ensure that control options will be delivered
reliably and in order to the remote TCP, even if the control options
are on segments without user-data. By using TCP's existing stream
Briscoe Expires September 10, 2015 [Page 33]
Internet-Draft Inner Space for all TCP Options March 2015
delivery mechanisms, it adds no extra protocol processing, no extra
packets and no extra bits.
The sender can even choose to place control options on a segment
without user-data, e.g. to reliably re-key TCP-level encryption on a
connection currently sending no data in one direction. The sender
can even add an InSpace Option without further Inner Options except a
no-op Suffix option. Then it can ensure that the segment will
automatically be delivered reliably and in order to the remote TCP,
even though it carries no user-data or other TCP control options,
e.g. for a test probe, a tail-loss probe or a keep-alive.
Figure 4a) illustrates control options arriving reliably and in order
at the receiving TCP stack in comparison with the traditional
approach shown in Figure 4b), in which control options are outside
the sequence space. In the traditional approach, during a period
when the remote TCP is sending no user-data, the local TCP may
receive control options E, B and D without ever knowing that they are
out of order, and without ever knowing that C is missing.
a) __ ____ _______ _ __
|__|____|_______|_| |__| control
:E : D : C :B: :A :
________________: : : : :__________________: :
|________________| |__________________| data
b) __
|__| E
|_|__ B __
|____|D |__|A control
\ / \ /
________________\/__________________\/
|________________||__________________| data
!
!drop
____!__
|_______|C
Figure 4: Control options a) inside vs. b) outside TCP sequence
space`
By including Inner Options within the sequence space, each control
option is automatically bound to the start of a particular byte in
the data stream, which makes it easy to switch behaviour at a
specific point mid-stream (e.g. re-keying or switching to a different
control mode). With traditional TCP options, a bespoke reliable and
ordered binding to the data stream would have to be developed for
each TCP option that needs this capability (e.g. co-ordinating use
Briscoe Expires September 10, 2015 [Page 34]
Internet-Draft Inner Space for all TCP Options March 2015
of new keys in TCP-AO [RFC5925] or tcpcrypt
[I-D.bittau-tcpinc-tcpcrypt]).
Including Inner Options in sequence also allows the receiver to tell
the sender the exact point at which it encountered an unrecognised
TCP option using only TCP's pre-existing byte-granularity
acknowledgement scheme.
Middleboxes exist that rewrite TCP sequence and acknowledgement
numbers, and they also rewrite options that refer to sequence numbers
(at least those known when the middlebox was produced, such as SACK,
but not any introduced afterwards). If Inner Options were not
included in sequence, the number of bytes beyond the TCP Data Offset
in each segment would not match the sequence number increment between
segments. Then, such middleboxes could unintentionally corrupt the
user-data and options by 'normalising' sequence or acknowledgement
numbering. Fortunately, including Inner Options in sequence improves
robustness against such middleboxes.
3.2.6.2. Fire-and-Forget Options
The Inner Space protocol allows Fire-and-Forget Options to be
tunnelled within the TCP Data so that they can traverse middleboxes
that would otherwise strip them or somehow normalise their contents.
Two question then arise: i) should Fire-and-Forget Objects (padding,
the InSpace Option and Prefix Options) consume sequence space and ii)
should they be covered by flow control? The answers to these
questions will also be re-usable to multiplex streams within one TCP
connection:
Sequence Space: Ideally, fire-and-forget objects would not consume
sequence space, because they do not need to be retransmitted.
However, many middleboxes expect the TCP sequence number to
increment consistently with the amount of TCP Data. For instance,
a split connection would be likely to 'normalise' sequence
numbers, being unaware that certain items in the datastream might
be exempt from sequence space consumption.
Therefore, although it is not elegant, the sender has to consume
sequence space for fire-and-forget objects, but it implicitly
considers these octets to be immediately acknowledged. And the
receiver does not have to immediately acknowledge sequence space
consumed solely by fire-and-forget objects; it can defer until it
acknowledges reliably delivered flow-controlled objects--when it
does no harm to cumulatively acknowledge intervening fire-and-
forget objects as well. This is the underlying principle behind
the normative rules given on sequence space consumption and ACK
withholding in Section 2.3.1.3 and Section 2.3.1.5.
Briscoe Expires September 10, 2015 [Page 35]
Internet-Draft Inner Space for all TCP Options March 2015
Flow Control: The sender does not need to count Fire-and-Forget
Objects against the receive window ("Rcv.Wind"), just as it does
not count Outer TCP Options against "Rcv.Wind".This should work
because It is impossible for middleboxes to 'normalise' the
receive window and flow control, because they cannot know when the
application is releasing data from the receive buffer. Also the
receiver always processes Fire-and-Forget Objects immediately
without buffering them; it could be considered that the receiver
effectively subtracts their size from "Rcv.Wind" then immediately
restores "Rcv.Wind" to its former value.
In fact, as shall now be explained, it has to be _mandatory_ for
the sender not to count fire-and-forget objects against
"Rcv.Wind". It is important for deadlock avoidance that certain
TCP options never consume "Rcv.Wind". Some TCP options
acknowledge data, e.g. SACK or the Data ACK within the Data
Sequence Signal (DSS) sub-option of MPTCP. Other TCP options need
to be applied to all ACKs, e.g. the MAC of tcpcrypt. If an
acknowledgement were to need sufficient advertised receive window
before it could be sent, there would always be a risk of deadlock
if the receiver ever needed the acknowledgement before it could
release more receiver buffer [Raiciu12].
The rule above concerning sequence space is a compromise needed to
traverse middleboxes. So, perhaps predictably, this begets further
compromises. The rule concerning flow-control is principled. So
perhaps predictably, it has to be compromised to traverse certain
middleboxes. The rationale for these compromises is explained below,
referring to the normative rules in the protocol specification where
appropriate:
Sequence Space: If the sender does not retransmit unacknowledged
data after a RTO, some middleboxes will mimic TCP's retransmission
timeout (RTO) and resend the fire-and-forget data themselves,
which could lead to an ACK storm. Therefore, Section 2.3.1.5
allows a receiver to emit a pure ACK every round trip, just to
keep such middleboxes quiet. In general, allowing TCP to ACK an
ACK can lead to an ACK storm. However, in this case, all that is
allowed is a Pure ACK in response to an Impure ACK, which
immediately terminates any potential for a vicious circle. This
solution even works in the case where both TCP hosts ignore ACKs
unless they are authenticated (which the pure ACK will not be).
No harm will arise if the remote host ignores the pure ACK,
because it is only for the benefit of a middlebox anyway.
If a sequence of one or more Impure ACKs is lost the receiver
cannot suppress retransmission, because it can only decide whether
it needed in-sequence data once it arrives. Therefore, loss of
Briscoe Expires September 10, 2015 [Page 36]
Internet-Draft Inner Space for all TCP Options March 2015
fire-and-forget data causes a retransmission that may prove to be
unnecessary. By the rules in Section 2.3.1.5, an ACK would only
include fire-and-forget data in the first place if it was actually
necessary. Therefore, normally retransmission of Impure ACKs will
be required and useful. However, sometimes, the Prefix Option(s)
within the Impure ACK(s) might have become unnecessary. This
inefficiency could just be ignored, or partial reliability could
be added to TCP to address it. The Inner Space protocol does not
prevent partial reliability being added, but it does not require
it either.
Flow Control: Some middleboxes attempt to mitigate scanning or DoS
attacks by reading the window field in the main TCP header (and
the Window Scale outer TCP option if present) and discarding
segments that they calculate contain data that is out-of-window.
Section Section 2.3.1.4 requires the two endpoints to tacitly
agree that the fire-and-forget portion of the TCP Data is exempt
from flow control. A legacy middlebox will not know this, so it
might think data is out-of-window when the endpoints have agreed
it is in-window. Section Section 2.3.1.4 provides a solution to
this problem, which is only necessary if a TCP implementation is
deployed where there is a risk of encountering such middleboxes.
The solution involves the TCP sender denying itself the use of the
bottom of the buffer advertised by the receiver. Normally the
sender stops sending when it calculates the remaining receive
window is zero. Instead, the modified sender sets itself a
threshold (Snd.Wind.Min) to allow for the Fire-and-Forget Objects
it might need in flight, and it stops sending before the receive
window drops below this threshold.
Snd.Wind.Min bytes at the 'left-hand' end of the receive buffer
are wasted by this solution (to be fair, the middlebox behaviour
is really to blame). An alternative was considered where the
sender and receiver use a new Inner TCP Option to agree a window
offset between themselves, so that middleboxes are not party to
their agreement. Although, this would not waste any of the left-
hand end of the receive buffer, it would reduce the maximum
advertised buffer at the right-hand end by the same amount.
Therefore the sender-only solution was chosen, given it is much
simpler, and the sender can continuously adapt how much allowance
it sets aside throughout the connection, rather than having to
commit to a necessarily conservative estimate at the start.
Briscoe Expires September 10, 2015 [Page 37]
Internet-Draft Inner Space for all TCP Options March 2015
3.3. Deployment Approach
3.3.1. Substrate Protocol: TCP vs. UDP
Inner Space uses TCP as a substrate protocol, i.e. on the wire, the
headers look like an RFC793-compliant TCP, and there is only a
difference if one looks inside the TCP Data. Other transport
extensibility approaches have used UDP as a substrate protocol, for
instance, to carry SCTP through middleboxes.
In design and implementation terms, it is much easier to turn UDP
into a reliable protocol, than it is to selectively turn TCP into an
unreliable protocol. However, UDP is already blocked on about 15% of
Internet paths {ToDo: ref}, whereas vanilla TCP is still universally
permitted. Therefore, because the goal is middlebox traversal, not
just ease of implementation, Inner Space uses TCP as a substrate.
It may well turn out that Inner Space cannot reach some places that
UDP can. It is expected that applications (or even the TCP stack)
might sometimes have to resort to tryinging UDP as a substrate in
such cases.
3.3.2. Kernel-Space vs. User-Space
At an earlier stage in the specification of the Inner Space protocol
[I-D.briscoe-tcpm-inner-space] before unordered delivery of Inner
Options was introduced, Inner Options could all be processed in
either user-space or kernel-space. The only exception was the
interactions controlling the handshake on the first segment in each
direction. However, with the addition of unordered delivery of
Prefix Options, the protocol has to be implemented in the kernel,
because the protocol modifies the behaviour of TCP, not just its
payload.
3.4. Rationale for the InSpace Option Format
The format of the InSpace Option (Figure 3) does not necessarily have
to comply with the RFC 793 format for TCP options, because it is not
intended to ever appear in a sequence of TCP options. In particular,
it does not need an Option Kind, because the option is always in a
known location. In effect the magic number serves as a multi-octet
Option Kind for the first InSpace Option, and the location of each
subsequent option is always known by the marker in the InSpace option
as well as by the offset from the previous one, using the Sent Data
Size field.
Other aspects of the layout are justified as follows:
Briscoe Expires September 10, 2015 [Page 38]
Internet-Draft Inner Space for all TCP Options March 2015
Length: Whatever the size of the InSpace Option, the right-hand edge
of the Length field is always located 8 octets from the left-hand
edge of the marker that starts the InSpace Option. From the
Length, the receiver can always determine the layout of the rest
of the option. The length is in 4-octet words because the InSpace
option is always a multiple of 4 octets long, so that any
subsequent Inner TCP Options comply with TCP's option alignment
requirements.
Sent Data Size: This field is 16 bits wide, which is reasonable
given segment size cannot exceed the limits set by the Total
Length field in the IPv4 header and the Payload Length field in
the IPv6 header, both of which are 16 bits wide.
If the sender were to use a jumbogram [RFC2675], it could use the
Jumbo InSpace Option defined in Appendix C.2, which offers a
32-bit Sent Data Size field. The Jumbo InSpace Option is not
mandatory to implement for the present experimental specification.
Even if it is implemented, it is only defined when SYN=0, given
use of a jumbogram for a SYN or SYN/ACK would significantly exceed
other limits that TCP sets for these segments.
Inner Options Offset: This field is in units of 4-octet words, so
its width is 14-bits. Then, if necessary Suffix Options can be as
large as a maximum sized segment (given 4 * 2^14 = 2^16 octets).
Suffix Options Offset: The InOO field is the same 14-bit width as
the SOO field, and for the same reason. Both the SOO and InOO
fields are aligned 2 bits to the left of a word boundary so that
they can be used directly in units of octets by masking out the
2-bit field to the right.
When SYN=1 the layout of the InSpace Option includes:
Magic Numbers: The 32-bit size of Magic Number A is not enough to
reduce the probability of mistaking the start of an Ordinary SYN
Payload for the start of the Inner Space protocol. A 64-bit magic
number could have been provided by using the next 4-octet word,
but this would be unnecessarily large. Therefore, when SYN=1,
Magic Number B provides 16 more bits of magic number. Otherwise,
these 16-bits would only have to be used for padding to align with
the next 4-octet word boundary anyway.
When SYN=0, the following further considerations determined the
layout of the InSpace Option:
ZOMBI: The ZOMBI field holds an offset that has to be sufficiently
wide to span the extent of a maximum-sized segment of 2^16 bits.
Briscoe Expires September 10, 2015 [Page 39]
Internet-Draft Inner Space for all TCP Options March 2015
Given the offset is measured in 2-octet units, this means the
ZOMBI field has to be at least 15 bits wide (see Appendix C.2 for
the size of the ZOMBI field for a jumbogram).
Marker: Given occurrences of the marker are replaced by offsets of
the size of the ZOMBI field, the marker has to be at least as wide
as the ZOMBI field. However, a 16-bit marker is used, because it
is more efficient than having to replace 15-bit markers.
Currently Unused (CU): There are three CU fields in the InSpace
option when SYN=0 that fill odd corners of space. Unfortunately,
this is necessary to ensure 4-octet alignment of the first Inner
Options.
Prefix (P) flag: When there are solely Prefix Options, or solely
Suffix Options, a short-form InSpace Option can be used (Len = 1)
by omitting the last 4-octet word. Then the P flag determines
whether there are solely Prefix Options or solely Suffix Options
in the Inner Options field. Whenever both Prefix and a Suffix
Option are needed on the same segment, even though only 14 more
bits of framing information are needed, the InSpace option has to
grow in steps of 32 bits to maintain 4-octet alignment. Therefore
18 bits have to be assigned as Currently Unused (CU).
4. Protocol Overhead
The overhead of the Inner Space protocol is quantified as follows:
Dual Handshake:
Latency:
Upgraded Server : zero;
Legacy Server: worst latency of the two, if dual handshakes
are used.
Connection Rate: The typical connection rate will inflate by P*D,
where:
P [0-100%] is the proportion of connections that use extra
option space;
D [0-100%] is the proportion of these that use a dual
handshake (the remainder use a single handshake, e.g. by
caching knowledge of upgraded servers).
Briscoe Expires September 10, 2015 [Page 40]
Internet-Draft Inner Space for all TCP Options March 2015
For example, if P=80% and D=10%, the connection rate will
inflate by 8%. P is difficult to predict. D is likely to be
small, and in the longer term it should reduce to the
proportion of connections to remaining legacy servers, which
are likely to be the less frequently accessed ones. In the
worst case if both P & D are 100%, the maximum that the
connection rate can inflate by is 100% (i.e. to twice present
levels).
Connection State: Connection state on servers and middleboxes
will inflate by P*D/R, where
R is the average hold time of connection state measured in
round trip times
This is because a server or middlebox only holds dual
connection state for one round trip, until the RST on one of
the two connections. For example, keeping P & D as they were
in the above example, if R = 3 round trips {ToDo: TBA},
connection state would inflate by 2.7%. In the longer term, any
extra connection state would be focused on legacy servers, with
none on upgraded servers. Therefore, if memory for dual
handshake flow state was a problem, upgrading the server to
support the Inner Space protocol would solve the problem.
Network Traffic: The network traffic overhead is 2*H*P*D/J
counting in bytes or 2*P*D/K counting in packets, where
H is (h+60B+12B) where h is the IP header size (assuming the
Ordinary SYN and SYN/ACK have a TCP header packed to the
maximum of 60B with TCP options, they have no TCP Payload,
their IP headers have no extensions and the InSpace Option
in the SYN-U and SYN/ACK-U is 12B). That is H will be 92B
for IPv4 or 112B for IPv6;
J is the average number of bytes per TCP connection (in both
directions)
K is the average number of packets per TCP connection (in both
directions);
For example, keeping and P & D as they were in the above
example, if J = 50KiB for IPv4 and K = 70 packets (ToDo: TBA),
traffic overhead would be 0.03% counting in bytes or 0.2%
counting in packets.
Processing: {ToDo: Implementation tests}
Briscoe Expires September 10, 2015 [Page 41]
Internet-Draft Inner Space for all TCP Options March 2015
InSpace Option on every non-empty SYN=0 segment:
Network Traffic: The traffic overhead is P*Q*8/F, where
Q is the proportion of Inner Space connections that leave the
protocol enabled after the initial handshake;
F is the average frame size in bytes (assuming one segment per
frame).
This assumes an InSpace option adds 8B per segment (i.e. both
Prefix and Suffix Options together on every segment will be
rare). For example, keeping P as it was in the above example
and taking Q=10% and F=750B, the traffic overhead is 0.09%. It
is as difficult to predict Q as it is to predict P.
Processing: {ToDo: Implementation tests}
5. Interaction with Pre-Existing TCP Implementations
5.1. Compatibility with Pre-Existing TCP Variants
It is believed that all TCP options that were designed as Outer
Options can be relocated without alteration as Prefix Options,
because the unreliable unordered semantics are the same as TCP Outer
Options. However, some yet-to-be-defined TCP options might be better
suited to the reliable ordered semantics of Suffix Options.
Specifically, existing or proposed TCP options fall into the
following categories:
Segment-Related: Concerned with the delivery of individual segments
as they arrive at the receiver. Therefore these options MUST NOT
be located as Suffix Options:
* Timestamp [RFC7323] on SYN=0 segments;
* SACK [RFC2018];
* The Data ACK part of the DSS option of Multipath TCP [RFC6824];
* TCP-AO [RFC5925] if covering TCP Options;
Stream-Related: Controlling delivery of an ordered stream.
Therefore these options SHOULD be located as Suffix Options:
* The tcpcrypt CRYPT sub-options [I-D.bittau-tcpinc-tcpcrypt].
Briscoe Expires September 10, 2015 [Page 42]
Internet-Draft Inner Space for all TCP Options March 2015
Connection-Related: Controlling the parameters of a connection.
These options can be located either as Suffix, Prefix or Outer
Options:
* No-op and end of option list [RFC0793];
* Maximum Segment Size (MSS) [RFC0793];
* SACK-ok [RFC2018];
* The timestamp when used on SYN=1 segments to indicate support
for timestamps [RFC7323];
* Window Scale [RFC7323];
* Multipath TCP [RFC6824], except the Data ACK part of the Data
Sequence Signal (DSS) option;
* TCP Fast Open [I-D.ietf-tcpm-fastopen];
{ToDo: The above list is not authoritative. Some TCP options include
suboptions, some of which are discussed below, but others remain to
be fully assessed.}
The specification of any future TCP option MUST state whether it is
designed as a Suffix Option (reliable ordered) or as a Prefix / Outer
Option (unreliable unordered) or "Don't Care". A TCP option MUST by
default only be used as an Outer or Prefix Option, unless it is
explicitly specified that it can (or must) be used as a Suffix
Option.
The Inner Space protocol supports TCP Fast Open, by constraining the
client to obey the rules in Section 2.3.1.1).
All the sub-types of the MPTCP option [RFC6824] except one could be
located as Suffix or Prefix Options. That is, MP_CAPABLE, MP_JOIN,
ADD_ADDR(2), REMOVE_ADDR, MP_PRIO, MP_FAIL, MP_FASTCLOSE. The Data
Sequence Signal (DSS) of MPTCP consists of four separable parts: i)
the Data ACK; ii) the mapping between the Data Sequence Number and
the Subflow Sequence Number over a Data-Level Length; iii) the
Checksum; and iv) the DATA_FIN flag. If MPTCP were re-factored to
take advantage of the Inner Space protocol, all these parts except
the Data ACK could be located as Suffix Options (the Checksum would
not be necessary).
The MPTCP Data ACK has to remain as a Prefix or Outer Option
otherwise there would be a risk of flow control deadlock, as pointed
out in [Raiciu12]. For instance, a Web client might pipeline
Briscoe Expires September 10, 2015 [Page 43]
Internet-Draft Inner Space for all TCP Options March 2015
multiple requests that fill a Web server's receive buffer, while the
Web server might be busy sending a large response to the first
request before it reads the second request. If the Data ACK were a
Suffix Option, the Web client would have to stop acknowledging the
first response from the server (due to lack of receive window). Then
the server would not be able to move on to the next request--a
classic deadlock.
The TCP authentication option can be configured either to cover TCP
Options or not (when it was defined only Outer Options existed). If
it covers any TCP Options it has to be located as an Outer or Prefix
Option to prevent the possibility of flow-control deadlock (because
it would consume receive window on pure ACKs if it were located as a
Suffix Option).
All sub-options of the tcpcrypt CRYPT option could be located as
Suffix Options. However, as long as the tcpcrypt MAC option covers
the TCP header and Outer Options, it has to be located as an Outer
Option for the same deadlock reason as TCP-AO.
An Upgraded Server can support SYN Cookies [RFC4987] for Ordinary
Connections. For Upgraded Connections Section 2.5 defines a new
EchoCookie TCP option that is a prerequisite for InSpace
implementations, and provides sufficient space for the more extensive
connection state requirements of an InSpace server.
{ToDo: TCP States and Transitions, Connectionless Resets, ICMP
Handling, Forward-Compatibility.}
5.2. Interaction with Middleboxes
The interaction with the assumptions about TCP made by middleboxes is
covered extensively elsewhere:
o Section 2.3.3 specifies forwarding behaviour for Inner Options;
o The following sections explain the Inner Space protocol approach
to middlebox traversal:
* Section 3.2.1 justifies extending TCP within the TCP Data;
* Section 3.2.2 justifies the magic number approach;
* Section 3.2.3 explains why the protocol will remain robust as
middlboxes evolve;
* Section 3.2.6 justifies including Inner Options in sequence;
Briscoe Expires September 10, 2015 [Page 44]
Internet-Draft Inner Space for all TCP Options March 2015
* Section 3.2.5) explains how the protocol will remain robust to
resegmentation.
5.3. Interaction with the Pre-Existing TCP API
An aim of the Inner Space protocol is for legacy applications to
continue to just work without modification. Therefore it is expected
that the dual handshaking logic and placement of options within the
TCP Data will be implemented beneath the well-known socket interface.
Inner Space implementations will need to comply with the following
behaviours to ensure that legacy applications continue to receive
predictable behaviour from the socket interface:
Querying local port (TCP client): If an application calls
"getsockname()" while the TCP client behind the socket is engaged
in a dual TCP handshake, the call SHOULD block until the local TCP
has aborted one of the connections so it knows which of the two
ports will continue to be used.
Binding to an explicit port: If an application specifies that it
wants the TCP client to use a specific port, the Inner Space
capability can be used, but the dual handshake MUST be disabled,
because the dual handshake has to try two ports. Therefore, if
the app binds to a specific port, the upgraded SYN MUST be tried
first on its own, then if that reveals that the server is not
upgraded, the stack will abort that connection with a RST and use
the same port to send an ordinary SYN. Use of a specific port
might be necessary, for example in the FTP protocol, in a port-
testing application or if the application wants to explicitly
control all the handshaking logic of the Inner Space protocol
itself.
Logging: The dual handshake will show up as a specific signature in
logs of network activity. Log formats might not be able to record
two local ports against one socket, so logs might contain
unexpected or erroneous data. Even if logs correctly track both
connection attempts, log analysis software might not expect to see
one socket attempt to use two different ports. {ToDo: All this
needs to be turned into a predictability requirement.}
Note that Inner Space has no impact on queries for the remote port
from a TCP server. If an application calls "getpeername()" while the
TCP server behind the socket is (unwittingly) engaged in a dual
handshake, it will return the port of the remote client, even though
this connection might subsequently be aborted. This is because a TCP
server is not aware of whether it is part of a dual handshake.
Briscoe Expires September 10, 2015 [Page 45]
Internet-Draft Inner Space for all TCP Options March 2015
Some applications interrogate the TCP stack to determine the path max
transmission unit (PMTU), e.g. in order to optimize application
message boundaries within the datastream. From the viewpoint of such
applications, TCP options subtract the same amount from the PMTU
whether they are Outer or Inner Options. However, the 8 (or 12)
octet InSpace header and the alignment padding represent extra
overhead. Therefore, for such applications, the TCP stack as seem
through the socket API will seem similar to a tunnel that reduces the
useful size of the PMTU. This could lead to fragmentation until such
applications are updated. Nonetheless, most such applications
already include code to adapt to PMTU reduction by tunnels.
It would be appropriate to enable the Inner Space protocol on a per-
host or per-user basis. The necessary configuration switch does not
need to be standardised, but it might allow the following three
states:
Enabled: The stack will enable Inner Space on any TCP connection
that that needs Inner Space for its TCP options. The stack might
still disable the Inner Space protocol autonomously after the
initial handshake if it is not needed.
Forwarding: The Forwarding mode is for TCP implementations on
middleboxes that implement split TCP connections, as discussed in
Section 2.3.3. Forwarding mode is similar to Disabled, except it
forwards data in SYN without deferring it until the incoming
connection is established.
Disabled: Inner Space is not enabled by default on any connections,
except those that specifically request it.
The socket API might also need to be extended for future applications
that want to control the Inner Space protocol explicitly. Experience
will determine the best API, so these ideas are merely informational
suggestions at this stage:
Enabling/disabling Inner Space: As well as the above per-host or
per-user switches, the extended API might need to allow an
application to disable Inner Options on a per-socket basis (e.g.
for testing). A socket might need to be opened in one of three
possible Inner Space modes: i) Enabled; ii) Enabled initially but
can be disabled autonomously by the stack if redundant; iii)
Enabled initially, then disables itself after the SYN/ACK; and iv)
Disabled. It also ought to be possible for an application to
disable Inner Options on-demand mid-connection.
Querying support for Inner Space: An application might need to be
able to determine whether the host supports Inner Space and in
Briscoe Expires September 10, 2015 [Page 46]
Internet-Draft Inner Space for all TCP Options March 2015
which mode it is enabled on a particular socket. For instance, an
application might need to choose different socket options
depending on how much space is available, which depends on whether
Inner Space is enabled.
Latency vs Efficiency: A socket that prefers efficient use of
connection state over latency might use the optional explicit
variant of the dual handshake (Appendix D). It is unlikely that a
new option specific to Inner Space would be needed to express this
preference, as many operating systems already offer a similar
socket option.
Logging: Log formats and log analysis software might need to be
extended to distinguish between the deliberate RST within the dual
handshake and an unexpected connection RST.
6. IANA Considerations
This specification requires IANA to allocate values from the TCP
Option Kind name-space against the following names:
o "Inner Option Space Upgraded (InSpaceU)"
o "Inner Option Space Ordinary (InSpaceO)"
o "ModeSwitch"
Early implementation before the IANA allocation MUST follow [RFC6994]
and use experimental option 254 and respective Experiment IDs:
o 0xUUUU (16 bits);
o 0xOOOO (16 bits);
o 0xMMMM (16 bits);
{ToDo: Values TBA and register them with IANA} then migrate to the
assigned option after allocation.
7. Security Considerations
Certain cryptographic functions have different coverage rules for the
TCP Header and TCP Payload. Placing some TCP options beyond the Data
Offset could mean that they are treated differently from regular TCP
options. This is a deliberate feature of the protocol, but
application developers will need to be aware that this is the case.
Briscoe Expires September 10, 2015 [Page 47]
Internet-Draft Inner Space for all TCP Options March 2015
A malicious host can send bogus SYN segments with a spoofed source IP
address (a SYN flood attack). The Inner Space protocol does not
alter the feasibility of this attack. However, the extra space for
TCP options on a SYN allows the attacker to include more TCP options
on a SYN than before, so it can make a server do more option
processing before replying with a SYN/ACK. To mitigate this problme,
a server under stress could deprioritise SYNs with longer option
fields to focus its resources on SYNs that require less processing.
Each SYN in a SYN flood attack causes a TCP server to consume memory.
The Inner Space protocol allows a potentially large amount of TCP
option state to be negotiated during the SYN exchange, which could
allow attackers to exhaust the TCP server's memory more easily. The
EchoCookie TCP option (see Section 2.5) allows the server to place
this state in a cookie and send it on the SYN/ACK to the purported
address of the client--rather than hold it in memory. Then, as long
as the client returns the cookie on the acknowledgement and the
server verifies it, the server can recover its full record of all the
TCP options it negotiated and continue the connection without delay.
On the other hand, the server's responses to SYNs from spoofed
addresses will scatter to those spoofed addresses and the server will
not have consumed any memory while waiting in vain for them to reply.
See the Security Considerations in [I-D.briscoe-tcpm-echo-cookie] for
how the EchoCookie facility protects against reflection and
amplification attacks.
Some security devices block data in an initial SYN segment,
classifying it as the signature of an attack. Attackers might indeed
use data-in-SYN to strengthen the force of a SYN flood attack, but it
has also always been valid for clients to use data-in-SYN for low
latency service as well (today data-in-SYN is used by TCP Fast Open,
but data-in-SYN has been permitted for similar reasons right back to
the days of RFC 793). On its own, data-in-SYN MUST NOT be considered
a sufficient signature of an attack. It can only be considered an
attack signature if seen in combination with other symptoms of a SYN
flood attack. The logic that led to data-in-SYN alone being
considered an attack was probably well-intentioned, but it actually
turns a security device into an attack on innocent low latency
services.
The optional extension for DPI traversal specified in Appendix C.3
might create a new attack vector. The attack was originally proposed
(by David Mazieres) when an earlier draft required the optional
extension to be applied at the start of both half-connections. As
long as the DPI traversal extension no longer applies in the server-
client direction the attack seems less feasible. Nonetheless, the
attack in the server-client direction is described here anyway (in
Briscoe Expires September 10, 2015 [Page 48]
Internet-Draft Inner Space for all TCP Options March 2015
case it prompts someone to think of a similar feasible attack in the
client-server direction):
Attack that used to be feasible in the server-client direction: An a
ttacker could have crafted content (e.g. a binary file such as a
graphics object) such that it included the appropriate bits in the
correct positions to match the Inner Space magic numbers and the
expected format of some TCP options. It could have then uploaded
this content to a legacy server for download by other clients
(e.g. a public image archive). Then, if an upgraded Inner Space
TCP client had accessed this legacy server, it would have seemed
as if the server was upgraded. So the attacker could have
theoretically conscripted the server into sending TCP options of
its choice. Although the attacker would have been limited to TCP
options relevant to those previously proposed by the client, some
harm might have been possible. The attacker might also have been
able to contrive the remainder of the content (after removing the
apparent TCP options) to be some form of script or executable.
If the DPI traversal solution is to be used, and a feasible attack is
developed in the client-server direction, a couple of directions to
prevent such an attack could be explored:
o the magic number would somehow have to be complemented by another
signal, perhaps out of band;
o the magic number would need to somehow include a cryptographic
hash of material sent by the client, so that an attacker could not
predict it.
8. Acknowledgements
The idea of this approach grew out of discussions with Joe Touch
while developing draft-touch-tcpm-syn-ext-opt, and with Jana Iyengar
and Olivier Bonaventure. Jana Iyengar also suggested the sender-only
flow-control offset. The idea that it is architecturally preferable
to place a protocol extension within a higher layer, and code its
location into upgraded implementations of the lower layer, was
originally articulated by Rob Hancock. {ToDo: Ref?} The following
people provided useful comments: Joe Touch, Yuchung Cheng, John
Leslie, Mirja Kuehlewind, Andrew Yourtchenko, Costin Raiciu, Marcelo
Bagnulo Braun, Julian Chesterfield, Jaime Garcia, Ted Hardie and
David Mazieres, Tim Shepard, Mark Handley.
Bob Briscoe's contribution is part-funded by the European Community
under its Seventh Framework Programme through the Trilogy 2 project
(ICT-317756) and the Reducing Internet Transport Latency (RITE)
Briscoe Expires September 10, 2015 [Page 49]
Internet-Draft Inner Space for all TCP Options March 2015
project (ICT-317700). The views expressed here are solely those of
the author.
9. References
9.1. Normative References
[I-D.ietf-tcpm-fastopen]
Cheng, Y., Chu, J., Radhakrishnan, S., and A. Jain, "TCP
Fast Open", draft-ietf-tcpm-fastopen-10 (work in
progress), September 2014.
[RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
[RFC6994] Touch, J., "Shared Use of Experimental TCP Options", RFC
6994, August 2013.
9.2. Informative Reference
[Briscoe14]
Briscoe, B., "Tunnelling through Inner Space", IAB
Workshop on Stack Evolution in a Middlebox Internet ,
January 2015.
[Cheshire97]
Cheshire, S. and M. Baker, "Consistent Overhead Byte
Stuffing", Proc. ACM SIGCOMM'97, Computer Communication
Review 27(4):209--220, October 1997.
[Honda11] Honda, M., Nishida, Y., Raiciu, C., Greenhalgh, A.,
Handley, M., and H. Tokuda, "Is it Still Possible to
Extend TCP?", Proc. ACM Internet Measurement Conference
(IMC'11) 181--192, November 2011.
[I-D.bittau-tcpinc-tcpcrypt]
Bittau, A., Boneh, D., Hamburg, M., Handley, M., Mazieres,
D., and Q. Slack, "Cryptographic protection of TCP Streams
(tcpcrypt)", draft-bittau-tcpinc-tcpcrypt-00 (work in
progress), October 2014.
[I-D.briscoe-tcpm-echo-cookie]
Briscoe, B., "The Echo Cookie TCP Option", draft-briscoe-
tcpm-echo-cookie-00 (work in progress), October 2014.
Briscoe Expires September 10, 2015 [Page 50]
Internet-Draft Inner Space for all TCP Options March 2015
[I-D.briscoe-tcpm-inner-space]
Briscoe, B., "Inner Space for TCP Options", draft-briscoe-
tcpm-inner-space-01 (work in progress), October 2014.
[I-D.ietf-httpbis-http2]
Belshe, M., Peon, R., and M. Thomson, "Hypertext Transfer
Protocol version 2", draft-ietf-httpbis-http2-17 (work in
progress), February 2015.
[I-D.iyengar-minion-protocol]
Jana, J., Cheshire, S., and J. Graessley, "Minion - Wire
Protocol", draft-iyengar-minion-protocol-02 (work in
progress), October 2013.
[I-D.touch-tcpm-tcp-syn-ext-opt]
Touch, J. and T. Faber, "TCP SYN Extended Option Space
Using an Out-of-Band Segment", draft-touch-tcpm-tcp-syn-
ext-opt-01 (work in progress), September 2014.
[I-D.wing-tsvwg-happy-eyeballs-sctp]
Wing, D. and P. Natarajan, "Happy Eyeballs: Trending
Towards Success with SCTP", draft-wing-tsvwg-happy-
eyeballs-sctp-02 (work in progress), October 2010.
[RFC2018] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP
Selective Acknowledgment Options", RFC 2018, October 1996.
[RFC2675] Borman, D., Deering, S., and R. Hinden, "IPv6 Jumbograms",
RFC 2675, August 1999.
[RFC4987] Eddy, W., "TCP SYN Flooding Attacks and Common
Mitigations", RFC 4987, August 2007.
[RFC5925] Touch, J., Mankin, A., and R. Bonica, "The TCP
Authentication Option", RFC 5925, June 2010.
[RFC6555] Wing, D. and A. Yourtchenko, "Happy Eyeballs: Success with
Dual-Stack Hosts", RFC 6555, April 2012.
[RFC6824] Ford, A., Raiciu, C., Handley, M., and O. Bonaventure,
"TCP Extensions for Multipath Operation with Multiple
Addresses", RFC 6824, January 2013.
[RFC7323] Borman, D., Braden, B., Jacobson, V., and R.
Scheffenegger, "TCP Extensions for High Performance", RFC
7323, September 2014.
Briscoe Expires September 10, 2015 [Page 51]
Internet-Draft Inner Space for all TCP Options March 2015
[Raiciu12]
Raiciu, C., Paasch, C., Barre, S., Ford, A., Honda, M.,
Duchene, F., Bonaventure, O., and M. Handley, "How Hard
Can It Be? Designing and Implementing a Deployable
Multipath TCP", Proc. USENIX Symposium on Networked
Systems Design and Implementation , April 2012.
Appendix A. Zero Overhead Message Boundary Insertion (ZOMBI)
This appendix is normative and mandatory to implement for the Inner
Space protocol. This encoding is relegated to an appendix merely
because it is applicable more generally than for just Inner Space.
Therefore, in a future revision, this appendix might be removed and
replaced by a reference to a stand-alone document.
The Inner Space protocol requires the sender to add a marker in every
segment at the first 4-octet aligned word from the start of the
datastream. Then, even if the stream is subsequently resegmented,
the receiver can recover segments and their associated TCP options as
they were sent. The sender uses the value 0x0000 as the 2-octet
marker at the start of the InSpace option header. It uses the ZOMBI
encoding to remove all other occurrences of 0x0000, treating the
segment as a sequence of 2-octet shorts. Then, a marker will
unambiguously locate the InSpace option at the start of each segment.
From this InSpace option, the receiver can find the length of the
segment. Then it can decode the ZOMBI encoding to return the segment
to its original form.
The sender applies the ZOMBI encoding as follows:
1. It places 0x0000 in the Marker and the ZOMBI fields of the
InSpace option, and fills all the other fields of the InSpace
option with the relevant sizes and offsets.
2. Treating the stream as a sequence of 2-octet shorts,starting from
the ZOMBI field, it replaces each occurrence of 0x0000 with the
offset (in shorts) to the next occurrence of 0x0000, or to just
beyond the end of the segment when there are no more occurrences
of 0x0000.
Because an offset can never be zero, this process naturally removes
all occurrences of 0x0000 from the segment.
The receiver reverses the above encoding, assuming the worst case of
a resegmented stream unless it finds otherwise:
1. If it is buffering undecoded bytes either side of the newly
arrived segment in the sequence space, it coalesces them.
Briscoe Expires September 10, 2015 [Page 52]
Internet-Draft Inner Space for all TCP Options March 2015
2. Scanning two octets at a time aligned on even numbers of octets
from the ISN, it locates the next occurrence of an InSpace option
by locating the next occurrence of 0x0000 in a segment.
3. Starting at the ZOMBI field, it points a variable (e.g. "ptr") to
a position in the stream, reads the short at that location,
writes 0x0000 into the stream to replace it, then increments
"ptr" by the value just read. It continually repeats the same
read, replace and increment operations at each new location
pointed to by "ptr".
4. The receiver knows the size of the sent segment from the SDS
field, so that it knows when to stop decoding. If the end of the
received segment is reached before this, it implies the stream
has been resegmented and the next segment has not been buffered
yet. In this case, the receiver stores how much decoding is
left.
5. If there are more undecoded octets buffered, the process repeats
from step 1.
Below an implementation of the ZOMBI encode and decode algorithms is
given in C. The decode algorithm would be preceded by marker-
scanning code to find the location of the ZOMBI and SDS fields within
the InSpace option. The SDS field will always be non-zero, therefore
it will never be changed by the encoding, so the receiver can read it
before starting to decode. In case length is odd, a non-zero pseudo-
padding octet is considered to be appended to the segment while
encoding or decoding (but it is not actually transmitted).
Briscoe Expires September 10, 2015 [Page 53]
Internet-Draft Inner Space for all TCP Options March 2015
/* {ToDo: Test}
* ZombiEncode encodes "length" bytes of data
* starting directly after the marker pointed to by "ptr", where:
* length = sds - pad.
*/
void ZombiEncode(unsigned short *ptr, unsigned short length)
{
const unsigned short *end = ptr + ++length>>1; % /2 rounded up
unsigned short *code_ptr = ++ptr; % point to ZOMBI
unsigned short code = 0x0001;
while (++ptr < end) { % initialise after ZOMBI
if (*ptr == 0) {
*code_ptr = code;
code_ptr = ptr;
code = 0x0001;
} else
code++;
}
}
/* {ToDo: Test}
* ZombiDecode decodes "length" bytes of data
* starting after the marker pointed to by "ptr", where
* length = sds - pad.
* Returns number of shorts still to decode.
*/
short ZombiDecode(unsigned short *ptr, unsigned short length)
{
const unsigned short *end = ptr++ + ++length>>1; % /2 rounded up
while (ptr < end) { % initialise to ZOMBI
code = *ptr;
*ptr = 0;
ptr += code;
}
return (ptr - end);
}
The ZOMBI encoding always uses a marker that is larger than the
maximum possible segment size. Therefore, for a jumbo segment
Appendix C.2, the sender uses 0x00000000 (4 octets of zeros) as the
marker; it pads the segment to a multiple of 4 octets; and it scans
the stream in 4-octet words, replacing any occurrences of the marker
with the offset in 4-octet words to the next marker.
Briscoe Expires September 10, 2015 [Page 54]
Internet-Draft Inner Space for all TCP Options March 2015
The ZOMBI encoding is similar to consistent overhead byte stuffing
(COBS [Cheshire97]). The main difference is that COBS markers are
only one octet. Therefore, in COBS, whenever the distance between
zero-bytes is greater than 0xFE, it has to insert an extra byte into
the stream with the special value of 0xFF. When decoding, 0xFF is
removed rather than replaced by 0x00. Therefore, as well as 2 extra
delimiting octets, COBS introduces a variable number of extra octets,
but no more than 1 in 254 (a more accurate name would have been
_capped_ overhead byte stuffing, because the overhead is variable,
not consistent).
In contrast, ZOMBI introduces a predictable overhead of 4 delimiting
octets per segment (or 5 for odd length segments), with no
unpredictable variation. Therefore, space for the known overhead can
be set aside in the InSpace option, and the ZOMBI encode and decode
operation can be zero-copy, which is not possible with COBS. A more
accurate name for ZOMBI would have been _constant_ overhead message
boundary insertion. Nonetheless, the encoding to replace markers
once the message boundaries have been inserted actually is zero
overhead, so the cool acronym is not totally contrived.
Appendix B. Generic Connection Mode Switching
This appendix is normative and mandatory to implement for the Inner
Space protocol. This encoding is relegated to an appendix merely
because, in a future revision, this appendix might be removed and
replaced by a reference to a stand-alone document. It defines the
new ModeSwitch TCP option illustrated in Figure 5. This option
provides a facility to disable the Inner Space protocol for the
remainder of a connection. It also provides a general-purpose
facility for a TCP connection to co-ordinate between the endpoints
before switching into a yet-to-be-defined mode.
0 1 2
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
+---------------+---------------+-----------+-+-+
| ModeSwitch | Length=3 |Flags (CU) |I|R|
+---------------+---------------+-----------+-+-+
Figure 5: The ModeSwitch TCP Option
The Option Kind is ModeSwitch, the value of which is to be allocated
by IANA {ToDo: Value TBA}. ModeSwitch MUST be used only as an Inner
Option, because it uses the reliable ordered delivery property of
Inner Options. Therefore implementation of the Inner Space protocol
is REQUIRED for an implementation of ModeSwitch. Nonetheless,
ModeSwitch is a generic facility for switching a connection between
Briscoe Expires September 10, 2015 [Page 55]
Internet-Draft Inner Space for all TCP Options March 2015
yet-to-be-defined modes that do not have to relate to extra option
space.
The sender MUST set the option Length to 3 (octets). The Length
field MUST be forwarded unchanged by other nodes, even if its value
is different.
The Flags field is available for defining modes of the connection.
Only two connection modes are currently defined. The first 6 bits of
the Flags field are Currently Unused (CU) and the sender MUST set
them to zero. The CU flags MUST be ignored and forwarded unchanged
by other nodes, even if their value is non-zero.
The two 1-bit connection mode flags that are currently defined have
the following meanings:
o R: Request flag if 1. Request mode is a special mode that allows
the hosts to co-ordinate a change to any other mode(s);
o I: Inner Space mode: Enabled if 1, Disabled if 0.
The default Inner Space mode at the start of a connection is I=1,
meaning Inner Space is in enabled mode.
The procedure for changing a mode or modes is as follows:
o The host that wants to change modes (the requester) sends a
ModeSwitch message as an Inner Option with R=1 and with the other
flag(s) set to the mode(s) it wants to change to. The requester
does not change modes yet.
o The responder echoes the mode flag(s) it is willing to change to,
with the request flag R=0.
o The half-connection from the responder changes to the mode(s) it
confirms directly after the end of the segment that echoes its
confirmation, i.e. after the last octet of the TCP Payload
following the ModeSwitch option that echoes its confirmation.
Therefore it sends the segment carrying the confirmation in the
prior mode(s) of the connection.
o Once the requester receives the responder's confirmation message,
it re-echoes its confirmation of the responder's confirmation,
with the mode(s) set to those that both hosts agree on and R=0.
o The half-connection from the requester changes to the mode(s) it
confirms directly after the end of the segment that re-echoes its
Briscoe Expires September 10, 2015 [Page 56]
Internet-Draft Inner Space for all TCP Options March 2015
confirmation. Therefore it sends the segment carrying the
confirmation in the prior mode(s) of the connection.
o The responder can refuse a request to change into a mode in any
one of three ways:
* either implicitly by never confirming it;
* or explicitly by sending a message with R=0 and the opposite
mode;
* or explicitly be sending a counter-request to switch to the
opposite mode (that the connection is already in) with R=1.
The regular TCP sequence numbers and acknowledgement numbers of
requests or confirmations can be used to disambiguate overlapping
requests or responses.
Once a host switches to Disabled mode, it MUST NOT send any further
InSpace Options. Therefore it can send no further Inner Options and
it cannot switch back to Enabled mode for the rest of the connection.
To temporarily reduce InSpace overhead without permanently disabling
the protocol, the sender can use a value of 0xFFFF in the Sent Data
Size (see Section 2.4).
Appendix C. Protocol Extension Specifications
This appendix specifies protocol extensions that are OPTIONAL while
the specification is experimental. If an implementation includes an
extension, this section gives normative specification requirements.
However, if the extension is not implemented, the normative
requirements can be ignored.
{Temporary note: The IETF may wish to consider making some of these
extensions mandatory to implement if early testing shows they are
useful or even necessary. Or it may wish to make at least the
receiving side mandatory to implement to ensure that two-ended
experiments are more feasible.}
C.1. Dual Handshake: The Explicit Variant
This appendix is normative. It is separated from the body of the
specification because it is OPTIONAL to implement while the Inner
Space protocol is experimental. It is not mandatory to implement
because it will be more useful once the Inner Space protocol has
become accepted widely enough that fewer middleboxes will discard SYN
segments carrying this option (see Appendix D for when best to deploy
Briscoe Expires September 10, 2015 [Page 57]
Internet-Draft Inner Space for all TCP Options March 2015
it). It only works if both ends support it, but it can be deployed
one end at a time, so there is no need for support in early
experimental implementations.
{Temporary note: The choice between the explicit handshake in the
present section or the handshake in Section 2.1.1 is a tradeoff
between robustness against middlebox interference and minimal server
state. During the IETF review process, one might be chosen as the
only variant to go forward, at which point the other will be deleted.
Alternatively, the IETF could require a server to understand both
variants and a client could be implemented with either, or both. If
both, the application could choose which to use at run-time. Then we
will need a section describing the necessary API.}
This explicit dual handshake is similar to that in Section 2.1.1,
except the SYN that the Upgraded Client sends on the Ordinary
Connection is explicitly distinguishable from the SYN that would be
sent by a Legacy Client. Then, if the server actually is an Upgraded
Server, it can reset the Ordinary Connection itself, rather than
creating connection state for at least a round trip until the client
resets the connection.
For an explicit dual handshake, the TCP client still sends two
alternative SYNs: a SYN-O intended for Legacy Servers and a SYN-U
intended for Upgraded Servers. The two SYNs MUST have the same
network addresses and the same destination port, but different source
ports. Once the client establishes which type of server has
responded, it continues the connection appropriate to that server
type and aborts the other. The SYN intended for Upgraded Servers
includes additional options within the TCP Data (the SYN-U defined as
before in Section 2.2.1).
Table 2 summarises the TCP 3-way handshake exchange for each of the
two SYNs in the two right-hand columns, between an Upgraded TCP
Client (the active opener) and either:
1. a Legacy Server, in the top half of the table (steps 2-4), or
2. an Upgraded Server, in the bottom half of the table (steps 2-4)
The table uses the same layout and symbols as Table 1, which has
already been explained in Section 2.1.1.
Briscoe Expires September 10, 2015 [Page 58]
Internet-Draft Inner Space for all TCP Options March 2015
+------+------------------+--------------------+--------------------+
| | | Ordinary | Upgraded |
| | | Connection | Connection |
+------+------------------+--------------------+--------------------+
| 1 | Upgraded Client | >SYN-O | >SYN-U |
| | | | |
| /\/\ | /\/\/\/\/\/\/\/\ | /\/\/\/\/\/\/\/\/\ | /\/\/\/\/\/\/\/\/\ |
| 2 | Legacy Server | <SYN/ACK | <SYN/ACK |
| | | | |
| 3a | Upgraded Client | Waits for response | |
| | | to both SYNs | |
| | | | |
| 3b | " | >ACK | >RST |
| | | | |
| 4 | | Cont... | |
| | | | |
| /\/\ | /\/\/\/\/\/\/\/\ | /\/\/\/\/\/\/\/\/\ | /\/\/\/\/\/\/\/\/\ |
| 2 | Upgraded Server | <RST | <SYN/ACK-U |
| | | | |
| 3 | Upgraded Client | | >ACK |
| | | | |
| 4 | | | Cont... |
+------+------------------+--------------------+--------------------+
Table 2: Explicit Variant of Dual 3-Way Handshake in Two Server
Scenarios
As before, an Upgraded Server MUST respond to a SYN-U with a SYN/ACK-
U. Then, the client recognises that it is talking to an Upgraded
Server.
Unlike before, an Upgraded Server MUST respond to a SYN-O with a RST.
However, the client cannot rely on this behaviour, because a
middlebox might be stripping Outer TCP Options which would turn the
SYN-O into a regular SYN before it reached the server. Then the
handshake would effectively revert to the implicit variant.
Therefore the client's behaviour still depends on which SYN-ACK
arrives first, so its response to SYN-ACKs has to follow the rules
specified for the implicit handshake variant in Section 2.1.1.
The rules for processing TCP options are also unchanged from those in
Section 2.3.
C.1.1. SYN-O Structure
The SYN-O is merely a SYN with an extra InSpaceO Outer TCP Option as
shown in Figure 6. It merely identifies that the SYN is opening an
Briscoe Expires September 10, 2015 [Page 59]
Internet-Draft Inner Space for all TCP Options March 2015
Ordinary Connection, but explicitly identifies that the client
supports the Inner Space protocol.
0 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+---------------+---------------+
| Kind=InSpaceO | Length=2 |
+---------------+---------------+
Figure 6: An InSpaceO TCP Option Flag
An InSpaceO TCP Option has Option Kind InSpaceO with value {ToDo:
Value TBA} and MUST have Length = 2 octets.
To use this option, the client MUST place it with the Outer TCP
Options. A Legacy Server will just ignore this TCP option, which is
the normal behaviour for an option that TCP does not recognise
[RFC0793].
C.1.2. Retransmission Behaviour - Explicit Variant
If the client receives a RST on one connection, but a short while
after that {ToDo: duration TBA} the response to the SYN-U has not
arrived, it SHOULD retransmit the SYN-U. If latency is more
important than the extra TCP option space, in parallel to any
retransmission, or instead of any retransmission, the client MAY send
a SYN without any InSpace TCP Option, in case this is the cause of
the black-hole. However, the presence of the RST implies that the
SYN with the InSpaceO TCP Option (the SYN-O) probably reached the
server, therefore it is more likely (but not certain) that the lack
of response on the other connection is due to transmission loss or
congestion loss.
If the client receives no response at all to either the SYN-O or the
SYN-U, it SHOULD solely retransmit one or the other, not both. If
latency is more important than the extra TCP option space, it SHOULD
send a SYN without an InSpaceO TCP Option. Otherwise it SHOULD
retransmit the SYN-U. It MUST NOT retransmit both segments, because
the lack of response could be due to severe congestion.
C.1.3. Corner Cases
There is a small but finite possibility that the Explicit Dual
Handshake might encounter the cases below. The Implicit Handshake
(Section 2.1.1) is robust to these possibilities, but the Explicit
Handshake is not, unless the following additional rules are followed:
Briscoe Expires September 10, 2015 [Page 60]
Internet-Draft Inner Space for all TCP Options March 2015
Both successful: This could occur if one load-sharing replica of a
server is upgraded, while another is not. This could happen in
either order but, in both cases, the client aborts the last
connection to respond:
* The client completes the Ordinary Handshake (because it
receives a SYN/ACK), but then, before it has aborted the
Upgraded Connection, it receives a SYN/ACK-U on it. In this
case, the client MUST abort the Upgraded Connection even though
it would work. Otherwise the client will have opened both
connections, one with Inner TCP Options and one without. This
could confuse the application.
* The client completes the Upgraded Connection after receiving a
SYN/ACK-U, but then it receives a SYN/ACK in response to the
SYN-O. In this case, the client MUST abort the connection it
initiated with the SYN-O.
Both aborted: The client might receive a RST in response to its SYN-
O, then an Ordinary SYN/ACK on its Upgraded Connection in response
to its SYN-U. This could occur i) if a split connection middlebox
actively forwards unknown options but holds back or discards data
in a SYN; or ii) if one load-sharing replica of a server is
upgraded, while another is not.
Whatever the likely cause, the client MUST still respond with a
RST on its Upgraded Connection. Otherwise, its Inner TCP Options
will be passed as user-data to the application by a Legacy Server.
If confronted with this scenario where both connections are
aborted, the client will not be able to include extra options on a
SYN, but it might still be able to set up a connection with extra
option space on all the other segments in both directions using
the approach in Appendix C.1.4. If that doesn't work either, the
client's only recourse is to retry a new dual handshake on
different source ports, or ultimately to fall-back to sending an
Ordinary SYN.
C.1.4. Workround if Data in SYN is Blocked
If a path either holds back or discards data in a SYN-U, but there is
evidence that the server is upgraded from a RST response to the SYN-
O, the strategy below might at least allow a connection to use extra
option space on all the segments except the SYN.
It is assumed that the symptoms described in the 'both aborted' case
(Appendix C.1.3) have occurred, i.e. the server has responded to the
SYN-O with a RST, but it has responded to the SYN-U with an Ordinary
Briscoe Expires September 10, 2015 [Page 61]
Internet-Draft Inner Space for all TCP Options March 2015
SYN/ACK not a SYN/ACK-U, so the client has had to RST the Upgraded
Connection as well. In this case, the client SHOULD attempt the
following (alternatively it MAY give up and fall back to opening an
Ordinary TCP connection).
The client sends an 'Alternative SYN-U' by including an InSpaceU
Outer TCP Option (Figure 7). This Alternative SYN-U merely flags
that the client is attempting to open an Upgraded Connection. The
client MUST NOT include any Inner Options or InSpace Option or Magic
Number. If the previous aborted SYN/ACK-U acknowledged the data that
the client sent within the original SYN-U, the client SHOULD resend
the TCP Payload data in the Alternative SYN-U, otherwise it might as
well defer it to the first data segment.
0 1
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+---------------+---------------+
| Kind=InSpaceU | Length=2 |
+---------------+---------------+
Figure 7: An InSpaceU Flag TCP option
An InSpaceU Flag TCP Option has Option Kind InSpaceU with value
{ToDo: Value TBA} and MUST have Length = 2 octets.
To use this option, the client MUST place it with the Outer TCP
Options. A Legacy Server will just ignore this TCP option, which is
the normal behaviour for an option that TCP does not recognise
[RFC0793]. Because the client has received a RST from the server in
response to the SYN-O it can assume that the server is upgraded. So
the client probably only needs to send a single Alternative SYN-U in
this repeat attempt. Nonetheless, the RST might have been spurious.
Therefore the client MAY also send an Ordinary SYN in parallel, i.e.
using the Implicit Dual Handshake (Section 2.1.1).
If an Upgraded Server receives a SYN carrying the InSpaceU option, it
MUST continue the rest of the connection as if it had received a full
SYN-U (Section 2.2), i.e. by processing any Outer Options in the
SYN-U and responding with a SYN/ACK-U.
C.2. Jumbo InSpace TCP Option (only if SYN=0)
This appendix is normative. It defines the format of the InSpace
Option necessary to support jumbograms. It is separated from the
body of the specification because it is OPTIONAL to implement while
the Inner Space protocol is experimental. In experimental
implementations, it will be sufficient to implement the required
Briscoe Expires September 10, 2015 [Page 62]
Internet-Draft Inner Space for all TCP Options March 2015
behaviour for when the Length of a received InSpace Option is not
recognised (Section 2.4).
If the IPv6 Jumbo extension header is used, a sender MUST use the
InSpace Option format defined in Figure 8.
All the fields have the same meanings as defined in Section 2.2.2,
except Sent Data Size (SDS), the Inner Options Offset (InOO) and the
Suffix Options Offset (SOO) use more bits, respectively 32, 30 and
30. The Length (Len) field can be either 2, 3 or 4, where binary 00
represents 4.
If Len=3: the last 4-octet word is omitted and the value of SOO is
determined by the P flag as already described in Section 2.2.2.
If Len=2: it is assumed InOO = SOO = 0.
When reading a segment, the Jumbo InSpace Option could be present in
a packet that is not a jumbogram (e.g. due to resegmentation).
Therefore a receiver MUST use the Jumbo InSpace Option to work along
the stream irrespective of whether arriving packets are jumbo sized
or not.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------------------------------------------------------+
| Marker |
+-----------------------------------------------------------+---+
| ZOMBI |Len|
+-----------------------------------------------------------+---+
| Sent Data Size (SDS) |
+-----------------------------------------------------------+-+-+
| Inner Options Offset (InOO) CU|P|
+-----------------------------------------------------------+-+-+
| Suffix Options Offset (SOO) |CU |
+-----------------------------------------------------------+---+
Figure 8: InSpace Option for a Jumbo Datagram
C.3. Optional Segment Structure to Traverse DPI boxes
This appendix is normative. It is separated from the body of the
specification because it is OPTIONAL to implement while the Inner
Space protocol is experimental.
In experiments conducted between 2010 and 2011, [Honda11] reported
that 7 of 142 paths (about 5%) blocked access to port 80 if the
payload was not parsable as valid HTTP. This extension to the
Briscoe Expires September 10, 2015 [Page 63]
Internet-Draft Inner Space for all TCP Options March 2015
specification has been defined in case experiments prove that it
significantly improves traversal of such deep packet inspection (DPI)
boxes.
This extension places the expected app-layer headers at the start of
the TCP Data in the SYN and in the first data segment in the client-
to-server direction:
SYN=1: The sender uses the structure in Figure 9a) on the SYN. The
sender right-aligns the 12-octet InSpace Option at the end of the
segment. Then it right-aligns the Inner Options against the
InSpace Option, all after the end of the TCP Payload and any
padding necessary to align the options on a 4-octet word boundary.
Magic Number A starts 4*3=12 octets from the end of the segment
{ToDo: Magic Number A could be placed at the end of the segment
instead.}. A receiver implementation of this optional extension
MUST check whether Magic Number A is present within the InSpace
option if it does not first find it at the start of the segment.
The start of the Inner Options is therefore 4 * (InOO +3) octets
before the end of the segment, where InOO is read from within the
InSpace Option. Although the InnerOptions are located at the end
of the TCP Payload, they are considered to be applied before the
first octet of the TCP Payload.
SYN=0: The structure of the first non-SYN segment that contains any
TCP Data is shown in Figure 9b).
The receiver will find the second InSpace Option (InSpace#2)
located SDS#1 octets from the start of the segment (plus possible
padding), where SDS#1 is the value of Sent Data Size that was read
from the InSpace Option in the previous (SYN=1) segment that
started the half-connection. Although the Inner Options are
shifted, they are still considered to be applied at the start of
the TCP Data in this second segment.
From the second InSpace Option onwards, the structure of the stream
reverts to that already defined in Section 2.2.1. So the value of
Sent Data Size (SDS#2) in the second InSpace Option (InSpace #2)
defines the length of the remaining TCP Data before the end of the
first data segment, as shown.
Briscoe Expires September 10, 2015 [Page 64]
Internet-Draft Inner Space for all TCP Options March 2015
TCP Data
.---------------------------'------------------.
| Inner Options |
a) SYN=1 | .---------'---------. |
+--------+----------+-------------+-+---------+---------+----------+
| BaseHdr| OuterOpts| Payload | | PrefOpts| SuffOpts|InSpace#1 |
+--------+----------+-------------+-+---------+---------+----------+
| DO | | | SOO | | |
`------------------>| |P`-------->| | Len = 3 |
| | |a| InOO |<---------'
|d|<------------------' |
b) First SYN=0 segment
+--------+----------+--------+-+---------+--------+--------+-------+
| BaseHdr| OuterOpts|Payload | |InSpace#2|PrefOpts|SuffOpts|Payload|
+--------+----------+--------+-+---------+--------+--------+-------+
| DO | | | Len | SOO | |
`------------------>| |P`-------->`------->| | |
| |a| | InOO | |
| |d| `---------------->| |
| SDS#1 | SDS#2 |
`------->`------------------------------------>|
| | |
All offsets are specified in 4-octet (32-bit) words, except SDS and
Pad, which are in octets.
Figure 9: Segment Structures to Traverse DPI boxes (not to scale)
It is recognised that having to work from the end of the first
segment makes segment processing more involved. Experimental
implementation of this approach will determine whether the extra
complexity improves DPI box traversal sufficiently to make it
worthwhile.
If it does work, it is believed that this extension will only be
necessary on the initial SYN and the first data segment sent in the
direction from TCP client to server. Therefore, the SYN/ACK and data
segments sent by the TCP server will continue to use the regular
Inner Space segment structure illustrated in Figure 2.
If a TCP client that implements this extension opens a connection
with a server that does not, the client will fall back to ordinary
TCP even though the server would have supported the Inner Space
protocol without the DPI traversal extension. This is because the
server does not look for the magic number at the end of the SYN, so
it behaves like a legacy TCP server responding with an ordinary SYN/
Briscoe Expires September 10, 2015 [Page 65]
Internet-Draft Inner Space for all TCP Options March 2015
ACK, which in turn makes the client fall back to ordinary TCP. Such
limited fall-back is considered sufficient to support experiments to
see whether the DPI traversal extension is useful. If it is useful,
a future standards track specification could make support for this
DPI traversal extension mandatory for an Inner Space TCP server, but
still optional for an Inner Space TCP client.
Appendix D. Comparison of Alternatives
D.1. Implicit vs Explicit Dual Handshake
In the body of this specification, two variants of the dual handshake
are defined:
1. The implicit dual handshake (Section 2.1.1) starting with just an
Ordinary SYN (no InSpaceO flag option) on the Ordinary
Connection;
2. The explicit dual handshake (Appendix C.1) starting with a SYN-O
(InSpaceO flag option) on the Ordinary Connection.
Both schemes double up connection state (for a round trip) on the
Legacy Server. But only the implicit scheme doubles up connection
state (for a round trip) on the Upgraded Server as well. On the
other hand, the explicit scheme risks delay accessing a Legacy Server
if a middlebox discards the SYN-O (some firewalls and middleboxes
discard packets with unrecognised TCP options [Honda11]). Table 3
summarises these points.
+----------------------------------+---------------+----------------+
| | SYN | SYN-L |
| | (Implicit) | (Explicit) |
+----------------------------------+---------------+----------------+
| Minimum state on Upgraded Server | - | + |
| | | |
| Minimum risk of delay to Legacy | + | - |
| Server | | |
+----------------------------------+---------------+----------------+
Table 3: Comparison of Implicit vs. Explicit Dual Handshake on the
Ordinary Connection
There is no need for the IETF to choose between these. If the
specification allows either or both, the tradeoff can be left to
implementers at build-time, or to the application at run-time.
Initially clients might choose the Implicit Dual Handshake to
minimise delays due to middlebox interference. But later, perhaps
Briscoe Expires September 10, 2015 [Page 66]
Internet-Draft Inner Space for all TCP Options March 2015
once more middleboxes support the scheme, clients might choose the
Explicit scheme, to minimise state on Upgraded Servers.
Appendix E. Protocol Design Issues (to be Deleted before Publication)
This appendix is informative, not normative. It records outstanding
issues with the protocol design that will need to be resolved before
publication.
Data in SYN middlebox traversal: Certain middleboxes do not forward
data in a SYN. The scheme can detect this (by the lack of
acknowledgement of the data on the SYN/ACK). However, it would be
ideal to be able to work round this problem in all circumstances,
not just those in Appendix C.1.4.
Options that alter the main TCP header semantics: Need to include
text to ensure Inner options are used with care where middleboxes
are known to use a main header field, particularly if the
middlebox also understands how a TCP option alters its semantics.
Examples:
WScale: Easiest to only locate this as an Outer Option - too
many TCP normalisers that check whether a segment is in window
use WS to interpret the Window field.
SACK: A similar but different example is where a middlebox shifts
the ISN, and also shifts all seqno values including in TCP
options, e.g. SACK. Here, if SACK were placed as an Inner
Option, another 'ISN' option would be needed to detect and
allow for the ISN shift.
Flow-control deadlock: It needs to be proved whether the solution to
flow-control deadlock for acknowledgement-related options also
avoids the risk of deadlock across one or more connection-
splitting middleboxes.
Simultaneous open: If host A sends a SYN-U from port S to D, it
might receive a SYN rather than a SYN/ACK on port S from port D.
Whether the SYN is upgraded or not, it is believed that it will be
possible to define all the cases necessary to fully specify the
simultaneous open case. The number of combinations that have to
be considered becomes quite tiresome, especially if the case of
simulataneous dual handshakes is included. Therefore, these
corner-cases will be addressed in a later revision.
TCP offload: The protocol design is intended to ensure that new TCP
extensions will survive segmentation offload. The InSpace Options
are also intended to provide a robust way for an Inner Space TCP
Briscoe Expires September 10, 2015 [Page 67]
Internet-Draft Inner Space for all TCP Options March 2015
to offload the generation or ingestion of TCP segments without
breaking extensibility, but whether it is the best way to
interwork with offload hardware is yet to be determined.
Appendix F. Change Log (to be Deleted before Publication)
A detailed version history can be accessed at
<http://datatracker.ietf.org/doc/draft-briscoe-tcpm-inner-space/
history/>
From briscoe-...-inner-space-01 to briscoe-...-inner-space-sink-00:
Technical changes:
* Added choice of in-order and out-of-order TCP option delivery
* Added padding for 4-octet alignment of options
* Made InSpace Options for SYN=0 or SYN=1 have the same structure
by i) including magic no / message boundary marker as prefix to
InSpace option and ii) allowing Prefix (out-of-order or fire-
and-forget) Options in all segments.
* Changed Sent Payload Size (SPS) field to Sent Data Size (SDS),
to minimise framing arithmetic.
* Allowed space in the InSpace Option for the SOO field on all
segments (not just SYN=1). Also allowed a choice of Len=1 or 2
when SYN=0 and introduced the P flag if Len=1 to state whether
the Inner Options are all Prefix or all Suffix.
* Added the Marker and ZOMBI fields to the InSpace Option when
SYN=0.
* Extended Sequence Space Consumption rules to require the
sequence space of fire-and-forget objects to be coinsidered
implicitly acknowledged.
* Removed Fire-and-Forget Options from flow control coverage.
* New rules for new concept of Impure ACKs.
* Defined Construction Order for writing TCP Data.
* Extensive changes to processing order when reading Inner
Options with SYN=0.
* 'Compatibility with Pre-Existing TCP Variants' now categorises
existing TCP options by whether they must be Prefix, Suffix or
Briscoe Expires September 10, 2015 [Page 68]
Internet-Draft Inner Space for all TCP Options March 2015
either, and requires future option definitions to make this
distinction. Also added some previously overlooked options
(no-op & EOL) and re-categorised TCP-AO, with explanation
* When explicit port binding needed, recommended dual handshakes
in series rather than disabling Inner Space.
* Defined behaviour when app attempts to determine PMTU.
* Added security recommendation not to block data-in-SYN unless
other signs of SYN flood attack.
* Discussed the potential new attack vector in the optional DPI
traversal approach, and why it is probably not a concern now
that the approach is only used in the client-server direction.
* Made ModeSwitch mandatory, not optional.
* Restructured the InSpace Option for a jumbogram
* Specified that the optional DPI traversal extension would only
be used in the client-server direction, and restructured to
remain consistent with the changes to the regular InSpace
Option structure.
* Cleared all Protocol Design Issues, and added some new ones.
Editorial changes:
* Changes to document structure:
+ Added Wider Implications subsection to Intro, looking
forward to i) a structured control channel for end-to-middle
interaction and ii) new transport services such as
Multiplexed streams, compression and encryption;
+ Added 'Flow Control Coverage' and 'Construction Order for
TCP Data' subsections to 'Writing Inner TCP Options'
section;
+ Added 'Header Extension by Encapsulation' and 'Framing
Segments' subsections to rationale for Inner Option Space;
+ Split 'Control Options Within Data Sequence Space' into two
subsections: i) 'In-Order Flow-Controlled Options' using the
existing text and a new 'Fire-and-Forget Options'
subsection;
Briscoe Expires September 10, 2015 [Page 69]
Internet-Draft Inner Space for all TCP Options March 2015
+ Added 'Deployment Approach', including 'Substrate Protocol:
TCP vs. UDP', and ''User-Space vs. Kernel-Space' to
Rationale section;
+ Promoted Protocol Overhead subsection.
+ Added appendix for 'Zero Overhead Message Boundary Insertion
(ZOMBI)';
* Abstract & Introduction: primary goal changed to redesign of
TCP's extensibility mechanism (ie middlebox traversal as well
as option space).
* Introduction:
+ Rewrote Introduction to introduce the two difficult
questions that tunnelling TCP options raises: i) immediate
(out-of-order) delivery of certain options and ii)
bootstrapping the inner control channel;
+ Made examples in Intro consistent with those in TCP
Compatibility section (i.e. TCP-AO removed from Inner
Option list).
+ Added MPTCP & tcpinc to 'Motivation for Adoption Now'
* Terminology: Added definitions of Pure ACKs, Impure ACKs and
Flow-Controlled ACKs.
* Protocol Spec
+ Upgraded Segment Structure and Format: Reflected technical
changes as above
+ Inner TCP Option Processing: Introduced distinction between
flow-controlled and fire-and-forget options at the start
* Acknowledged more helpful people.
* Added refs related to Minion/COBS, HTTP2 and an architectural
paper on Inner Space.
* Appendices: Expanded rationale for optional DPI traversal fall-
back if not supported by both ends.
From briscoe-...-inner-space-00 to briscoe-...-inner-space-01:
Technical changes:
Briscoe Expires September 10, 2015 [Page 70]
Internet-Draft Inner Space for all TCP Options March 2015
* Corrected DO to 4 * DO (twice)
* Confirmed that receive window applies to Inner Options
* Generalised the cause of decryption/decompression from a
previous TCP option to any previous control message
* Added requirement for a middlebox not to defer data on SYN
* Latency of dual handshake is worst of two
* Completed "Interaction with Pre-Existing TCP Implementations"
section, covering other TCP variants, TCP in middleboxes and
the TCP API. Shifted some TCP options to Outer only, because
of RWND deadlock problem
* Added two outstanding issues: i) ossifies reliable ordered
delivery; ii) Ideally Outer in Inner.
Editorial changes:
* Removed section on Echo TCP option to a separate I-D that is
mandatory to implement for inner-space, and shifted some SYN
flood discussion in Security Considerations
* Clarifications throughout
* Acknowledged more review comments
From draft-briscoe-tcpm-syn-op-sis-02 to draft-briscoe-tcpm-inner-
space-00:
The Inner Space protocol is a development of a proposal called the
SynOpSis (Sister SYN options) protocol. Most of the elements of
Inner Space were in SynOpSis, such as the implicit and explicit
dual handshakes; the use of a magic number to flag the existence
of the option; the various header offsets; and the option
processing rules.
The main technical differences are: Inner Space extends option
space on any segment, not just the SYN; this advance requires the
introduction of the Sent Payload Size field and a general
rearrangement and simplification of the protocol format; the
option processing rules have been extended to assure compatibility
with TFO and one degree of recursion has been introduced to cater
for encryption or compression of Inner Options; The Echo option
has been added to provide a SYN-cookie-like capability. Also, the
default protocol has been pared down to the bare bones and
optional extensions relegated to appendices.
Briscoe Expires September 10, 2015 [Page 71]
Internet-Draft Inner Space for all TCP Options March 2015
The main editorial differences are: The emphasis of the Abstract
and Introduction has expanded from a focus on just extra space
using the dual handshake to include much more comprehensive
middlebox traversal. A comprehensive Design Rationale section has
been added.
Author's Address
Bob Briscoe
BT
B54/77, Adastral Park
Martlesham Heath
Ipswich IP5 3RE
UK
Phone: +44 1473 645196
Email: bob.briscoe@bt.com
URI: http://bobbriscoe.net/
Briscoe Expires September 10, 2015 [Page 72]
