RTGWG E. Nordmark (ed) Internet-Draft Arista Networks Intended status: Informational A. Tian Expires: September 10, 2015 Ericsson Inc. J. Gross VMware J. Hudson Brocade Communications Systems, Inc. L. Kreeger Cisco Systems, Inc. P. Garg Microsoft P. Thaler Broadcom Corporation T. Herbert Google March 9, 2015 Encapsulation Considerations draft-rtg-dt-encap-01 Abstract The IETF Routing Area director has chartered a design team to look at common issues for the different data plane encapsulations being discussed in the NVO3 and SFC working groups and also in the BIER BoF, and also to look at the relationship between such encapsulations in the case that they might be used at the same time. The purpose of this design team is to discover, discuss and document considerations across the different encapsulations in the different WGs/BoFs so that we can reduce the number of wheels that need to be reinvented in the future. Status of this Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference Nordmark (ed), et al. Expires September 10, 2015 [Page 1] Internet-Draft Encaps Considerations March 2015 material or to cite them other than as "work in progress." This Internet-Draft will expire on September 10, 2015. Copyright Notice Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Nordmark (ed), et al. Expires September 10, 2015 [Page 2] Internet-Draft Encaps Considerations March 2015 Table of Contents 1. Design Team Charter . . . . . . . . . . . . . . . . . . . . . 4 2. Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Common Issues . . . . . . . . . . . . . . . . . . . . . . . . 6 4. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 5. Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . 7 6. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 7 7. Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 8. Next-protocol indication . . . . . . . . . . . . . . . . . . . 9 9. MTU and Fragmentation . . . . . . . . . . . . . . . . . . . . 10 10. OAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 11. Security Considerations . . . . . . . . . . . . . . . . . . . 12 11.1. Encapsulation-specific considerations . . . . . . . . . . 12 11.2. Virtual network isolation . . . . . . . . . . . . . . . . 14 11.3. Packet level security . . . . . . . . . . . . . . . . . . 15 12. QoS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 13. Congestion Considerations . . . . . . . . . . . . . . . . . . 16 14. Header Protection . . . . . . . . . . . . . . . . . . . . . . 18 15. Extensibility Considerations . . . . . . . . . . . . . . . . . 19 15.1. Next-protocol . . . . . . . . . . . . . . . . . . . . . . 22 16. Layering Considerations . . . . . . . . . . . . . . . . . . . 22 17. Service model . . . . . . . . . . . . . . . . . . . . . . . . 23 18. Hardware Friendly . . . . . . . . . . . . . . . . . . . . . . 24 18.1. Considerations for NIC offload . . . . . . . . . . . . . 26 19. Middlebox Considerations . . . . . . . . . . . . . . . . . . . 29 20. Related Work . . . . . . . . . . . . . . . . . . . . . . . . . 30 21. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 31 22. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 31 23. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32 23.1. Normative References . . . . . . . . . . . . . . . . . . 32 23.2. Informative References . . . . . . . . . . . . . . . . . 33 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 36 Nordmark (ed), et al. Expires September 10, 2015 [Page 3] Internet-Draft Encaps Considerations March 2015 1. Design Team Charter There have been multiple efforts over the years that have resulted in new or modified data plane behaviors involving encapsulations. That includes IETF efforts like MPLS, LISP, and TRILL but also industry efforts like VXLAN and NVGRE. These collectively can be seen as a source of insight into the properties that data planes need to meet. The IETF is currently working on potentially new encapsulations in NVO3 and SFC and considering working on BIER. In addition there is work on tunneling in the INT area. This is a short term design team chartered to collect and construct useful advice to parties working on new or modified data plane behaviors that include additional encapsulations. The goal is for the group to document useful advice gathered from interacting with ongoing efforts. An Internet Draft will be produced for IETF92 to capture that advice, which will be discussed in RTGWG. Data plane encapsulations face a set of common issues such as: o How to provide entropy for ECMP o Issues around packet size and fragmentation/reassembly o OAM - what support is needed in an encapsulation format? o Security and privacy. o QoS o Congestion Considerations o IPv6 header protection (zero UDP checksum over IPv6 issue) o Extensibility - e.g., for evolving OAM, security, and/or congestion control o Layering of multiple encapsulations e.g., SFC over NVO3 over BIER The design team will provide advice on those issues. The intention is that even where we have different encapsulations for different purposes carrying different information, each such encapsulation doesn't have to reinvent the wheel for the above common issues. The design team will look across the routing area in particular at SFC, NVO3 and BIER. It will not be involved in comparing or analyzing any particular encapsulation formats proposed in those WGs and BoFs but instead focus on common advice. 2. Overview The references provide background information on NVO3, SFC, and BIER. In particular, NVO3 is introduced in [RFC7364], [RFC7365], and [I-D.ietf-nvo3-arch]. SFC is introduced in [I-D.ietf-sfc-architecture] and [I-D.ietf-sfc-problem-statement]. Finally, the information on BIER is in [I-D.shepherd-bier-problem-statement], Nordmark (ed), et al. Expires September 10, 2015 [Page 4] Internet-Draft Encaps Considerations March 2015 [I-D.wijnands-bier-architecture], and [I-D.wijnands-mpls-bier-encapsulation]. We assume the reader has some basic familiarity with those proposed encapsulations. The Related Work section points at some prior work that relates to the encapsulation considerations in this document. Encapsulation protocols typically have some unique information that they need to carry. In some cases that information might be modified along the path and in other cases it is constant. The in-flight modifications has impacts on what it means to provide security for the encapsulation headers. o NVO3 carries a VNI Identifier edge to edge which is not modified. There has been OAM discussions in the WG and it isn't clear whether some of the OAM information might be modified in flight. o SFC carries service meta-data which might be modified or unmodified as the packets follow the service path. SFC talks of some loop avoidance mechanism which is likely to result in modifications for for each hop in the service chain even if the meta-data is unmodified. o BIER carries a bitmap of egress ports to which a packet should be delivered, and as the packet is forwarded down different paths different bits are cleared in that bitmap. Even if information isn't modified in flight there might be devices that wish to inspect that information. For instance, one can envision future NVO3 security devices which filter based on the virtual network identifier. The need for extensibility is different across the protocols o NVO3 might need some extensions for OAM and security. o SFC is all about carrying service meta-data along a path, and different services might need different types and amount of meta- data. o BIER might need variable number of bits in their bitmaps, or other future schemes to scale up to larger network. The extensibility needs and constraints might be different when considering hardware vs. software implementations of the encapsulation headers. NIC hardware might have different constraints than switch hardware. As the IETF designs these encapsulations the different WGs solve the issues for their own encapsulation. But there are likely to be future cases when the different encapsulations are combined in the same header. For instance, NVO3 might be a "transport" used to carry SFC between the different hops in the service chain. Most of the issues discussed in this document are not new. The IETF and industry as specified and deployed many different encapsulation Nordmark (ed), et al. Expires September 10, 2015 [Page 5] Internet-Draft Encaps Considerations March 2015 or tunneling protocols over time, ranging from simple IP-in-IP and GRE encapsulation, IPsec, pseudo-wires, session-based approached like L2TP, and the use of MPLS control and data planes. IEEE 802 has also defined layered encapsulation for Provider Backbone Bridges (PBB) and IEEE 802.1Qbp (ECMP). This document tries to leverage what we collectively have learned from that experience and summarize what would be relevant for new encapsulations like NVO3, SFC, and BIER. 3. Common Issues [This section is mostly a repeat of the charter but with a few modifications and additions.] Any new encapsulation protocol would need to address a large set of issues that are not central to the new information that this protocol intends to carry. The common issues explored in this document are: o How to provide entropy for Equal Cost MultiPath (ECMP) routing o Issues around packet size and fragmentation/reassembly o Next header indication - each encapsulation might be able to carry different payloads o OAM - what support is needed in an encapsulation format? o Security and privacy o QoS o Congestion Considerations o Header protection o Extensibility - e.g., for evolving OAM, security, and/or congestion control o Layering of multiple encapsulations e.g., SFC over NVO3 over BIER o Importance of being friendly to hardware and software implementations 4. Scope It is important to keep in mind what we are trying to cover and not cover in this document and effort. This is o A look across the three new encapsulations, while taking lots of previous work into account o Focus on the class of encapsulations that would run over IP/UDP. That was done to avoid being distracted by the data-plane and control-plane interaction, which is more significant for protocols that are designed to run over "transports" that maintain session or path state. o We later expanded the scope somewhat to consider how the encapsulations would play with MPLS "transport", which is important because SFC and BIER seem to target being independent of the underlying "transport" Nordmark (ed), et al. Expires September 10, 2015 [Page 6] Internet-Draft Encaps Considerations March 2015 However, this document and effort is NOT intended to: o Design some new encapsulation header to rule them all o Design yet another new NVO3 encapsulation header o Try to select the best encapsulation header o Evaluate any existing and proposed encapsulations 5. Assumptions The design center for the new encapsulations is a well-managed network. That network can be a datacenter network (plus datacenter interconnect) or a service provider network. Based on the existing and proposed encapsulations in those environment it is reasonable to make these assumptions: o The MTU is carefully managed and configured. Hence an encapsulation protocol can make the packets bigger without resulting in a requirement for fragmentation and reassembly between ingress and egress. (However, it might be useful to detecting MTU misconfigurations.) o In general an encapsulation needs some approach for congestion management. But the assumptions are different than for arbitrary Internet paths in that the underlay might be well-provisioned and better policed at the edge, and due to multi-tenancy, the congestion control in the endpoints might be even less trusted than on the Internet at large. The goal is to implement these encapsulations in hardware and software hence we can't assume that the needs of either implementation approach can trump the needs of the other. In particular, around extensibility the needs and constraints might be quite different. 6. Terminology The capitalized keyword MUST is used as defined in http://en.wikipedia.org/wiki/Julmust TBD: Refer to existing documents for at least NVO3 and SFC terminology. We use at least the VNI ID in this document. 7. Entropy In many cases the encapsulation format needs to enable ECMP in unmodified routers. Those routers might use different fields in TCP/ UDP packets to do ECMP without a risk of reordering a flow. Nordmark (ed), et al. Expires September 10, 2015 [Page 7] Internet-Draft Encaps Considerations March 2015 The common way to do ECMP-enabled encapsulation over IP today is to add a UDP header and to use UDP with the UDP source port carrying entropy from the inner/original packet headers as in LISP [RFC6830]. The total entropy consists of 14 bits in the UDP source port (using the ephemeral port range) plus the outer IP addresses which seems to be sufficient for entropy; using outer IPv6 headers would give the option for more entropy should it be needed in the future. In some environments it might be fine to use all 16 bits of the port range. However, middleboxes might make assumptions about the system ports or user ports. But they should not make any assumptions about the ports in the Dynamic and/or Private Port range, which have the two MSBs set to 11b. The UDP source port might change over the lifetime of an encapsulated flow, for instance for DoS mitigation or re-balancing load across ECMP. There is some interaction between entropy and OAM and extensibility mechanism. It is desirable to be able to send OAM packets to follow the same path as network packets. Hence OAM packets should use the same entropy mechanism as data packets. While routers might use information in addition the entropy field and outer IP header, they can not use arbitrary parts of the encapsulation header since that might result in OAM frames taking a different path. Likewise if routers look past the encapsulation header they need to be aware of the extensibility mechanism(s) in the encapsulation format to be able to find the inner headers in the presence of extensions; OAM frames might use some extensions e.g. for timestamps. Architecturally the entropy and the next header field are really part of enclosing delivery header. UDP with entropy goes hand-in-hand with the outer IP header. Thus the UDP entropy is present for the underlay IP routers the same way that an MPLS entropy label is present for LSRs. The entropy above is all about providing entropy for the outer delivery of the encapsulated packets. It has been suggested that when IPv6 is used it would not be necessary to add a UDP header for entropy, since the IPv6 flow label can be used for entropy. (This assumes that there is an IP protocol number for the encaps in addition to a UDP destination port number since UDP would be used with IPv4 underlay. And any use of UDP checksums would need to be replaced by an encaps-specific checksum or secure hash.) While such an approach would save 8 bytes of headers when the underlay is IPv6, it does assume that the underlay routers use the flow label for ECMP, and it also would make the IPv6 approach different than the IPv4 approach. Currently the leaning is towards recommending using the UDP encaps for both IPv4 and IPv6 underlay. Nordmark (ed), et al. Expires September 10, 2015 [Page 8] Internet-Draft Encaps Considerations March 2015 The IPv6 flow label can be used for additional entropy if need be. Note that in the proposed BIER encapsulation [I-D.wijnands-mpls-bier-encapsulation], there is an an 8-bit field which specifies an entropy value that can be used for load balancing purposes. This entropy is for the BIER forwarding decisions, which is independent of any outer delivery ECMP between BIER routers. Thus it is not part of the delivery ECMP discussed in this section. [Note: For any given bit in BIER (that identifies an exit from the BIER domain) there might be multiple immediate next hops. The BIER entropy field is used to select that next hop as part of BIER processing. The BIER forwarding process may do equal cost load balancing, but the load balancing procedure MUST choose the same path for any two packets have the same entropy value.] 8. Next-protocol indication The transport delivery mechanism for the encapsulations we discuss in this document need some way to indicate which encapsulation header (or other payload) comes next in the packet. Some encapsulations might be identified by a UDP port; others might be identified by an Ethernet type or IP protocol number. Which approach is used is a function of the preceding header the same was as IPv4 being identified by both an Ethernet type and an IP protocol number (for IP-in-IP). In some cases the header type is implicit in some session (L2TP) or path (MPLS) setup. But this is largely beyond the control of the encapsulation protocol. For instance, if there is a requirement to carry the encapsulation after an Ethernet header, then an Ethernet type is needed. If required to be carried after an IP/ UDP header, then a UDP port number is needed. The encapsulation needs to indicate the type of its payload, which is in scope for the design of the encapsulation. We have existing protocols which use Ethernet types (such as GRE). Here each encapsulation header can potentially makes its own choices between: o Reuse Ethernet types - makes it easy to carry existing L2 and L3 protocols o Reuse IP protocol numbers - makes it easy to carry e.g., ESP but brings in all existing protocol numbers many of which would never be used directly on top of the encapsulation protocol. o Define their own next-protocol number space, which can use fewer bits than an Ethernet type and give more flexibility, but at the cost of administering that numbering space. If the IETF ends up defining multiple encapsulations at about the same time, and there is some chance that multiple such encapsulations can be combined in the same packet, there is a question whether it Nordmark (ed), et al. Expires September 10, 2015 [Page 9] Internet-Draft Encaps Considerations March 2015 makes sense to use a common approach and numbering space for the encapsulation across the different protocols. A common approach might not be beneficial as long as there is only one way to indicate e.g., SFC inside NVO3. 9. MTU and Fragmentation A common approach today is to assume that the underlay have sufficient MTU to carry the encapsulated packets without any fragmentation and reassembly at the tunnel endpoints. That is sufficient when the operator of the ingress and egress have full control of the paths between those endpoints. And it makes for simpler (hardware) implementations if fragmentation and reassembly can be avoided. However, even under that assumption it would be beneficial to be able to detect when there is some misconfiguration causing packets to be dropped due to MTU issues. One way to do this is to have the encapsulator set the don't-fragment (DF) flag in the outer IPv4 header and receive and log any received ICMP "packet too big" (PTB) errors. Note that no flag needs to be set in an outer IPv6 header [RFC2460]. Encapsulations could also define an optional tunnel fragmentation and reassembly mechanism which would be useful in the case when the operator doesn't have full control of the path. Such a mechanism would be required if the underlay might have a path MTU which makes it impossible to carry at least 1518 bytes (if offering Ethernet service), or at least 1280 (if offering IPv6 service). The use of such a protocol mechanism could be triggered by receiving a PTB. But such a mechanism might not be implemented by all encaps and decaps nodes. [Aerolink is one example of such a protocol.] Depending on the payload carried by the encapsulation there are some additional possibilities: o If payload is IPv4/6 then the underlay path MTU could be used to report end-to-end path MTU. o If the payload service is Ethernet/L2, then there is no such per destination reporting mechanism. However, there is a LLDP TLV for reporting max frame size; might be useful to report minimum to end stations, but unmodified end stations would do nothing with that TLV since they assume that the MTU is at least 1518. 10. OAM The OAM area is seeing active development in the IETF with Nordmark (ed), et al. Expires September 10, 2015 [Page 10] Internet-Draft Encaps Considerations March 2015 discussions (at least) in NVO3 and SFC working groups, plus the new LIME WG looking at architecture and YANG models. The design team has take a narrow view of OAM to explore the potential OAM implications on the encapsulation format. In terms of what we have heard from the various working groups there seem to be needs to: o Be able to send out-of-band OAM messages - that potentially should follow the same path through the network as some flow of data packets. * Such OAM messages should not accidentally be decapsulated and forwarded to the end stations. * Be able to add OAM information to data packets that are encapsulated. Discussions have been around * Using a bit in the OAM to synchronize sampling of counters between the encapsulator and decapsulator. * Optional timestamps, sequence numbers, etc for more detailed measurements between encapsulator and decapsulator. o Usable for both proactive monitoring (akin to BFD) and reactive checks (akin to traceroute to pin-point a failure) To ensure that the OAM messages can follow the same path the OAM messages need to get the same ECMP (and LAG hashing) results as a given data flow. An encaps can choose between one of: o Limit ECMP hashing to not look past the UDP header i.e. the entropy needs to be in the source/destination IP and UDP ports o Make OAM packets look the same as data packets i.e. the initial part of the OAM payload has the inner Ethernet, IP, TCP/UDP headers as a payload. (This approach was taken in TRILL out of necessity since there is no UDP header.) OAM bit in encaps must in any case be excluded from the entropy. There can be several ways to prevent OAM packets from accidentally being forwarded to hosts using: o A bit in the frame (as in TRILL) indicating OAM o A next protocol indication with a designated value for "none" or "oam". This assumes that the bit or next protocol, respectively, would not affect entropy/ECMP in the underlay. There has been suggestions that one (or more) marker bits in the encaps header would be useful in order to delineate measurement epochs on the encapsulator and decapsulator and use that to compare counters to determine packet loss. A result of the above is that OAM is likely to evolve and needs some degree of extensibility from the encapsulation format; a bit or two Nordmark (ed), et al. Expires September 10, 2015 [Page 11] Internet-Draft Encaps Considerations March 2015 plus the ability to define additional larger extensions. An open question is how to handle error messages or other reports relating to OAM. One can think if such reporting as being associated with the encaps the same way ICMP is associated with IP. Would it make sense for the IETF to develop a common Encaps Error Reporting Protocol as part of OAM, which can be used for different encapsulations? And if so, what are the technical challenges. For instance, how to avoid it being filtered as ICMP often is? A potential additional consideration for OAM is the possible future existence of gateways that "stitch" together different dataplane encapsulations and might want to carry OAM end-to-end across the different encapsulations. 11. Security Considerations Different encapsulation use cases will have different requirements around security. For instance, when encapsulation is used to build overlay networks for network virtualization, isolation between virtual networks may be paramount. BIER support of multicast may entail different security requirements than encapsulation for unicast. In real deployment, the security of the underlying network may be considered for determining the level of security needed in the encapsulation layer. However for the purposes of this discussion, we assume that network security is out of scope and that the underlying network does not itself provide adequate or as least uniform security mechanisms for encapsulation. There are at least three considerations for security: o Anti-spoofing/virtual network isolation o Interaction with packet level security such as IPsec o Privacy (e.g., VNI ID confidentially for NVO3) This section uses a VNI ID in NVO3 as an example. A SFC or BIER encapsulation is likely to have fields with similar security and privacy requirements. 11.1. Encapsulation-specific considerations Some of these considerations appear for a new encapsulation, and others are more specific to network virtualization in datacenters. o New attack vectors: Nordmark (ed), et al. Expires September 10, 2015 [Page 12] Internet-Draft Encaps Considerations March 2015 * DDOS on specific queued/paths by attempting to reproduce the 5-tuple hash for targeted connections. * Entropy in outer 5-tuple may be too little or predictable. * Leakage of identifying information in the encapsulation header for an encrypted payload. * Vulnerabilities of using global values in fields like VNI ID. o Trusted versus untrusted tenants in network virtualization: * The criticality of virtual network isolation depends on whether tenants are trusted or untrusted. In the most extreme cases, tenants might not only be untrusted but may be considered hostile. * For a trusted set of users (e.g. a private cloud) it may be sufficient to have just a virtual network identifier to provide isolation. Packets inadvertently crossing virtual networks should be dropped similar to a TCP packet with a corrupted port being received on the wrong connection. * In the presence of untrusted users (e.g. a public cloud) the virtual network identifier must be adequately protected against corruption and verified for integrity. This case may warrant keyed integrity. o Different forms of isolation: * Isolation could be blocking all traffic between tenants (or except as allowed by some firewall) * Could also be about performance isolation i.e. one tenant can overload the network in a way that affects other tenants * Physical isolation of traffic for different tenants in network may be required, as well as required restrictions that tenants may have on where their packets may be routed. o New attack vectors from untrusted tenants: * Third party VMs with untrusted tenants allows internally borne attacks within data centers * Hostile VMs inside the system may exist (e.g. public cloud) * Internally launched DDOS * Passive snooping for mis-delivered packets * Mitigate damage and detection in event that a VM is able to circumvent isolation mechanisms o Tenant-provider relationship: * Tenant might not trust provider, hypervisors, network * Provider likely will need to provide SLA or a least a statement on security * Tenant may implement their own additional layers of security * Regulation and certification consuderations o Trend towards tighter security: * Tenants' data in network increases in volume and value, attacks become more sophisticated * Large DCs already encrypt everything on disk Nordmark (ed), et al. Expires September 10, 2015 [Page 13] Internet-Draft Encaps Considerations March 2015 * DCs likely to encrypt inter-DC traffic at this point, use TLS to Internet. * Encryption within DC is becoming more commonplace, becomes ubiquitous when cost is low enough. * Cost/performance considerations. Cost of support for strong security has made strong network security in DCs prohibitive. * Are there lessons from MacSec? 11.2. Virtual network isolation The first requirement is isolation between virtual networks. Packets sent in one virtual network should never be illegitimately received by a node in another virtual network. Isolation should be protected in the presence of malicious attacks or inadvertent packet corruption. The second requirement is sender authentication. Sender identity is authenticated to prevent anti-spoofing. Even if an attacker has access to the packets in the network, they cannot send packets into a virtual network. This may have two possibilities: o Pairwise sender authentication. Any two communicating hosts negotiate a shared key. o Group authentication. A group of hosts share a key (this may be more appropriate for multicast of encapsulation). Possible security solutions: o Security cookie: This is similar to L2TP cookie mechanism [RFC3931]. A shared plain text cookie is shared between encapsulator and decapsulator. A receiver validates a packet by evaluating if the cookie is correct for the virtual network and address of a sender. Validation function is F(cookie, VNI ID, source addr). If cookie matches, accept packet, else drop. Since cookie is plain text this method does not protect against an eavesdropping. Cookies are set and may be rotated out of band. o Secure hash: This is a stronger mechanism than simple cookies that borrows from IPsec and PPP authentication methods. In this model security field contains a secure hash of some fields in the packet using a shared key. Hash function may be something like H(key, VNI ID, addrs, salt). The salt ensures the hash is not the same for every packet, and if it includes a sequence number may also protect against replay attacks. In any use of a shared key, periodic re-keying should be allowed. This could include use of techniques like generation numbers, key windows, etc. See [I-D.farrelll-mpls-opportunistic-encrypt] for an example application. We might see firewalls that are aware of the encaps and can provide Nordmark (ed), et al. Expires September 10, 2015 [Page 14] Internet-Draft Encaps Considerations March 2015 some defense in depth combined with the above example anti-spoofing approaches. An example would be an NVO3-aware firewall being able to check the VNI ID. Separately and in addition to such filtering, there might be a desire to completely block an encapsulation protocol at certain places in the network, e.g., at the edge of a datacenter. Using a fixed standard UDP destination port number for each encapsulation protocol would facilitate such blocking. 11.3. Packet level security An encapsulated packet may itself be encapsulated in IPsec (e.g. ESP). This should be straightforward and in fact is what would happen today in security gateways. In this case, there is no special consideration for the fact that packet is encapsulated, however since the encapsulation layer headers are included (part of encrypted data for instance) we lose visibility in the network of the encapsulation. The more interesting case is when security is applied to the encapsulation payload. This will keep the encapsulation headers in the outer header and visible to the network (for instance in nvo3 we may want to firewall based on VNI ID even if packet is encrypted). In this model protocol stack may be something like IP|UDP|Encap|ESP|IP in tunnel mode, but there's nothing that prevents using transport mode (this looks a lot like ESP/UDP [RFC3948]). The encapsulation and security are probably done together at encapsulator and resolved at decapsulator. Since the encapsulation header is outside of the security coverage, this may itself require security also (like described above). In both of the above the security associations (SAs) may be between physical hosts, so for instance in nvo3 we can have packets of different virtual networks using the same SA-- this should not be an issue since it is the VNI ID that ensures isolation (which needs to be secured also). In this case of security applied to encap payload, this does present a bit of protocol layer inversion in the header (encapsulation refers to overlay, but ESP operates on underlay), but this should be okay as long as semantics are clear and processing is deterministic. 12. QoS In the Internet architecture we support QoS using the Differentiated Services Code Points (DSCP) in the formerly named Type-of-Service field in the IPv4 header, and in the Traffic-Class field in the IPv6 header. The ToS and TC fields also contain the two ECN bits. Nordmark (ed), et al. Expires September 10, 2015 [Page 15] Internet-Draft Encaps Considerations March 2015 We have existing specifications how to process those bits. See [RFC2983] for diffserv handling, which specifies how the received DSCP value is used to set the DSCP value in an outer IP header when encapsulating. (There are also existing specifications how DSCP can be mapped to layer2 priorities.) Those specifications apply whether or not there is some intervening headers (e.g., for NVO3 or SFC) between the inner and outer IP headers. Thus the encapsulation considerations in this area are mainly about applying the framework in [RFC2983]. There are some other considerations specific to doing OAM for encapsulations. If OAM messages are used to measure latency, it would make sense to treat them the same as data payloads. Thus they need to have the same outer DSCP value as the data packets which they wish to measure. Due to OAM there are constraints on middleboxes in general. If middleboxes inspect the packet past the outer IP+UDP and encaps header and look for inner IP and TCP/UDP headers, that might violate the assumption that OAM packets will be handled the same as regular data packets. That issue is broader than just QoS - applies to firewall filters etc. 13. Congestion Considerations Additional encapsulation headers does not introduce anything new for Explicit Congestion Notification. It is just like IP-in-IP and IPsec tunnels which is specified in [RFC6040] in terms of how the ECN bits in the inner and outer header are handled when encapsulating and decapsulating packets. Thus new encapsulations can more or less include that by reference. There are additional considerations around carrying non-congestion controlled traffic. These details have been worked out in [I-D.ietf-mpls-in-udp]. As specified in [RFC5405]: "IP-based traffic is generally assumed to be congestion-controlled, i.e., it is assumed that the transport protocols generating IP-based traffic at the sender already employ mechanisms that are sufficient to address congestion on the path Consequently, a tunnel carrying IP-based traffic should already interact appropriately with other traffic sharing the path, and specific congestion control mechanisms for the tunnel are not necessary". For this reason, where an encaps is used to carry IP traffic that is known to be congestion controlled, the UDP tunnels does not create an additional need for congestion control. Internet IP traffic is Nordmark (ed), et al. Expires September 10, 2015 [Page 16] Internet-Draft Encaps Considerations March 2015 generally assumed to be congestion-controlled. Similarly, in general Layer 3 VPNs are carrying IP traffic that is similarly assumed to be congestion controlled. However, some of the encapsulations (at least NVO3) will be able to carry arbitrary Layer 2 packets to provide an L2 service, in which case one can not assume that the traffic is congestion controlled. One could handle this by adding some congestion control support to the encapsulation header (one instance of which would end up looking like DCCP). However, if the underlay is well-provisioned and managed as opposed to being arbitrary Internet path, it might be sufficient to have a slower reaction to congestion induced by that traffic. There is work underway on a notion of "circuit breakers" for this purpose. See See [I-D.ietf-tsvwg-circuit-breaker]. Encapsulations which carry arbitrary Layer 2 packets want to consider that ongoing work. If the underlay is provisioned in such a way that it can guarantee sufficient capacity for non-congestion controlled Layer 2 traffic, then such circuit breakers might not be needed. Two other considerations appear in the context of these encapsulations as applied to overlay networks: o Protect against malicious end stations o Ensure fairness and/or measure resource usage across multiple tenants Those issues are really orthogonal to the encapsulation, in that they are present even when no new encapsulation header is in use. However, the application of the new encapsulations are likely to be in environments where those issues are becoming more important. Hence it makes sense to consider them. One could make the encapsulation header be extensible to that it can carry sufficient information to be able to measure resource usage, delays, and congestion. The suggestions in the OAM section about a single bit for counter synchronization, and optional timestamps and/or sequence numbers, could be part of such an approach. There might also be additional congestion-control extensions to be carried in the encapsulation. Overall this results in a consideration to be able to have sufficient extensibility in the encapsulation to be handle to handle potential future developments in this space. Coarse measurements are likely to suffice, at least for circuit- breaker-like purposes, see [I-D.wei-tsvwg-tunnel-congestion-feedback] and [I-D.briscoe-conex-data-centre] for examples on active work in this area via use of ECN. [RFC6040] Appendix C is also relevant. The outer ECN bits seem sufficient (at least when everything uses Nordmark (ed), et al. Expires September 10, 2015 [Page 17] Internet-Draft Encaps Considerations March 2015 ECN) to do this course measurements. Needs some more study for the case when there are also drops; might need to exchange counters between ingress and egress to handle drops. Circuit breakers are not sufficient to make a network with different congestion control when the goal is to provide a predictable service to different tenants. The fallback would be to rate limit different traffic. 14. Header Protection Many UDP based encapsulations such as VXLAN [RFC7348] either discourage or explicitly disallow the use of UDP checksums. The reason is that the UDP checksum covers the entire payload of the packet and switching ASICs are typically optimized to look at only a small set of headers as the packet passes through the switch. In these case, computing a checksum over the packet is very expensive. (Software endpoints and the NICs used with them generally do not have the same issue as they need to look at the entire packet anyways.) The lack a header checksum creates the possibility that bit errors can be introduced into any information carried by the new headers. Specifically, in the case of IPv6, the assumption is that a transport layer checksum - UDP in this case - will protect the IP addresses through the inclusion of a pseudoheader in the calculation. This is different from IPv4 on which many of these encapsulation protocols are initially deployed which contains its own header checksum. In addition to IP addresses, the encapsulation header often contains its own information which is used for addressing packets or other high value network functions. Without a checksum, this information is potentially vulnerable - an issue regardless of whether the packet is carried over IPv4 or IPv6. Several protocols cite [RFC6935] and [RFC6936] as an exemption to the IPv6 checksum requirements. However, these are intended to be tailored to a fairly narrow set of circumstances - primarily relying on sparseness of the address space to detect invalid values and well managed networks - and are not a one size fits all solution. In these cases, an analysis should be performed of the intended environment, including the probability of errors being introduced and the use of ECC memory in routing equipment. Conceptually, the ideal solution to this problem is a checksum that covers only the newly added headers of interest. There is little value in the portion of the UDP checksum that covers the encapsulated packet because that would generally be protected by other checksums and this is the expensive portion to compute. In fact, this solution Nordmark (ed), et al. Expires September 10, 2015 [Page 18] Internet-Draft Encaps Considerations March 2015 already exists in the form of UDP-Lite and UDP based encapsulations could be easily ported to run on top of it. Unfortunately, the main value in using UDP as part of the encapsulation header is that it is recognized by already deployed equipment for the purposes of ECMP, RSS, and middlebox operations. As UDP-Lite uses a different protocol number than UDP and it is not widely implemented in middleboxes, this value is lost. A possible solution is to incorporate the same partial-checksum concept as UDP-Lite or other header checksum protection into the encapsulation header and continue using UDP as the outer protocol. One potential challenge with this approach is the use of NAT or other form of translation on the outer header will result in an invalid checksum as the translator will not know to update the encapsulation header. The method chosen to protect headers is often related to the security needs of the encapsulation mechanism. On one hand, the impact of a poorly protected header is not limited to only data corruption but can also introduce a security vulnerability in the form of misdirected packets to an unauthorized recipient. Conversely, high security protocols that already include a secure hash over the valuable portion of the header (such as by encrypting the entire IP packet using IPsec, or some secure hash of the encap header) do not require additional checksum protection as the hash provides stronger assurance than a simple checksum. 15. Extensibility Considerations Protocol extensibility is the concept that a networking protocol may be extended to include new use cases or functionality that were not part of the original protocol specification. Extensibility may be used to add security, control, management, or performance features to a protocol. A solution may allow private extensions for customization or experimentation. Extending a protocol often implies that a protocol header must carry new information. There are two usual methods to accomplish this: 1. Define or redefine the meaning of existing fields in a protocol header. 2. Add new (optional) fields to the protocol header. It is also possible to create a new protocol version, but this is more associated with defining a protocol than extending it (IPv6 being a successor to IPv4 is an example of protocol versioning). Many protocol definitions include some number of reserved fields or bits which can be used for future extension. VXLAN is an example of a protocol that includes reserved bits which are subsequently being allocated for new purposes. Another technique employed is to Nordmark (ed), et al. Expires September 10, 2015 [Page 19] Internet-Draft Encaps Considerations March 2015 repurpose existing header fields with new meanings. A classic example of this is the definition of DSCP code point which redefines the ToS field originally specified in IPv4. When a field is redefined, some mechanism may be needed to ensure that all interested parties agree on the meaning of the field. The techniques of defining meaning for reserved bits or redefining existing fields have the advantage that a protocol header can be kept a fixed length. The disadvantage is that the extensibility is limited. For instance, the number reserved bits in a fixed protocol header is limited. For standard protocols the decision to commit to a definition for a field can be wrenching since it is difficult to retract later. Also, it is difficult to predict a priori how many reserved fields or bits to put into a protocol header to satisfy the extensions create over the lifetime of the protocol. Extending a protocol header with new fields can be done in several ways. o TLVs are a very popular method used in such protocols as IP and TCP. Depending on the type field size and structure, TLVs can offer a virtually unlimited range of extensions. A disadvantage of TLVs is that processing them can be verbose, quite complicated, several validations must often be done for each TLV, and there is no deterministic ordering for a list of TLVs. TCP serves as an example of a protocol where TLVs have been successfully used (i.e. required for protocol operation). IP is an example of a protocol that allows TLVs but are rarely used in practice (router fast paths usually that assume no IP options). Note that TCP TLVs are implemented in software as well as (NIC) hardware handling various forms of TCP offload. o Extension headers are closely related to TLVs. These also carry type/value information, but instead of being a list of TLVs within a single protocol header, each one is in its own protocol header. IPv6 extension headers and SFC NSH are examples of this technique. Similar to TLVs these offer a wide range of extensibility, but have similarly complex processing. Another difference with TLVs is that each extension header is idempotent. This is beneficial in cases where a protocol implements a push/pop model for header elements like service chaining, but makes it more difficult group correlated information within one protocol header. o A particular form of extension headers are the tags used by IEEE 802 protocols. Those are similar to e.g., IPv6 extension headers but with the key difference that each tag is a fixed length header where the length is implicit in the tag value. Thus as long as a receiver can be programmed with a tag value to length map, it can skip those new tags. o Flag-fields are a non-TLV like method of extending a protocol header. The basic idea is that the header contains a set of flags, where each set flags corresponds to optional field that is Nordmark (ed), et al. Expires September 10, 2015 [Page 20] Internet-Draft Encaps Considerations March 2015 present in the header. GRE is an example of a protocol that employs this mechanism. The fields are present in the header in the order of the flags, and the length of each field is fixed. Flag-fields are simpler to process compared to TLVs, having fewer validations and the order of the optional fields is deterministic. A disadvantage is that range of possible extensions with flag- fields is smaller than TLVs. The requirements for receiving unknown or unimplemented extensible elements in an encapsulation protocol (flags, TLVs, optional fields) need to be specified. There are two parties to consider, middle boxes and terminal endpoints of encapsulation (at the decapsulator). A protocol may allow or expect nodes in a path to modify fields in an encapsulation (example use of this is BIER). In this case, the middleboxes should follow the same requirements as nodes terminating the encapsulation. In the case that middle boxes do not modify the encapsulation, we can assume that they may still inspect any fields of the encapsulation. Missing or unknown fields should be accepted per protocol specification, however it is permissible for a site to implement a local policy otherwise (e.g. a firewall may drop packets with unknown options). For handling unknown options at terminal nodes, there are two possibilities: drop packet or accept while ignoring the unknown options. Many Internet protocols specify that reserved flags must be set to zero on transmission and ignored on reception. L2TP is example data protocol that has such flags. GRE is a notable exception to this rule, reserved flag bits 1-5 cannot be ignored [RFC2890]. For TCP and IPv4, implementations must ignore optional TLVs with unknown type; however in IPv6 if a packet contains an unknown extension header (unrecognized next header type) the packet must be dropped with an ICMP error message returned. The IPv6 options themselves (encoded inside the destinations options or hop- by-hop options extension header) have more flexibility. There bits in the option code are used to instruct the receiver whether to ignore, silently drop, or drop and send error if the option is unknown. Some protocols define a "mandatory bit" that can is set with TLVs to indicate that an option must not be ignored. Conceptually, optional data elements can only be ignored if they are idempotent and do not alter how the rest of the packet is parsed or processed. Depending on what type of protocol evolution one can predict, it might make sense to have an way for a sender to express that the packet should be dropped by a terminal node which does not understand the new information. In other cases it would make sense to have the receiver silently ignore the new info. The former can be expressed Nordmark (ed), et al. Expires September 10, 2015 [Page 21] Internet-Draft Encaps Considerations March 2015 by having a version field in the encapsulation, or a notion of "mandatory bit" as discussed above. A security mechanism which use some form secure hash over the encaps header would need to be able to know which extensions can be changed in flight. 15.1. Next-protocol [Note that there is editorial duplication between this section and the Next Protocol Indication section earlier in the document] Choices: o Payload type implied (by UDP port for instance). ESP, L2TP, MPLS, over UDP are example, Linux Foo-Over-UDP is example implementation. This model is simple, however allocating a port for every possible protocol might be difficult given the trend towards port conservation as described in [I-D.ietf-tsvwg-port-use]. o Encapsulation contains a next header field. Possibilities: * EtherType: GRE protocol field is example. Allows encapsulation of any EtherType (including IPv4, IPv6, end Ethernet). Disadvantages are that it is 16 bits for less than 100 needed values, and the number space is controlled by the IEEE 802 RAC. * IP protocol: IPv6 extension headers, ESP are examples. Allows encapsulation of any IP protocol including Ethernet via ETHERIP, IPv4, IPv6, IPsec protocols, UDP (transport mode considerations needed). IANA managed eight bit values, presumably more difficult to get an assigned number than to get a transport port assignment. * Protocol specific number space. Example PPP. Could be 8 or 16 bits and would be IANA controlled. Primary advantage is that it might be easier to define protocols for encapsulation that are not defined in other number spaces (802.11 for instance). Disadvantage is that it represents yet another number space to be managed and doesn't leverage existing ones. 16. Layering Considerations One can envision that SFC might use NVO3 as a delivery/transport mechanism. With more imagination that in turn might be delivered using BIER. Thus it is useful to think about what things look like when we have BIER+NVO3+SFC+payload. Also, if NVO3 is widely deployed there might be cases of NVO3 nesting where a customer uses NVO3 to provide network virtualization e.g., across departments. That customer uses a service provider which happens to use NVO3 to provide transport for their customers.Thus NVO3 in NVO3 might happen. Nordmark (ed), et al. Expires September 10, 2015 [Page 22] Internet-Draft Encaps Considerations March 2015 A key question we set out to answer is what the packets might look like in such a case, and in particular whether we would end up with multiple UDP headers for entropy. Based on the discussion in the Entropy section, the entropy is associated with the outer delivery IP header. Thus if there are multiple IP headers there would be a UDP header for each one of the IP headers. But SFC does not require its own IP header. So a case of NVO3+SFC would be IP+UDP+NVO3+SFC. A nested NVO3 encapsulation would have independent IP+UDP headers. The layering also has some implications for middleboxes. o A device on the path between the ingress and egress is allowed to transparently inspect all layers of the protocol stack and drop or forward, but not transparently modify anything but the layer in which they operate. What this means is that an IP router is allowed modify the outer IP ttl and ECN bits, but not the encaps header or inner headers and payload. And a BIER router is allowed to modify the BIER header. o Alternatively such a device can become visible at a higher layer. E.g., a middlebox could become an decaps + function + encaps which means it will generate a new encaps header. The design team asked itself some additional questions: o Would it make sense to have a common encaps base header (for OAM, security?, etc) and then followed by the specific information for NVO3, SFC, BIER? Given that there are separate proposals and the set of information needing to be carried differs, and the extensibility needs might be different, it would be difficult and not that useful to have a common base header. o With a base header in place, one could view the different functions (NVO3, SFC, and BIER) as different extensions to that base header resulting in encodings which are more space optimal by not repeating the same base header. The base header would only be repeated when there is an additional IP (and hence UDP) header. That could mean a single length field (to skip to get to the payload after all the encaps headers). That might be technically feasible, but it would create a lot of dependencies between different WGs making it harder to make progress. Compare with the potential savings in packet size. 17. Service model The IP service is lossy and subject to reordering. In order to avoid a performance impact on transports like TCP the handling of packets is designed to avoid reordering packets that are in the same transport flow (which is typically identified by the 5-tuple). But Nordmark (ed), et al. Expires September 10, 2015 [Page 23] Internet-Draft Encaps Considerations March 2015 across such flows the receiver can see different ordering for a given sender. That is the case for a unicast vs. a multicast flow from the same sender. There is a general tussle between the desire for high capacity utilization across a multipath network and the import on packet ordering within the same flow (which results in lower transport protocol performance). That isn't affected by the introduction of an encapsulation. However, the encapsulation comes with some entropy, and there might be cases where folks want to change that in response to overload or failures. For instance, might want to change UDP source port to try different ECMP route. Such changes can result in packet reordering within a flow, hence would need to be done infrequently and with care e.g., by identifying packet trains. There might be some applications/services which are not able to handle reordering across flows. The IETF has defined pseudo-wires [RFC3985] which provides the ability to ensure ordering (implemented using sequence numbers and/or timestamps). Architectural such services would make sense, but as a separate layer on top of an encapsulation protocol. They could be deployed between ingress and egress of a tunnel which uses some encaps. Potentially the tunnel control points in the form of an ingress and egress will become a platform for fixing suboptimal behavior elsewhere in the network. For example, this document suggests that some congestion handling might be needed to handle non-congestion controlled traffic that gets tunneled, and also that fairness/QoS policing can be deployed on those devices. Others have suggested that tunnels is one way to deploy ECN without having to add ECN support in the endpoints [I-D.briscoe-conex-data-centre]. But the tunnels could potentially do more like increase reliability (retransmissions, FEC) or load spreading using e.g. MP-TCP between ingress and egress. 18. Hardware Friendly Hosts, switches and routers often leverage capabilities in the hardware to accelerate packet encapsulation, decapsulation and forwarding. Some design considerations in encapsulation that leverage these hardware capabilities may result in more efficiently packet processing and higher overall protocol throughput. While "hardware friendliness" can be viewed as unnecessary Nordmark (ed), et al. Expires September 10, 2015 [Page 24] Internet-Draft Encaps Considerations March 2015 considerations for a design, part of the motivation for considering this is ease of deployment; being able to leverage existing NIC and switch chips for at least a useful subset of the functionality that the new encapsulation provides. The other part is the ease of implementing new NICs and switch/router chips that support the encapsulation at ever increasing line rates. [disclaimer] There are many different types of hardware in any given network, each maybe better at some tasks while worse at others. We would still recommend protocol designers to examine the specific hardware that are likely to be used in their networks and make decisions on a case by case basis. Some considerations are: o Keep the encap header small. Switches and routers usually only read the first small number of bytes into the fast memory for quick processing and easy manipulation. The bulk of the packets are usually stored in slow memory. A big encap header may not fit and additional read from the slow memory will hurt the overall performance and throughput. o Put important information at the beginning of the encapsulation header. The reasoning is similar as explained in the previous point. If important information are located at the beginning of the encapsulation header, the packet may be processed with smaller number of bytes to be read into the fast memory and improve performance. o Separation of NVO3 header from SFC header such that an encap can also be processed by forwarding hardware (who can only process network virtualization and pass the service chaining function to another device specialized in service offering) o Avoid full packet checksums in the encapsulation if possible. Most of the switch/router hardware make switching/forwarding decisions by reading and examining only the first certain number of bytes in the packet. Most of the body of the packet do not need to be processed normally. if we are concerned of preventing packet to be misdelivered due to memory errors, consider only perform header checksums. Note that NIC chips can typically already do full packet checksums for TCP/UDP, while adding a header checksum might require adding some hardware support. o Place important information at fixed offset in the encapsulation header. Packet processing hardware may be capable of parallel processing. If important information can be found at fixed offset, different part of the encapsulation header may be processed by different hardware units in parallel (for example multiple table lookups may be launched in parallel). Hardware can handle optional information as long as when the information is present it is found in one and only one place in the header. Typical TLV encoding of options does not have that property since Nordmark (ed), et al. Expires September 10, 2015 [Page 25] Internet-Draft Encaps Considerations March 2015 the order of TLVs is unconstrained. o Limit the number of header combinations. In many cases the hardware can explore different combinations of headers in parallel, however there is some added cost for this. 18.1. Considerations for NIC offload This section provides guidelines to provide support of common offloads for encapsulation in Network Interface Cards (NICs). Offload mechanisms are techniques that are implemented separately from the normal protocol implementation of a host networking stack and are intended to optimize or speed up protocol processing. Hardware offload is performed within a NIC device on behalf of a host. There are three basic offload techniques of interest: o Receive multi queue o Checksum offload o Segmentation offload 18.1.1. Receive multi-queue Contemporary NICs support multiple receive descriptor queues (multi- queue). Multi-queue enables load balancing of network processing for a NIC across multiple CPUs. On packet reception, a NIC must select the appropriate queue for host processing. Receive Side Scaling (RSS) is a common method which uses the flow hash for a packet to index an indirection table where each entry stores a queue number. UDP encapsulation, where the source port is used for entropy, should be compatible with multi-queue NICs that support five-tuple hash calculation for UDP/IP packets as input to RSS. The source port ensures classification of the encapsulated flow even in the case that the outer source and destination addresses are the same for all flows (e.g. all flows are going over a single tunnel). 18.1.2. Checksum offload Many NICs provide capabilities to calculate standard ones complement payload checksum for packets in transmit or receive. When using encapsulation over UDP there are at least two checksums that may be of interest: the encapsulated packet's transport checksum, and the UDP checksum in the outer header. 18.1.2.1. Transmit checksum offload NICs may provide a protocol agnostic method to offload transmit checksum (NETIF_F_HW_CSUM in Linux parlance) that can be used with Nordmark (ed), et al. Expires September 10, 2015 [Page 26] Internet-Draft Encaps Considerations March 2015 UDP encapsulation. In this method the host provides checksum related parameters in a transmit descriptor for a packet. These parameters include the starting offset of data to checksum, the length of data to checksum, and the offset in the packet where the computed checksum is to be written. The host initializes the checksum field to pseudo header checksum. In the case of encapsulated packet, the checksum for an encapsulated transport layer packet, a TCP packet for instance, can be offloaded by setting the appropriate checksum parameters. NICs typically can offload only one transmit checksum per packet, so simultaneously offloading both an inner transport packet's checksum and the outer UDP checksum is likely not possible. In this case setting UDP checksum to zero (per above discussion) and offloading the inner transport packet checksum might be acceptable. There is a proposal in [I-D.herbert-remotecsumoffload] to leverage NIC checksum offload when an encapsulator is co-resident with a host. 18.1.2.2. Receive checksum offload Protocol encapsulation is compatible with NICs that perform a protocol agnostic receive checksum (CHECKSUM_COMPLETE in Linux parlance). In this technique, a NIC computes a ones complement checksum over all (or some predefined portion) of a packet. The computed value is provided to the host stack in the packet's receive descriptor. The host driver can use this checksum to "patch up" and validate any inner packet transport checksum, as well as the outer UDP checksum if it is non-zero. Many legacy NICs don't provide checksum-complete but instead provide an indication that a checksum has been verified (CHECKSUM_UNNECESSARY in Linux). Usually, such validation is only done for simple TCP/IP or UDP/IP packets. If a NIC indicates that a UDP checksum is valid, the checksum-complete value for the UDP packet is the "not" of the pseudo header checksum. In this way, checksum-unnecessary can be converted to checksum-complete. So if the NIC provides checksum- unnecessary for the outer UDP header in an encapsulation, checksum conversion can be done so that the checksum-complete value is derived and can be used by the stack to validate an checksums in the encapsulated packet. 18.1.3. Segmentation offload Segmentation offload refers to techniques that attempt to reduce CPU utilization on hosts by having the transport layers of the stack operate on large packets. In transmit segmentation offload, a transport layer creates large packets greater than MTU size (Maximum Nordmark (ed), et al. Expires September 10, 2015 [Page 27] Internet-Draft Encaps Considerations March 2015 Transmission Unit). It is only at much lower point in the stack, or possibly the NIC, that these large packets are broken up into MTU sized packet for transmission on the wire. Similarly, in receive segmentation offload, small packets are coalesced into large, greater than MTU size packets at a point low in the stack receive path or possibly in a device. The effect of segmentation offload is that the number of packets that need to be processed in various layers of the stack is reduced, and hence CPU utilization is reduced. 18.1.3.1. Transmit Segmentation Offload Transmit Segmentation Offload (TSO) is a NIC feature where a host provides a large (larger than MTU size) TCP packet to the NIC, which in turn splits the packet into separate segments and transmits each one. This is useful to reduce CPU load on the host. The process of TSO can be generalized as: o Split the TCP payload into segments which allow packets with size less than or equal to MTU. o For each created segment: 1. Replicate the TCP header and all preceding headers of the original packet. 2. Set payload length fields in any headers to reflect the length of the segment. 3. Set TCP sequence number to correctly reflect the offset of the TCP data in the stream. 4. Recompute and set any checksums that either cover the payload of the packet or cover header which was changed by setting a payload length. Following this general process, TSO can be extended to support TCP encapsulation UDP. For each segment the Ethernet, outer IP, UDP header, encapsulation header, inner IP header if tunneling, and TCP headers are replicated. Any packet length header fields need to be set properly (including the length in the outer UDP header), and checksums need to be set correctly (including the outer UDP checksum if being used). To facilitate TSO with encapsulation it is recommended that optional fields should not contain values that must be updated on a per segment basis-- for example an encapsulation header should not include checksums, lengths, or sequence numbers that refer to the payload. If the encapsulation header does not contain such fields then the TSO engine only needs to copy the bits in the encapsulation header when creating each segment and does not need to parse the encapsulation header. Nordmark (ed), et al. Expires September 10, 2015 [Page 28] Internet-Draft Encaps Considerations March 2015 18.1.3.2. Large Receive Offload Large Receive Offload (LRO) is a NIC feature where packets of a TCP connection are reassembled, or coalesced, in the NIC and delivered to the host as one large packet. This feature can reduce CPU utilization in the host. LRO requires significant protocol awareness to be implemented correctly and is difficult to generalize. Packets in the same flow need to be unambiguously identified. In the presence of tunnels or network virtualization, this may require more than a five-tuple match (for instance packets for flows in two different virtual networks may have identical five-tuples). Additionally, a NIC needs to perform validation over packets that are being coalesced, and needs to fabricate a single meaningful header from all the coalesced packets. The conservative approach to supporting LRO for encapsulation would be to assign packets to the same flow only if they have identical five-tuple and were encapsulated the same way. That is the outer IP addresses, the outer UDP ports, encapsulated protocol, encapsulation headers, and inner five tuple are all identical. 19. Middlebox Considerations This document has touched upon middleboxes in different section. The reason for this is as encapsulations get widely deployed one would expect different forms of middleboxes might become aware of the encapsulation protocol just as middleboxes have been made aware of other protocols where there are business and deployment opportunities. Such middleboxes are likely to do more than just drop packets based on the UDP port number used by an encapsulation protocol. We note that various forms of encapsulation gateways that stitch one encapsulation protocol together with another form of protocol could have similar effects. An example of a middlebox that could see some use would be an NVO3- aware firewall that would filter on the VNI IDs to provide some defense in depth inside or across NVO3 datacenters. A question for the IETF is whether we should document what to do or what not to do in such middleboxes. This document touches on areas of OAM and ECMP as it relates to middleboxes and it might make sense to document how encaps-aware middleboxes should avoid unintended consequences in those (and perhaps other) areas. Nordmark (ed), et al. Expires September 10, 2015 [Page 29] Internet-Draft Encaps Considerations March 2015 20. Related Work The IETF and industry has defined encapsulations for a long time, with examples like GRE [RFC2890], VXLAN [RFC7348], and NVGRE [I-D.sridharan-virtualization-nvgre] being able to carry arbitrary Ethernet payloads, and various forms of IP-in-IP and IPsec encapsulations that can carry IP packets. As part of NVO3 there has been additional proposals like Geneve [I-D.gross-geneve] and GUE [I-D.herbert-gue] which look at more extensibility. NSH [I-D.quinn-sfc-nsh] is an example of an encapsulation that tries to provide extensibility mechanisms which target both hardware and software implementations. There is also a large body of work around MPLS encapsulations [RFC3032]. The MPLS-in-UDP work [I-D.ietf-mpls-in-udp] and GRE over UDP [I-D.ietf-tsvwg-gre-in-udp-encap] have worked on some of the common issues around checksum and congestion control. MPLS also introduced a entropy label [RFC6790]. There is also a proposal for MPLS encryption [I-D.farrelll-mpls-opportunistic-encrypt]. The idea to use a UDP encapsulation with a UDP source port for entropy for the underlay routers' ECMP dates back to LISP [RFC6830]. The pseudo-wire work [RFC3985] is interesting in the notion of layering additional services/characteristics such as ordered delivery or timely deliver on top of an encapsulation. That layering approach might be useful for the new encapsulations as well. For instance, the control word [RFC4385]. Both MPLS and L2TP [RFC3931] rely on some control or signaling to establish state (for the path/labels in the case of MPLS, and for the session in the case of L2TP). The NVO3, SFC, and BIER encapsulations will also have some separation between the data plane and control plane, but the type of separation appears to be different. IEEE 802.1 has defined encapsulations for L2 over L2, in the form of Provider backbone Bridging (PBB) [IEEE802.1Q-2014] and Equal Cost Multipath (ECMP) [IEEE802.1Q-2014]. The latter includes something very similar to the way the UDP source port is used as entropy: "The flow hash, carried in an F-TAG, serves to distinguish frames belonging to different flows and can be used in the forwarding process to distribute frames over equal cost paths" TRILL, which is also a L2 over L2 encapsulation, took a different approach to entropy but preserved the ability for OAM frames [RFC7174] to use the same entropy hence ECMP path as data frames. In [I-D.ietf-trill-oam-fm] there 96 bytes of headers for entropy in the OAM frames, followed by the actual OAM content. This ensures that Nordmark (ed), et al. Expires September 10, 2015 [Page 30] Internet-Draft Encaps Considerations March 2015 any headers, which fit in those 96 bytes except the OAM bit in the TRILL header, can be used for ECMP hashing. As encapsulations evolve there might be a desire to fit multiple inner packets into one outer packet. The work in [I-D.saldana-tsvwg-simplemux] might be interesting for that purpose. 21. Acknowledgements The authors acknowledge the comments from David Black, Andy Malis, and Radia Perlman. 22. Open Issues o Middleboxes: * Due to OAM there are constraints on middleboxes in general. If middleboxes inspect the packet past the outer IP+UDP and encaps header and look for inner IP and TCP/UDP headers, that might violate the assumption that OAM packets will be handled the same as regular data packets. That issue is broader than just QoS - applies to firewall filters etc. * Firewalls looking at inner payload? How does that work for OAM frames? Even if it only drops ... TRILL approach might be an option? Would that encourage more middleboxes making the network more fragile? * Editorially perhaps we should pull the above two into a separate section about middlebox considerations? o Next protocol indication - should it be common across different encapsulation headers? We will have different ways to indicate the presence of the first encapsulation header in a packet (could be a UDP destination port, an Ethernet type, etc depending on the outer delivery header). But for the next protocol past an encapsulation header one could envision creating or adoption a common scheme. Such a would also need to be able to identify following headers like Ethernet, IPv4/IPv6, ESP, etc. o Common OAM error reporting protocol? o There is discussion about timestamps, sequence numbers, etc in three different parts of the document. OAM, Congestion Considerations, and Service Model, where the latter argues that a pseudo-wire service should really be layered on top of the encaps using its own header. Those recommendations seem to be at odds with each other. Do we envision sequence numbers, timestamps, etc as potential extensions for OAM and CC? If so, those extensions could be used to provide a service which doesn't reorder packets. Nordmark (ed), et al. Expires September 10, 2015 [Page 31] Internet-Draft Encaps Considerations March 2015 23. References 23.1. Normative References [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, December 1998. [RFC2890] Dommety, G., "Key and Sequence Number Extensions to GRE", RFC 2890, September 2000. [RFC2983] Black, D., "Differentiated Services and Tunnels", RFC 2983, October 2000. [RFC3032] Rosen, E., Tappan, D., Fedorkow, G., Rekhter, Y., Farinacci, D., Li, T., and A. Conta, "MPLS Label Stack Encoding", RFC 3032, January 2001. [RFC3931] Lau, J., Townsley, M., and I. Goyret, "Layer Two Tunneling Protocol - Version 3 (L2TPv3)", RFC 3931, March 2005. [RFC3948] Huttunen, A., Swander, B., Volpe, V., DiBurro, L., and M. Stenberg, "UDP Encapsulation of IPsec ESP Packets", RFC 3948, January 2005. [RFC3985] Bryant, S. and P. Pate, "Pseudo Wire Emulation Edge-to- Edge (PWE3) Architecture", RFC 3985, March 2005. [RFC4385] Bryant, S., Swallow, G., Martini, L., and D. McPherson, "Pseudowire Emulation Edge-to-Edge (PWE3) Control Word for Use over an MPLS PSN", RFC 4385, February 2006. [RFC5405] Eggert, L. and G. Fairhurst, "Unicast UDP Usage Guidelines for Application Designers", BCP 145, RFC 5405, November 2008. [RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion Notification", RFC 6040, November 2010. [RFC6790] Kompella, K., Drake, J., Amante, S., Henderickx, W., and L. Yong, "The Use of Entropy Labels in MPLS Forwarding", RFC 6790, November 2012. [RFC6830] Farinacci, D., Fuller, V., Meyer, D., and D. Lewis, "The Locator/ID Separation Protocol (LISP)", RFC 6830, January 2013. [RFC6935] Eubanks, M., Chimento, P., and M. Westerlund, "IPv6 and UDP Checksums for Tunneled Packets", RFC 6935, April 2013. Nordmark (ed), et al. Expires September 10, 2015 [Page 32] Internet-Draft Encaps Considerations March 2015 [RFC6936] Fairhurst, G. and M. Westerlund, "Applicability Statement for the Use of IPv6 UDP Datagrams with Zero Checksums", RFC 6936, April 2013. [RFC7174] Salam, S., Senevirathne, T., Aldrin, S., and D. Eastlake, "Transparent Interconnection of Lots of Links (TRILL) Operations, Administration, and Maintenance (OAM) Framework", RFC 7174, May 2014. [RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger, L., Sridhar, T., Bursell, M., and C. Wright, "Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks", RFC 7348, August 2014. [RFC7364] Narten, T., Gray, E., Black, D., Fang, L., Kreeger, L., and M. Napierala, "Problem Statement: Overlays for Network Virtualization", RFC 7364, October 2014. [RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y. Rekhter, "Framework for Data Center (DC) Network Virtualization", RFC 7365, October 2014. 23.2. Informative References [I-D.briscoe-conex-data-centre] Briscoe, B. and M. Sridharan, "Network Performance Isolation in Data Centres using Congestion Policing", draft-briscoe-conex-data-centre-02 (work in progress), February 2014. [I-D.farrelll-mpls-opportunistic-encrypt] Farrel, A. and S. Farrell, "Opportunistic Security in MPLS Networks", draft-farrelll-mpls-opportunistic-encrypt-04 (work in progress), January 2015. [I-D.gross-geneve] Gross, J., Sridhar, T., Garg, P., Wright, C., Ganga, I., Agarwal, P., Duda, K., Dutt, D., and J. Hudson, "Geneve: Generic Network Virtualization Encapsulation", draft-gross-geneve-02 (work in progress), October 2014. [I-D.herbert-gue] Herbert, T., Yong, L., and O. Zia, "Generic UDP Encapsulation", draft-herbert-gue-03 (work in progress), March 2015. [I-D.herbert-remotecsumoffload] Nordmark (ed), et al. Expires September 10, 2015 [Page 33] Internet-Draft Encaps Considerations March 2015 Herbert, T., "Remote checksum offload for encapsulation", draft-herbert-remotecsumoffload-01 (work in progress), November 2014. [I-D.ietf-mpls-in-udp] Xu, X., Sheth, N., Yong, L., Callon, R., and D. Black, "Encapsulating MPLS in UDP", draft-ietf-mpls-in-udp-11 (work in progress), January 2015. [I-D.ietf-nvo3-arch] Black, D., Hudson, J., Kreeger, L., Lasserre, M., and T. Narten, "An Architecture for Overlay Networks (NVO3)", draft-ietf-nvo3-arch-02 (work in progress), October 2014. [I-D.ietf-sfc-architecture] Halpern, J. and C. Pignataro, "Service Function Chaining (SFC) Architecture", draft-ietf-sfc-architecture-07 (work in progress), March 2015. [I-D.ietf-sfc-problem-statement] Quinn, P. and T. Nadeau, "Service Function Chaining Problem Statement", draft-ietf-sfc-problem-statement-13 (work in progress), February 2015. [I-D.ietf-trill-oam-fm] Senevirathne, T., Finn, N., Salam, S., Kumar, D., Eastlake, D., Aldrin, S., and L. Yizhou, "TRILL Fault Management", draft-ietf-trill-oam-fm-11 (work in progress), October 2014. [I-D.ietf-tsvwg-circuit-breaker] Fairhurst, G., "Network Transport Circuit Breakers", draft-ietf-tsvwg-circuit-breaker-00 (work in progress), September 2014. [I-D.ietf-tsvwg-gre-in-udp-encap] Crabbe, E., Yong, L., Xu, X., and T. Herbert, "GRE-in-UDP Encapsulation", draft-ietf-tsvwg-gre-in-udp-encap-05 (work in progress), March 2015. [I-D.ietf-tsvwg-port-use] Touch, J., "Recommendations for Transport Port Number Uses", draft-ietf-tsvwg-port-use-07 (work in progress), January 2015. [I-D.quinn-sfc-nsh] Quinn, P., Guichard, J., Surendra, S., Smith, M., Henderickx, W., Nadeau, T., Agarwal, P., Manur, R., Nordmark (ed), et al. Expires September 10, 2015 [Page 34] Internet-Draft Encaps Considerations March 2015 Chauhan, A., Halpern, J., Majee, S., Elzur, U., Melman, D., Garg, P., McConnell, B., Wright, C., and K. Kevin, "Network Service Header", draft-quinn-sfc-nsh-07 (work in progress), February 2015. [I-D.saldana-tsvwg-simplemux] Saldana, J., "Simplemux. A generic multiplexing protocol", draft-saldana-tsvwg-simplemux-02 (work in progress), January 2015. [I-D.shepherd-bier-problem-statement] Shepherd, G., Dolganow, A., and a. arkadiy.gulko@thomsonreuters.com, "Bit Indexed Explicit Replication (BIER) Problem Statement", draft-shepherd-bier-problem-statement-02 (work in progress), February 2015. [I-D.sridharan-virtualization-nvgre] Garg, P. and Y. Wang, "NVGRE: Network Virtualization using Generic Routing Encapsulation", draft-sridharan-virtualization-nvgre-07 (work in progress), November 2014. [I-D.wei-tsvwg-tunnel-congestion-feedback] Wei, X., Zhu, L., and L. Deng, "Tunnel Congestion Feedback", draft-wei-tsvwg-tunnel-congestion-feedback-03 (work in progress), October 2014. [I-D.wijnands-bier-architecture] Wijnands, I., Rosen, E., Dolganow, A., Przygienda, T., and S. Aldrin, "Multicast using Bit Index Explicit Replication", draft-wijnands-bier-architecture-05 (work in progress), March 2015. [I-D.wijnands-mpls-bier-encapsulation] Wijnands, I., Rosen, E., Dolganow, A., Tantsura, J., and S. Aldrin, "Encapsulation for Bit Index Explicit Replication in MPLS Networks", draft-wijnands-mpls-bier-encapsulation-02 (work in progress), December 2014. [I-D.xu-bier-encapsulation] Xu, X., Somasundaram, S., Jacquenet, C., and R. Raszuk, "BIER Encapsulation", draft-xu-bier-encapsulation-02 (work in progress), February 2015. [IEEE802.1Q-2014] IEEE, "IEEE Standard for Local and metropolitan area Nordmark (ed), et al. Expires September 10, 2015 [Page 35] Internet-Draft Encaps Considerations March 2015 networks--Bridges and Bridged Networks", IEEE Std 802.1Q- 2014, 2014, . (Access Controlled link within page) Authors' Addresses Erik Nordmark Arista Networks 5453 Great America Parkway Santa Clara, CA 95054 USA Email: nordmark@arista.com Albert Tian Ericsson Inc. 300 Holger Way San Jose, California 95134 USA Email: albert.tian@ericsson.com Jesse Gross VMware 3401 Hillview Ave. Palo Alto, CA 94304 USA Email: jgross@vmware.com Jon Hudson Brocade Communications Systems, Inc. 130 Holger Way San Jose, CA 95134 USA Email: jon.hudson@gmail.com Nordmark (ed), et al. Expires September 10, 2015 [Page 36] Internet-Draft Encaps Considerations March 2015 Lawrence Kreeger Cisco Systems, Inc. 170 W. Tasman Avenue San Jose, CA 95134 USA Email: kreeger@cisco.com Pankaj Garg Microsoft 1 Microsoft Way Redmond, WA 98052 USA Email: pankajg@microsoft.com Patricia Thaler Broadcom Corporation 3151 Zanker Road San Jose, CA 95134 USA Email: pthaler@broadcom.com Tom Herbert Google 1600 Amphitheatre Parkway Mountain View, CA USA Email: therbert@google.com Nordmark (ed), et al. Expires September 10, 2015 [Page 37]