Internet DRAFT - draft-ietf-nvo3-geneve

draft-ietf-nvo3-geneve







Network Working Group                                      J. Gross, Ed.
Internet-Draft
Intended status: Standards Track                           I. Ganga, Ed.
Expires: September 8, 2020                                         Intel
                                                         T. Sridhar, Ed.
                                                                  VMware
                                                          March 07, 2020


          Geneve: Generic Network Virtualization Encapsulation
                       draft-ietf-nvo3-geneve-16

Abstract

   Network virtualization involves the cooperation of devices with a
   wide variety of capabilities such as software and hardware tunnel
   endpoints, transit fabrics, and centralized control clusters.  As a
   result of their role in tying together different elements in the
   system, the requirements on tunnels are influenced by all of these
   components.  Flexibility is therefore the most important aspect of a
   tunnel protocol if it is to keep pace with the evolution of the
   system.  This document describes Geneve, an encapsulation protocol
   designed to recognize and accommodate these changing capabilities and
   needs.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on September 8, 2020.

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as the
   document authors.  All rights reserved.





Gross, et al.           Expires September 8, 2020               [Page 1]

Internet-Draft               Geneve Protocol                  March 2020


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   4
     1.2.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   4
   2.  Design Requirements . . . . . . . . . . . . . . . . . . . . .   6
     2.1.  Control Plane Independence  . . . . . . . . . . . . . . .   7
     2.2.  Data Plane Extensibility  . . . . . . . . . . . . . . . .   7
       2.2.1.  Efficient Implementation  . . . . . . . . . . . . . .   8
     2.3.  Use of Standard IP Fabrics  . . . . . . . . . . . . . . .   8
   3.  Geneve Encapsulation Details  . . . . . . . . . . . . . . . .   9
     3.1.  Geneve Packet Format Over IPv4  . . . . . . . . . . . . .   9
     3.2.  Geneve Packet Format Over IPv6  . . . . . . . . . . . . .  11
     3.3.  UDP Header  . . . . . . . . . . . . . . . . . . . . . . .  13
     3.4.  Tunnel Header Fields  . . . . . . . . . . . . . . . . . .  14
     3.5.  Tunnel Options  . . . . . . . . . . . . . . . . . . . . .  15
       3.5.1.  Options Processing  . . . . . . . . . . . . . . . . .  17
   4.  Implementation and Deployment Considerations  . . . . . . . .  18
     4.1.  Applicability Statement . . . . . . . . . . . . . . . . .  18
     4.2.  Congestion Control Functionality  . . . . . . . . . . . .  19
     4.3.  UDP Checksum  . . . . . . . . . . . . . . . . . . . . . .  19
       4.3.1.  UDP Zero Checksum Handling with IPv6  . . . . . . . .  19
     4.4.  Encapsulation of Geneve in IP . . . . . . . . . . . . . .  21
       4.4.1.  IP Fragmentation  . . . . . . . . . . . . . . . . . .  21
       4.4.2.  DSCP, ECN and TTL . . . . . . . . . . . . . . . . . .  22
       4.4.3.  Broadcast and Multicast . . . . . . . . . . . . . . .  23
       4.4.4.  Unidirectional Tunnels  . . . . . . . . . . . . . . .  23
     4.5.  Constraints on Protocol Features  . . . . . . . . . . . .  24
       4.5.1.  Constraints on Options  . . . . . . . . . . . . . . .  24
     4.6.  NIC Offloads  . . . . . . . . . . . . . . . . . . . . . .  25
     4.7.  Inner VLAN Handling . . . . . . . . . . . . . . . . . . .  25
   5.  Transition Considerations . . . . . . . . . . . . . . . . . .  26
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .  26
     6.1.  Data Confidentiality  . . . . . . . . . . . . . . . . . .  27
       6.1.1.  Inter-Data Center Traffic . . . . . . . . . . . . . .  27
     6.2.  Data Integrity  . . . . . . . . . . . . . . . . . . . . .  28
     6.3.  Authentication of NVE peers . . . . . . . . . . . . . . .  29
     6.4.  Options Interpretation by Transit Devices . . . . . . . .  29



Gross, et al.           Expires September 8, 2020               [Page 2]

Internet-Draft               Geneve Protocol                  March 2020


     6.5.  Multicast/Broadcast . . . . . . . . . . . . . . . . . . .  29
     6.6.  Control Plane Communications  . . . . . . . . . . . . . .  29
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  30
   8.  Contributors  . . . . . . . . . . . . . . . . . . . . . . . .  31
   9.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  32
   10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  33
     10.1.  Normative References . . . . . . . . . . . . . . . . . .  33
     10.2.  Informative References . . . . . . . . . . . . . . . . .  34
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  37

1.  Introduction

   Networking has long featured a variety of tunneling, tagging, and
   other encapsulation mechanisms.  However, the advent of network
   virtualization has caused a surge of renewed interest and a
   corresponding increase in the introduction of new protocols.  The
   large number of protocols in this space, for example, ranging all the
   way from VLANs [IEEE.802.1Q_2018] and MPLS [RFC3031] through the more
   recent VXLAN [RFC7348] (Virtual eXtensible Local Area Network) and
   NVGRE [RFC7637] (Network Virtualization Using Generic Routing
   Encapsulation), often leads to questions about the need for new
   encapsulation formats and what it is about network virtualization in
   particular that leads to their proliferation.  Note that the list of
   protocols presented above is non-exhaustive.

   While many encapsulation protocols seek to simply partition the
   underlay network or bridge between two domains, network
   virtualization views the transit network as providing connectivity
   between multiple components of a distributed system.  In many ways
   this system is similar to a chassis switch with the IP underlay
   network playing the role of the backplane and tunnel endpoints on the
   edge as line cards.  When viewed in this light, the requirements
   placed on the tunnel protocol are significantly different in terms of
   the quantity of metadata necessary and the role of transit nodes.

   Work such as [VL2] (A Scalable and Flexible Data Center Network) and
   the NVO3 Data Plane Requirements
   [I-D.ietf-nvo3-dataplane-requirements] have described some of the
   properties that the data plane must have to support network
   virtualization.  However, one additional defining requirement is the
   need to carry metadata (e.g. system state) along with the packet
   data; example use cases of metadata are noted below.  The use of some
   metadata is certainly not a foreign concept - nearly all protocols
   used for network virtualization have at least 24 bits of identifier
   space as a way to partition between tenants.  This is often described
   as overcoming the limits of 12-bit VLANs, and when seen in that
   context, or any context where it is a true tenant identifier, 16
   million possible entries is a large number.  However, the reality is



Gross, et al.           Expires September 8, 2020               [Page 3]

Internet-Draft               Geneve Protocol                  March 2020


   that the metadata is not exclusively used to identify tenants and
   encoding other information quickly starts to crowd the space.  In
   fact, when compared to the tags used to exchange metadata between
   line cards on a chassis switch, 24-bit identifiers start to look
   quite small.  There are nearly endless uses for this metadata,
   ranging from storing input port identifiers for simple security
   policies to sending service based context for advanced middlebox
   applications that terminate and re-encapsulate Geneve traffic.

   Existing tunnel protocols have each attempted to solve different
   aspects of these new requirements, only to be quickly rendered out of
   date by changing control plane implementations and advancements.
   Furthermore, software and hardware components and controllers all
   have different advantages and rates of evolution - a fact that should
   be viewed as a benefit, not a liability or limitation.  This draft
   describes Geneve, a protocol which seeks to avoid these problems by
   providing a framework for tunneling for network virtualization rather
   than being prescriptive about the entire system.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

1.2.  Terminology

   The NVO3 Framework [RFC7365] defines many of the concepts commonly
   used in network virtualization.  In addition, the following terms are
   specifically meaningful in this document:

   Checksum offload.  An optimization implemented by many NICs (Network
   Interface Controller) which enables computation and verification of
   upper layer protocol checksums in hardware on transmit and receive,
   respectively.  This typically includes IP and TCP/UDP checksums which
   would otherwise be computed by the protocol stack in software.

   Clos network.  A technique for composing network fabrics larger than
   a single switch while maintaining non-blocking bandwidth across
   connection points.  ECMP is used to divide traffic across the
   multiple links and switches that constitute the fabric.  Sometimes
   termed "leaf and spine" or "fat tree" topologies.

   ECMP.  Equal Cost Multipath.  A routing mechanism for selecting from
   among multiple best next hop paths by hashing packet headers in order




Gross, et al.           Expires September 8, 2020               [Page 4]

Internet-Draft               Geneve Protocol                  March 2020


   to better utilize network bandwidth while avoiding reordering of
   packets within a flow.

   Geneve.  Generic Network Virtualization Encapsulation.  The tunnel
   protocol described in this document.

   LRO.  Large Receive Offload.  The receive-side equivalent function of
   LSO, in which multiple protocol segments (primarily TCP) are
   coalesced into larger data units.

   LSO.  Large Segmentation Offload.  A function provided by many
   commercial NICs that allows data units larger than the MTU to be
   passed to the NIC to improve performance, the NIC being responsible
   for creating smaller segments of size less than or equal to the MTU
   with correct protocol headers.  When referring specifically to TCP/
   IP, this feature is often known as TSO (TCP Segmentation Offload).

   Middlebox.  The term middlebox in the context of this document refers
   to network service functions or appliances for service interposition
   that would typically implement NVE functionality, which terminate or
   re-encapsulate Geneve traffic.

   NIC.  Network Interface Controller.  Also called as Network Interface
   Card or Network Adapter.  A NIC could be part of a tunnel endpoint or
   transit device and can either process Geneve packets or aid in the
   processing of Geneve packets.

   Transit device.  A forwarding element (e.g. router or switch) along
   the path of the tunnel making up part of the Underlay Network.  A
   transit device may be capable of understanding the Geneve packet
   format but does not originate or terminate Geneve packets.

   Tunnel endpoint.  A component performing encapsulation and
   decapsulation of packets, such as Ethernet frames or IP datagrams, in
   Geneve headers.  As the ultimate consumer of any tunnel metadata,
   tunnel endpoints have the highest level of requirements for parsing
   and interpreting tunnel headers.  Tunnel endpoints may consist of
   either software or hardware implementations or a combination of the
   two.  Tunnel endpoints are frequently a component of an NVE (Network
   Virtualization Edge) but may also be found in middleboxes or other
   elements making up an NVO3 Network.

   VM.  Virtual Machine.








Gross, et al.           Expires September 8, 2020               [Page 5]

Internet-Draft               Geneve Protocol                  March 2020


2.  Design Requirements

   Geneve is designed to support network virtualization use cases for
   data center environments, where tunnels are typically established to
   act as a backplane between the virtual switches residing in
   hypervisors, physical switches, or middleboxes or other appliances.
   An arbitrary IP network can be used as an underlay although Clos
   networks composed using ECMP links are a common choice to provide
   consistent bisectional bandwidth across all connection points.  Many
   of the concepts of network virtualization overlays over Layer 3 IP
   networks are described in the NVO3 Framework [RFC7365].  Figure 1
   shows an example of a hypervisor, top of rack switch for connectivity
   to physical servers, and a WAN uplink connected using Geneve tunnels
   over a simplified Clos network.  These tunnels are used to
   encapsulate and forward frames from the attached components such as
   VMs or physical links.

     +---------------------+           +-------+  +------+
     | +--+  +-------+---+ |           |Transit|--|Top of|==Physical
     | |VM|--|       |   | | +------+ /|Router |  | Rack |==Servers
     | +--+  |Virtual|NIC|---|Top of|/ +-------+\/+------+
     | +--+  |Switch |   | | | Rack |\ +-------+/\+------+
     | |VM|--|       |   | | +------+ \|Transit|  |Uplink|   WAN
     | +--+  +-------+---+ |           |Router |--|      |=========>
     +---------------------+           +-------+  +------+
            Hypervisor

                 ()===================================()
                         Switch-Switch Geneve Tunnels

                    Figure 1: Sample Geneve Deployment

   To support the needs of network virtualization, the tunnel protocol
   should be able to take advantage of the differing (and evolving)
   capabilities of each type of device in both the underlay and overlay
   networks.  This results in the following requirements being placed on
   the data plane tunneling protocol:

   o  The data plane is generic and extensible enough to support current
      and future control planes.

   o  Tunnel components are efficiently implementable in both hardware
      and software without restricting capabilities to the lowest common
      denominator.

   o  High performance over existing IP fabrics.





Gross, et al.           Expires September 8, 2020               [Page 6]

Internet-Draft               Geneve Protocol                  March 2020


   These requirements are described further in the following
   subsections.

2.1.  Control Plane Independence

   Although some protocols for network virtualization have included a
   control plane as part of the tunnel format specification (most
   notably, VXLAN [RFC7348] prescribed a multicast learning- based
   control plane), these specifications have largely been treated as
   describing only the data format.  The VXLAN packet format has
   actually seen a wide variety of control planes built on top of it.

   There is a clear advantage in settling on a data format: most of the
   protocols are only superficially different and there is little
   advantage in duplicating effort.  However, the same cannot be said of
   control planes, which are diverse in very fundamental ways.  The case
   for standardization is also less clear given the wide variety in
   requirements, goals, and deployment scenarios.

   As a result of this reality, Geneve is a pure tunnel format
   specification that is capable of fulfilling the needs of many control
   planes by explicitly not selecting any one of them.  This
   simultaneously promotes a shared data format and reduces the chance
   of obsolescence by future control plane enhancements.

2.2.  Data Plane Extensibility

   Achieving the level of flexibility needed to support current and
   future control planes effectively requires an options infrastructure
   to allow new metadata types to be defined, deployed, and either
   finalized or retired.  Options also allow for differentiation of
   products by encouraging independent development in each vendor's core
   specialty, leading to an overall faster pace of advancement.  By far
   the most common mechanism for implementing options is Type-Length-
   Value (TLV) format.

   It should be noted that while options can be used to support non-
   wirespeed control packets, they are equally important on data packets
   as well to segregate and direct forwarding (for instance, the
   examples given before of input port based security policies and
   terminating/re-encapsulating service interposition both require tags
   to be placed on data packets).  Therefore, while it would be
   desirable to limit the extensibility to only control packets for the
   purposes of simplifying the datapath, that would not satisfy the
   design requirements.






Gross, et al.           Expires September 8, 2020               [Page 7]

Internet-Draft               Geneve Protocol                  March 2020


2.2.1.  Efficient Implementation

   There is often a conflict between software flexibility and hardware
   performance that is difficult to resolve.  For a given set of
   functionality, it is obviously desirable to maximize performance.
   However, that does not mean new features that cannot be run at a
   desired speed today should be disallowed.  Therefore, for a protocol
   to be efficiently implementable means that a set of common
   capabilities can be reasonably handled across platforms along with a
   graceful mechanism to handle more advanced features in the
   appropriate situations.

   The use of a variable length header and options in a protocol often
   raises questions about whether it is truly efficiently implementable
   in hardware.  To answer this question in the context of Geneve, it is
   important to first divide "hardware" into two categories: tunnel
   endpoints and transit devices.

   Tunnel endpoints must be able to parse the variable header, including
   any options, and take action.  Since these devices are actively
   participating in the protocol, they are the most affected by Geneve.
   However, as tunnel endpoints are the ultimate consumers of the data,
   transmitters can tailor their output to the capabilities of the
   recipient.

   Transit devices may be able to interpret the options, however, as
   non-terminating devices, transit devices do not originate or
   terminate the Geneve packet, hence MUST NOT modify Geneve headers and
   MUST NOT insert or delete options, which is the responsibility of
   tunnel endpoints.  Options, if present in the packet, MUST only be
   generated and terminated by tunnel endpoints.  The participation of
   transit devices in interpreting options is OPTIONAL.

   Further, either tunnel endpoints or transit devices MAY use offload
   capabilities of NICs such as checksum offload to improve the
   performance of Geneve packet processing.  The presence of a Geneve
   variable length header should not prevent the tunnel endpoints and
   transit devices from using such offload capabilities.

2.3.  Use of Standard IP Fabrics

   IP has clearly cemented its place as the dominant transport mechanism
   and many techniques have evolved over time to make it robust,
   efficient, and inexpensive.  As a result, it is natural to use IP
   fabrics as a transit network for Geneve.  Fortunately, the use of IP
   encapsulation and addressing is enough to achieve the primary goal of
   delivering packets to the correct point in the network through
   standard switching and routing.



Gross, et al.           Expires September 8, 2020               [Page 8]

Internet-Draft               Geneve Protocol                  March 2020


   In addition, nearly all underlay fabrics are designed to exploit
   parallelism in traffic to spread load across multiple links without
   introducing reordering in individual flows.  These equal cost
   multipathing (ECMP) techniques typically involve parsing and hashing
   the addresses and port numbers from the packet to select an outgoing
   link.  However, the use of tunnels often results in poor ECMP
   performance without additional knowledge of the protocol as the
   encapsulated traffic is hidden from the fabric by design and only
   tunnel endpoint addresses are available for hashing.

   Since it is desirable for Geneve to perform well on these existing
   fabrics, it is necessary for entropy from encapsulated packets to be
   exposed in the tunnel header.  The most common technique for this is
   to use the UDP source port, which is discussed further in
   Section 3.3.

3.  Geneve Encapsulation Details

   The Geneve packet format consists of a compact tunnel header
   encapsulated in UDP over either IPv4 or IPv6.  A small fixed tunnel
   header provides control information plus a base level of
   functionality and interoperability with a focus on simplicity.  This
   header is then followed by a set of variable options to allow for
   future innovation.  Finally, the payload consists of a protocol data
   unit of the indicated type, such as an Ethernet frame.  Section 3.1
   and Section 3.2 illustrate the Geneve packet format transported (for
   example) over Ethernet along with an Ethernet payload.

3.1.  Geneve Packet Format Over IPv4

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   Outer Ethernet Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Outer Destination MAC Address                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Outer Destination MAC Address |   Outer Source MAC Address    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                   Outer Source MAC Address                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Optional Ethertype=C-Tag 802.1Q|  Outer VLAN Tag Information   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |       Ethertype=0x0800        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Outer IPv4 Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Version|  IHL  |Type of Service|          Total Length         |



Gross, et al.           Expires September 8, 2020               [Page 9]

Internet-Draft               Geneve Protocol                  March 2020


      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |         Identification        |Flags|      Fragment Offset    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |  Time to Live |Protocol=17 UDP|         Header Checksum       |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                     Outer Source IPv4 Address                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                   Outer Destination IPv4 Address              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Outer UDP Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |       Source Port = xxxx      |       Dest Port = 6081        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           UDP Length          |        UDP Checksum           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Geneve Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Ver|  Opt Len  |O|C|    Rsvd.  |          Protocol Type        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |        Virtual Network Identifier (VNI)       |    Reserved   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                    Variable Length Options                    |
      ~                                                               ~
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Inner Ethernet Header (example payload):
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Inner Destination MAC Address                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Inner Destination MAC Address |   Inner Source MAC Address    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                   Inner Source MAC Address                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Optional Ethertype=C-Tag 802.1Q|  Inner VLAN Tag Information   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Payload:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Ethertype of Original Payload |                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
      |                                  Original Ethernet Payload    |
      |                                                               |
      | (Note that the original Ethernet Frame's Preamble, Start Frame|
      |  Delimiter(SFD) & Frame Check Sequence(FCS) are not included  |
      |  and the Ethernet Payload need not be 4-byte aligned)         |



Gross, et al.           Expires September 8, 2020              [Page 10]

Internet-Draft               Geneve Protocol                  March 2020


      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Frame Check Sequence:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   New Frame Check Sequence (FCS) for Outer Ethernet Frame     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

3.2.  Geneve Packet Format Over IPv6

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   Outer Ethernet Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Outer Destination MAC Address                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Outer Destination MAC Address |   Outer Source MAC Address    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                   Outer Source MAC Address                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Optional Ethertype=C-Tag 802.1Q|  Outer VLAN Tag Information   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |       Ethertype=0x86DD        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Outer IPv6 Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Version| Traffic Class |           Flow Label                  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |         Payload Length        | NxtHdr=17 UDP |   Hop Limit   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +                     Outer Source IPv6 Address                 +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +                  Outer Destination IPv6 Address               +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Outer UDP Header:



Gross, et al.           Expires September 8, 2020              [Page 11]

Internet-Draft               Geneve Protocol                  March 2020


      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |       Source Port = xxxx      |       Dest Port = 6081        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           UDP Length          |        UDP Checksum           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Geneve Header:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Ver|  Opt Len  |O|C|    Rsvd.  |          Protocol Type        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |        Virtual Network Identifier (VNI)       |    Reserved   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                    Variable Length Options                    |
      ~                                                               ~
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Inner Ethernet Header (example payload):
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                 Inner Destination MAC Address                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Inner Destination MAC Address |   Inner Source MAC Address    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                   Inner Source MAC Address                    |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |Optional Ethertype=C-Tag 802.1Q|  Inner VLAN Tag Information   |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Payload:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      | Ethertype of Original Payload |                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
      |                                  Original Ethernet Payload    |
      |                                                               |
      | (Note that the original Ethernet Frame's Preamble, Start Frame|
      |  Delimiter(SFD) & Frame Check Sequence(FCS) are not included  |
      |  and the Ethernet Payload need not be 4-byte aligned)         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Frame Check Sequence:
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |   New Frame Check Sequence (FCS) for Outer Ethernet Frame     |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+








Gross, et al.           Expires September 8, 2020              [Page 12]

Internet-Draft               Geneve Protocol                  March 2020


3.3.  UDP Header

   The use of an encapsulating UDP [RFC0768] header follows the
   connectionless semantics of Ethernet and IP in addition to providing
   entropy to routers performing ECMP.  The header fields are therefore
   interpreted as follows:

   Source port:  A source port selected by the originating tunnel
      endpoint.  This source port SHOULD be the same for all packets
      belonging to a single encapsulated flow to prevent reordering due
      to the use of different paths.  To encourage an even distribution
      of flows across multiple links, the source port SHOULD be
      calculated using a hash of the encapsulated packet headers using,
      for example, a traditional 5-tuple.  Since the port represents a
      flow identifier rather than a true UDP connection, the entire
      16-bit range MAY be used to maximize entropy.  In addition to
      setting the source port, for IPv6, flow label MAY also be used for
      providing entropy.  For an example of using IPv6 flow label for
      tunnel use cases, see [RFC6438].

      If Geneve traffic is shared with other UDP listeners on the same
      IP address, tunnel endpoints SHOULD implement a mechanism to
      ensure ICMP return traffic arising from network errors is directed
      to the correct listener.  The definition of such a mechanism is
      beyond the scope of this document.

   Dest port:  IANA has assigned port 6081 as the fixed well-known
      destination port for Geneve.  Although the well-known value should
      be used by default, it is RECOMMENDED that implementations make
      this configurable.  The chosen port is used for identification of
      Geneve packets and MUST NOT be reversed for different ends of a
      connection as is done with TCP.  It is the responsibility of the
      control plane for any reconfiguration of the assigned port and its
      interpretation by respective devices.  The definition of the
      control plane is beyond the scope of this document.

   UDP length:  The length of the UDP packet including the UDP header.


   UDP checksum:  In order to protect the Geneve header, options and
      payload from potential data corruption, UDP checksum SHOULD be
      generated as specified in [RFC0768] and [RFC1112] when Geneve is
      encapsulated in IPv4.  To protect the IP header, Geneve header,
      options and payload from potential data corruption, the UDP
      checksum MUST be generated by default as specified in [RFC0768]
      and [RFC8200] when Geneve is encapsulated in IPv6, except for
      certain conditions, which are outlined in the next paragraph.
      Upon receiving such packets with non-zero UDP checksum, the



Gross, et al.           Expires September 8, 2020              [Page 13]

Internet-Draft               Geneve Protocol                  March 2020


      receiving tunnel endpoints MUST validate the checksum.  If the
      checksum is not correct, the packet MUST be dropped, otherwise the
      packet MUST be accepted for decapsulation.

      Under certain conditions, the UDP checksum MAY be set to zero on
      transmit for packets encapsulated in both IPv4 and IPv6 [RFC8200].
      See Section 4.3 for additional requirements that apply when using
      zero UDP checksum with IPv4 and IPv6.  Disabling the use of UDP
      checksums is an operational consideration that should take into
      account the risks and effects of packet corruption.

3.4.  Tunnel Header Fields

   Ver (2 bits):  The current version number is 0.  Packets received by
      a tunnel endpoint with an unknown version MUST be dropped.
      Transit devices interpreting Geneve packets with an unknown
      version number MUST treat them as UDP packets with an unknown
      payload.

   Opt Len (6 bits):  The length of the options fields, expressed in
      four byte multiples, not including the eight byte fixed tunnel
      header.  This results in a minimum total Geneve header size of 8
      bytes and a maximum of 260 bytes.  The start of the payload
      headers can be found using this offset from the end of the base
      Geneve header.

      Transit devices MUST maintain consistent forwarding behavior
      irrespective of the value of 'Opt Len', including ECMP link
      selection.

   O (1 bit):  Control packet.  This packet contains a control message.
      Control messages are sent between tunnel endpoints.  Tunnel
      endpoints MUST NOT forward the payload and transit devices MUST
      NOT attempt to interpret it.  Since control messages are less
      frequent, it is RECOMMENDED that tunnel endpoints direct these
      packets to a high priority control queue (for example, to direct
      the packet to a general purpose CPU from a forwarding ASIC or to
      separate out control traffic on a NIC).  Transit devices MUST NOT
      alter forwarding behavior on the basis of this bit, such as ECMP
      link selection.

   C (1 bit):  Critical options present.  One or more options has the
      critical bit set (see Section 3.5).  If this bit is set then
      tunnel endpoints MUST parse the options list to interpret any
      critical options.  On tunnel endpoints where option parsing is not
      supported the packet MUST be dropped on the basis of the 'C' bit
      in the base header.  If the bit is not set tunnel endpoints MAY
      strip all options using 'Opt Len' and forward the decapsulated



Gross, et al.           Expires September 8, 2020              [Page 14]

Internet-Draft               Geneve Protocol                  March 2020


      packet.  Transit devices MUST NOT drop packets on the basis of
      this bit.

   Rsvd. (6 bits):  Reserved field, which MUST be zero on transmission
      and MUST be ignored on receipt.

   Protocol Type (16 bits):  The type of the protocol data unit
      appearing after the Geneve header.  This follows the EtherType
      [ETYPES] convention; with Ethernet itself being represented by the
      value 0x6558.

   Virtual Network Identifier (VNI) (24 bits):  An identifier for a
      unique element of a virtual network.  In many situations this may
      represent an L2 segment, however, the control plane defines the
      forwarding semantics of decapsulated packets.  The VNI MAY be used
      as part of ECMP forwarding decisions or MAY be used as a mechanism
      to distinguish between overlapping address spaces contained in the
      encapsulated packet when load balancing across CPUs.

   Reserved (8 bits):  Reserved field which MUST be zero on transmission
      and ignored on receipt.

3.5.  Tunnel Options

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |          Option Class         |      Type     |R|R|R| Length  |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                      Variable Option Data                     |
      ~                                                               ~
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                               Geneve Option

   The base Geneve header is followed by zero or more options in Type-
   Length-Value format.  Each option consists of a four byte option
   header and a variable amount of option data interpreted according to
   the type.

   Option Class (16 bits):  Namespace for the 'Type' field.  IANA will
      be requested to create a "Geneve Option Class" registry to
      allocate identifiers for organizations, technologies, and vendors
      that have an interest in creating types for options.  Each
      organization may allocate types independently to allow
      experimentation and rapid innovation.  It is expected that over
      time certain options will become well known and a given



Gross, et al.           Expires September 8, 2020              [Page 15]

Internet-Draft               Geneve Protocol                  March 2020


      implementation may use option types from a variety of sources.  In
      addition, IANA will be requested to reserve specific ranges for
      allocation by IETF Review and for Experimental Use (see
      Section 7).

   Type (8 bits):  Type indicating the format of the data contained in
      this option.  Options are primarily designed to encourage future
      extensibility and innovation and so standardized forms of these
      options will be defined in separate documents.

      The high order bit of the option type indicates that this is a
      critical option.  If the receiving tunnel endpoint does not
      recognize this option and this bit is set then the packet MUST be
      dropped.  If this bit is set in any option then the 'C' bit in the
      Geneve base header MUST also be set.  Transit devices MUST NOT
      drop packets on the basis of this bit.  The following figure shows
      the location of the 'C' bit in the 'Type' field:

      0 1 2 3 4 5 6 7 8
      +-+-+-+-+-+-+-+-+
      |C|    Type     |
      +-+-+-+-+-+-+-+-+

      The requirement to drop a packet with an unknown option with the
      'C' bit set applies to the entire tunnel endpoint system and not a
      particular component of the implementation.  For example, in a
      system comprised of a forwarding ASIC and a general purpose CPU,
      this does not mean that the packet must be dropped in the ASIC.
      An implementation may send the packet to the CPU using a rate-
      limited control channel for slow-path exception handling.

   R (3 bits):  Option control flags reserved for future use.  These
      bits MUST be zero on transmission and MUST be ignored on receipt.

   Length (5 bits):  Length of the option, expressed in four byte
      multiples excluding the option header.  The total length of each
      option may be between 4 and 128 bytes.  A value of 0 in the Length
      field implies an option with only an option header and no variable
      option data.  Packets in which the total length of all options is
      not equal to the 'Opt Len' in the base header are invalid and MUST
      be silently dropped if received by a tunnel endpoint that
      processes the options.

   Variable Option Data:  Option data interpreted according to 'Type'.







Gross, et al.           Expires September 8, 2020              [Page 16]

Internet-Draft               Geneve Protocol                  March 2020


3.5.1.  Options Processing

   Geneve options are intended to be originated and processed by tunnel
   endpoints.  However, options MAY be interpreted by transit devices
   along the tunnel path.  Transit devices not interpreting Geneve
   headers (which may or may not include options) MUST handle Geneve
   packets as any other UDP packet and maintain consistent forwarding
   behavior.

   In tunnel endpoints, the generation and interpretation of options is
   determined by the control plane, which is beyond the the scope of
   this document.  However, to ensure interoperability between
   heterogeneous devices some requirements are imposed on options and
   the devices that process them:

   o  Receiving tunnel endpoints MUST drop packets containing unknown
      options with the 'C' bit set in the option type.  Conversely,
      transit devices MUST NOT drop packets as a result of encountering
      unknown options, including those with the 'C' bit set.

   o  The contents of the options and their ordering MUST NOT be
      modified by transit devices.

   o  If a tunnel endpoint receives a Geneve packet with 'Opt Len'
      (total length of all options) that exceeds the options processing
      capability of the tunnel endpoint then the tunnel endpoint MUST
      drop such packets.  An implementation may raise an exception to
      the control plane of such an event.  It is the responsibility of
      the control plane to ensure the communicating peer tunnel
      endpoints have the processing capability to handle the total
      length of options.  The definition of the control plane is beyond
      the scope of this document.

   When designing a Geneve option, it is important to consider how the
   option will evolve in the future.  Once an option is defined it is
   reasonable to expect that implementations may come to depend on a
   specific behavior.  As a result, the scope of any future changes must
   be carefully described upfront.

   Architecturally, options are intended to be self-descriptive and
   independent.  This enables parallelism in option processing and
   reduces implementation complexity.  However, the control plane may
   impose certain ordering restrictions as described in Section 4.5.1.

   Unexpectedly significant interoperability issues may result from
   changing the length of an option that was defined to be a certain
   size.  A particular option is specified to have either a fixed
   length, which is constant, or a variable length, which may change



Gross, et al.           Expires September 8, 2020              [Page 17]

Internet-Draft               Geneve Protocol                  March 2020


   over time or for different use cases.  This property is part of the
   definition of the option and conveyed by the 'Type'.  For fixed
   length options, some implementations may choose to ignore the length
   field in the option header and instead parse based on the well known
   length associated with the type.  In this case, redefining the length
   will impact not only parsing of the option in question but also any
   options that follow.  Therefore, options that are defined to be fixed
   length in size MUST NOT be redefined to a different length.  Instead,
   a new 'Type' should be allocated.  Actual definition of the option
   type is beyond the scope of this document.  The option type and its
   interpretation should be defined by the entity that owns the option
   class.

   Options may be processed by NIC hardware utilizing offloads (e.g.
   LSO and LRO) as described in Section 4.6.  Careful consideration
   should be given to how the offload capabilities outlined in
   Section 4.6 impact an option's design.

4.  Implementation and Deployment Considerations

4.1.  Applicability Statement

   Geneve is a network virtualization overlay encapsulation protocol
   designed to establish tunnels between NVEs over an existing IP
   network.  It is intended for use in public or private data center
   environments, for deploying multi-tenant overlay networks over an
   existing IP underlay network.

   Geneve is a UDP based encapsulation protocol transported over
   existing IPv4 and IPv6 networks.  Hence, as a UDP based protocol,
   Geneve adheres to the UDP usage guidelines as specified in [RFC8085].
   The applicability of these guidelines are dependent on the underlay
   IP network and the nature of Geneve payload protocol (example TCP/IP,
   IP/Ethernet).

   Geneve is intended to be deployed in a data center network
   environment operated by a single operator or adjacent set of
   cooperating network operators that fits with the definition of
   controlled environments in [RFC8085].  A network in a controlled
   environment can be managed to operate under certain conditions
   whereas in the general Internet this cannot be done.  Hence
   requirements for a tunnel protocol operating under a controlled
   environment can be less restrictive than the requirements of the
   general Internet.

   For the purpose of this document, a traffic-managed controlled
   environment (TMCE) is defined as an IP network that is traffic-
   engineered and/or otherwise managed (e.g., via use of traffic rate



Gross, et al.           Expires September 8, 2020              [Page 18]

Internet-Draft               Geneve Protocol                  March 2020


   limiters) to avoid congestion.  The concept of TMCE is outlined in
   [RFC8086].  Significant portions of the text in Section 4.1 through
   Section 4.3 are based on [RFC8086] as applicable to Geneve.

   It is the responsibility of the operator to ensure that the
   guidelines/requirements in this section are followed as applicable to
   their Geneve deployment(s).

4.2.  Congestion Control Functionality

   Geneve does not natively provide congestion control functionality and
   relies on the payload protocol traffic for congestion control.  As
   such Geneve MUST be used with congestion controlled traffic or within
   a network that is traffic managed to avoid congestion (TMCE).  An
   operator of a traffic managed network (TMCE) may avoid congestion by
   careful provisioning of their networks, rate-limiting of user data
   traffic and traffic engineering according to path capacity.

4.3.  UDP Checksum

   In order to provide integrity of Geneve headers, options and payload,
   (for example to avoid misdelivery of payload to different tenant
   systems) in case of data corruption, the outer UDP checksum SHOULD be
   used with Geneve when transported over IPv4.  The UDP checksum
   provides a statistical guarantee that a payload was not corrupted in
   transit.  These integrity checks are not strong from a coding or
   cryptographic perspective and are not designed to detect physical-
   layer errors or malicious modification of the datagram (see
   Section 3.4 of [RFC8085]).  In deployments where such a risk exists,
   an operator SHOULD use additional data integrity mechanisms such as
   offered by IPsec (see Section 6.2).

   An operator MAY choose to disable UDP checksums and use zero
   checksums if Geneve packet integrity is provided by other data
   integrity mechanisms such as IPsec or additional checksums or if one
   of the conditions in Section 4.3.1 a, b, c are met.

   By default, UDP checksums MUST be used when Geneve is transported
   over IPv6.  A tunnel endpoint MAY be configured for use with zero UDP
   checksum if additional requirements in Section 4.3.1 are met.

4.3.1.  UDP Zero Checksum Handling with IPv6

   When Geneve is used over IPv6, the UDP checksum is used to protect
   IPv6 headers, UDP headers and Geneve headers, options and payload
   from potential data corruption.  As such by default Geneve MUST use
   UDP checksums when transported over IPv6.  An operator MAY choose to
   configure to operate with zero UDP checksum if operating in a traffic



Gross, et al.           Expires September 8, 2020              [Page 19]

Internet-Draft               Geneve Protocol                  March 2020


   managed controlled environment as stated in Section 4.1 if one of the
   following conditions are met.

   a.  It is known that the packet corruption is exceptionally unlikely
       (perhaps based on knowledge of equipment types in their underlay
       network) and the operator is willing to take a risk of undetected
       packet corruption

   b.  It is judged through observational measurements (perhaps through
       historic or current traffic flows that use non zero checksum)
       that the level of packet corruption is tolerably low and where
       the operator is willing to take the risk of undetected
       corruption.

   c.  Geneve payload is carrying applications that are tolerant of
       misdelivered or corrupted packets (perhaps through higher layer
       checksum validation and/or reliability through retransmission)

   In addition Geneve tunnel implementations using zero UDP checksum
   MUST meet the following requirements:

   1.  Use of UDP checksum over IPv6 MUST be the default configuration
       for all Geneve tunnels.

   2.  If Geneve is used with zero UDP checksum over IPv6 then such
       tunnel endpoint implementation MUST meet all the requirements
       specified in Section 4 of [RFC6936] and requirement 1 as
       specified in Section 5 of [RFC6936] as that is relevant to
       Geneve.

   3.  The Geneve tunnel endpoint that decapsulates the tunnel SHOULD
       check the source and destination IPv6 addresses are valid for the
       Geneve tunnel that is configured to receive zero UDP checksum and
       discard other packets for which such check fails.

   4.  The Geneve tunnel endpoint that encapsulates the tunnel MAY use
       different IPv6 source addresses for each Geneve tunnel that uses
       zero UDP checksum mode in order to strengthen the decapsulator's
       check of the IPv6 source address (i.e the same IPv6 source
       address is not to be used with more than one IPv6 destination
       address, irrespective of whether that destination address is a
       unicast or multicast address).  When this is not possible, it is
       RECOMMENDED to use each source address for as few Geneve tunnels
       that use zero UDP checksum as is feasible.

       Note that (for requirements 3 and 4) the receiving tunnel
       endpoint can apply these checks only if it has out-of-band
       knowledge that the encapsulating tunnel endpoint is applying the



Gross, et al.           Expires September 8, 2020              [Page 20]

Internet-Draft               Geneve Protocol                  March 2020


       indicated behavior.  One possibility to obtain this out-of-band
       knowledge is through signaling by the control plane.  The
       definition of the control plane is beyond the scope of this
       document.

   5.  Measures SHOULD be taken to prevent Geneve traffic over IPv6 with
       zero UDP checksum from escaping into the general Internet.
       Examples of such measures include employing packet filters at the
       gateways or edge of Geneve network and/or keeping logical or
       physical separation of the Geneve network from networks carrying
       the general Internet traffic.

   The above requirements do not change either the requirements
   specified in [RFC8200] or the requirements specified in [RFC6936].

   The use of the source IPv6 address in addition to the destination
   IPv6 address, plus the recommendation against reuse of source IPv6
   addresses among Geneve tunnels collectively provide some mitigation
   for the absence of UDP checksum coverage of the IPv6 header.  A
   traffic-managed controlled environment that satisfies at least one of
   three conditions listed at the beginning of this section provides
   additional assurance.

   Editorial Note (The following paragraph to be removed by the RFC
   Editor before publication)

   It was discussed during TSVART early review if the level of
   requirement for using different IPv6 source addresses for different
   tunnel destinations would need to be "MAY" or "SHOULD".  The
   discussion concluded that it was appropriate to keep this as "MAY",
   since it was considered not realistic for control planes having to
   maintain a high level of state on a per tunnel destination basis.  In
   addition, the text above provides sufficient guidance to operators
   and implementors on possible mitigations.

4.4.  Encapsulation of Geneve in IP

   As an IP-based tunnel protocol, Geneve shares many properties and
   techniques with existing protocols.  The application of some of these
   are described in further detail, although in general most concepts
   applicable to the IP layer or to IP tunnels generally also function
   in the context of Geneve.

4.4.1.  IP Fragmentation

   It is strongly RECOMMENDED that Path MTU Discovery ([RFC1191],
   [RFC8201]) be used to prevent or minimize fragmentation.  The use of
   Path MTU Discovery on the transit network provides the encapsulating



Gross, et al.           Expires September 8, 2020              [Page 21]

Internet-Draft               Geneve Protocol                  March 2020


   tunnel endpoint with soft-state about the link that it may use to
   prevent or minimize fragmentation depending on its role in the
   virtualized network.  The NVE can maintain this state (the MTU size
   of the tunnel link(s) associated with the tunnel endpoint), so if a
   tenant system sends large packets that when encapsulated exceed the
   MTU size of the tunnel link, the tunnel endpoint can discard such
   packets and send exception messages to the tenant system(s).  If the
   tunnel endpoint is associated with a routing or forwarding function
   and/or has the capability to send ICMP messages, the encapsulating
   tunnel endpoint MAY send ICMP fragmentation needed [RFC0792] or
   Packet Too Big [RFC4443] messages to the tenant system(s).  When
   determining the MTU size of a tunnel link, maximum length of options
   MUST be assumed as options may vary on a per-packet basis.  For
   example, recommendations/guidance for handling fragmentation in
   similar overlay encapsulation services like PWE3 are provided in
   Section 5.3 of [RFC3985].

   Note that some implementations may not be capable of supporting
   fragmentation or other less common features of the IP header, such as
   options and extension headers.  For example, some of the issues
   associated with MTU size and fragmentation in IP tunneling and use of
   ICMP messages is outlined in Section 4.2 of
   [I-D.ietf-intarea-tunnels].

4.4.2.  DSCP, ECN and TTL

   When encapsulating IP (including over Ethernet) packets in Geneve,
   there are several considerations for propagating DSCP and ECN bits
   from the inner header to the tunnel on transmission and the reverse
   on reception.

   [RFC2983] provides guidance for mapping DSCP between inner and outer
   IP headers.  Network virtualization is typically more closely aligned
   with the Pipe model described, where the DSCP value on the tunnel
   header is set based on a policy (which may be a fixed value, one
   based on the inner traffic class, or some other mechanism for
   grouping traffic).  Aspects of the Uniform model (which treats the
   inner and outer DSCP value as a single field by copying on ingress
   and egress) may also apply, such as the ability to remark the inner
   header on tunnel egress based on transit marking.  However, the
   Uniform model is not conceptually consistent with network
   virtualization, which seeks to provide strong isolation between
   encapsulated traffic and the physical network.

   [RFC6040] describes the mechanism for exposing ECN capabilities on IP
   tunnels and propagating congestion markers to the inner packets.
   This behavior MUST be followed for IP packets encapsulated in Geneve.




Gross, et al.           Expires September 8, 2020              [Page 22]

Internet-Draft               Geneve Protocol                  March 2020


   Though Uniform or Pipe models could be used for TTL (or Hop Limit in
   case of IPv6) handling when tunneling IP packets, the Pipe model is
   more aligned with network virtualization.  [RFC2003] provides
   guidance on handling TTL between inner IP header and outer IP
   tunnels; this model is more aligned with the Pipe model and is
   RECOMMENDED for use with Geneve for network virtualization
   applications.

4.4.3.  Broadcast and Multicast

   Geneve tunnels may either be point-to-point unicast between two
   tunnel endpoints or may utilize broadcast or multicast addressing.
   It is not required that inner and outer addressing match in this
   respect.  For example, in physical networks that do not support
   multicast, encapsulated multicast traffic may be replicated into
   multiple unicast tunnels or forwarded by policy to a unicast location
   (possibly to be replicated there).

   With physical networks that do support multicast it may be desirable
   to use this capability to take advantage of hardware replication for
   encapsulated packets.  In this case, multicast addresses may be
   allocated in the physical network corresponding to tenants,
   encapsulated multicast groups, or some other factor.  The allocation
   of these groups is a component of the control plane and therefore is
   beyond the scope of this document.

   When physical multicast is in use, devices with heterogeneous
   capabilities may be present in the same group.  Some options may only
   be interpretable by a subset of the devices in the group.  Other
   devices can safely ignore such options unless the 'C' bit is set to
   mark the unknown option as critical.  Requirements outlined in
   Section 3.4 apply for critical options.

   In addition, [RFC8293] provides examples of various mechanisms that
   can be used for multicast handling in network virtualization overlay
   networks.

4.4.4.  Unidirectional Tunnels

   Generally speaking, a Geneve tunnel is a unidirectional concept.  IP
   is not a connection oriented protocol and it is possible for two
   tunnel endpoints to communicate with each other using different paths
   or to have one side not transmit anything at all.  As Geneve is an
   IP-based protocol, the tunnel layer inherits these same
   characteristics.

   It is possible for a tunnel to encapsulate a protocol, such as TCP,
   which is connection oriented and maintains session state at that



Gross, et al.           Expires September 8, 2020              [Page 23]

Internet-Draft               Geneve Protocol                  March 2020


   layer.  In addition, implementations MAY model Geneve tunnels as
   connected, bidirectional links, such as to provide the abstraction of
   a virtual port.  In both of these cases, bidirectionality of the
   tunnel is handled at a higher layer and does not affect the operation
   of Geneve itself.

4.5.  Constraints on Protocol Features

   Geneve is intended to be flexible to a wide range of current and
   future applications.  As a result, certain constraints may be placed
   on the use of metadata or other aspects of the protocol in order to
   optimize for a particular use case.  For example, some applications
   may limit the types of options which are supported or enforce a
   maximum number or length of options.  Other applications may only
   handle certain encapsulated payload types, such as Ethernet or IP.
   This could be either globally throughout the system or, for example,
   restricted to certain classes of devices or network paths.

   These constraints may be communicated to tunnel endpoints either
   explicitly through a control plane or implicitly by the nature of the
   application.  As Geneve is defined as a data plane protocol that is
   control plane agnostic, definition of such mechanisms are beyond the
   scope of this document.

4.5.1.  Constraints on Options

   While Geneve options are flexible, a control plane may restrict the
   number of option TLVs as well as the order and size of the TLVs
   between tunnel endpoints to make it simpler for a data plane
   implementation in software or hardware to handle
   [I-D.ietf-nvo3-encap].  For example, there may be some critical
   information such as a secure hash that must be processed in a certain
   order to provide lowest latency or there may be other scenarios where
   the options must be processed in a certain order due to protocol
   semantics.

   A control plane may negotiate a subset of option TLVs and certain TLV
   ordering, as well may limit the total number of option TLVs present
   in the packet, for example, to accommodate hardware capable of
   processing fewer options [I-D.ietf-nvo3-encap].  Hence, a control
   plane needs to have the ability to describe the supported TLVs subset
   and their order to the tunnel endpoints.  In the absence of a control
   plane, alternative configuration mechanisms may be used for this
   purpose.  Such mechanisms are beyond the scope of this document.







Gross, et al.           Expires September 8, 2020              [Page 24]

Internet-Draft               Geneve Protocol                  March 2020


4.6.  NIC Offloads

   Modern NICs currently provide a variety of offloads to enable the
   efficient processing of packets.  The implementation of many of these
   offloads requires only that the encapsulated packet be easily parsed
   (for example, checksum offload).  However, optimizations such as LSO
   and LRO involve some processing of the options themselves since they
   must be replicated/merged across multiple packets.  In these
   situations, it is desirable to not require changes to the offload
   logic to handle the introduction of new options.  To enable this,
   some constraints are placed on the definitions of options to allow
   for simple processing rules:

   o  When performing LSO, a NIC MUST replicate the entire Geneve header
      and all options, including those unknown to the device, onto each
      resulting segment unless an option allows an exception.
      Conversely, when performing LRO, a NIC may assume that a binary
      comparison of the options (including unknown options) is
      sufficient to ensure equality and MAY merge packets with equal
      Geneve headers.

   o  Options MUST NOT be reordered during the course of offload
      processing, including when merging packets for the purpose of LRO.

   o  NICs performing offloads MUST NOT drop packets with unknown
      options, including those marked as critical, unless explicitly
      configured.

   There is no requirement that a given implementation of Geneve employ
   the offloads listed as examples above.  However, as these offloads
   are currently widely deployed in commercially available NICs, the
   rules described here are intended to enable efficient handling of
   current and future options across a variety of devices.

4.7.  Inner VLAN Handling

   Geneve is capable of encapsulating a wide range of protocols and
   therefore a given implementation is likely to support only a small
   subset of the possibilities.  However, as Ethernet is expected to be
   widely deployed, it is useful to describe the behavior of VLANs
   inside encapsulated Ethernet frames.

   As with any protocol, support for inner VLAN headers is OPTIONAL.  In
   many cases, the use of encapsulated VLANs may be disallowed due to
   security or implementation considerations.  However, in other cases
   trunking of VLAN frames across a Geneve tunnel can prove useful.  As
   a result, the processing of inner VLAN tags upon ingress or egress
   from a tunnel endpoint is based upon the configuration of the tunnel



Gross, et al.           Expires September 8, 2020              [Page 25]

Internet-Draft               Geneve Protocol                  March 2020


   endpoint and/or control plane and not explicitly defined as part of
   the data format.

5.  Transition Considerations

   Viewed exclusively from the data plane, Geneve is compatible with
   existing IP networks as it appears to most devices as UDP packets.
   However, as there are already a number of tunnel protocols deployed
   in network virtualization environments, there is a practical question
   of transition and coexistence.

   Since Geneve builds on the base data plane functionality provided by
   the most common protocols used for network virtualization (VXLAN,
   NVGRE) it should be straightforward to port an existing control plane
   to run on top of it with minimal effort.  With both the old and new
   packet formats supporting the same set of capabilities, there is no
   need for a hard transition - tunnel endpoints directly communicating
   with each other can use any common protocol, which may be different
   even within a single overall system.  As transit devices are
   primarily forwarding packets on the basis of the IP header, all
   protocols appear similar and these devices do not introduce
   additional interoperability concerns.

   To assist with this transition, it is strongly suggested that
   implementations support simultaneous operation of both Geneve and
   existing tunnel protocols as it is expected to be common for a single
   node to communicate with a mixture of other nodes.  Eventually, older
   protocols may be phased out as they are no longer in use.

6.  Security Considerations

   As encapsulated within a UDP/IP packet, Geneve does not have any
   inherent security mechanisms.  As a result, an attacker with access
   to the underlay network transporting the IP packets has the ability
   to snoop, alter or inject packets.  Compromised tunnel endpoints or
   transit devices may also spoof identifiers in the tunnel header to
   gain access to networks owned by other tenants.

   Within a particular security domain, such as a data center operated
   by a single service provider, the most common and highest performing
   security mechanism is isolation of trusted components.  Tunnel
   traffic can be carried over a separate VLAN and filtered at any
   untrusted boundaries.

   When crossing an untrusted link, such as the general Internet, VPN
   technologies such as IPsec [RFC4301] should be used to provide
   authentication and/or encryption of the IP packets formed as part of
   Geneve encapsulation (See Section 6.1.1).



Gross, et al.           Expires September 8, 2020              [Page 26]

Internet-Draft               Geneve Protocol                  March 2020


   Geneve does not otherwise affect the security of the encapsulated
   packets.  As per the guidelines of BCP 72 [RFC3552], the following
   sections describe potential security risks that may be applicable to
   Geneve deployments and approaches to mitigate such risks.  It is also
   noted that not all such risks are applicable to all Geneve deployment
   scenarios, i.e., only a subset may be applicable to certain
   deployments.  So an operator has to make an assessment based on their
   network environment and determine the risks that are applicable to
   their specific environment and use appropriate mitigation approaches
   as applicable.

6.1.  Data Confidentiality

   Geneve is a network virtualization overlay encapsulation protocol
   designed to establish tunnels between NVEs over an existing IP
   network.  It can be used to deploy multi-tenant overlay networks over
   an existing IP underlay network in a public or private data center.
   The overlay service is typically provided by a service provider, for
   example a cloud services provider or a private data center operator,
   this may or not may be the same provider as an underlay service
   provider.  Due to the nature of multi-tenancy in such environments, a
   tenant system may expect data confidentiality to ensure its packet
   data is not tampered with (active attack) in transit or a target of
   unauthorized monitoring (passive attack) for example by other tenant
   systems or underlay service provider.  A compromised network node or
   a transit device within a data center may passively monitor Geneve
   packet data between NVEs; or route traffic for further inspection.  A
   tenant may expect the overlay service provider to provide data
   confidentiality as part of the service or a tenant may bring its own
   data confidentiality mechanisms like IPsec or TLS to protect the data
   end to end between its tenant systems.  The overlay provider is
   expected to provide cryptographic protection in cases where the
   underlay provider is not the same as the overlay provider to ensure
   the payload is not exposed to the underlay.

   If an operator determines data confidentiality is necessary in their
   environment based on their risk analysis, for example as in multi-
   tenant environments, then an encryption mechanism SHOULD be used to
   encrypt the tenant data end to end between the NVEs.  The NVEs may
   use existing well established encryption mechanisms such as IPsec,
   DTLS, etc.

6.1.1.  Inter-Data Center Traffic

   A tenant system in a customer premises (private data center) may want
   to connect to tenant systems on their tenant overlay network in a
   public cloud data center or a tenant may want to have its tenant
   systems located in multiple geographically separated data centers for



Gross, et al.           Expires September 8, 2020              [Page 27]

Internet-Draft               Geneve Protocol                  March 2020


   high availability.  Geneve data traffic between tenant systems across
   such separated networks should be protected from threats when
   traversing public networks.  Any Geneve overlay data leaving the data
   center network beyond the operator's security domain SHOULD be
   secured by encryption mechanisms such as IPsec or other VPN
   technologies to protect the communications between the NVEs when they
   are geographically separated over untrusted network links.
   Specification of data protection mechanisms employed between data
   centers is beyond the scope of this document.

   The principles described in Section 4 regarding controlled
   environments still apply to the geographically separated data center
   usage outlined in this section.

6.2.  Data Integrity

   Geneve encapsulation is used between NVEs to establish overlay
   tunnels over an existing IP underlay network.  In a multi-tenant data
   center, a rogue or compromised tenant system may try to launch a
   passive attack such as monitoring the traffic of other tenants, or an
   active attack such as trying to inject unauthorized Geneve
   encapsulated traffic such as spoofing, replay, etc., into the
   network.  To prevent such attacks, an NVE MUST NOT propagate Geneve
   packets beyond the NVE to tenant systems and SHOULD employ packet
   filtering mechanisms so as not to forward unauthorized traffic
   between tenant systems in different tenant networks.  An NVE MUST NOT
   interpret Geneve packets from tenant systems other than as frames to
   be encapsulated.

   A compromised network node or a transit device within a data center
   may launch an active attack trying to tamper with the Geneve packet
   data between NVEs.  Malicious tampering of Geneve header fields may
   cause the packet from one tenant to be forwarded to a different
   tenant network.  If an operator determines the possibility of such
   threat in their environment, the operator may choose to employ data
   integrity mechanisms between NVEs.  In order to prevent such risks, a
   data integrity mechanism SHOULD be used in such environments to
   protect the integrity of Geneve packets including packet headers,
   options and payload on communications between NVE pairs.  A
   cryptographic data protection mechanism such as IPsec may be used to
   provide data integrity protection.  A data center operator may choose
   to deploy any other data integrity mechanisms as applicable and
   supported in their underlay networks, although non-cryptographic
   mechanisms may not protect the Geneve portion of the packet from
   tampering.






Gross, et al.           Expires September 8, 2020              [Page 28]

Internet-Draft               Geneve Protocol                  March 2020


6.3.  Authentication of NVE peers

   A rogue network device or a compromised NVE in a data center
   environment might be able to spoof Geneve packets as if it came from
   a legitimate NVE.  In order to mitigate such a risk, an operator
   SHOULD use an authentication mechanism, such as IPsec to ensure that
   the Geneve packet originated from the intended NVE peer, in
   environments where the operator determines spoofing or rogue devices
   is a potential threat.  Other simpler source checks such as ingress
   filtering for VLAN/MAC/IP address, reverse path forwarding checks,
   etc., may be used in certain trusted environments to ensure Geneve
   packets originated from the intended NVE peer.

6.4.  Options Interpretation by Transit Devices

   Options, if present in the packet, are generated and terminated by
   tunnel endpoints.  As indicated in Section 2.2.1, transit devices may
   interpret the options.  However, if the packet is protected by tunnel
   endpoint to tunnel endpoint encryption, for example through IPsec,
   transit devices will not have visibility into the Geneve header or
   options in the packet.  In such cases transit devices MUST handle
   Geneve packets as any other IP packet and maintain consistent
   forwarding behavior.  In cases where options are interpreted by
   transit devices, the operator MUST ensure that transit devices are
   trusted and not compromised.  The definition of a mechanism to ensure
   this trust is beyond the scope of this document.

6.5.  Multicast/Broadcast

   In typical data center networks where IP multicasting is not
   supported in the underlay network, multicasting may be supported
   using multiple unicast tunnels.  The same security requirements as
   described in the above sections can be used to protect Geneve
   communications between NVE peers.  If IP multicasting is supported in
   the underlay network and the operator chooses to use it for multicast
   traffic among tunnel endpoints, then the operator in such
   environments may use data protection mechanisms such as IPsec with
   multicast extensions [RFC5374] to protect multicast traffic among
   Geneve NVE groups.

6.6.  Control Plane Communications

   A Network Virtualization Authority (NVA) as outlined in [RFC8014] may
   be used as a control plane for configuring and managing the Geneve
   NVEs.  The data center operator is expected to use security
   mechanisms to protect the communications between the NVA to NVEs and
   use authentication mechanisms to detect any rogue or compromised NVEs
   within their administrative domain.  Data protection mechanisms for



Gross, et al.           Expires September 8, 2020              [Page 29]

Internet-Draft               Geneve Protocol                  March 2020


   control plane communication or authentication mechanisms between the
   NVA and the NVEs are beyond the scope of this document.

7.  IANA Considerations

   IANA has allocated UDP port 6081 in the Service Name and Transport
   Protocol Port Number Registry [IANA-SN] as the well-known destination
   port for Geneve based on early registration.

   Upon publication of this document, this registration will have its
   reference changed to cite this document [RFC-to-be] and inline with
   [RFC6335] the assignee and contact of the port entry should be
   changed to IESG <iesg@ietf.org> and IETF Chair <chair@ietf.org>
   respectively:

   Service Name: geneve
   Transport Protocol(s): UDP
   Assignee: IESG <iesg@ietf.org>
   Contact: IETF Chair <chair@ietf.org>
   Description: Generic Network Virtualization Encapsulation (Geneve)
   Reference: [RFC-to-be]
   Port Number: 6081

   In addition, IANA is requested to create a new "Geneve Option Class"
   registry to allocate Option Classes.  This registry is to be placed
   under a new Network Virtualization Overlay (NVO3) protocols page (to
   be created) in IANA protocol registries [IANA-PR].  The Geneve Option
   Class registry shall consist of 16-bit hexadecimal values along with
   descriptive strings, assignee/contact information and references.
   The registration rules for the new registry are (as defined by
   [RFC8126]):

               +----------------+-------------------------+
               | Range          | Registration Procedures |
               +----------------+-------------------------+
               | 0x0000..0x00FF | IETF Review             |
               | 0x0100..0xFEFF | First Come First Served |
               | 0xFF00..0xFFFF | Experimental Use        |
               +----------------+-------------------------+

   Initial registrations in the new registry are as follows:










Gross, et al.           Expires September 8, 2020              [Page 30]

Internet-Draft               Geneve Protocol                  March 2020


   +----------------+------------------+------------------+------------+
   | Option Class   | Description      | Assignee/Contact | References |
   +----------------+------------------+------------------+------------+
   | 0x0100         | Linux            |                  |            |
   | 0x0101         | Open vSwitch     |                  |            |
   |                | (OVS)            |                  |            |
   | 0x0102         | Open Virtual     |                  |            |
   |                | Networking (OVN) |                  |            |
   | 0x0103         | In-band Network  |                  |            |
   |                | Telemetry (INT)  |                  |            |
   | 0x0104         | VMware, Inc.     |                  |            |
   | 0x0105         | Amazon.com, Inc. |                  |            |
   | 0x0106         | Cisco Systems,   |                  |            |
   |                | Inc.             |                  |            |
   | 0x0107         | Oracle           |                  |            |
   |                | Corporation      |                  |            |
   | 0x0108..0x0110 | Amazon.com, Inc. |                  |            |
   +----------------+------------------+------------------+------------+

8.  Contributors

   The following individuals were authors of an earlier version of this
   document and made significant contributions:




























Gross, et al.           Expires September 8, 2020              [Page 31]

Internet-Draft               Geneve Protocol                  March 2020


   Pankaj Garg
   Microsoft Corporation
   1 Microsoft Way
   Redmond, WA  98052
   USA

   Email: pankajg@microsoft.com

   Chris Wright
   Red Hat Inc.
   1801 Varsity Drive
   Raleigh, NC  27606
   USA

   Email: chrisw@redhat.com

   Kenneth Duda
   Arista Networks
   5453 Great America Parkway
   Santa Clara, CA  95054
   USA

   Email: kduda@arista.com

   Dinesh G. Dutt
   Independent

   Email: didutt@gmail.com

   Jon Hudson
   Independent

   Email: jon.hudson@gmail.com

   Ariel Hendel
   Facebook, Inc.
   1 Hacker Way
   Menlo Park, CA  94025
   USA

   Email: ahendel@fb.com

9.  Acknowledgements

   The authors wish to acknowledge Puneet Agarwal, David Black, Sami
   Boutros, Scott Bradner, Martin Casado, Alissa Cooper, Roman Danyliw,
   Bruce Davie, Anoop Ghanwani, Benjamin Kaduk, Suresh Krishnan, Mirja
   Kuhlewind, Barry Leiba, Daniel Migault, Greg Mirksy, Tal Mizrahi,



Gross, et al.           Expires September 8, 2020              [Page 32]

Internet-Draft               Geneve Protocol                  March 2020


   Kathleen Moriarty, Magnus Nystrom, Adam Roach, Sabrina Tanamal, Dave
   Thaler, Eric Vyncke, Magnus Westerlund and many other members of the
   NVO3 WG for their reviews, comments and suggestions.

   The authors would like to thank Sam Aldrin, Alia Atlas, Matthew
   Bocci, Benson Schliesser, and Martin Vigoureux for their guidance
   throughout the process.

10.  References

10.1.  Normative References

   [RFC0768]  Postel, J., "User Datagram Protocol", STD 6, RFC 768,
              DOI 10.17487/RFC0768, August 1980,
              <https://www.rfc-editor.org/info/rfc768>.

   [RFC0792]  Postel, J., "Internet Control Message Protocol", STD 5,
              RFC 792, DOI 10.17487/RFC0792, September 1981,
              <https://www.rfc-editor.org/info/rfc792>.

   [RFC1112]  Deering, S., "Host extensions for IP multicasting", STD 5,
              RFC 1112, DOI 10.17487/RFC1112, August 1989,
              <https://www.rfc-editor.org/info/rfc1112>.

   [RFC1191]  Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191,
              DOI 10.17487/RFC1191, November 1990,
              <https://www.rfc-editor.org/info/rfc1191>.

   [RFC2003]  Perkins, C., "IP Encapsulation within IP", RFC 2003,
              DOI 10.17487/RFC2003, October 1996,
              <https://www.rfc-editor.org/info/rfc2003>.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC4443]  Conta, A., Deering, S., and M. Gupta, Ed., "Internet
              Control Message Protocol (ICMPv6) for the Internet
              Protocol Version 6 (IPv6) Specification", STD 89,
              RFC 4443, DOI 10.17487/RFC4443, March 2006,
              <https://www.rfc-editor.org/info/rfc4443>.

   [RFC6040]  Briscoe, B., "Tunnelling of Explicit Congestion
              Notification", RFC 6040, DOI 10.17487/RFC6040, November
              2010, <https://www.rfc-editor.org/info/rfc6040>.





Gross, et al.           Expires September 8, 2020              [Page 33]

Internet-Draft               Geneve Protocol                  March 2020


   [RFC6936]  Fairhurst, G. and M. Westerlund, "Applicability Statement
              for the Use of IPv6 UDP Datagrams with Zero Checksums",
              RFC 6936, DOI 10.17487/RFC6936, April 2013,
              <https://www.rfc-editor.org/info/rfc6936>.

   [RFC7365]  Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
              Rekhter, "Framework for Data Center (DC) Network
              Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
              2014, <https://www.rfc-editor.org/info/rfc7365>.

   [RFC8085]  Eggert, L., Fairhurst, G., and G. Shepherd, "UDP Usage
              Guidelines", BCP 145, RFC 8085, DOI 10.17487/RFC8085,
              March 2017, <https://www.rfc-editor.org/info/rfc8085>.

   [RFC8126]  Cotton, M., Leiba, B., and T. Narten, "Guidelines for
              Writing an IANA Considerations Section in RFCs", BCP 26,
              RFC 8126, DOI 10.17487/RFC8126, June 2017,
              <https://www.rfc-editor.org/info/rfc8126>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8200]  Deering, S. and R. Hinden, "Internet Protocol, Version 6
              (IPv6) Specification", STD 86, RFC 8200,
              DOI 10.17487/RFC8200, July 2017,
              <https://www.rfc-editor.org/info/rfc8200>.

   [RFC8201]  McCann, J., Deering, S., Mogul, J., and R. Hinden, Ed.,
              "Path MTU Discovery for IP version 6", STD 87, RFC 8201,
              DOI 10.17487/RFC8201, July 2017,
              <https://www.rfc-editor.org/info/rfc8201>.

10.2.  Informative References

   [ETYPES]   The IEEE Registration Authority, "IEEE 802 Numbers",
              <https://www.iana.org/assignments/ieee-802-numbers>.

   [I-D.ietf-intarea-tunnels]
              Touch, J. and M. Townsley, "IP Tunnels in the Internet
              Architecture", draft-ietf-intarea-tunnels-10 (work in
              progress), September 2019.

   [I-D.ietf-nvo3-dataplane-requirements]
              Bitar, N., Lasserre, M., Balus, F., Morin, T., Jin, L.,
              and B. Khasnabish, "NVO3 Data Plane Requirements", draft-
              ietf-nvo3-dataplane-requirements-03 (work in progress),
              April 2014.



Gross, et al.           Expires September 8, 2020              [Page 34]

Internet-Draft               Geneve Protocol                  March 2020


   [I-D.ietf-nvo3-encap]
              Boutros, S., "NVO3 Encapsulation Considerations", draft-
              ietf-nvo3-encap-05 (work in progress), February 2020.

   [IANA-PR]  IANA, "Protocol Registries",
              <https://www.iana.org/protocols>.

   [IANA-SN]  IANA, "Service Name and Transport Protocol Port Number
              Registry", <https://www.iana.org/assignments/service-
              names-port-numbers>.

   [IEEE.802.1Q_2018]
              IEEE, "IEEE Standard for Local and Metropolitan Area
              Networks--Bridges and Bridged Networks", IEEE 802.1Q-2018,
              DOI 10.1109/ieeestd.2018.8403927, July 2018,
              <http://ieeexplore.ieee.org/servlet/
              opac?punumber=8403925>.

   [RFC2983]  Black, D., "Differentiated Services and Tunnels",
              RFC 2983, DOI 10.17487/RFC2983, October 2000,
              <https://www.rfc-editor.org/info/rfc2983>.

   [RFC3031]  Rosen, E., Viswanathan, A., and R. Callon, "Multiprotocol
              Label Switching Architecture", RFC 3031,
              DOI 10.17487/RFC3031, January 2001,
              <https://www.rfc-editor.org/info/rfc3031>.

   [RFC3552]  Rescorla, E. and B. Korver, "Guidelines for Writing RFC
              Text on Security Considerations", BCP 72, RFC 3552,
              DOI 10.17487/RFC3552, July 2003,
              <https://www.rfc-editor.org/info/rfc3552>.

   [RFC3985]  Bryant, S., Ed. and P. Pate, Ed., "Pseudo Wire Emulation
              Edge-to-Edge (PWE3) Architecture", RFC 3985,
              DOI 10.17487/RFC3985, March 2005,
              <https://www.rfc-editor.org/info/rfc3985>.

   [RFC4301]  Kent, S. and K. Seo, "Security Architecture for the
              Internet Protocol", RFC 4301, DOI 10.17487/RFC4301,
              December 2005, <https://www.rfc-editor.org/info/rfc4301>.

   [RFC5374]  Weis, B., Gross, G., and D. Ignjatic, "Multicast
              Extensions to the Security Architecture for the Internet
              Protocol", RFC 5374, DOI 10.17487/RFC5374, November 2008,
              <https://www.rfc-editor.org/info/rfc5374>.






Gross, et al.           Expires September 8, 2020              [Page 35]

Internet-Draft               Geneve Protocol                  March 2020


   [RFC6335]  Cotton, M., Eggert, L., Touch, J., Westerlund, M., and S.
              Cheshire, "Internet Assigned Numbers Authority (IANA)
              Procedures for the Management of the Service Name and
              Transport Protocol Port Number Registry", BCP 165,
              RFC 6335, DOI 10.17487/RFC6335, August 2011,
              <https://www.rfc-editor.org/info/rfc6335>.

   [RFC6438]  Carpenter, B. and S. Amante, "Using the IPv6 Flow Label
              for Equal Cost Multipath Routing and Link Aggregation in
              Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011,
              <https://www.rfc-editor.org/info/rfc6438>.

   [RFC7348]  Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
              L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
              eXtensible Local Area Network (VXLAN): A Framework for
              Overlaying Virtualized Layer 2 Networks over Layer 3
              Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
              <https://www.rfc-editor.org/info/rfc7348>.

   [RFC7637]  Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network
              Virtualization Using Generic Routing Encapsulation",
              RFC 7637, DOI 10.17487/RFC7637, September 2015,
              <https://www.rfc-editor.org/info/rfc7637>.

   [RFC8014]  Black, D., Hudson, J., Kreeger, L., Lasserre, M., and T.
              Narten, "An Architecture for Data-Center Network
              Virtualization over Layer 3 (NVO3)", RFC 8014,
              DOI 10.17487/RFC8014, December 2016,
              <https://www.rfc-editor.org/info/rfc8014>.

   [RFC8086]  Yong, L., Ed., Crabbe, E., Xu, X., and T. Herbert, "GRE-
              in-UDP Encapsulation", RFC 8086, DOI 10.17487/RFC8086,
              March 2017, <https://www.rfc-editor.org/info/rfc8086>.

   [RFC8293]  Ghanwani, A., Dunbar, L., McBride, M., Bannai, V., and R.
              Krishnan, "A Framework for Multicast in Network
              Virtualization over Layer 3", RFC 8293,
              DOI 10.17487/RFC8293, January 2018,
              <https://www.rfc-editor.org/info/rfc8293>.

   [VL2]      "VL2: A Scalable and Flexible Data Center Network", ACM
              SIGCOMM Computer Communication Review,
              DOI 10.1145/1594977.1592576, 2009,
              <https://www.sigcomm.org/sites/default/files/ccr/
              papers/2009/October/1594977-1592576.pdf>.






Gross, et al.           Expires September 8, 2020              [Page 36]

Internet-Draft               Geneve Protocol                  March 2020


Authors' Addresses

   Jesse Gross (editor)

   Email: jesse@kernel.org


   Ilango Ganga (editor)
   Intel Corporation
   2200 Mission College Blvd.
   Santa Clara, CA  95054
   USA

   Email: ilango.s.ganga@intel.com


   T. Sridhar (editor)
   VMware, Inc.
   3401 Hillview Ave.
   Palo Alto, CA  94304
   USA

   Email: tsridhar@vmware.com




























Gross, et al.           Expires September 8, 2020              [Page 37]