DMM Working Group S. Homma Internet-Draft NTT Intended status: Informational T. Miyasaka Expires: May 7, 2020 KDDI Research S. Matsushima SoftBank D. Voyer Bell Canada November 4, 2019 User Plane Protocol and Architectural Analysis on 3GPP 5G System draft-ietf-dmm-5g-uplane-analysis-03 Abstract This document analyzes the mobile user plane protocol and the architecture specified in 3GPP 5G documents. The analysis work is to clarify those specifications, extract protocol and architectural requirements and derive evaluation aspects for user plane protocols on IETF side. This work is corresponding to the User Plane Protocol Study work on 3GPP side. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on May 7, 2020. Copyright Notice Copyright (c) 2019 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of Homma, et al. Expires May 7, 2020 [Page 1] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Current Status of Mobile User Plane for 5G . . . . . . . 3 1.2. Our Way of Analysis Work . . . . . . . . . . . . . . . . 4 2. Terms and Abbreviations . . . . . . . . . . . . . . . . . . . 4 3. GTP-U Specification and Observation . . . . . . . . . . . . . 6 3.1. GTP-U Tunnel . . . . . . . . . . . . . . . . . . . . . . 6 3.2. GTP-U Header Format . . . . . . . . . . . . . . . . . . . 10 3.3. Control Plane Protocol for GTP-U . . . . . . . . . . . . 12 3.4. GTP-U message . . . . . . . . . . . . . . . . . . . . . . 13 3.5. Packet Format . . . . . . . . . . . . . . . . . . . . . . 14 3.6. Observations Summary . . . . . . . . . . . . . . . . . . 16 4. 5GS Architectural Requirements for User Plane Protocols . . . 16 4.1. Overview of 5G System Architecture . . . . . . . . . . . 16 4.1.1. UPF Functionalities . . . . . . . . . . . . . . . . . 18 4.1.2. UP Traffic Detection . . . . . . . . . . . . . . . . 19 4.1.3. User Plane Configuration . . . . . . . . . . . . . . 21 4.2. Architectural Requirements for User Plane Protocols . 22 5. Evaluation Aspects . . . . . . . . . . . . . . . . . . . . . 29 5.1. Supporting PDU Session Type Variations . . . . . . . . . 29 5.2. Nature of Data Path . . . . . . . . . . . . . . . . . . . 30 5.3. Supporting Transport Variations . . . . . . . . . . . . . 30 5.4. Data Path Management . . . . . . . . . . . . . . . . . . 31 5.5. QoS Control . . . . . . . . . . . . . . . . . . . . . . . 32 5.6. Traffic Detection and Flow Handling . . . . . . . . . . . 32 5.7. Supporting Network Slicing Diversity . . . . . . . . . . 32 5.8. Reliable Communication support . . . . . . . . . . . . . 33 6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 34 7. Security Consideration . . . . . . . . . . . . . . . . . . . 34 8. Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . 34 9. Informative References . . . . . . . . . . . . . . . . . . . 35 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 39 1. Introduction This document analyzes the mobile user plane protocol and the architecture specified by 3GPP 5G documents. The background of the work is that 3GPP requests through a liaison statement that the IETF to provide any information for the User Plane Protocol Study work in Homma, et al. Expires May 7, 2020 [Page 2] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 3GPP [CP-180116-3GPP]. Justification and the objectives of the study can be found from [CP-173160-3GPP]. We understand that the current user plane protocol, GTP-U [TS.29.281-3GPP], has been well developed in 3GPP, and deployed very widely as the successor of legacy network technologies, such as TDM circuit, or ATM virtual circuit. That GTP-U success seems based on IP overlay technique that is dramatically scaled compare to the previous ones because it successfully isolates mobile session states from the user plane transport network. Even after that big success, it is definitely worth that 3GPP has decided to revisit user plane which seems to response to IPv6 deployment growth and [IAB-Statement] that encourages the industry to develop strategies for IPv6-only operation. It can be seen from the justification section in [CP-173160-3GPP]. The study description mentions that the study would be based on Release 16 requirement while only Release 15 specifications has been available now. However we believe that to provide adequate information for 3GPP, we need to clearly understand what the current user plane protocol is in Release 15, and architectural requirements for the user plane. As the liaison statement indicates 3GPP specifications related to user plane, those documents should be a good start point to clarify their specifications and to extract protocol and architectural requirements from them. 1.1. Current Status of Mobile User Plane for 5G 3GPP RAN and CT4 decided to use GTP-U as the 5G user plane encapsulation protocol over N3 and N9 that respectively described in[TS.38.300-3GPP] and [TR.29.891-3GPP]. N3 is an interface between RAN and UPF and N9 is an interface between different UPFs [TS.23.501-3GPP]. In [TR.29.891-3GPP], it captured user plane requirements and concluded that GTP-U is adopted for the user plane protocol. It seems that GTP-U was only option to be chose and it focused on how to carry 5G specific QoS information between UPF and access networks. That is described in section 5.2 and 11.2 of [TR.29.891-3GPP]. Another aspects of user plane requirements couldn't be found. Homma, et al. Expires May 7, 2020 [Page 3] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 1.2. Our Way of Analysis Work First, we analyze [TS.29.281-3GPP] for clarifying it as the current user plane protocol in the 5G system. [TR.29.891-3GPP] describes how GTP-U is selected as the user plane protocol for 5G in 3GPP. Clarified characteristics of the protocol are described in Section 3. Then, to clarify what are required to the user plane protocol in architecture level, we analyze [TS.23.501-3GPP] as the 5G system architecture specification. [TS.23.502-3GPP] is the specification of system procedures that helps us to understand how the system works in the architecture. [TS.23.503-3GPP] is also helpful to find the role of user plane in the architecture that influences user plane protocol. Extracted architectural requirements are described in Section 4. Based on the results of above, we identify some aspects where there might be gap between the current user plane protocol and the architectural requirements on which [TR.29.891-3GPP] does not discuss. That aspects are discussed Section 5. That's what we intend to be as a part of the reply to 3GPP. CT4 WG in 3GPP can utilize it as an input to evaluate the candidate protocols for user plane to the 5G system including the current protocol. 2. Terms and Abbreviations This section describes terms of functions and interfaces relevant to user plane protocol which we extract from the 3GPP specifications since this document focuses on user plane. In those specifications, there are so many unique terms and abbreviations in the 3GPP context which IETF community seems not familiar with. We will try to bring those terms with brief explanations to make sure common understanding for them. GTP: GPRS Tunneling Protocol GTP-U: User Plane part of GTP Noted that GTP version 1 (GTPv1-U) is the user-plane protocol specification which is defined in [TS.29.281-3GPP]. Unless there is no specific annotation, we refer GTP-U to GTPv1-U in this document. PDU: Protocol Data Unit of end-to-end user protocol packet. Noted that the PDU in 3GPP includes IP header in case that PDU session type is IPv4 or IPv6. In contrast, in IETF it is supposed Homma, et al. Expires May 7, 2020 [Page 4] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 that PDU is the payload of IP packet so that it doesn't include IP/TCP/UDP header in end-to-end. T-PDU: Transport PDU. G-PDU: GTP encapsulated user Plane Data Unit. GTP-U has above two notions on PDU. T-PDU is a PDU that GTP-U header encapsulates. G-PDU is a PDU that includes GTP-U header. A G-PDU may include a T-PDU. G-PDU can be sent without T-PDU, but just with extension headers or TLV elements. It can be used for OAM related operations. PDU session: Association between the UE and a Data Network that provides a PDU connectivity service. Data Network (DN): The network of operator services, Internet access or 3rd party services. User Plane (UP): Encapsulating user end-to-end PDU. In fact, we can't find exact text that defines UP in the architecture specification. However when we see the figure 8.3.1-1 in [TS.23.501-3GPP], we specify UP as the layer right under PDU that directly encapsulates PDU. Underneath layers of UP are UP transport, such as IP/UDP, L2 and L1. However 3GPP is consistent to use the term user plane when they indicate that layer. In IETF, we can see the terms data plane, or forwarding plane as variations which often makes us tend to be confused in terminology. QFI: QoS Flow Identifier UPF: User Plane Function SMF: Session Management Function SMF is a control plane function which provides session management service that handling PDU sessions in the control plane. SMF allocates tunnels corresponding to the PDU sessions and configure the tunnel to the UPF. PFCP: Packet Forwarding Control Protocol PFCP is used on N4 interface between SMF and UPF to configure the rules of packet detection, forwarding action, QoS enforcement, usage report and buffering for each PDU session. Homma, et al. Expires May 7, 2020 [Page 5] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 PDR: Packet Detection Rule FAR: Fowarding Action Rule RAN: Radio Access Network Noted that UP protocol provides a RAN to connect UPF. But the UP protocol is not appeared on the air in the RAN. 3. GTP-U Specification and Observation In this section we analyze the GTP-U specification and summarize clarified characteristic of GTP-U to see if GTP-U meets the requirements of 5G architecture for user plane in later section. 3.1. GTP-U Tunnel GTP-U is a tunneling protocol between given a pair of GTP-U tunnel endpoint nodes and encapsulates T-PDU from/to UE on top of IP/UDP. A Tunnel Endpoint Identifier (TEID) value allocated on each end point indicates which tunnel a particular T-PDU belongs to. The receiving endpoint individually allocate a TEID and the sender tunnel endpoint node encapsulates the IP packet from/to UE with the TEID which is present in GTP-U header on top of IPv4 or IPv6, and UDP. That is described in section 4.2.1 of [TS.29.281-3GPP]. [GTP-U-1]: GTP-U is an unidirectional Point-to-Point tunneling protocol. Figure 1 shows an example of GTP-U protocol stack for uplink (UL) and downlink (DL) traffic flow. Two GTP-U tunnels are required to form one bi-directional tunnel. UL: From RAN to UPF1 (TEID=1), and from UPF1 to UPF2 (TEID=2) DL: From UPF2 to UPF1 (TEID=3), and from UPF1 to RAN (TEID=4) In 5GS, GTP-U tunnel is established at following interfaces to provide PDU Session between UE and 5GC. N3: Between RAN and UPF N9: Between different UPFs GTP-U allows one tunnel endpoint node to send out a G-PDU to be received by multiple tunnel endpoints by utilizing IP multicast capability of underlay IP networks. That is described in section Homma, et al. Expires May 7, 2020 [Page 6] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 4.2.6 of [TS.29.281-3GPP]. It looks GTP-U has Point-to-Multipoint (P2MP) tunneling capability. The P2MP tunneling is used for MBMS (Multimedia Broadcast Multicast Service) through GTP-U tunnel. [GTP-U-2]: GTP-U supports Point-to-Multipoint tunneling. UDP is utilized for GTP-U encapsulation and UDP destination port is 2152 which is assigned by IANA. Allocation of UDP source port depends on sender tunnel endpoint node and GTP-U supports dynamic allocation of UDP source port for load balancing objective. The specification of this dynamic allocation is described in section 4.4.2.0 of [TS.29.281-3GPP], however specific procedure, e.g., 5-tuple hashing, is not described in the document and depends on the implementation of GTP-U tunnel endpoint node. [GTP-U-3]: GTP-U supports load balancing by using dynamic UDP source port allocation. Homma, et al. Expires May 7, 2020 [Page 7] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 [Uplink] .- .-+--------+ - - - - +--------+ - - - - +--------+ | | |Payload | |Payload | |Payload | | PDU < +--------+ +--------+ +--------+ |(3GPP)| |Inner IP| |Inner IP| |Inner IP| | '-+--------+ - - - - +--------+ - - - - +--------+ | | GTP-U | | GTP-U | | L2 | | |(TEID=1)| |(TEID=2)| +--------+ GTP< +--------+ +--------+ | L1 | Pkt | | UDP | | UDP | +--------+ | +--------+ +--------+ | |Outer IP| |Outer IP| | +--------+ +--------+ | | L2 | | L2 | | +--------+ +--------+ | | L1 | | L1 | '- +--------+ - - - - +--------+ ================ Traffic Direction =======================> [Downlink] .- .-+--------+ - - - - +--------+ - - - - +--------+ | | |Payload | |Payload | |Payload | | PDU < +--------+ +--------+ +--------+ |(3GPP)| |Inner IP| |Inner IP| |Inner IP| | '-+--------+ - - - - +--------+ - - - - +--------+ | | GTP-U | | GTP-U | | L2 | | |(TEID=4)| |(TEID=3)| +--------+ GTP< +--------+ +--------+ | L1 | Pkt | | UDP | | UDP | +--------+ | +--------+ +--------+ | |Outer IP| |Outer IP| | +--------+ +--------+ | | L2 | | L2 | | +--------+ +--------+ | | L1 | | L1 | '- +--------+ - - - - +--------+ <=============== Traffic Direction ======================== +-----+ N3 +------+ N9 +------+ N6 -----+ RAN +-----------+ UPF1 +------------+ UPF2 +---------- +-----+ +------+ +------+ Figure 1: Protocol Stack by GTPv1-U for Uplink and Downlink Traffic Flow Homma, et al. Expires May 7, 2020 [Page 8] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 IPv6 flow label [RFC6437] is also candidate method for load balancing especially for IP-in-IPv6 tunnel [RFC6438] like GTP-U. However, how to use IPv6 flow label of GTP-U is not described in [TS.29.281-3GPP]. Though this method is limited to a case of IPv6 encapsulated GTP-U tunnel, it is worth to study usage of IPv6 flow label in 3GPP. [GTP-U-4]: GTP-U does not support IPv6 flow label for load balancing in case of IPv6 transport. GTP-U supports both IPv4 and IPv6 as underlying transport layer protocol. As for IPv6, GTP-U specification refers [RFC2460], which is described in section 4.2.3 of [TS.29.281-3GPP]. As [RFC2460] does not allow the tunnel protocols on top of UDP to set the checksum value to zero, the GTP-U specification inherits it while the IPv4 transport for GTP-U case allows UDP zero checksum. It is noted that the IPv6 specification in IETF has been updated to [RFC8200] which allows UDP zero checksum for the tunnel. [RFC6935] describes benefits of zero checksum for tunnel protocol over UDP. If GTP-U support UFP zero checksum in future version, possible interoperability issue between previous generations which does not support zero checksum may raise. [GTP-U-5]: UDP zero checksum is not available in case of IPv6 transport. "Unnecessary fragmentation should be avoided" is recommended and to avoid the fragmentation operator should configure MTU size at UE [TS.29.281-3GPP]. However, there's no reference and specification of Path MTU Discovery for IPv6 transport. If encapsulated IPv6 packet is too big on a network link between tunnel endpoint nodes, UE may not receive ICMPv6 Packet Too Big message and causes Path MTU Discovery black hole. [GTP-U-6]: GTP-U does not support to response ICMP PTB for Path MTU Discovery. Section 9.3 of [TS.23.060-3GPP] specifies advertisement of inner IPv6 link MTU size for UE by IPv6 RA message [RFC4861]. However, this document doesn't specify a procedure to measure MTU size in mobile network system and mobile network operator need to calculate MTU size for UE like Annex C of [TS.23.060-3GPP]. If link MTU of a router in a transport network is accidentally modified, UE cannot detect the event and send packet with initial MTU size, which may cause service disruption due to MTU exceed in the router link. Homma, et al. Expires May 7, 2020 [Page 9] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 3.2. GTP-U Header Format Figure 2 shows general and mandatory GTP-U header and Figure 3 shows extension GTP-U header. [GTP-U-7]: GTP-U supports sequence number option in the header, but it is not recommended to be used by almost GTP-U entities. GTP-U header has Sequence Number field to reorder incoming packets based on the sequence number. If Sequence Number Flag is set to '1' it indicates that Sequence Number Filed exists in GTP-U header and examined at receiving tunnel endpoint node to reorder incoming packets. However, the sequence number flag is set to '1' only for RAT HO procedure and sequence number flag should be set to '0' in normal case. Therefore, in normal case receiver tunnel endpoint node doesn't examine sequence number and can't reorder GTP-U packets based on the sequence number. This specification is described in section 5.1 of [TS.29.281-3GPP]. In 3GPP, sequential delivery is required only during handover procedure and is used by only RAN entities. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ver |P|R|E|S|N| Message Type | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Tunnel Endpoint Identifier | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number | N-PDU Number | Next-Ext-Hdr | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 2: GTP-U Header o Ver: Version field (Set to '1') o P: Protocol Type (Set to '1') o R: Reserved bit (Set to '0') o E: Extension Header Flag (Set to '1' if extension header exists) o S: Sequence Number Flag (Set to '1' if sequence number exists) o N: N-PDU Number Flag (Set to '1' if N-PDU number exists) o Message Type: Indicates the type of GTP-U message o Length: Indicates the length in octets of the payload Homma, et al. Expires May 7, 2020 [Page 10] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 o Tunnel Endpoint Identifier (TEID) o Sequence Number: Indicates increasing sequence number for T-PDUs is transmitted via GTP-U tunnels o N-PDU Number: It is used only for inter SGSN, 2G-3G handover case, etc. o Next-Ext-Hdr: Indicates following extension header type 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Ext-Hdr Length| | +-+-+-+-+-+-+-+-+ | | Extension Header Content | . . . +-+-+-+-+-+-+-+-+ | | Next-Ext-Hdr | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 3: Extension GTP-U Header o Ext-Hdr Length: Represents the length of the Extension header in units of 4 octets o Extension Header Content: Contains 3GPP related information o Next-Ext-Hdr: Indicates following extension header type The extension GTP-U header is a variable-length and extendable header and contains 3GPP specific information. Following list summarizes every extension header which is used for user plane protocol. These extension headers are defined in [TS.29.281-3GPP]. In this list Next-Ext-Hdr is represented in binary. o No more extension headers (Next-Ext-Hdr = 00000000) o Service Class Indicator (Next-Ext-Hdr = 00100000) o UDP Port (Next-Ext-Hdr = 01000000) o RAN Container (Next-Ext-Hdr = 10000001) o Long PDCP PDU Number (Next-Ext-Hdr = 10000010) o Xw RAN Container (Next-Ext-Hdr = 10000011) Homma, et al. Expires May 7, 2020 [Page 11] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 o NR RAN Container (Next-Ext-Hdr = 10000100) o PDU Session Container (Next-Ext-Hdr = 10000101) o PDCP PDU Number (Next-Ext-Hdr = 11000000) [GTP-U-8]: GTP-U supports carrying QoS Identifiers transparently for Access Networks in an extension header. GTP-U is designed to carry 3GPP specific information with extension headers. 3GPP creates PDU Session Container extension header for NGRAN of 5G to carry QFI. It is described in section 5.2.2.7 of [TS.29.281-3GPP]. [GTP-U-9]: GTP-U supports DSCP marking based on the QFI. DSCP marking on outer IPv4 or IPv6 shall be set by sender tunnel endpoint node based on the QFI. This specification is described in section 4.4.1 of [TS.29.281-3GPP]. [GTP-U-10]: GTP-U does not specify extension header order. In general, multiple GTP-U extension headers are able to contained in one GTP-U packet and the order of those extension headers is not specified by [TS.29.281-3GPP]. Thereby the receiving endpoint can't predict exact position where the target extension headers are. This could impact on header lookup performance on the node. As for PDU Session Container extension header, there is a note in [TS.29.281-3GPP] as "For a G-PDU with several Extension Headers, the PDU Session Container should be the first Extension Header". This note was added at the version 15.3.0 of [TS.29.281-3GPP] which is published on June 2018 in order to accelerate the processing of GTP-U packet at UPF and RAN. It is only one rule regarding the extension header order. [GTP-U-11]: GTP-U does not support to indicate next protocol type. When Next-Ext-Hdr is set to 0x00 it indicates that no more extension headers follow. As GTP is designed to indicate protocol types for T-PDU by control-plane signaling, GTP-U doesn't have Next-Protocol- Header field to indicate the T-PDU type in the header. 3.3. Control Plane Protocol for GTP-U Control plane protocol for GTP-U signals TEID between tunnel endpoint nodes. GTPv2-C [TS.29.274-3GPP] is the original control plane Homma, et al. Expires May 7, 2020 [Page 12] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 protocol tied with GTP-U in previous generation architectures before CUPS (Control and User Plane Separation). 3GPP decided to use extended PFCP (Packet Forwarding Control Protocol) [TS.29.244-3GPP] for N4 interface [TR.29.891-3GPP] to signal tunnel states from SMF to UPF. 3.4. GTP-U message GTP-U supports in-band messaging to signal OAM. Currently GTP-U supports following messages [TS.29.281-3GPP]. o Echo Request o Echo Response o Supported Extension Headers Notification o Error Indication o End Marker [GTP-U-12]: GTP-U supports active OAM as a path management message "Echo Request/Response". A GTP-U tunnel endpoint node sends a Echo Request message to another nodes for keep-alive and received node sends a Echo Response message to sender node as acknowledgment. Echo Request message and Echo Response message are described in section 7.2.1 and section 7.2.2 of [TS.29.281-3GPP] respectively. [TR.29.891-3GPP] recommends not to send Echo Request message more often than 60s on each path. Supported Extension Headers Notification message indicates a list of supported that tunnel endpoint node can support. This message is sent only in case a tunnel endpoint node receives GTP-U packet with unsupported extension header. [GTP-U-13]: GTP-U supports tunnel management messages "Error Indication". GTP-U has Error Indication message to notify that the receiving endpoint discard packets of which no session exist to the sending endpoint. Error Indication message is described in section 7.3.1 of [TS.29.281-3GPP]. [GTP-U-14]: GTP-U supports tunnel management messages "End Marker". Homma, et al. Expires May 7, 2020 [Page 13] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 GTP-U has End Marker message to indicate the end of the payload stream that needs to be sent on a GTP-U tunnel. End Marker message is described in section 7.3.2 of [TS.29.281-3GPP]. 3.5. Packet Format Figure 4 shows a packet format example of GTP-U over IPv6 that carries an extension header for QFI and an IPv6 PDU. All values in the example are illustration purpose only. The encoding of PDU Session Container for QFI refers to [TS.38.415-3GPP]. Outer IPv6 Header's DSCP value(EF) in Figure 4 is marked at sender tunnel endpoint node based on QFI value which is contained in GTP-U Extension Header (PDU Session Container). 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 Outer IPv6 Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| DSCP=EF | Flow Label | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Payload Length | NxtHdr=17(UDP)| Hop Limit | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + + | | + Source IPv6 Address + | 2001:db8:1:1::1 | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + + | | + Destination IPv6 Address + | 2001:db8:1:2::1 | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Outer UDP Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Port = xxxx | Dest Port = 2152 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | UDP Length | UDP Checksum (Non-zero) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ GTP-U header Homma, et al. Expires May 7, 2020 [Page 14] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0x1 |1|0|1|0|0| 0xff | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | TEID = 1654 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Sequence Number = 0 |N-PDU Number=0 |NextExtHdr=0x85| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ GTP-U Extension Header (PDU Session Container) +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | ExtHdrLen=2 |Type=0 | Spare |0|0| QFI | PPI | Spare | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Padding |NextExtHdr=0x0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Inner IPv6 Header +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| DSCP=0 | Flow Label | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Payload Length | NexttHdr | Hop Limit | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + + | | + Source IPv6 Address + | 2001:db8:2:1::1 | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + + | | + Destination IPv6 Address + | 2001:db8:3:1::1 | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Payload +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | | . TCP/UDP/etc., Data . . . | | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Homma, et al. Expires May 7, 2020 [Page 15] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 Figure 4: GTP-U Protocol Stack Example 3.6. Observations Summary [GTP-U-1]: An unidirectional Point-to-Point tunneling protocol. [GTP-U-2]: Supports Point-to-Multipoint tunneling. [GTP-U-3]: Supports load balancing by using dynamic UDP port allocation. [GTP-U-4]: Does not support IPv6 flow label for load balancing in case of IPv6 transport. [GTP-U-5]: UDP zero checksum is not available in case of IPv6 transport. [GTP-U-6]: Does not support to response ICMP PTB for Path MTU Discovery. [GTP-U-7]: Supports sequence number option and sequence number flag in the header, but it is not recommended to be used by almost GTP-U entities. [GTP-U-8]: Supports carrying QoS Identifiers transparently for Access Networks in extension headers. [GTP-U-9]: Supports DSCP marking based on the QFI. [GTP-U-10]: Does not specify the rule for the extension header order. [GTP-U-11]: Does not support an indication of next-header type. [GTP-U-12]: Supports active OAM as a path management message "Echo Request/Response". [GTP-U-13]: Supports tunnel management messages "Error Indication". [GTP-U-14]: Supports tunnel management messages "End Marker". 4. 5GS Architectural Requirements for User Plane Protocols 4.1. Overview of 5G System Architecture The 5G system is designed for applying to diverse devices and services due to factors such as the diffusion of IoT devices, and the UP protocol is required to have capabilities for satisfying their requirements. Homma, et al. Expires May 7, 2020 [Page 16] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 As a principle of the 5G system, User Plane (UP) functions are separated from the Control Plane (CP) functions for allowing independent scalability, evolution and flexible deployments. Network slicing is also one of the fundamental concepts of the 5G system, and it provides logical network separation. In terms of user plane, multiple network slices can be comprised of UPFs on top of same physical network resources. Allocated resources and structures may be differentiated among the slices by which the required features or capabilities. The 3GPP 5G architecture [TS.23.501-3GPP] defines slice types which are eMBB, URLLC and MIoT from Rel-15. In addition to that, V2X slice type is defined from Rel-16. The architecture overview is shown in Figure 5. The details of functions are described in [TS.23.501-3GPP]. A UPF handles UP paths on N3, N9 and N6 interface, and the setup is controlled by SMF via N4 interface. A UP path will be manipulated based on application requirements for the PDU session corresponding to the path. An SMF is also capable to receive information regarding routing path with API from AF via NEF, PCF, and SMF. Homma, et al. Expires May 7, 2020 [Page 17] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ |NSSF | | NEF | | NRF | | PCF | | UDM | | AF | +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ +--+--+ Nnssf| Nnef| Nnrf| Npcf| Nudm| |Naf ---+--------+--+-----+--+--------+--+-----+--------+- Nausf| Namf| Nsmf| +--+--+ +--+--+ +--+-------+ |AUSR | | AMF | | SMF | +-----+ ++-+--+ +--+-----+-+ / | | \ Control Plane N1/ |N2 |N4 \N4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ User Plane / | | \ +--++ +--+--+ N3 +--+--+ N9 +-+---+ N6 +-----+ |UE +--+(R)AN+-----+ UPF +-----+ UPF +----+ DN | +---+ +-----+ +--+--+ +-----+ +-----+ |N6 +--+--+ | DN | +-----+ Figure 5: 5GS Architecture and Service-based Interfaces This document mainly focuses on requirements for N9 interface as relevant to UP protocol of 5G system. 4.1.1. UPF Functionalities UPF has a role to handle UP traffic, and provides functionalities to look up user data traffic and enforce the appropriate policies to it. The followings are defined as UPF functionalities defined in the section 6.2.3 of [TS.23.501-3GPP] o Anchor point for Intra-/Inter-RAT mobility (when applicable). o External PDU Session point of interconnect to Data Network. o Packet routing and forwarding (e.g. support of Uplink classifier to route traffic flows to an instance of a data network, support of Branching point to support multi-homed PDU Session). o Packet inspection (e.g. Application detection based on service data flow template and the optional PFDs received from the SMF in addition). Homma, et al. Expires May 7, 2020 [Page 18] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 o User Plane part of policy rule enforcement, e.g. Gating, Redirection, Traffic steering). o Lawful intercept (UP collection). o Traffic usage reporting. o QoS handling for user plane, e.g. UL/DL rate enforcement, Reflective QoS marking in DL. o Uplink Traffic verification (SDF to QoS Flow mapping). o Transport level packet marking in the uplink and downlink. o Downlink packet buffering and downlink data notification triggering. o Sending and forwarding of one or more "end marker" to the source NG-RAN node. o ARP proxying and / or IPv6 Neighbour Solicitation Proxying for the Ethernet PDUs. o Packet duplication in downlink direction and elimination in uplink direction in UP protocol layer. o TSN Translator functionality to hold and forward user plane packets for de-jittering when 5G System is integrated as a bridge with the TSN network. 4.1.2. UP Traffic Detection The traffic detection is described in the section 5.8.2.4 of [TS.23.501-3GPP]. In 3GPP UP packet forwarding model, UPF detects UP traffic flow which belong to a N4 session configured by SMF. The protocol of N4 interface, PFCP, brings a set of traffic detection information from SMF to UPF as Packet Detection Information (PDI) in a PDR to establish/modify the N4 PFCP session. It is defined in section 7.5.2.2 of [TS.29.244-3GPP]. Combination of the following information is used for the traffic detection: o For IPv4 or IPv6 PDU Session type * CN tunnel info (Tunnel ID and the endpoint IP address of 5G Core) Homma, et al. Expires May 7, 2020 [Page 19] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 * Network instance * QFI * IP Packet Filter Set * Application Identifier: The Application ID is an index to a set of application detection rules configured in UPF o For Ethernet PDU Session type * CN tunnel info(Tunnel ID and the endpoint IP address of 5G Core) * Network instance * QFI * Ethernet Packet Filter Set It is noted that Network Instance is encoded as Octet String in PFCP, and is NOT appeared in UP packet over the wire. It is expected like an attribute of the receiving IP interface of the UPF. It supports UPF to be able to connect to different IP domains of N3, N9 or N6, which run each independent policy in routing and addressing. The UPF detects traffic flow with Network Instance which the receiving interface attributed to. The IP Packet Filter Set and Ethernet Packet Filter Set defined in clause 5.7.6 of [TS.23.501-3GPP] are following: o IP Packet Filter Set: * Source/destination IP address or IPv6 prefix * Source/destination port number * Protocol ID of the protocol above IP/Next header type * Type of Service (TOS) (IPv4) / Traffic class (IPv6) and Mask. * Flow Label (IPv6) * Security parameter index * Packet filter direction o Ethernet Packet Filter Set: Homma, et al. Expires May 7, 2020 [Page 20] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 * Source/destination MAC address * Ethertype as defined in IEEE 802.3 * Customer-VLAN tag(C-TAG) and/or Service-VLAN tag(S-TAG) VID fields as defined in IEEE 802.1Q * Customer-VLAN tag(C-TAG) and/or Service-VLAN tag(S-TAG) PCP/DEI fields as defined in IEEE 802.1Q * IP Packet Filter Set, in case Ethertype indicates IPv4/IPv6 payload * Packet filter direction 4.1.3. User Plane Configuration User Plane configuration on a UPF is managed by an SMF through PFCP [TS.29.244-3GPP]. The SMF establishes PFCP sessions on the UPF per PDU session basis. The UPF maintains each configured PFCP session states during the sessions exist. A PFCP session consists of the rules of packet detection, forwarding action, QoS enforcement, usage reporting and buffering action. Figure 6 depicts overview of the PFCP session state structure. The listed information in Section 4.1.2 indicates packet detection information of packet detection rule for that the rest of related rules within the PFCP session to be derived. All rules are per session unique and no rules are shared with other sessions. PFCP-Session* [F-SEID] +- F-SEID(Full Qualified Session Endpoint ID) uint64 +- PDU-Session-Type [IPv4|IPv6|IPv4v6|Ether|Unstrct] +- DNN(Data Network Name) +- PDR(Packet Detection Rule)* [PDR-ID] | +- PDR-ID uint16 | +- PDI (Packet Detection Information) | | +- Traffic-Endpoint-ID? -> Traffic-Endpoint-ID reference | | +- .... | +- FAR/URR/QER-ID -> FAR/URR/QER-ID references +- FAR(Forwarding Action Rule)* [FAR-ID] | +- FAR-ID uint32 | +- Forwarding-Parameters | | +- Network-Instance? Octet String | | +- Outer-Header-Creation | | | +- Outer-Hdr-Creation-Desc [GTPoUDP/IPv4|IPv6, etc.,] Homma, et al. Expires May 7, 2020 [Page 21] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 | | | +- TEID, outer IP-Address for N3/N9 | | | +- C/S-TAG, UDP Port-number for N6 | | +- Forwarding-Policy-ID? Octet String | | +- .... | +- Duplicating-Parameters | | +- .... | +- BAR-ID? -> BAR-ID reference +- QER(QoS Enforcement Rule)* [QER-ID] | +- QER-ID uint32 | +- MBR(Maximum Bit Rate) | | +- UL/DL-MBR? bitrate_in_kbps (0..10000000) | +- GBR(Guaranteed Bit Rate) | | +- UL/DL-GBR? bitrate_in_kbps (0..10000000) | +- QoS-flow-identifier? QFI value(6-bits) | +- Reflective-QoS? boolean | +- Paging-Policy-Indicator? PPI value(3-bits) | +- .... +- URR(Usage Reporting Rule)* [URR-ID] | +- URR-ID uint32 | +- Measurement-Method, Period, Reporting-Triggers? | +- Volume/Event/Time Threshold, Quota? | +- Quota-Holding-Time? | +- FAR-ID for Quota action? -> FAR-ID reference | +- .... +- BAR(Buffering Action Rule)* [BAR-ID] | +- BAR-ID uint8 | +- Suggested-Buffering-Packets-Count +- Traffic-Endpoint* [Traffic-Endpoint-ID] +- Traffic-Endpoint-ID uint8 +- TEID, Tunnle IP Address, UE Address...? Figure 6: User Plane Configuration Model 4.2. Architectural Requirements for User Plane Protocols This section lists the requirements for the UP protocol on the 5G system. The requirements are picked up from [TS.23.501-3GPP]. In addition, some of service requirements described in [TS.22.261-3GPP] are referred to clarify the originations of architectural requirements. According to [TS.23.501-3GPP], the specifications potentially have assumptions that the UP protocol is a tunnel representing a single TEID between a pair of UPFs and it is corresponding to a single PDU session. In short, the UP protocol is a tunnel and it is assumed to be managed under per PDU session handling. Also, it should be a stateful tunnel in the UPFs along with the PDU session. Homma, et al. Expires May 7, 2020 [Page 22] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 The requirements for UP protocols are described below: ARCH-Req-1: Supporting IPv4, IPv6, Ethernet and Unstructured PDU The 5G system defines four types of PDU session as IPv4, IPv6, Ethernet, and Unstructured. Therefore, UP protocol must support to convey all of these PDU session types. This is described in [TS.23.501-3GPP]. Note: In TS 23.501 v15.2.0, IPv4v6 is added as a PDU session type. ARCH-Req-2: Supporting IP connectivity for N3, N6, and N9 interfaces The 5G system requires IP connectivity for N3, N6, and N9 interfaces. The IP connectivity is assumed that it comprises of IP routing and L1/L2 transport networks which are outside of 3GPP specifications. It is desirable that the IP connectivity built on IPv6 networks when it comes to address space for end-to-end user plane coverage. But it is expected to take certain time. During the IPv6 networks are not deployed for all the coverage, UP protocol should support RANs and DNs running on IPv4 transport connect to UPF running on IPv6 transport. Furthermore, on N6 interface, point-to-point tunneling based on UDP/ IPv6 may be used to deliver unstructured PDU type data. Then, the content information of the PDU may be mapped into UDP port number, and the UDP port numbers is pre-configured in the UPF and DN. This is described in the section 9.2 of [TS.29.561-3GPP]. ARCH-Req-3: Supporting deployment of multiple UPFs as anchors for a single PDU session The 5G system allows to deploy multiple UPFs as anchors for a single PDU session, and supports multihoming of a single PDU session for such anchor UPFs. Multihoming is provided with Branching Point (BP). BP provides forwarding of UL traffic towards the different PDU Session Anchors based on the source IPv6 prefixes and merge of DL traffic to the UE. IPv6 multihoming only means multiple source IPv6 prefixes are used for a PDU session. It is identical to one classified as scenario 1 in [RFC7157]. Up link classifier (UL CL) is to forward uplink packets to multiple anchor UPFs based on the destination IP of the T-PDU regardless of the source IP address. Noted that single source IP address/prefix Homma, et al. Expires May 7, 2020 [Page 23] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 PDU session is not defined as multihoming PDU session in 5GCS even though a PDU session has multiple anchor UPFs. On UL side, P2P tunnels are established per destination anchor UPFs basis from one UL CL UPF to the anchor UPFs for the PDU session. On DL side, one single multipoint-to-point (MP2P) tunnel exists from the source anchor UPFs to the destination BP UPF for the PDU session. It means that the paths from the anchor UPFs are merged into just one tunnel state at the destination BP UPF. Multiple P2P paths on DL could also be used for multihoming. However it should be the multiple PDU sessions multihoming case where the destination gNB or UPF needs to maintain multiple tunnel states under the one PDU session to one UP tunnel architectural principle. It causes increase of load on tunnel states management in UPF due to increment of the anchor UPF for the PDU session. However, P2P tunneling could increase explosively the number of states in UPF as the anchor UPF/DN incremented to the PDU session. Thereby single PDU session multihoming with MP2P path should be a better option for multihoming in terms of reducing total number of tunnel states. SSC mode 3 for session continuity in hand-over case uses a single PDU multihoming with BP to make sure make-before-break. It is described in the section 5.6.4 and 5.6.9 of [TS.23.501-3GPP]. Multihoming is also assumed to be used for edge computing scenario. Edge computing enables some services to be hosted close to the UE's access point of attachment, and achieves an efficient service delivery through the reduced end-to-end latency and load on the transport network. In edge computing, local user's traffic is routed or steered to application in the local DN by UPF. This refers the section 5.13 of [TS.23.501-3GPP]. ARCH-Req-4: Supporting flexible UPF selection for PDU The appropriate UPFs are selected for a PDU session based on parameters and information such as UPF's dynamic load or UE location information. Examples of parameters and information are described in the section 6.3.3 of [TS.23.501-3GPP]. This means that it is possible to make routing on user plane more efficient in the 5GS. For example, in case that UPFs are distributed geographically, decision of the destination UPF based on locations of end hosts (e.g., UE or NF in DN) enables to forward PDUs with a route connecting between UPFs nearby the hosts directly. This would be Homma, et al. Expires May 7, 2020 [Page 24] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 useful UE-to-UE or UE-to-local_DN communication, and such usage is described in the section 6.5 of [TS.22.261-3GPP]. The 5GS allows operators to select parameters used for UPF selection. (In other words, any specific schemes on UPF selection are not defined in the current 3GPP documents.) ARCH-Req-5: No limitation for number of UPFs in a data path The number of UPF in the data path is not constrained by 3GPP specifications. This specification is described in the section 8.3.1 of [TS.23.501-3GPP]. Putting multiple UPFs, which provides specific function, in a data path enables flexible function deployment to make sure load distribution optimizations, etc. Meanwhile, each UPF in a data path shall be controlled by an SMF via N4 interface. Thus putting an excess of UPF for data paths might cause increase of load of an SMF. Pragmatically, the number of UPF put in a data path is one or two (e.g., for MEC or roaming cases), and, at most, it would be three (e.g., for case where UE moves during a session). It is expected that multiple UPFs with per session tunnel handling for a PDU session becomes complicated task more and more for a SMF by increasing number of UPFs. ARCH-Req-6: Supporting aggregation of multiple QoS Flow indicated with QFI into a PDU Session Against to the previous generation, 5G enables UPF to multiplex QoS Flows, equivalent with IP-CAN bearers in the previous generation, into one single PDU session. That means that a single tunnel includes multiple QFIs contrast to just one QoS Flow (a bearer) to one tunnel before 5G. In even the 5GS, each flow is forwarded based on the appropriate QoS rules. QoS rules are configured by SMF as QoS profiles to UP components and these components perform QoS controls to PDUs based on rules. In downlink, a UPF pushes QFI into an extension header, and transmits the PDU to RAN or another UPF. Then, such UPF may perform transport level QoS packet marking (e.g., DSCP marking in the outer header). In uplink, each UE obtains the QoS rule from SMF, and transmit PDUs with QFI containing the QoS rules to the RAN. The following RAN and UPFs perform enforcement of QoS control and charging based on the QFI. Homma, et al. Expires May 7, 2020 [Page 25] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 This specification is described in 5.7.1 of [TS.23.501-3GPP]. ARCH-Req-7: Supporting network slicing The 5GS fundamentally supports network slicing for provision the appropriate end-to-end communication to various services. In the relevant documents (e.g., [TS.23.501-3GPP], [TS.28.530-3GPP]), a network slice is defined as virtual network and it is structured with 5GS NF instances, such as SMF, UPF including IP transport connectivity between RANs and DNS. Each network slice is independent and its user plane (including network functions and links) should be noninteractive against the others. The 5G architecture specification has been updated with that Network Instance is defined as the glue of network slice between 5G slice and corresponding IP transport slice in addition to the original role of separating IP domains, which is described in Section 4.1.2. It has been appeared from version 15.2.0 of [TS.23.501-3GPP] in section 5.6.12. UP underlay transport networks and UPFs may be shared by 5G slices, as described in section 4 of [TS.28.530-3GPP]. The data model defined in [TS.29.510-3GPP] allows that a Network Instance, a UPF and its interfaces can belong to multiple slices as same as other type of NFs. UP endpoint IP prefix/address of an interface can also be shared with multiple interfaces on the UPF as the model doesn't make them slice unique. The slice lifecycle managements is described in the relevant documents: [TS.28.531-3GPP], [TS.28.532-3GPP], and [TS.28.533-3GPP]. ARCH-Req-8: End Marker support The construction of End Marker packets specified in [TS.23.501-3GPP] may either be done in the CP/UP functions for indicating the end of the payload stream on a given UP tunnel. PDU packets arrive after an End Marker message on the tunnel may be silently discarded. For example, End Maker is used for handover procedures, and it can prevent reordering of arriving packets due to switch of anchor UPFs. ARCHI-Req-9: Supporting Redundant UP transmission for URLLC The 5G is expected to support services which are latency sensitive and require high reliability. Communication to realize such services is called Ultra-Reliable and Low-Latency Communication or URLLC. In URLLC, redundancy of QoS flows is required for providing highly reliable communication. For instance, a set of UP NFs (e.g., UPF or Homma, et al. Expires May 7, 2020 [Page 26] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 gNB) and interfaces between UE and DN are redundant, and packets are replicated and forwarded via each route. UEs and DN support dual connectivity and drop duplicated received packets. The scheme of packet dropping at UE is out of responsibility of 3GPP. The overview is shown in Figure 7. ---+-----------+----------+--- Namf| Nsmf| Nsmf| +--+--+ +--+--+ +--+--+ | AMF | | SMF1| | SMF3| +-+---+ +--+--+ +-+---+ / / / N2/ N4/ N4/ / / / +----++ N3 +--+---+ / N6 +----+ |M-RAN+--------+ UPF1 +--------------+ | ++-+--+ +-+--+-+ / | | / | | | / | | +----+ / | +--+ / | | | UE +< |Xn N9 / | DN | +----+ \ | / | | \ | / | | ++-+--+ N3 +----+-+ N6 | | |S-RAN+--------+ UPF2 +--------------+ | +-----+ +-+--+-+ +----+ | | +--+ N9 *Legends M-RAN: Master RAN S-RAN: Secondary RAN Figure 7: Redundant UP paths using dual connectivity Otherwise, in case that RAN nodes and UPFs have enough reliability and they are not redundant by dual devices, reliable connectivity of QoS flows is provided by dual N3 tunnels between RAN and UPFs. Such tunnels are treated as individual ones, but they have the same sequence number. UP NFs identifies the duplication of PDU packets based on sequence number content in the UP tunnel headers. For uplink packets, a RAN node replicates each packet from a UE. An anchor UPF receives the duplicated packets, and drops ones which reach later in each duplicated packet pare. On the other hand, for downlink packets, a UPF replicates packets received from DN, and a RAN node drops the duplicated packets as well. The overviews of the ways are shown in Figure 8. Homma, et al. Expires May 7, 2020 [Page 27] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 ----+-----------+-----------+--- Namf| Nsmf| Npcf| +--+--+ +--+--+ +--+--+ | AMF | | SMF | | PCF | +-+---+ +--+--+ +-----+ / | N2/ N4| / N3 Tunnel1 | +----+ +--+--+__________+--+--+ N6 +----+ | UE +----+ RAN |__________| UPF |-----+ DN | +----+ +-----+ +-----+ +----+ N3 Tunnel2 Figure 8: Redundant UP transmission with two N3 tunnels In addition, there is a case that two intermediate UPFs (I-UPFs) between anchor UPF and RAN are used to support the redundant transmission based on two N3 and N9 tunnels between single anchor UPF and RAN node. The RAN node and anchor UPF support the packet replication and dropping of duplicated packets as described above. As described above, anchor UPF and RAN node detect packet duplication with sequence number of UP tunnels, and thus I-UPFs would forward the packets with the same sequence number on N3 and N9 tunnels. The overview is shown in Figure 9. ----+-------------+--------------+--- Namf| Nsmf| Npcf| +--+--+ +---+---+ +--+--+ | AMF | | SMF + | PCF | +--+--+ +-+-+-+-+ +-----+ / | | | N2 / N4| | +----------------+ / | | | / N3 Tunnel1 +-+-+---+ N9 Tunnel1 | / +-----+I-UPF1 +----+ | +----+ +--+--+______| +-------+ |______+--+--+ N6 +----+ | UE +---+ RAN |______ | ______| UPF +-----+ DN | +----+ +-----+ N3 | +---+---+ | N9 +-----+ +----+ +-----+I-UPF2 +----+ N3 Tunnel2 +-------+ N9 Tunnel2 Figure 9: Redundant UP transmission with two I-UPF and N3/N9 tunnels ARCHI-Req-10: Supporting QoS Monitoring for URLLC QoS monitoring is also required for URLLC. It means that the user plane should be able to measure packet delay between anchor UPF and Homma, et al. Expires May 7, 2020 [Page 28] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 UE. The measurement would be in various granularities, in the basis of per QoS Flow per UE, or per UP path for example. To help the measurement at anchor UPF and RAN, UP protocol requires to have capability to convey necessary information to do that; such as time information at sending or reception of a measurement packet. That information should exist in per F-TEID and QFI basis which indicates QoS Flow of the packet. UP protocol should also be able to indicate which packets include the corresponding information for each measurement. The QoS monitoring requirement has been appeared in section 5.33.3 of [TS.23.501-3GPP] from Rel-16, version 16.2.0. 5. Evaluation Aspects This section provides UP protocol evaluation aspects that are mainly we derived from the architectural requirements described in Section 4. Those aspects are not prioritized by the order here. Expected deployment scenarios explain the evaluations purpose in the corresponding aspects. As we were noticed that the gaps between GTP-U specifications and 5G architectural requirements through the analysis, those each gap are briefly described in the evaluation aspect associated to it. Since it is obvious that 5G system should be able to interwork with existing previous generation based systems, any aspects from coexisting and interworking point of view are not particularly articulated here. It may be described in a next version. 5.1. Supporting PDU Session Type Variations Given that UP protocol is required to support all PDU session types: IPv4, IPv6, Ethernet, and Unstructured. However, it is expected that some deployment cases allow candidate protocol to adopt only one or few PDU session type(s) for simplicity of operations. As we can expect that IPv4 connectivity services will be available through IPv6-only PDU session that enabled by bunch of IPv6 transition solutions already available in the field. For this, the expected evaluation points from this aspect should be whether there is substitutional means to cover other PDU session types. And how much it makes simple the system than deploying original PDU session types. Homma, et al. Expires May 7, 2020 [Page 29] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 5.2. Nature of Data Path As it is described in Section 4.2, the single PDU session multi- homing case requires multipoint-to-point (MP2P) data path. It should be much scalable than multi-homing with multiple PDU sessions because number of required path states in the UPFs are reduced as closed to egress endpoint. Against that point-to-point (P2P) protocol requires same number of states in each UPF throughout the path, and it could increase explosively the load on management of tunnel states. From this point of view, the expected evaluation points from this aspect is whether the nature of candidate UP protocols are to utilize MP2P data path. Supporting MP2P data path by GTP-U could be a gap since GTP-U is a point-to-point tunneling protocol as it is described in Section 3. Noted that 3GPP CT WG4 pointed out GTP-U was already required to allow one single tunnel endpoint to receive packets from multiple source endpoints ([C4-185491-3GPP]). It was an architectural requirement of 3GPP system from a previous generation. It means that MP2P data path requirement for UP protocol has been existed before the 5G system. 5.3. Supporting Transport Variations The 5G system will be expected that the new radio spectrums in high frequency bands require operators to deploy their base stations much dense for much wider areas compare to previous generation footprints. To make sure that density and coverage, all available types of transport in the field must be employed between RAN to UPF, or UPF to UPF. It is also expected that MTU size of each transport could be varied. Because one could be own fiber which the operator configure the MTU size as they like while others are third-party provided L2/L3 VPN lines which MTU size can't be controlled by the operators. The MTU between RAN and UPF can be discovered by offline means and the operator takes into account the MTU that is transferable on the radio interface and based on this the operator configures the right MTU to be used. That is then signaled to the UE either via PCO (for IPv4 case) or the IPv6 RA message (for IPv6 case). In addition, for cases that third-parties provide VPN lines, it would be recommended MTU size discovery for each data path and dynamic MTU size adjustment mechanisms, while GTP-U does not support those mechanisms. Homma, et al. Expires May 7, 2020 [Page 30] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 As the study item in 3GPP mentioned, IPv6 is preferable address family not only for UEs, but also for the UP transport, in terms of size of available address space to support dense and wide footprint of base stations. However it increases header size from 20bytes to 40bytes compare to IPv4. It could be a problem if the MTU size is uncontrollable, or only limited MTU size available to carry committed PDU size on the user plane. The expected evaluation points from this aspect should be that the candidate protocols are able to dynamically adjust path MTU size with appropriate MTU size discovery mechanism. It also should be that how the candidate protocols leverage IPv6 to deal with header size increasing. 5.4. Data Path Management As Section 4.2 described, the 5G systems allows user plane that flexible UPF selection, multiple anchor UPFs, and no limit on how many UPFs chained for the data path of the PDU session. UPF deployments in the field will thereby be distributed to be able to optimize the data path based on various logics and service scenarios. That powerful user plane capability could make data path management through the control plane, or operation support systems (OSS) be complicated and difficult. Perhaps it could be the case where the UP protocol nature is P2P and it only supports per session base data path handling. Therefore it would be better that UP protocol could support to aggregate several PDU sessions into a tunnel or shall be a session-less tunnel. Because it increases data path states by number of sessions, and number of endpoints of UPFs that makes data path handling much hectic and the control plane tend to be overloaded by not only usual attach/detach/hand-over operations, but also existing session manipulation triggered by UPF and transport nodes/paths restoration, etc., The expected evaluation points from this aspect should be that how much the candidate protocols can reduce data path management loads both on the control plane NFs and UPFs compare to the per session based handling for P2P paths. It could possibly include N3 and N6 in addition to N9 while it supports flexible user plane data path optimizations for some example scenarios. Homma, et al. Expires May 7, 2020 [Page 31] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 5.5. QoS Control The QoS model is based on QoS flows to which QFI indicates in the 5G system that allows multiple QoS flows are aggregated into a single PDU session. So that it is given that the UP protocol should convey QFIs for a PDU session and the UPF needs to lookup them. It makes sure that reflects QoS policy in the 5G system to corresponding forwarding policy in the user plane IP transports. The expected evaluation points from this aspect should be whether the candidate protocols can provide stable ID space for QFI which shouldn't be a big deal since QFI just requires 6-bits space. As we pointed out in Section 3.2, the lookup process could impact UPF performance if the QFI container position in the header is unpredictable. It could happen many times along the path because the each UPFs should do it again and again in case that various different QoS policies are deployed in the networks under the UP as we discussed in Section 5.3. As [TS.29.281-3GPP] updated in version 15.3.0, it is recommended that the first extension header is the PDU session container in which QFI is present. 5.6. Traffic Detection and Flow Handling As described in Section 4.1.1, UPF need to detect traffic flow specified by SMF within a PDU session, and enforce some processes to the PDU based on the pre-configured policy rule. As similar with QoS flow lookup described in Section 5.5, UPFs along the path are repeatedly detecting an specified traffic flow in inner PDU. It could increase redundant flow detection load on every UPFs that could be avoided if the upstream UPF put some identifier which abstracts the detected flow into the packets. It enables following UPFs just find the ID to detect the indicated flow from the packet. The expected evaluation points from this aspect should be whether the candidate protocols can provide means to reduce that redundant flow detection that could be enough bits space on stable ID space to put abstracted detected flow identifier. 5.7. Supporting Network Slicing Diversity Network Instance has been defined as the glue of network slice between 5G and IP transport in addition to IP domain separation, as described in Section 4.1.2. It is expected that SMF is able to configure UPF to send UP packet to corresponding transport slice by Homma, et al. Expires May 7, 2020 [Page 32] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 indicating Network Instance in FAR so that UPF can determine outgoing interface for the UP packet. It is assumed that IP transport networks are Network Instance agnostic, i.e., transport slices are independently instantiated and not bound to specific IP address space in the 5GC, for preventing increase of routing table size. As a transport slice may be shared with multiple IP domains, Network Instance could be instantiated for all combination of IP domains and transport slice. To indicate those combination in UP packet over the wire, the 5G architecture expects VPN solutions as described in section 5.6.12 of [TS.23.501-3GPP]. Binding Network Instance with corresponding VPN would be varied per VPN solutions and FAR is not able to do. Hence it is out of scope of 3GPP and it may be covered by IETF, or other SDOs. Apart from binding, if it is the case where MPLS based VPNs, such as [RFC4364] and [RFC4664] are expected as the existing VPN solution which bound to Network Instance, there are some avaiable deployment options, such as 1). PE router integrates UPF, 2). CE router integrates UPF, 3). UPF connects to the VPN behind the CE router. Option 1 could work since all legacy MPLS or Segment Routing [RFC8402] based solution are available for both VPN and transport slicing at the UPF integrated PE router. However it is hard to expect it in multi-vendor deployment case, where the PE routers providing vendor is different from the vendor who provides UPFs, for example. Option 2 and 3 are expected as IP domain separation, but it is hard to see that it is able to indicate transport slice in addition to the IP domain. Other L2 and tunneling solutions should be same with those options. The expected evaluation points from this aspect should be whether the candidate protocols can contain forwarding information associated to the assigned IP domain and transport slice for all possible deployment cases. 5.8. Reliable Communication support As Section 4.2 described, more than two UP paths are required for a QoS flow of a PDU session between the anchor UPF and gNB. Those UP paths are to convey redundant duplicated packets. Homma, et al. Expires May 7, 2020 [Page 33] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 To support reliable communication with above requirements, UPF and gNB must replicate the sending UP packets and eliminate the received duplicated UP packets. Not to mention that UP protocol should be able to make sure that the paths are not in fate sharing, the UP packet must have sequence number to indicate duplicate packets per QoS flow basis. The expected evaluation points from this aspect should be whether the candidate protocols can indicate packet sequence and diversed paths in the context of QoS flow, not in UP tunnel context. The packet sequence information should be transparent through I-UPF(s) exist in the middle of the path even in case that the UP tunnels are terminated at the I-UPF(s). 6. Conclusion We analyzed the 3GPP specifications of the 5G architecture in terms of user plane and the current protocol adopted to the user plane. After the analysis work, we believe that the results described in this document shows that we reach at certain level of understanding on the 5G systems and ready to provide our inputs to 3GPP. We clarified GTP-U through the analysis and listed observed characteristics in Section 3.6. We also clarified the architectural requirements for UP protocol described in Section 4.2. Our conclusion here is that it is hopefull if the evaluation aspects described in Section 5 help for the study progress. It is worth to study possible candidate UP protocols for the 5G system including current one based from the aspects. 7. Security Consideration TBD 8. Acknowledgement The authors would like to thank Tom Herbert, Takashi Ito, John Leddy, Pablo Camarillo, Daisuke Yokota, Satoshi Watanabe, Koji Tsubouchi and Miya Kohno for their detailed reviews, comments, and contributions. A special thank you goes to Arashmid Akhavain for his technical review and feedback. Lastly, the authors would like to thank 3GPP CT WG4 folks for their review and feedback. Homma, et al. Expires May 7, 2020 [Page 34] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 9. Informative References [C4-185491-3GPP] 3rd Generation Partnership Project (3GPP), "LS OUT on User Plane Analysis", July 2018, . [CP-173160-3GPP] 3rd Generation Partnership Project (3GPP), "New Study Item on User Plane Protocol in 5GC", December 2017, . [CP-180116-3GPP] 3rd Generation Partnership Project (3GPP), "LS on user plane protocol study", March 2018, . [IAB-Statement] Internet Architecture Board (IAB), "IAB Statement on IPv6", November 2016, . [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460, December 1998, . [RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February 2006, . [RFC4664] Andersson, L., Ed. and E. Rosen, Ed., "Framework for Layer 2 Virtual Private Networks (L2VPNs)", RFC 4664, DOI 10.17487/RFC4664, September 2006, . [RFC4861] Narten, T., Nordmark, E., Simpson, W., and H. Soliman, "Neighbor Discovery for IP version 6 (IPv6)", RFC 4861, DOI 10.17487/RFC4861, September 2007, . [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, "IPv6 Flow Label Specification", RFC 6437, DOI 10.17487/RFC6437, November 2011, . Homma, et al. Expires May 7, 2020 [Page 35] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 [RFC6438] Carpenter, B. and S. Amante, "Using the IPv6 Flow Label for Equal Cost Multipath Routing and Link Aggregation in Tunnels", RFC 6438, DOI 10.17487/RFC6438, November 2011, . [RFC6935] Eubanks, M., Chimento, P., and M. Westerlund, "IPv6 and UDP Checksums for Tunneled Packets", RFC 6935, DOI 10.17487/RFC6935, April 2013, . [RFC7157] Troan, O., Ed., Miles, D., Matsushima, S., Okimoto, T., and D. Wing, "IPv6 Multihoming without Network Address Translation", RFC 7157, DOI 10.17487/RFC7157, March 2014, . [RFC8200] Deering, S. and R. Hinden, "Internet Protocol, Version 6 (IPv6) Specification", STD 86, RFC 8200, DOI 10.17487/RFC8200, July 2017, . [RFC8402] Filsfils, C., Ed., Previdi, S., Ed., Ginsberg, L., Decraene, B., Litkowski, S., and R. Shakir, "Segment Routing Architecture", RFC 8402, DOI 10.17487/RFC8402, July 2018, . [TR.29.891-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TR 29.891 (V15.0.0): 5G System Phase 1, CT WG4 Aspects", December 2017, . [TS.22.261-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 22.261 (V15.7.0): Service requirements for 5G system stage 1", December 2018, . [TS.23.060-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 23.060 (V15.3.0): General Packet Radio Service (GPRS); Service description; Stage 2", June 2018, . Homma, et al. Expires May 7, 2020 [Page 36] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 [TS.23.501-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 23.501 (V16.2.0): System Architecture for 5G System; Stage 2", September 2019, . [TS.23.502-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 23.502 (V15.4.0): Procedures for 5G System; Stage 2", December 2018, . [TS.23.503-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 23.503 (V15.4.0): Policy and Charging Control System for 5G Framework; Stage 2", December 2018, . [TS.28.530-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 28.530 (V15.1.0): Management and orchestration of networks and network slicing; Concepts, use cases and requirements (work in progress)", December 2018, . [TS.28.531-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 28.531 (V15.1.0): Management and orchestration of networks and network slicing; Provisioning; Stage 1 (Release 15)", December 2018, . [TS.28.532-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 28.532 (V15.1.0): Management and orchestration of networks and network slicing; Provisioning; Stage 2 and stage 3 (Release 15)", Decempber 2018, . Homma, et al. Expires May 7, 2020 [Page 37] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 [TS.28.533-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 28.533 (V15.1.0): Management and orchestration of networks and network slicing; Management and orchestration architecture (Release 15)", December 2018, . [TS.29.244-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 29.244 (V15.1.0): Interface between the Control Plane and the User Plane Nodes; Stage 3", December 2018, . [TS.29.274-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 29.274 (V15.4.0): 3GPP Evolved Packet System (EPS); Evolved General Packet Radio Service (GPRS) Tunneling Protocol for Control plane (GTPv2-C); Stage 3", June 2018, . [TS.29.281-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 29.281 (V15.5.0): GPRS Tunneling Protocol User Plane (GTPv1-U)", December 2018, . [TS.29.510-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 29.510 (V15.2.0): 5G System; Network Function Repository Services; Stage 3", December 2018, . [TS.29.561-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 29.561 (V15.1.0): 5G System; Interworking between 5G Network and external Data Networks; Stage 3", September 2018, . [TS.38.300-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 38.300 (v15.4.0): NR and NG-RAN Overall Description; Stage 2", December 2018, . Homma, et al. Expires May 7, 2020 [Page 38] Internet-Draft draft-ietf-dmm-5g-uplane-analysis-02 November 2019 [TS.38.401-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 38.401 (v15.4.0): NG-RAN; Architecture Description", December 2018, . [TS.38.415-3GPP] 3rd Generation Partnership Project (3GPP), "3GPP TS 38.415 (v15.2.0): NG-RAN; PDU Session User Plane protocol", December 2018, . Authors' Addresses Shunsuke Homma NTT Email: homma.shunsuke@lab.ntt.co.jp Takuya Miyasaka KDDI Research Email: ta-miyasaka@kddi-research.jp Satoru Matsushima SoftBank Email: satoru.matsushima@g.softbank.co.jp Daniel Voyer Bell Canada Email: daniel.voyer@bell.ca Homma, et al. Expires May 7, 2020 [Page 39]