QUIC Q. An Internet-Draft Y. Liu Intended status: Standards Track Y. Ma Expires: April 25, 2021 Alibaba Inc. Z. Li ICT-CAS October 22, 2020 Multipath Extension for QUIC draft-an-multipath-quic-00 Abstract This document specifies multipath extension for the QUIC protocol to enable the simultaneous usage of multiple paths for a single connection. The extension is compliant with the single-path QUIC design. The design principle is to support multipath by adding limited extension to QUIC-Transport [I-D.ietf-quic-transport]. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on April 25, 2021. Copyright Notice Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents An, et al. Expires April 25, 2021 [Page 1] Internet-Draft multipath-quic October 2020 carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Notational Conventions . . . . . . . . . . . . . . . . . . . 3 3. Sub-Connection . . . . . . . . . . . . . . . . . . . . . . . 4 4. Enable Multipath QUIC - Handshake . . . . . . . . . . . . . . 5 5. Sub-connection Management . . . . . . . . . . . . . . . . . . 5 5.1. Multipath QUIC Interaction . . . . . . . . . . . . . . . 5 5.2. Path validation and sub-connection ID negotiation . . . . 8 5.3. New sub-connection establishment . . . . . . . . . . . . 9 5.4. Close sub-connection . . . . . . . . . . . . . . . . . . 9 5.5. Sub-connection Lookup . . . . . . . . . . . . . . . . . . 10 5.6. Sub-connection migration . . . . . . . . . . . . . . . . 10 5.7. Sub-connection state machine management . . . . . . . . . 10 5.8. Sender and Receiver Connection (Sub-connection) States . 11 5.9. Use load balancer in Multipath QUIC . . . . . . . . . . . 11 6. Packet scheduling . . . . . . . . . . . . . . . . . . . . . . 11 6.1. Overview . . . . . . . . . . . . . . . . . . . . . . . . 11 6.2. Basic Static Scheduling Strategy . . . . . . . . . . . . 12 6.3. Dynamic (feedback-based) Scheduling Strategy . . . . . . 13 6.4. Application Policy-Awareness . . . . . . . . . . . . . . 14 6.4.1. Per-connection Policy . . . . . . . . . . . . . . . . 15 6.4.2. Per-stream Policy . . . . . . . . . . . . . . . . . . 17 7. Congestion control and loss detect . . . . . . . . . . . . . 18 7.1. Congestion control . . . . . . . . . . . . . . . . . . . 18 7.2. Packet number space and acknowledgements . . . . . . . . 18 7.3. Flow control . . . . . . . . . . . . . . . . . . . . . . 18 8. New frames . . . . . . . . . . . . . . . . . . . . . . . . . 18 8.1. MP_SUB_CONN_NEW frame . . . . . . . . . . . . . . . . . . 19 8.2. MP_SUB_CONN_ACCEPT frame . . . . . . . . . . . . . . . . 19 8.3. MP_SUB_CONN_CLOSE frame . . . . . . . . . . . . . . . . . 19 8.4. MP_ACK frame . . . . . . . . . . . . . . . . . . . . . . 20 8.5. MP_ADD_ADDRESS frame . . . . . . . . . . . . . . . . . . 21 8.6. MP_REMOVE_ADDRESS frame . . . . . . . . . . . . . . . . . 21 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 10. Informative References . . . . . . . . . . . . . . . . . . . 21 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 22 An, et al. Expires April 25, 2021 [Page 2] Internet-Draft multipath-quic October 2020 1. Introduction In this document, we propose an extension to the current QUIC design to enable the simultaneous usage of multiple paths for a single connection. This proposal differs from past proposals [I-D.deconinck-quic-multipath] in two fundamental perspectives: o The multi-path QUIC is built on top of the concept of the bidirectional sub-connection, which readily fits into the nature of both cellular and wifi links that cover the majority of multi- path applications in QUIC while keeping the design simple and easy to implement. In doing so, we are able to re-use most of the current QUIC transport design with the sole addition of six new frames. o The multi-path QUIC design enables feedback-based dynamic scheduling strategy. As the major goal of multi-path QUIC is to enhance performance in mobile applications, where the sender and receiver may have different viewpoints about the fast-changing wireless connectivity, especially in high-mobility scenarios, the proposed design allows the sender and receiver to synchronize their viewpoints via message exchange in ACK packet in order to maximize performance. This document is organized as follows. It first provides definition of sub-connection in Section 3. It then specifies how to enable multipath QUIC during handshake in Section 4, and sub-connection management in Section 5. It discusses packet scheduling in Section 6, and congestion control in Section 7. It specifies the new frames in Section 8. 2. Notational Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. We assume that the reader is familiar with the terminology used in [I-D.ietf-quic-transport]. In addition, we define the following terms: o Path: A sequence of links between a sender and a receiver, defined in this context by a 4-tuple of source and destination address/ port pairs[RFC8684]. An, et al. Expires April 25, 2021 [Page 3] Internet-Draft multipath-quic October 2020 o Sub-Connection: Sub-connection is bidirectional and provides reliable transmission between client and server. A connection can contain one or multiple sub-connections. A sub-connection is identified by an internal identifier, called Sub-Connection Index (SCI). Each sub-connection has its unique Source Connection ID and Destination Connection ID. The Connection ID is mapped with the 2-tuple of IP address and port. o (Multipath QUIC) Connection: A set of one or more sub-connections, over which an application can communicate between two host. 3. Sub-Connection A connection can contain one or multiple sub-connections which are bidirectional and provides reliable transmission between client and server. Sub-connection is identified by Sub-Connection Index (SCI). If a connection contains at least 2 sub-connections, then the first established sub-connection is called Initial sub-connection. The rest sub-connections are called supplementary sub-connections. Every sub-connection has its own unique CID pair that is associated with the 4-tuple (source IP, source port, destination IP, destination port) of the underlying network path currently used by the sub- connection. The Connection ID negotiation process is specified in Section 5.1. In case of sub-connection migration, the CID pair will be renegotiated following the connection migration procedure specified in [I-D.ietf-quic-transport]. Endpoints can find which sub-connection a received packet belongs to according to the CID pair of the packet. Endpoints can find the context of a sub-connection by its' CID pair or SCI. In the context of a sub-connection, a reference pointer MUST be provided to access the context of the multipath QUIC connection that the sub-connection belongs to. Each sub-connection has its independent Packet Number Space. And all sub-connections in the same connection share the same 1-RTT encryption key which is generated during the connection's cryptographic handshake. Note: The reason of using SCI to identify a Sub-connection: acknowledgements may not be transferred via the same sub-connection where the packets were sent, therefore the MP_ACK frame SHOULD contain field that can uniquely identify the sub-connection, and the same logic applies to other new MP frames. If we use Connection ID to identify a sub-connection in MP frames, the length of Connection ID is too long and will add more overhead in the frames. An, et al. Expires April 25, 2021 [Page 4] Internet-Draft multipath-quic October 2020 4. Enable Multipath QUIC - Handshake The connection handshake flow follows QUIC-Transport [I-D.ietf-quic-transport], using the transport parameter to negotiate multipath feature. The negotiation mechanism is similar to the negotiation in [I-D.deconinck-quic-multipath] Section 5.1, while the semantic of the transport parameter is different. A new transport parameter is defined: o name: max_sub_conn_index (0x40) o value: a variable-length integer (1 to 8 bytes) The value range and definition: o 0: MP feature disabled o [1, 2^31-1]: the maximum number of sub-connections The value of SCI(sub-connection index) starts from 1 and increases by 1 when a new sub-connection is created. The value range of SCI is [1, max_sub_conn_index]. The SCI of initial sub-connection is 1. A multipath QUIC connection MUST NOT reuse any used SCI for new sub- connections in its' lifetime. If the peer does not carry the max_sub_conn_index(0x40) transport parameter, which means the peer does NOT support multipath, endpoint MUST fallback to QUIC-Transport [I-D.ietf-quic-transport] with single path, and MUST NOT send any MP frames in the following packets. 5. Sub-connection Management This section describes the details of sub-connection management. 5.1. Multipath QUIC Interaction Figure 1 illustrates the Multipath QUIC interaction process. Server Client Server Path 2 Path 1 Initial (DCID=A, SCID=B, ClientHello(max_sub_conn_index=4)) -> Handshake(DCID=B, SCID=C, <- EE(max_sub_conn_index=4)) An, et al. Expires April 25, 2021 [Page 5] Internet-Draft multipath-quic October 2020 Pkt(DCID=B, SCID=C, <- frames=[New_connection_id(E)]) Pkt(DCID=C, SCID=B, frames=[New_connection_id(E)]) -> (Detect new path 2) Pkt(DCID=E, frames=[MP_SUB_CONN_NEW( <- SCI=2, X)]) Pkt(DCID=F, frames=[MP_SUB_CONN_ACCEPT( SCI=2, X)], PATH_CHALLENGE(Y)) -> Pkt(DCID=E, <- frames=[PATH_CHALLENGE(Y)]) (Data transmission) Pkt(DCID=E, pktnum=N1, <- frames=[STREAM("Request 2")]) Pkt(DCID=F, pktnum=N2, frames=[MP_ACK(SCI=2, N1), STREAM("Response 2")]) -> Pkt(DCID=C, pktnum=M1, frames=[STREAM("Request 1")]) -> Pkt(DCOD=E, pktnum=N3, frames=[MP_ACK(SCI=2, N2), <- STREAM("Request 3")]) Pkt(DCID=B, pktnum=M2, frames=[MP_ACK(SCI=1, M1), <- STREAM("Response 1")]) Pkt(DCID=F, pktnum=N4, frames=[MP_ACK(SCI=2, N3), STREAM("Response 3")]) -> Pkt(DCID=C, pktnum=M3, frames=[MP_ACK(SCI=1, M2), MP_ACK(SCI=2, N4)]) -> An, et al. Expires April 25, 2021 [Page 6] Internet-Draft multipath-quic October 2020 Figure 1: Multipath QUIC interaction process The process is composed of four phases. A. Handshake negotiation During the QUIC-Transport [I-D.ietf-quic-transport] handshake, endpoints negotiate whether multipath feature is supported. The negotiation parameter (see Section 4) is carried within the transport parameters of TLS cryptographic handshake. After the handshake finished, the connection contains the initial sub-connection with SCI equals 1. In Figure 1, the maximum sub-connection index is four. B. Exchange unused Connection ID in advance After the two endpoints complete the connection establishment, they can exchange unused Connection IDs by NEW_CONNECTION_ID frame. Before an endpoint starts to create a new sub-connection, it SHOULD check if there are unused Connection IDs for both endpoints. Note: QUIC-Transport [I-D.ietf-quic-transport] requires Connection ID is uniquely mapped with 2-tuple of IP address and port. If client attempts to use a new 2-tuple as source address to establish a new sub-connection, a new Connection ID is required for client, and also a new Connection ID is required for server. C. New sub-connection establishment During this phase, a new sub-connection is established between client and server, and address validation is needed. When client detects a new network path, it MAY attempts to establish a new sub-connection by sending MP_NEW_SUB_CONN frame which carries a 64-bit random value and claims the new sub-connection's SCI (which is 2 in the example flow in Figure 1). The establishment of sub- connection is always initiated by client. After the server receives the MP_NEW_SUB_CONN frame from the client, it responds with MP_SUB_CONN_ACCEPT frame which carries the identical 64-bit random value from the received MP_NEW_SUB_CONN frame and agrees with the sub-connection's SCI (2 in the example). Server MUST also perform path validation following the procedure specified in QUIC-Transport [I-D.ietf-quic-transport]. Once the server successfully validates its' peers' address, the new sub-connection is established. D. Data transmission on new sub-connection An, et al. Expires April 25, 2021 [Page 7] Internet-Draft multipath-quic October 2020 As soon as sub-connections are established, endpoints can communicate with each others over the newly established sub-connections. All valid short header packets defined in QUIC-Transport [I-D.ietf-quic-transport] can be carried on these sub-connections. Every sub-connection has its' independent PNS. Thus, standard QUIC ACK frames defined in QUIC-Transport [I-D.ietf-quic-transport] only acknowledge packets that belong to the same PNS of the sub-connection on which the ACK frames were received. To enable endpoints reply acknowledgements on different sub- connections rather than the sub-connection where the corresponding packets were received, a new type of frame, MP_ACK, is defined. MP_ACK frames can also be replied over the same sub-connection on which data packets were received. In this case, MP_ACK frames serves very similar purposes as QUIC ACK frames do. MP_ACK frame contains the sub-connection index of the packets to be acknowledged. For example, in Figure 1, the packet (packet number is N4) is sent via the second sub-connection (SCI is 2), and its corresponding acknowledgement MP_ACK(Sub-Connection Index=2, N4) is sent via the initial sub-connection. 5.2. Path validation and sub-connection ID negotiation Before clients initiate new sub-connections by sending MP_SUB_CONN_NEW frames to servers through their additional network addresses, they MAY want to validate the reachability between their new network addresses and servers' addresses. In this case, clients can initiate a path validation procedure as specified in QUIC- Transport [I-D.ietf-quic-transport] per address pair. Path validation uses the PATH_CHALLENGE and PATH_RESPONSE frame defined in QUIC-Transport [I-D.ietf-quic-transport]. Each sub-connection MUST has a unique pair of SCID and DCID within a multipath QUIC connection. Thus, endpoints MUST NOT initiate or accept new sub-connections if they currently have no free CIDs supplied by their peers. In this case, endpoints SHOULD announce new free CIDs to their peers by exchanging NEW_CONNECTION_ID frames. To ensure that endpoints have free CIDs to create new sub-connections as soon as they get new network addresses, an endpoint SHOULD announce a least one free CID to its peer by sending NEW_CONNECTION_ID frame [I-D.ietf-quic-transport] over its initial sub-connection as soon as the handshake on the initial sub-connection is completed. Endpoints MAY also track the number of free CIDs that their peers can use and announce more free CIDs if needed. An, et al. Expires April 25, 2021 [Page 8] Internet-Draft multipath-quic October 2020 Sub-connection ID negotiation follows the Connection ID negotiation method in Connection Migration defined in QUIC-Transport [I-D.ietf-quic-transport], which is to let client and server claim its own unused Connection ID in advance by NEW_CONNECTION_ID frame. If there is no available unused Connection ID, then establishment of new sub-connection is not allowed. 5.3. New sub-connection establishment New sub-connection establishment is always initiated by client, by sending MP_NEW_SUB_CONN frame. Because source address(2-tuple of IP address and port) is usually different in the new network path, client needs to generate and claim new Source Connection IDs prior to the new sub-connection establishment. Client that sends the MP_SUB_CONN_NEW frame in 1-RTT packets with short headers, MUST use the unused Connection ID claimed in advance by server as Destination Connection ID. MP_SUB_CONN_NEW frame carries a 64-bit random value, and a SCI (increased progressively). After receiving the MP_SUB_CONN_NEW frame, server responds with MP_SUB_CONN_ACCEPT frame carrying the identical SCI and identical 64-bit random value from the received MP_NEW_SUB_CONN frame. Then, server sends PATH_CHALLENGE to verify the client address. After client receives the PATH_CHALLENGE frame, it replies with PATH_RESPONSE frame In the following 1-RTT packet (short header) to complete the address validation. After the address validation is completed, client and server can send and receive data unrestrictedly on the established sub-connection. Before the client's address validation is completed, server needs to limit the cumulative size of packets it sends to an unvalidated address to three times the size of packets it receives from that address in the new sub-connection (to prevent amplification attack). 5.4. Close sub-connection Both client and server can terminate a sub-connection, by sending MP_SUB_CONN_CLOSE frame that carries a SCI. In scenarios such as client detects the network environment change (client's 4G/Wi-Fi is turned off, Wi-Fi signal is fading to a threshold), or endpoints detect that the quality of RTT or loss rate is becoming worse, client or server can terminate a sub-connection immediately. An, et al. Expires April 25, 2021 [Page 9] Internet-Draft multipath-quic October 2020 MP_SUB_CONN_CLOSE frame can be sent via a different sub-connection instead of the sub-connection to be closed. 5.5. Sub-connection Lookup Endpoints use Connection IDs to find the context of a connection. Figure 2 illustrates the Connection context. Each sub-connection's Connection IDs can be mapped to the connection. Connection +--------------------------------------------------------------+ | | | Sub-Conn 1 | | | +------+ IP 1, Port 1 SCI 1 - SCID A, DCID B IP 3, Port 3 +------+ | |<----------------------------------------------------->| | |Client| |Server| | |<----------------------------------------------------->| | +------+ IP 2, Port 2 SCI 2 - SCID C, DCID D IP 3, Port 3 +------+ | | | Sub-Conn 2 | | | +--------------------------------------------------------------+ Figure 2: Connection context In the connection context, client and server can use SCI or Connection ID to find a sub-connection. Note that if sub-connection migration happens, sub-connection's Connection ID need to be renegotiated (See Section 5.6), while the SCI of sub-connection could remain unchanged. 5.6. Sub-connection migration Sub-connection migration follows the Connection Migration defined in QUIC-Transport [I-D.ietf-quic-transport]. When client experiences NAT rebinding (source address is changed), server needs to revalidate the client address. 5.7. Sub-connection state machine management TODO An, et al. Expires April 25, 2021 [Page 10] Internet-Draft multipath-quic October 2020 5.8. Sender and Receiver Connection (Sub-connection) States For each sender and receiver, the sub-connection states include: +---------+----------------------+----------+--------------+--------+ | Sender | SubConnectionIndex(S | CIDs(SCI | 4-tuple(sIP, | packet | | | CI) | D, DCID) | dIP, sPort, | number | | | | | dPort) | space | +---------+----------------------+----------+--------------+--------+ | Receive | SubConnectionIndex(S | CIDs(SCI | 4-tuple(sIP' | packet | | r | CI) | D, DCID) | , dIP', | number | | | | | sPort', | space | | | | | dPort') | | +---------+----------------------+----------+--------------+--------+ Table 1: Sender and Receiver Connection (Sub-connection) States 5.9. Use load balancer in Multipath QUIC This specification follows the Connection ID negotiation defined in QUIC-Transport [I-D.ietf-quic-transport]. For stateless or low-state load balancers supporting Multipath QUIC, implementations SHOULD use the specification of Connection ID generation and Load balancer routing defined in QUIC-LB [I-D.ietf-quic-load-balancers], guarantee that packets with Connection IDs belonging to the same connection, can be routed to same server. 6. Packet scheduling 6.1. Overview For an outgoing packet, the packet scheduler decides which sub- connection the packet shall be transmitted. The concept of packet scheduler in Multipath QUIC is similar to that in MPTCP. As long as more than one path's congestion controller allows for a new packet transmission, the packet scheduler is enabled. However, the proposed packet scheduler in this draft differs from past MPTCP proposals in the following aspects: o We enable dynamic (feedback-based) scheduling strategy with feedback in this proposal to further enhance quality of user experience(QoE) and to facilitate the expression of the application policy-awareness. Such a capability is made available by adding the QoE control signal length field and QoE control signal field in MP_ACK frame (see Section 8.4). With the help of such extension, a receiver is able to interact with a sender's scheduling strategy in real time. An, et al. Expires April 25, 2021 [Page 11] Internet-Draft multipath-quic October 2020 o Unlike MPTCP which send ACK packet over the same path, multipath QUIC allows a packet to be acknowledged over a different path, which allows multipath QUIC to better handle the uplink-downlink heterogeneity in wireless networks. o We support application policy-awareness in multipath QUIC. An application can implement both per-connection policies and per- stream policies. For example, a live streaming application is allowed to choose a different policy from a web application. The per-connection policy includes path mode, path preference, scheduling algorithm and packet redundancy strategy. A per-stream policy is associated with user-defined stream priorities to express the applications's intent. 6.2. Basic Static Scheduling Strategy A basic static scheduling strategy consists of four major components: o Path mode: A scheduler may want to decide which sub-connections shall be activated to transmit data. For instance, a scheduler can choose to use only one of the two sub-connections and completely ignore the other one. A scheduler marks the selected sub-connections to be in the "active" state and the un-selected ones in the "inactive" state. o Path preference: Due to the fact that costs of transmitting data over different sub-connections are not always equal. For example, the energy (battery) cost over a 5G sub-connection and a Wi-Fi sub-connection are very different, so a user may prefer the Wi-Fi sub-connection when his/her cell phone's battery is low. In another example, transmissions over a Wi-Fi sub-connection and a cellular sub-connection may incur different service charges per packet such that a user prefers to use the Wi-Fi sub-connection over the LTE one. Note that a user's preference may change over time. For instance, certain mobile carriers offer unlimited free data for a particular streaming app. Therefore, the sub- connection priority should be made available in the scheduler. o Sub-connection selection algorithm:A selection algorithm splits packets across different sub-connections and determines the order of sub-connections to be selected. The selection algorithm takes congestion controller states as inputs, such as smoothed RTTs (sRTTs), estimated bandwidths (eBWs) and congestion window sizes (CWNDs) as well as application-defined information such as sub-connection priorities and path states. The outputs of the algorithm is an ordered list of sub-connections to put a packet on. To name a few, some of the commonly used algorithms are: An, et al. Expires April 25, 2021 [Page 12] Internet-Draft multipath-quic October 2020 o * Round-Robin: There is no priority. it selects sub-connections one by one in order to transmit data. * Lowest-RTT: It first chooses the sub-connection with the lowest RTT and feeds packets to it until that sub-connection's congestion window is full. Then it chooses the sub-connection with the second lowest RTT. * Highest-Sending-Rate: It first chooses the sub-connection with the highest bandwidth and feeds packets to it until that sub- connection's congestion window is full. Then it chooses sub- connection with the second largest bandwidth. o Packet redundancy strategy: One major challenge in multi-path transmission is that a packet loss on the slow sub-connection might block the overall transmission when packets are split across fast-changing sub-connections. As the sub-connection selection algorithm takes inputs from congestion controllers which are basically rough predictions of the network and may not be accurate enough for fast-changing wireless channels, such an imprecise estimation could lead to network overuse/underuse. A solution to this problem is to implement packet redundancy strategy. A redundancy strategy can be applied to only ACK packets (partial redundancy) or all data packets (full redundancy). The multipath QUIC in this draft is designed to enable such flexible redundancy strategies. It is up to the application to determine whether, when, and on which packets to activate transmission redundancy. 6.3. Dynamic (feedback-based) Scheduling Strategy An important feature of this proposal is the capability of dynamic (feedback-based) scheduling. In a dynamic scheduling strategy, a receiver notifies its currently preferred scheduling strategy to a sender. Such feedback information is carried by QoE control signal in MP_ACK frames. The frequency of such feedback can be controlled to limit the amount of extra information. To do so, four types of MP_ACK frames are designed (Figure 8): o Type(i) = 0x22 , with no ECN Counts and no QoE Control Signals o Type(i) = 0x23 , with ECN Counts and no QoE Control Signals o Type(i) = 0x24 , with no ECN Counts and QoE Control Signals o Type(i) = 0x25 , with ECN Counts and QoE Control Signals An, et al. Expires April 25, 2021 [Page 13] Internet-Draft multipath-quic October 2020 The type 0x24 and 0x25 give the flexibility of carrying QoE control signals. Given that the sender and the receiver may have different views of the wireless environments, especially in high-mobility scenarios, the QoE control signal allows a synchronization between their viewpoints dynamically. It is up to the application to determine the interpretation of QoE control signal and its encoding method. 6.4. Application Policy-Awareness Applications may have completely different QoE requirements---the interactive applications are delay sensitive, while the video streaming applications are more throughput sensitive. There is thus a trend of cross-layer design that tries to take applications' demands into account when managing paths or scheduling packets. The static scheduling strategy and the dynamic scheduling strategy are used together to fully support application policy-awareness in multipath scheduling. To be more specifically, a 'control plane' is separated from a `data plane' as in software-defined networking. The 'control plane' takes applications' high-level demands (a.k.a intent) as input to generate the corresponding policies, which later are deployed on the 'data plane'. The 'data plane' maps users policies to the 'actions', which control the packet scheduler and other functionalities that the transport implements. To allow maximum design flexibility, the proposed multipath QUIC let applications to access/change every single logic of the packet scheduling and path management. The application policy consists of two layers: per- connection policy and per-stream policy. An, et al. Expires April 25, 2021 [Page 14] Internet-Draft multipath-quic October 2020 per-stream intent per-conn policy Control e.g. stream prioirity e.g. path preference, path mode, Plane deadline-aware etc | | -----------|----------------------------------------|------------- V V Data +--------+ - +------------------------+ Plane | | | | | +--------+ | | +-----+ | | | +->| ... | | +--------+ | | | +-----+ | | | | | | +-----+ | +--------+ | | +->| ... | | | | | +-----+ | +--------+ | +----------+ chunks |+----------+ | | | | ->| stream |------->|| packet |-+ ... | +--------+ | |scheduling| ||scheduling| | | ... | +----------+ |+----------+ | +-----+ | +--------+ | | +->| ... | | | | | | | +-----+ | +--------+ | | | +-----+ | | | +->| ... | | +--------+ | | +-----+ | | | | | path manager | +--------+ - +------------------------+ streams Figure 3: Application Policy Awareness in Multi-path QUIC framework 6.4.1. Per-connection Policy An application imposes per-connection policy through the primitives provided by the control plane. As described above, the policy is translated into indications on sub- connection states, sub-connection priorities, sub-connection selection algorithms and packet redundant strategies. The packet scheduler at the data plane will act based on these indications. We assume the policies are 'soft'---the policies are not a must. Instead, the data plane will follow the policies as much as possible. An, et al. Expires April 25, 2021 [Page 15] Internet-Draft multipath-quic October 2020 +-----+-----------------+-------------+------------+----------------+ | No. | Application | Application | Underlying | Underlying | | | defined policy: | defined | action: | action: Path | | | Path mode | policy: | Packet | mngm. | | | | Path | Scheduling | | | | | Preference | | | +-----+-----------------+-------------+------------+----------------+ | 1 | Wi-Fi=full, | Wi-Fi=1, | full | / | | | Cellular=full | Cellular=1 | redundant | | | | | | | | | 2 | Wi-Fi=full, | Wi-Fi=1, | full | activate | | | Cellular=backup | Cellular=1 | redundant | backup | | | | | | interfaces | | | | | | when the | | | | | | active one's | | | | | | performance is | | | | | | lower than X | | | | | | for 5s | | | | | | | | 3 | Wi-Fi=full, | Wi-Fi=2, | partially | / | | | Cellular=full | Cellular=1 | redundant | | +-----+-----------------+-------------+------------+----------------+ Table 2: Example policies of a real-time interaction application Let us take real-time interaction applications as an example to illustrate the basic idea. The applications are indeed delay sensitive but data volume is often low. 3 types of policies may be used by different applications, as shown in Table 2 where we assume only two paths are available (Wi-Fi and Cellular) The first type of policies would like to use two paths equally, and because the applications are delay sensitive, the actions will be to active 'full redundancy' for the packet redundancy strategy---two paths send the same data. The second type of policies, on the other hand, would like to use the Wi-Fi interface (possibly because of data charge) as much as possible, hence giving the Wi-Fi sub-connection a higher priority. But if two paths have to be activated at the same time due to the lower performance of Wi-Fi, then the two paths are set with same the priority which can be configured dynamically through QoE control signal in MP_ACK feedbacks. The third type of policies would like to use the two interfaces at the same time, but Wi-Fi is preferred twice as the cellular one. The actions will take this into consideration, by implementing a weighed round-robin sub- connection selection algorithm. An, et al. Expires April 25, 2021 [Page 16] Internet-Draft multipath-quic October 2020 Likewise, we can define a mapping between the policies of different types of applications and the actions in the data plane. We leave the design of such a mapping to the designers. 6.4.2. Per-stream Policy Per-stream intent is a unique feature provided by (MP)QUIC---it is implemented through the multiple streams in QUIC. Streams can be associated with priorities to implement applications intent. For instance, objects in a web page may be dependent on others and thus have different priorities [MPQUIC-Scheduler]. A stream priority- aware packet scheduling algorithm will improve the performance notably. High priority /\ +---------+ || | | || +---------+ || +---------+ || | | || +---------+ || ... User-defined priority || +---------+ Low priority || | | || +---------+ ----------------------------------------------------------- High priority /\ +---------+ || | | || +---------+ || +---------+ || | | || +---------+ || ... Default priority || +---------+ Low priority || | | || +---------+ Figure 4: Stream priority We envision a priority management scheme of two separated priority ranges (see Figure 4). The user-defined priority ranges are those streams that the applications explicitly designate the priorities, where the default priority ranges include the streams with no priority values set by the applications. Only when the streams in the user-defined ranges have no data sent, the data in the streams in the default priority ranges can be sent. In the same range, one can use the weighted round robin for scheduling---the higher-priority An, et al. Expires April 25, 2021 [Page 17] Internet-Draft multipath-quic October 2020 streams get more quantum for data sending in each round. One can also dynamically set/change the priorities of the streams in the default priority ranges to enable short stream first if needed. 7. Congestion control and loss detect 7.1. Congestion control Implementations MAY support coupled congestion controllers such as LIA [MPTCP-LIA], OLIA [MPTCP-OLIA]s, and etc., or support decoupled congestion controllers in environments using disjoint network paths. In decoupled congestion control, every sub-connection runs its own congestion controller without interacting with the congestion controllers of other sub-connections. That is to say, in the aspect of congestion control, a sub-connection behaves exactly the same as a normal QUIC connection over the same network path. Each sub-connection MAY choose congestion control algorithm independently. 7.2. Packet number space and acknowledgements Every sub-connection has its' own packet number space for transmitting 1-RTT packets. ACK frame [I-D.ietf-quic-transport] MUST be returned via the same sub-connection on which the corresponding packets were sent. MP_ACK frame can be returned via either a different sub-connection, or the same sub-connection, based on different strategies of sending MP_ACK frames. Note: Only MP_ACK frame returned via the same sub-connection can be used to calculate RTT(round trip time). 7.3. Flow control TODO 8. New frames All the MP frames MUST be sent in 1-RTT packet, and MUST NOT use other encryption levels. If an endpoint receives MP frames from packets of other encryption levels, it MAY return MP_PROTOCOL_VIOLATION as a connection error and close the connection. An, et al. Expires April 25, 2021 [Page 18] Internet-Draft multipath-quic October 2020 8.1. MP_SUB_CONN_NEW frame MP_SUB_CONN_NEW frame(type=0x2a) is used to establish a new sub- connection. The MP_SUB_CONN_NEW frame will specify a SCI and include a 64-bit random value. MP_SUB_CONN_NEW frames are formatted as shown in Figure 5. MP_SUB_CONN_NEW Frame { Type (i) = 0x2a, Sub_Connection_Index (i), Data (64), } Figure 5: MP_SUB_CONN_NEW Frame Format 8.2. MP_SUB_CONN_ACCEPT frame MP_SUB_CONN_ACCEPT frame (type=0x2b) is used by endto accept a new sub-connection, as a response to MP_NEW_SUB_CONN frame. MP_SUB_CONN_ACCEPT frames are formatted as shown in Figure 6, which is identical to the MP_NEW_SUB_CONN frame (Section 8.1). MP_SUB_CONN_ACCEPT Frame { Type (i) = 0x2b, Sub_Connection_Index (i), Data (64), } Figure 6: MP_SUB_CONN_ACCEPT Frame Format 8.3. MP_SUB_CONN_CLOSE frame MP_SUB_CONN_CLOSE frame(type=0x2c..0x2d) is used to close a sub- connection, which is formatted by adding a SCI field to QUIC- Transport [I-D.ietf-quic-transport] CONNECTION_CLOSE frame. The SCI is used to distinguish sub-connections, so each sub-connection can be closed independently. MP_SUB_CONN_CLOSE frames are formatted as shown in Figure 7. An, et al. Expires April 25, 2021 [Page 19] Internet-Draft multipath-quic October 2020 MP_SUB_CONN_CLOSE Frame { Type (i) = 0x2c..0x2d, Sub_Connection_Index (i), Error Code (i), [Frame Type (i)], Reason Phrase Length (i), Reason Phrase (..), } Figure 7: MP_SUB_CONN_CLOSE Frame Format 8.4. MP_ACK frame MP_ACK frame allows for acknowledgements on different sub- connections. MP_ACK frame is formatted by adding a SCI field and QoE signal fields to QUIC-Transport [I-D.ietf-quic-transport] ACK frame. MP_ACK frames are formatted as shown in Figure 8. MP_ACK Frame { Type (i) = 0x22..0x23..0x24..0x25, Sub_Connection_Index(i), Largest Acknowledged (i), ACK Delay (i), ACK Range Count (i), First ACK Range (i), ACK Range (..) ..., [ECN Counts (..)], [QoE Control Signals Length(8)], [QoE Control Signals (..)], } Figure 8: MP_ACK Frame Format Type(i) = 0x22 , with no ECN Counts and no QoE Control Signals Type(i) = 0x23 , with ECN Counts and no QoE Control Signals Type(i) = 0x24 , with no ECN Counts and QoE Control Signals Type(i) = 0x25 , with ECN Counts and QoE Control Signals An, et al. Expires April 25, 2021 [Page 20] Internet-Draft multipath-quic October 2020 8.5. MP_ADD_ADDRESS frame TODO 8.6. MP_REMOVE_ADDRESS frame TODO 9. IANA Considerations This document makes no request of IANA. 10. Informative References [I-D.deconinck-quic-multipath] De Coninck, Q. and O. Bonaventure, "Multipath Extensions for QUIC (MP-QUIC)", draft-deconinck-quic-multipath-05 (work in progress), August 2020. [I-D.ietf-quic-load-balancers] Duke, M. and N. Banks, "QUIC-LB: Generating Routable QUIC Connection IDs", draft-ietf-quic-load-balancers-04 (work in progress), August 2020. [I-D.ietf-quic-transport] Iyengar, J. and M. Thomson, "QUIC: A UDP-Based Multiplexed and Secure Transport", draft-ietf-quic-transport-32 (work in progress), October 2020. [MPQUIC-Scheduler] Wang, J., Gao, Y., and C. Xu, "A Multipath QUIC Scheduler for Mobile HTTP/2", Proceedings of the 3rd Asia-Pacific Workshop on Networking 2019 (APNet '19). Association for Computing Machinery, New York, NY, USA, 43-49., 2019, . [MPTCP-LIA] Raiciu, C., Handly, M., and D. Wischik, "Coupled Congestion Control for Multipath Transport Protocols", RFC 6365, October 2011, . [MPTCP-OLIA] Khalili, R., Gast, N., and J. Boudec, "Opportunistic Linked-Increases Congestion Control Algorithm for MPTCP", July 2014, . An, et al. Expires April 25, 2021 [Page 21] Internet-Draft multipath-quic October 2020 [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8684] Ford, A., Raiciu, C., Handley, M., Bonaventure, O., and C. Paasch, "TCP Extensions for Multipath Operation with Multiple Addresses", RFC 8684, March 2020, . Authors' Addresses Qing An Alibaba Inc. Email: anqing.aq@alibaba-inc.com Yanmei Liu Alibaba Inc. Email: miaoji.lym@alibaba-inc.com Yunfei Ma Alibaba Inc. Email: yunfei.ma@alibaba-inc.com Zhenyu Li ICT-CAS Email: zyli@ict.ac.cn An, et al. Expires April 25, 2021 [Page 22]