Internet DRAFT - draft-lachosrothenberg-alto-brokermdo

draft-lachosrothenberg-alto-brokermdo







ALTO WG                                                        D. Lachos
Internet-Draft                                             C. Rothenberg
Intended status: Informational                                   Unicamp
Expires: January 14, 2021                                  July 13, 2020


         ALTO-based Broker-assisted Multi-domain Orchestration
                draft-lachosrothenberg-alto-brokermdo-04

Abstract

   Evolving networking scenarios (e.g., 5G) demand new multiple
   administrative domain (aka multi-domain) orchestration models.  This
   document proposes a new broker-plane approach working on top of per-
   domain management and orchestration functions to coordinate the
   delivery of a multi-operator End-to-End Network Service (E2ENS).
   This proposed design resorts to the Application-Layer Traffic
   Optimization (ALTO) protocol to offer topology and resource
   information from different administrative domains.  The ALTO services
   with the proposed protocol extension offer aggregated views on
   various types of resources contributing to a more simple and scalable
   solution.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on January 14, 2021.

Copyright Notice

   Copyright (c) 2020 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (https://trustee.ietf.org/license-info) in effect on the date of



Lachos & Rothenberg     Expires January 14, 2021                [Page 1]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . .   4
   4.  Problem Statement and Challenges  . . . . . . . . . . . . . .   5
   5.  Proposed Approach . . . . . . . . . . . . . . . . . . . . . .   6
     5.1.  Inter-domain Resource (IdR) Component . . . . . . . . . .   7
     5.2.  Inter-domain Topology (IdT) Component . . . . . . . . . .   7
     5.3.  ALTO Server Functionalities . . . . . . . . . . . . . . .   7
     5.4.  Filtered Cost Map Extension . . . . . . . . . . . . . . .   8
       5.4.1.  Accept Input Parameters . . . . . . . . . . . . . . .   8
       5.4.2.  Response  . . . . . . . . . . . . . . . . . . . . . .   9
     5.5.  Examples of Message Exchange  . . . . . . . . . . . . . .   9
       5.5.1.  Property Map Service  . . . . . . . . . . . . . . . .   9
       5.5.2.  Filtered Cost Map Service . . . . . . . . . . . . . .  10
   6.  Discussion  . . . . . . . . . . . . . . . . . . . . . . . . .  13
     6.1.  Benefits  . . . . . . . . . . . . . . . . . . . . . . . .  14
     6.2.  Open Issues . . . . . . . . . . . . . . . . . . . . . . .  14
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  15
   8.  Security Considerations . . . . . . . . . . . . . . . . . . .  15
   9.  Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  15
   10. References  . . . . . . . . . . . . . . . . . . . . . . . . .  15
     10.1.  Normative References . . . . . . . . . . . . . . . . . .  15
     10.2.  Informative References . . . . . . . . . . . . . . . . .  16
     10.3.  URIs . . . . . . . . . . . . . . . . . . . . . . . . . .  18
   Appendix A.  Proof of Concept Use Case Implementation . . . . . .  18
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  22

1.  Introduction

   Envisioned 5G network architectures and related service models
   consider broader cooperation between stakeholders in order to provide
   flexible multi-operator multi-domain services.  These multi-provider
   orchestration operations will require the information exchange across
   Multi-domain Orchestrators (MdOs).  The key information to be
   exchanged between MdOs includes the abstract network topology,
   resource availability (e.g., CPUs, Memory, and Storage) and
   capability (e.g., supported network functions).





Lachos & Rothenberg     Expires January 14, 2021                [Page 2]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   This document presents a federation networking paradigm where a
   broker-plane works on top of the management and orchestration plane
   to assist and coordinate the creation of an End-to-End Network
   Service (E2ENS) spanning over multi-operator multi-domain networks.
   Our design resorts to the Application-Layer Traffic Optimization
   (ALTO) protocol [RFC7285] to address the lack of abstractions to
   discover and adequately represent in confidentiality-preserving
   fashion the resource and topology information from different
   administrative domains.  Moreover, this draft introduces an extension
   to the ALTO base protocol for inter-domain connectivity information
   discovery.

2.  Terminology

   We use the following definitions, as established in [ETSI-NFV-DEF]:

   Administrative Domain:  Collection of systems and networks operated
      by a single organization or administrative authority.

   Network Function (NF):  Functional block within a network
      infrastructure that has well-defined external interfaces and well-
      defined functional behaviour.

   Network Functions Virtualisation (NFV):  The principle of separating
      network functions from the hardware they run on by using virtual
      hardware abstraction.

   NF Forwarding Graph: (NFFG):  Graph of logical links connecting NF
      nodes for the purpose of describing traffic flow between these
      network functions.

   Network Service Orchestration (NSO):  Function responsible for
      network service lifecycle management.

   Resource Orchestration (RO):  Function responsible for global
      resource management governance.

   Our proof of concept implementation follows the architectural
   proposal of the 5GEx project [H2020.5GEX].  Some additional 5GEx
   terms commonly used in this document are defined below:

   Domain Orchestrator (DO):  Performs Resource Orchestration and/or
      Service Orchestration within the same administrative domain.

   Multi-domain Orchestrator (MdO):  Coordinates resource and/or service
      orchestration at multi-domain level, where multi-domain may refer
      to multiple DOs or multiple administrative domains.




Lachos & Rothenberg     Expires January 14, 2021                [Page 3]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   Resource Topology (RT):  Functional module that is responsible for
      keeping an updated global view of the underlying infrastructure
      topology exposed by DOs.

   Service Graph (SG):  A high-level data model for defining flexible
      network services (including traffic steering primitives).

   Service Access Point (SAP):  A named/tagged port supporting stitching
      (service to service, domain to domain, etc.)

3.  Scope

   Envisioned 5G scenarios are expected to work not only with
   heterogeneous technologies but also across different network
   operators.  Many ongoing standardization activities and research
   projects are addressing the multi-provider multi-domain orchestration
   challenges under different approaches.

   For example, within the IETF, [RFC8459] proposes a hierarchical
   Service Function Chaining (SFC) for multiple domains under the same
   administrative entity, and the document "Hybrid Hierarchical Multi-
   Domain SFC [DRAFT-HHSFC] describes SFC crossing different domains
   owned by various organizations or by a single organization with
   administration partitions.  In the NFVRG, the draft "Multi-domain
   Network Virtualization" [DRAFT-MD-VIRT] envisions a complete E2E
   logical network as stitching services offered by multiple domains
   from multiple providers.  Another initiative is the ETSI Industry
   Specification Group for Network Functions Virtualization (ETSI NFV
   ISG), the document [ETSI-NFV-IFA028] reports different NFV MANO
   architectural approaches with use cases related to network services
   provided using multiple administrative domains.

   In case of research projects,  [H2020.5GEX] [H2020-5G-TRANSFORMER]
   seek to integrate multiple administrations and technologies through
   the collaboration between operations.  Other studies are envisioned
   to use a centralized approach, where each domain advertises its
   capabilities to a federation layer which will act as a
   broker [VITAL][T-NOVA].  The proposed architecture in [ICAF] allows
   the creation of cloud services from different administrative domains,
   however, it is not related to the provisioning of NFV-based cross-
   domain network services.

   All such proposals described above envision the potential
   introduction of new business model approaches, including federation
   models [PPP-5:2013] among administrative domains.  In this context,
   this document considers each network operator involved in the
   community advertises its abstracted capabilities (e.g., software/
   hardware resources, physical/virtual network functions, etc.) to a



Lachos & Rothenberg     Expires January 14, 2021                [Page 4]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   broker (i.e., 3rd party).  This broker, in its turn, provides or
   assists coordinate E2E network services spanning multi-domain
   networks.

4.  Problem Statement and Challenges

   The provision of a complete E2E network service requires chaining
   services provided by multiple network operators with multiple
   technologies.  In this multi-domain environment, the orchestration
   process will require an advertisement mechanism through which
   individual domains can describe their capabilities, resources, and
   VNFs in an interoperable manner.  Moreover, a discovery mechanism is
   also necessary so that source domains can obtain candidate domains
   (with the corresponding connectivity information) which can provide a
   part of the service and/or slice in an E2ENS requirement.

   In order that the advertising and discovery process works in a proper
   way, a number of challenges can be identified:

   Lack of Abstractions:  Multiple vendors with heterogeneous
        technologies need an information model to adequately represent
        in confidentiality-preserving fashion the resource and topology
        information.

   Scalability:  Involves the distribution of topology and resource
        information in a peer-to-peer fashion (MdO-to-MdO).  Multi-
        operator multi-domain environments where the information
        distribution is advertised in a peer-to-peer model scales
        linearly.  It means more MdO interconnections one has, the more
        it "costs" to distribute.

   Flexibility:  Considers that a distributed approach does not allow
        domains without physical infrastructure (e.g., without BGP or
        BGP-LS) to advertise resource capabilities and networking
        resources.  Such procedures consist in deploying and configuring
        physical peering points for these domains.

   Complexity:  Refers to the discovery mechanism to pre-select
        candidate domains, accounting for resources and capabilities,
        necessary for an E2E network service deployment.  An intrinsic
        complexity exists in the process of assembling, logically
        organizing, and enabling abstraction views of different
        resources and capabilities in multi-domain scenarios.








Lachos & Rothenberg     Expires January 14, 2021                [Page 5]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


5.  Proposed Approach

   The primary design goal for ALTO-based Broker-assisted Multi-domain
   Orchestration is to discover resource and topology information from
   different administrative domains involved in the federation, while
   also safeguarding the privacy and autonomy of every domain.

   In the architectural proposal shown in Figure 1, a broker component
   is conceived to be working as coordinator of a set of MdOs.  In
   particular, the broker-assisted design consists of the following key
   components: (i) Inter-domain Resource (IdR), (ii) Inter-domain
   Topology (IdT), and (iii) ALTO Server.



                              BROKER COMPONENT
              +--------------------------------------------+
              |                                            |
              |             +-----------------+            |
              |             |                 |            |
          XXXXXXXXXXXXXXXXXXX ALTO SERVER(s)  |            |
          X   |             |                 +            |
          X   |             +----------------+\            |
          X   |            /                   \           |
          X   |           /                     \          |
          X   |  +------+/+-------+     +---------------+  |
          X   |  | Inter-domain   |     | Inter-domain  |  |
          X   |  | Topology (IdT) |     | Resource (IdR)|  |
          X   |  +-^------^-------+     +---^-------^---+  |
          X   |    .      .                 *       *      |
          X   +----.------.-----------------*-------*------+
          X        .      .                 *       *
          X        .      .                 *       *
       +--X--------.---+*********************       *
       |               |  .                         *
       |               |  .............+------------*---+
       |     MdO1      |               |                |
       |               +<------------->+                +---+
       +---------------+               |      MdO2      |   |
                                       |                |   |
   Legend:                             +-+--------------+   |
   XXX ALTO Protocol                     |                  |
   ... BGP/BGP-LS/REST                   |      MdO(n)      |
   *** UNIFY/TOSCA/ETSI-NFV              +------------------+



       Figure 1: Broker-assisted Multi-operator Network Architecture



Lachos & Rothenberg     Expires January 14, 2021                [Page 6]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


5.1.  Inter-domain Resource (IdR) Component

   It creates a hierarchical database that contains inter-domain
   resource information such as resource availability (i.e., CPU,
   memory, and storage), Virtual Network Functions (VNFs) and Physical
   Network Functions (PNFs) supported and Service Access Points (SAPs)
   to access those resources.  UNIFY [UNIFY.NFFG], TOSCA [TOSCA], ETSI-
   NFV [ETSI-NFV-MANO], among other data models can be used to create
   the interface between IdR and MdOs.

5.2.  Inter-domain Topology (IdT) Component

   A hierarchical TED (Traffic Engineering Database) that contains
   inter-domain network topology information including additional key
   parameters (e.g., throughput and latency of links).  This information
   can be retrieved from each MdO through BGP-LS or REST interfaces.

5.3.  ALTO Server Functionalities

   The ALTO server component is the core of the broker layer.  Multiple
   logically centralized ALTO servers use the information collected from
   IdR and IdT components to create and provide abstract maps with a
   simplified view, yet enough information about MdOs involved in the
   federation.  This information includes domain-level topology,
   resource availability (i.e., CPU, memory, and storage), PNF/VNF
   capabilities, and SAPs.

   As an ALTO client, each MdO sends ALTO service queries to the ALTO
   server.  This server provides aggregated inter-domain information
   exposed as set ALTO base services defined in [RFC7285], e.g., Network
   Map, Cost Map and ALTO extension services, e.g., Property
   Map [DRAFT-PM], Multi-Cost Map [RFC8189], Path Vector [DRAFT-PV].

   For example, when a source MdO receives a customer service request,
   it checks whether or not it can deliver the full service.  If it is
   unable to do so, the MdO consumes from the ALTO Server the Property
   Map service to have a clear global view of the resource information
   offered by other MdOs.  This information allows discovering which
   candidate MdOs may be contacted to deliver the remaining requirements
   of a requested end-to-end service deployment.  The connectivity
   information among discovered MdOs can be retrieved by a Cost Map
   service, responding, for instance, a path vector with the AS-level
   topology distance between the source MdO and candidate MdOs.








Lachos & Rothenberg     Expires January 14, 2021                [Page 7]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


5.4.  Filtered Cost Map Extension

   The ALTO server MUST provide connectivity information for every SG
   link in the SG path for an E2E requirement.  This information is the
   AS-level topology distance in the form of path vector, and it
   includes all possible ways for each (source node, destination node)
   pair in the SG link.

   In this section, we introduce a non-normative overview of the
   Filtered Cost Map defined in Section 6.1 of [DRAFT-PV] [1].

   The specifications for the "Media Types", "HTTP method",
   "Capabilities" and "Uses" (described in Section 6.1 of [DRAFT-PV]
   [2]) are unchanged.

5.4.1.  Accept Input Parameters

   The ReqFilteredCostMap object in Section 6.1.2 of [DRAFT-PV] [3] is
   extended as follow:


     object {
       NFFG sg;
     } ReqFilteredCostMap;

     object {
       JSONString nfs<1..*>;
       JSONString saps<1..*>;
       NextHops sg_links<1..*>;
       REQs reqs<1..*>;
     } NFFG;

     object {
       JSONNumber id;
       JSONString src-node;
       JSONString dst-node;
     } NextHops;

     object {
       JSONString id;
       JSONString src-node;
       JSONString dst-node;
       JSONNumber sg-path<1..*>;
     } REQs;







Lachos & Rothenberg     Expires January 14, 2021                [Page 8]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   sg:  If present, the ALTO Server MUST allow the request input to
      include an SG with a formatted body as an NFFG object.  An NFFG
      object contains NFs, SAPs, SG links representing logical
      connections between NFs, SAPs or both and E2E requirements as a
      list of ids of SG links.

   It is worth noting that further versions of this draft will define a
   more elaborated NFFG object to support extended parameters such as
   monitoring parameters, resource requirements, etc.

5.4.2.  Response

   If the ALTO client includes the path vector cost mode in the "cost-
   type" or "multi-cost-types" field of the input parameter, the
   response for each SG link in each E2E requirement MUST be encoded as
   a JSONArray of JSONArrays of JSONStrings.  Anyone of the sub-arrays
   indicates a potential candidate path calculated as the per-domain
   topological distance corresponding to the amount of traversing
   domains.

   Moreover, as defined in Section 6.3.6 of [DRAFT-PV] [4], If an ALTO
   client sends a request of the media type "application/alto-
   costmapfilter+json" and accepts "multipart/related", the ALTO server
   MUST provide path vector information along with the associated
   Property Map information (e.g., entry points of the corresponding
   foreign domains), in the same body of the response.

   Section 5.5.2 gives an example of the Filtered Cost Map query and the
   corresponding responses.

5.5.  Examples of Message Exchange

   This section list a couple of examples of the Property Map and
   Filtered Cost Map queries and the corresponding responses.  These
   responses are based on the information in Table 1 and Table 2 of a
   use case implementation described in Appendix A.

5.5.1.  Property Map Service

   In this example, the ALTO client wants to retrieve the entire
   Property Map for PID entities with the "entry-point", "cpu", "mem",
   "storage", "port" and "nf" properties.

   o  HTTP Request:







Lachos & Rothenberg     Expires January 14, 2021                [Page 9]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


     GET /propmap/full/inet-ucmspn HTTP/1.1
     Host: alto.example.com
     Accept: application/alto-propmap+json,application/alto-error+json


   o  HTTP Response:



     HTTP/1.1 200 OK
     Content-Length: ###
     Content-Type: application/alto-propmap+json
     {
       "property-map": {
           "pid:AS1": {
               "entry-point": [ "http://172.25.0.10:8888/escape" ],
               "cpu": [ "50.0" ],
               "mem": [ "60.0" ],
               "storage": [ "70.0" ],
               "port": [ "SAP1" ],
               "nf": [ "NF1", "NF3" ]
           },
           "pid:AS2": {
               "entry-point": [ "http://172.26.0.10:8888/escape" ],
               "cpu": [ "10.0" ],
               "mem": [ "20.0" ],
               "storage": [ "30.0" ],
               "nf": [ "NF2" ]
           },
           "pid:AS3": {
               "entry-point": [ "http://172.27.0.10:8888/escape" ],
               "cpu": [ "80.0" ],
               "mem": [ "90.0" ],
               "storage": [ "100.0" ],
               "port": [ "SAP2" ],
               "nf": [ "NF1", "NF3" ]
           }
       }
     }


5.5.2.  Filtered Cost Map Service

   The following example uses the Filtered Cost Map service to request
   the path vector for a given E2E requirement.  The SG request
   information in Table 2 is used to describe the service, and it is
   composed of three NFs (NF1, NF2, and NF3) and two SAPs (SAP1 and
   SAP2).  Links connecting the NFs and SAPs ("sg_links" tag) are also



Lachos & Rothenberg     Expires January 14, 2021               [Page 10]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   included, followed by an E2E requirement ("reqs" tag) with
   information about the order in which NFs are traversed from SAP1 to
   SAP2.

   Note that the request accepts "multipart/related" media type.  This
   means the ALTO server will include associated property information in
   the same response.

   o  HTTP Request:


       POST /costmap/pv HTTP/1.1
       Host: alto.example.com
       Accept: multipart/related, application/alto-costmap+json,
             application/alto-propmap+json, application/alto-error+json
       Content-Length: [TBD]
       Content-Type: application/alto-costmapfilter+json

       {
         "cost-type": {
           "cost-mode": "array",
           "cost-metric": "ane-path"
         },
         "sg": {
           "nfs": [ "NF1", "NF2", "NF3" ],
           "saps": [ "SAP1", "SAP2" ],
           "sg_links":[
             {
               "id": 2,
               "src-node": "SAP1",
               "dst-node": "NF1",

             },
             {
               "id": 2,
               "src-node": "NF1",
               "dst-node": "NF2",
             },
             {
               "id": 3,
               "src-node": "NF2",
               "dst-node": "NF3",
             },
             {
               "id": 4,
               "src-node": "NF3",
               "dst-node": "SAP2",
             }



Lachos & Rothenberg     Expires January 14, 2021               [Page 11]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


           ],
           "reqs": [
             {
               "id": 1,
               "src-node": "SAP1",
               "dst-node": "SAP2",
               "sg-path": [ 1, 2, 3, 4 ]
             }
           ]
         }
       }


   o  HTTP Response: The ALTO server returns connectivity information
      for the E2E requirement provided by the ALTO Client request of the
      above example.

      For each SG link in the E2E requirement (SAP1->NF1, NF1->NF2,
      NF2->NF3, NF3->SAP2), the ALTO server returns sub-arrays
      indicating potential candidate paths calculated as the AS-level
      topological distance corresponding to the amount of traversing
      domains.

      Also, the response includes Property Map information for each
      element in the path vector.  In this case, it is retrieved a
      Property Map with the "entry-point" property, i.e., the URL of the
      MdO entry point for the corresponding network.



     HTTP/1.1 200 OK
     Content-Length: [TBD]
     Content-Type: multipart/related; boundary=example

     --example
     Content-Type: application/alto-endpointcost+json

     {
       "meta": {
         "cost-type": {
            "cost-mode": "array",
            "cost-metric": "ane-path"
          },
       },

       "cost-map": {
         "SAP1": {
           "SAP2": {



Lachos & Rothenberg     Expires January 14, 2021               [Page 12]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


               "SAP1": {
                   "NF1": [
                     [ "AS1" ], [ "AS1", "AS2", "AS3" ]
                   ]
               },
               "NF1": {
                   "NF2": [
                     [ "AS1", "AS2" ], [ "AS3", "AS2" ]
                   ]
               },
               "NF2": {
                   "NF3": [
                     [ "AS2", "AS1" ], [ "AS2", "AS3" ]
                   ]
               },
               "NF3": {
                   "SAP2": [
                     [ "AS1", "AS2", "AS3" ], [ "AS3" ]
                   ]
               }
           }
         }
       }
     }

     --example
     Content-Type: application/alto-propmap+json

     {
       "property-map": {
         "pid:AS1": { "entry-point": "http://172.25.0.10:8888/escape" },
         "pid:AS2": { "entry-point": "http://172.26.0.10:8888/escape" },
         "pid:AS3": { "entry-point": "http://172.27.0.10:8888/escape" }
       }
     }

     --example--



6.  Discussion

   In this section, we analyze the benefits and open issues in our
   broker-assisted architecture.







Lachos & Rothenberg     Expires January 14, 2021               [Page 13]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


6.1.  Benefits

   The broker-assisted orchestration has numerous benefits, such as:

   o  Avoid the distribution of topology and resource information in a
      peer-to-peer fashion (MdO-to-MdO)

   o  The (abstracted) information and offered resources, services are
      maintained in each local MdO.

   o  Allow domains without physical infrastructure (hence without BGP
      or BGP-LS) to advertise their capabilities.

   o  An ALTO-based privacy-preserving information model to provide
      computing, storage, and networking resource info.

   o  An MdO discovery method to determine the underlying network graph
      and a potential set of paths before bilateral negotiation between
      MdOs is started.

6.2.  Open Issues

   Although the broker-assisted information exchange has several
   advantages, it also raises some questions which we try to answer from
   our lessons learned.

   o  What kind of organization will manage and support the operation of
      a broker entity?  If a broker is used to exchange information,
      then how does one ensure that the data delivered amongst the
      operators by this 3rd party has not been changed?

      *  The broker entity must be trusted by each operator since it
         stores and handles sensitive information.  For example, future
         deployment of SDN at IXPs can be used as a trusted third-party
         platform to support rich business models between different
         operators [DRAFT-HHSFC].

   o  In the case of peer-to-peer information exchange model, an MdO
      failure concerns only the domain where the failure occurs, other
      peers can perform the information exchange without any limitation.
      However, If any error occurs in the broker entity the information
      exchange among all involved ASes will be impacted.  How avoid this
      single point of failure?

      *  The broker entity maintains a centralized database.  Local
         restoration/replication options may be applied.





Lachos & Rothenberg     Expires January 14, 2021               [Page 14]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   o  The MdO information exchange depends on the policies.  Operators
      have a preference to share a different view about its compute and
      network resources towards different operators.  For example, a
      detailed view for the operators that are belonging to same
      operator group and a high-level information towards the other
      operators.  How is the fine-grained/coarse-grained information
      exchange handled?.

      *  It requires much more complex database handling and information
         exchange with the MdOs depending on the policies.

7.  IANA Considerations

   This document includes no request to IANA.

8.  Security Considerations

   TBD.

9.  Acknowledgments

   This work is supported by the Innovation Center of Ericsson S.A.,
   Brazil (grant agreement UNI.64).

   Thank you to Robert Szabo (Ericsson Research, Hungary) for the
   contribution and substantial feedback and suggestions in this
   document.

   Many thanks to Richard Yang, Dawn Chan, Jensen Zhang, Shawn Lin, Qiao
   Xiang, Sabine Randriamasy for their feedback on this draft.

10.  References

10.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997,
              <http://xml.resource.org/public/rfc/html/rfc2119.html>.

   [RFC7285]  Alimi, R., Ed., Penno, R., Ed., Yang, Y., Ed., Kiesel, S.,
              Previdi, S., Roome, W., Shalunov, S., and R. Woundy,
              "Application-Layer Traffic Optimization (ALTO) Protocol",
              RFC 7285, DOI 10.17487/RFC7285, September 2014,
              <https://www.rfc-editor.org/info/rfc7285>.







Lachos & Rothenberg     Expires January 14, 2021               [Page 15]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   [RFC8189]  Randriamasy, S., Roome, W., and N. Schwan, "Multi-Cost
              Application-Layer Traffic Optimization (ALTO)", RFC 8189,
              DOI 10.17487/RFC8189, October 2017,
              <https://www.rfc-editor.org/info/rfc8189>.

   [RFC8459]  Dolson, D., Homma, S., Lopez, D., and M. Boucadair,
              "Hierarchical Service Function Chaining (hSFC)", RFC 8459,
              DOI 10.17487/RFC8459, September 2018,
              <https://www.rfc-editor.org/info/rfc8459>.

10.2.  Informative References

   [DRAFT-HHSFC]
              Li, G., Li, G., Xu, Q., Zhou, H., and B. Feng, "Hybrid
              Hierarchical Multi-Domain Service Function chaining",
              draft-li-sfc-hhsfc-08 (work in progress), March 2020.

   [DRAFT-MD-VIRT]
              Bernardos, C., Contreras, L., Vaishnavi, I., Szabo, R.,
              Li, X., Paolucci, F., Sgambelluri, A., Martini, B.,
              Valcarenghi, L., Landi, G., Andrushko, D., and A. Mourad,
              "Multi-domain Network Virtualization", draft-bernardos-
              nfvrg-multidomain-05 (work in progress), September 2018.

   [DRAFT-PM]
              Roome, W., Randriamasy, S., Yang, Y., Zhang, J., and K.
              Gao, "Unified Properties for the ALTO Protocol", draft-
              ietf-alto-unified-props-new-11 (work in progress), March
              2020.

   [DRAFT-PV]
              Gao, K., Randriamasy, S., Yang, Y., and J. Zhang, "ALTO
              Extension: Path Vector", draft-ietf-alto-path-vector-10
              (work in progress), March 2020.

   [ETSI-NFV-DEF]
              ETSI, "Network Functions Virtualisation (NFV); Terminology
              for Main Concepts in NFV V1.3.1", Jan 2018,
              <https://docbox.etsi.org/isg/nfv/open/Publications_pdf/
              Specs-Reports/NFV%20003v1.3.1%20-%20GR%20-%20Terminology%2
              0for%20Main%20Concepts%20in%20NFV.pdf>.

   [ETSI-NFV-IFA028]
              ETSI, "Report on architecture options to support multiple
              administrative domains V3.1.1", Jan 2018,
              <http://www.etsi.org/deliver/etsi_gr/NFV-
              IFA/001_099/028/03.01.01_60/gr_NFV-IFA028v030101p.pdf>.




Lachos & Rothenberg     Expires January 14, 2021               [Page 16]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   [ETSI-NFV-MANO]
              ETSI, "Network Functions Virtualisation (NFV) Management
              and Orchestration V1.1.1", Dec 2014,
              <http://www.etsi.org/deliver/etsi_gs/NFV-
              MAN/001_099/001/01.01.01_60/gs_NFV-MAN001v010101p.pdf>.

   [H2020-5G-TRANSFORMER]
              H2020, "5G-Transformer -- 5G Mobile Transport Platform for
              Vertical", 2017, <http://5g-transformer.eu/>.

   [H2020.5GEX]
              Bernardos, C., Dugeon, O., Galis, A., Morris, D., Simon,
              C., and R. Szabo, "5G Exchange (5GEx)--Multi-domain
              Orchestration for Software Defined Infrastructures",
              focus vol. 4, no.5, p.2, 2015.

   [H2020.5GEX.ESCAPE]
              5GEx Project, "ESCAPE: Extensible Service ChAin
              Prototyping Environment", 2015,
              <https://github.com/5GExchange/escape>.

   [ICAF]     Demchenko, Y., Makkes, M., Strijkers, R., Ngo, C., and C.
              Laat, "Intercloud Architecture Framework for Heterogeneous
              Multi-Provider Cloud based Infrastructure Services
              Provisioning", International Journal of Next-Generation
              Computing vol. 4, no.2, 2013.

   [PPP-5:2013]
              5G-PPP, "Advanced 5G Network Infrastructure for the Future
              Internet", 2013, <https://5g-ppp.eu/wp-
              content/uploads/2014/02/Advanced-5G-Network-
              Infrastructure-PPP-in-H2020_Final_November-2013.pdf>.

   [T-NOVA]   FP7 project T-NOVA, "T-NOVA Project, Network Functions as
              a Service over Virtualised Infrastructures", 2014,
              <http://www.t-nova.eu/>.

   [TELEFONICA.NET.TOPO]
              Telefonica I+D, "Netphony-Topology", 2016,
              <https://github.com/telefonicaid/netphony-topology>.

   [TOSCA]    OASIS, "TOSCA: Topology and Orchestration Specification
              for Cloud Applications V1.0", 2013, <http://docs.oasis-
              open.org/tosca/TOSCA/v1.0/os/TOSCA-v1.0-os.pdf>.







Lachos & Rothenberg     Expires January 14, 2021               [Page 17]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   [UNIFY.NFFG]
              UNIFY Deliverable D3.2a, "Network Function Forwarding
              Graph specification", 2015, <http://www.fp7-
              unify.eu/files/fp7-unify-eu-docs/Results/Deliverables/
              UNIGY_D3.2a_NFFG%20Specification.pdf>.

   [VITAL]    VITAL PROJECT H2020, "VITAL -- VIrtualized hybrid
              satellite-TerrestriAl systems for resilient and fLexible
              future networks", 2015, <http://www.ict-vital.eu/>.

10.3.  URIs

   [1] https://tools.ietf.org/html/draft-ietf-alto-path-vector-
       02#section-6.1

   [2] https://tools.ietf.org/html/draft-ietf-alto-path-vector-
       02#section-6.1

   [3] https://tools.ietf.org/html/draft-ietf-alto-path-vector-
       02#section-6.1.2

   [4] https://tools.ietf.org/html/draft-ietf-alto-path-vector-
       02#section-6.3.6

Appendix A.  Proof of Concept Use Case Implementation

   A strawman use case scenario has been implemented following the
   architectural proposal of the 5GEx project [H2020.5GEX].  It refers
   to an E2ENS orchestration involving three administrative domains.
   For reproducibility purposes, all supporting codes are publicly
   available in our research group repository:
   https://intrig.dca.fee.unicamp.br:8865/intrig-unicamp/alto-based-
   broker-assisted-mdo

   As shown in Figure 2, each administrative domain has an MdO (MdO-AS1,
   MdO-AS2, and MdO-AS3) to coordinate resource and/or service
   orchestration at multi-operator level via interface I2 APIs.  For the
   orchestration within the same administrative domain, each MdO uses
   emulated DOs with emulated I3 interfaces, since no data-plane is
   present.  DOs use static configuration files to load local
   information about resources (I3-RC) and topology (I3-RT).  The
   different MdO components are based on existing open source tools such
   as ESCAPE [H2020.5GEX.ESCAPE] (Service/Resource Orchestrator) and
   Netphony-topology [TELEFONICA.NET.TOPO] (Resource Topology) and run
   in Docker containers on a single computer.  Besides, MdOs expose I1
   interfaces to the tenants who request services and/or slices which
   should follow a Network Function Forwarding Graph (NFFG) [UNIFY.NFFG]
   format.



Lachos & Rothenberg     Expires January 14, 2021               [Page 18]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   In case of the broker layer, the IdR and IdT components use a UNIFY
   Virtualizer API [UNIFY.NFFG] (broker-based I2-RC API) and a REST API
   (broker-based I2-RT API) respectively, in order to create the
   hierarchical databases.  Regarding the IdT, the administrative domain
   2 is a transit provider so that the domain-level topology computed
   is: AS1-AS2-AS3.  From the inter-domain information are created the
   two different ALTO Map Services: (i) Property Map and (ii) Cost Map.












































Lachos & Rothenberg     Expires January 14, 2021               [Page 19]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


                            +----------------------------------------+
                            |       +---------------+    BROKER LAYER|
                       XXXXXXXXXXXXXX  ALTO Server  |                |
                       X    |       +--------+----+-+                |
                       X    |               /      \                 |
                       X    | +-----------+/+--+  +-\-------------+  |
                       X    | | Inter-domain   |  | Inter-domain  |  |
       +-----------+   X    | | Topology (IdT) |  | Resource (IdR)|  |
       | Service   |   X    | +----------------+  +---------------+  |
       | Graph (SG)|   X    +---------^-^----------------^--^--------+
       | Request   |   X              * *    .............  .
       +-----+-----+   X              * *    . ..............
             |         X              * *    . .              MdO-AS3
           I1|         X              * *    . .      +--------------+
             |         X     MdO-AS1  * *    . .      |              |
       +-----|---------X-----------+  * *    . .      |     MdO-AS2  |
       |     |                     |  * *    . .   +---------------+ |
       | +---v-------------------+ |  * *    . .   | +-----------+ | |
       | |                       | |  * *    . .   | |           | | |
       | |  Network Service Orch.| |  * *    . .   | |   NSO     | | |
       | |  (NSO)                | |  * *    . .   | |           | | |
       | +-----------------------+ |  * *    . .   | +-----------+ | |
       |                           |  * *    . .   |               | |
       | +---------+               |  * *    . .   | +---+         | |
       | | Resource........................... .   | |   |         | |
       | | Topoloy |               |  * *      .......RT |         | |
       | | (RT)    | +-----------+ |  * *          | |   |         | |
       | +---------+ |Resource   | |  * *          | +---+   +---+ | |
       |             |Orch.      | |  * **********************   | | |
       |             |(RO)       ******            |         |RO | +-+
       |             +-----------+ |               |         |   | |
       |                           |<------------->|         +---+ |
       +---------------------------+      I2       +-----+---------+
                  /     \                                |
               I3/       \                               |I3
       +-------+---+  +-----------+                +-----------+
       | Domain    |  | Domain    |                | Domain    |
       | Orch (DO) |  | Orch (DO) |                | Orch (DO) |
       +-----------+  +-----------+                +-----------+
                                            Legend:
                                            XXX ALTO Protocol
                                            ... broker-based I2-RT API
                                            *** broker-based I2-RC API


               Figure 2: Broker-assisted 5GEx Info Exchange





Lachos & Rothenberg     Expires January 14, 2021               [Page 20]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


   The Property Map includes property values grouped by Autonomous
   System (AS).  Such values are SAPs, NFs and the 5GEx Entry Point
   (e.g., the URL of the ESCAPE orchestrator).  An example of the
   Property Map in our prototype is:

   +-----+------------+-------+--------------+-----+-----+-------+-----+
   |     |   Entry    |  Port | Capabilities | CPU | MEM |  Stor | ... |
   |     |   Point    |  SAP  |              |     |     |  age  |     |
   +-----+------------+-------+--------------+-----+-----+-------+-----+
   | AS1 | http://... |  SAP1 |  {NF1, NF3}  |  50 |  60 |   70  | ... |
   | AS2 | http://... |   -   |    {NF2}     |  10 |  20 |   30  | ... |
   | AS3 | http://... |  SAP2 |  {NF1, NF3}  |  80 |  90 |  100  | ... |
   +-----+------------+-------+--------------+-----+-----+-------+-----+

                        Table 1: ALTO Property Map

   The Cost Map defines a path vector as an array of ASes, representing
   the AS-level topological distance for a given E2ENS request.  Path
   vector constraints (as described in the Multi-Cost Map [RFC8189]) can
   be applied to restricts the response to costs that satisfy a list of
   simple predicates.

   Table 2 below shows a brief example of an SG request and its path
   vector response containing a list of potential providers to be
   traversed to deliver such service.  Every AS path is computed from
   the inter-domain topology information in the IdT module.  In our
   scenario, MdO-AS2 is a transit provider, so that the domain-level
   topology map is AS1<->AS2<->AS3.

   +--------------------+----------------------------------------------+
   | Service Graph (SG) |                Path(s) Vector                |
   |      Request       |                                              |
   +--------------------+----------------------------------------------+
   | SAP1->NF1->NF2->NF | 1:{AS1:SAP1->AS1:NF1->AS2:NF2->AS3:NF3->AS3: |
   |      3->SAP2       |                    SAP2}                     |
   |                    | 2:{AS1:SAP1->AS1:NF1->AS2:NF2->AS1:NF3->AS2- |
   |                    |                  >AS3:SAP2}                  |
   |                    | 3:{AS1:SAP1->AS2->AS3:NF1->AS2:NF2->AS3:NF3- |
   |                    |                  AS3:SAP2}                   |
   |                    | 4:{AS1:SAP1->AS2->AS3:NF1->AS2:NF2->AS1:NF3- |
   |                    |               >AS2->AS3:SAP2}                |
   +--------------------+----------------------------------------------+

                          Table 2: ALTO Cost Map







Lachos & Rothenberg     Expires January 14, 2021               [Page 21]

Internet-Draft    ALTO-based Multi-domain Orchestration        July 2020


Authors' Addresses

   Danny Alex Lachos Perez
   University of Campinas
   Av. Albert Einstein 400
   Campinas, Sao Paulo  13083-970
   Brazil

   Email: dlachosp@dca.fee.unicamp.br
   URI:   https://intrig.dca.fee.unicamp.br/danny-lachos/


   Christian Esteve Rothenberg
   University of Campinas
   Av. Albert Einstein 400
   Campinas, Sao Paulo  13083-970
   Brazil

   Email: chesteve@dca.fee.unicamp.br
   URI:   https://intrig.dca.fee.unicamp.br/christian/































Lachos & Rothenberg     Expires January 14, 2021               [Page 22]