Internet Research Task Force NFVRG G. Carrozzo, Ed. Internet-Draft Nextworks Intended status: Informational K. Pentikousis, Ed. Expires: January 7, 2016 EICT July 6, 2015 Recursive orchestration of federated virtual network functions draft-felix-nfvrg-recursive-orchestration-00 Abstract This document introduces a policy-based resource management and orchestration framework which aims at contributing towards the current namesake NFVRG near-term work items. It describes key points of the recursive resource orchestration framework developed within the wider research area of federated virtual network function orchestration. The document also relates this effort with respect to other orchestration frameworks, thus addressing both the NFV research and practitioner communities. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on January 7, 2016. Copyright Notice Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents Carrozzo & Pentikousis Expires January 7, 2016 [Page 1] Internet-Draft Recursive orchestration July 2015 carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Recursive orchestration in federated virtual environments . . 5 3.1. Problem Statement . . . . . . . . . . . . . . . . . . . . 5 3.2. Resource Orchestrator . . . . . . . . . . . . . . . . . . 6 4. Policy-Based Resource Management . . . . . . . . . . . . . . 9 4.1. Certificate-based authN/authZ (C-BAS) . . . . . . . . . . 10 4.2. Resource Managers . . . . . . . . . . . . . . . . . . . . 11 5. Positioning w.r.t. existing Orchestration Frameworks . . . . 14 5.1. Openstack orchestration . . . . . . . . . . . . . . . . . 14 5.2. OpenMANO . . . . . . . . . . . . . . . . . . . . . . . . 15 5.3. Other orchestration approaches: federated SDN infrastructures for research experimentation . . . . . . 15 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 16 7. Security Considerations . . . . . . . . . . . . . . . . . . . 17 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 17 9. Contributors . . . . . . . . . . . . . . . . . . . . . . . . 17 10. Informative References . . . . . . . . . . . . . . . . . . . 17 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 19 1. Introduction Today's Internet is a concatenation of IP networks interconnected by many distributed functions integrated into a plethora of highly specialized middleboxes. These elements implement complex network functions like firewalls, NATs, DPI, traffic scrubbing, etc. The product is a quite complex and rigid internetworking system in which network administrators and users cannot easily determine what is happening to traffic flows as they go toward destinations. In the last decade networks, servers, storage technologies, and applications have all undergone significant changes with the introduction of virtualization, network overlays, and orchestration. Such technologies have allowed network operators and service providers to easily introduce a variety of (proprietary) hardware-based appliances in order to improve their network manageability as well as rapidly launch new services, keeping up with the pace of their users demand. Therefore, the current Internet looks like a concatenation of networks with many distributed functions, implemented via a plethora of highly specialized middleboxes which implement firewalls, DPI, NAT, traffic scrubbing, etc. [middlebox]. Carrozzo & Pentikousis Expires January 7, 2016 [Page 2] Internet-Draft Recursive orchestration July 2015 Software Define Networking and programmable virtualized network functions for flow processing are rapidly changing the current scenario, extending the support of network functions by virtualized and chained appliances beyond the virtual L2 switching over IP networks (e.g. VXLAN, GRENV, STT) and the basic LAN based flow pinpointing. Virtual Functions of this kind, for network and non network (e.g. computing) tasks, are generally available in heterogeneous pools under different administrative domains, being them related to the hosting infrastructures in which they originate. It is emerging a need to interconnect, federate and implement policy control on these pools of virtual resources, in order to abstract different infrastructures, resources and functions, as well as procedures by physical operators and infrastructure owners. This can allow defining larger virtual overlays where different resources and functions are deployed, combined and handled in the form of virtual instances irrespective of the administrative domain and specific technology from which they originate. Examples of application contexts in which this federation of virtual function pools may occur are: o large scale experimentation over programmable networks, which allows to reserve slices of network and non-network resources from different federated providers to run experiments on network control, protocols and algorithms at large scale (e.g. [FELIX]); o virtual infrastructure operators, who intend to implement their network service offer over a completely virtual infrastructure in the form of virtual network nodes and functions, virtual servers and storage, etc., all procured as a service from physical providers. This document discusses key points of the recursive resource orchestration framework developed within the wider research area of federated virtual network function orchestration. The proposed architecture allows federation and integration of different network and computing resources residing in a multi-domain heterogeneous environment across different providers. To achieve this, the architecture uses a combination of recursive and hierarchical configurations for orchestration, request delegation and inter-domain dependency management, with resource orchestrating entities (Resource Orchestrators, RO) responsible for synchronization of resources available in particular administrative domains. This document is being discussed on the nfvrg@irtf.org mailing list. Carrozzo & Pentikousis Expires January 7, 2016 [Page 3] Internet-Draft Recursive orchestration July 2015 2. Terminology The following terminology is used in this document. Virtual Network Function(VNF). One or more virtual machines running different software and processes, on top of industry standard high volume servers, switches and storage, or even cloud computing infrastructure, and capable of implementing network functions traditionally implemented via custom hardware appliances and middleboxes (e.g. router, NAT, Firewall, load-balancer, etc.) VNF Island. A set of virtualized network functions and related network and IT resources under the same administrative ownership/ control. A VNF island could consist of multiple zones, each characterized by a specific set of control tools & interfaces. VNF Zone. A set of virtual network functions grouped for homogeneity of technologies and/or control tools and/or interfaces (e.g. L2 switching zone, optical switching zone, OF protocol controlled zone, other transit domain zone with a control interface). The major goal of defining SDN zones is to implement appropriate policies for increasing availability, scalability and control of the different resources of the island. Examples of zone definitions are available in popular Cloud Management Systems (CMS) like Cloudstack (e.g. refer to the Cloudstack Infrastructure partitioning into regions, zones, pods, etc., [cloudstack]) and OpenStack (e.g. refer to the infrastructure partitioning in availability zones and host aggregates [openstack]). Transit network domains. The network domains use a Bandwidth on Demand Interface to expose automatically and on-demand control of connectivity services and, optionally, inter-domain topology exchange. In order to federate resources belonging to distant facilities (i.e. islands/zones) it must be ensured that interconnectivity is provided on-demand and with a specific granularity. Slice. A user-defined subset of virtual networking and IT resources, created from the physical resources available in federated VNF Zones and VNF Islands. A VNF Slice has the basic property of being isolated from other slices defined over the same physical resources, and being dynamically extensible across multiple VNF Islands. Each VNF Slice instantiates the specific set of control tools of the specific zones it traverses. Resource Orchestrator (RO). Entity responsible for orchestrating end-to-end network service and resources reservation in terms of compute, storage and network functions over the infrastructure, as Carrozzo & Pentikousis Expires January 7, 2016 [Page 4] Internet-Draft Recursive orchestration July 2015 well as delegating end-to-end resource and service provisioning in a technology-agnostic way. Resource Managers (RMs). Entity responsible for controlling and managing different types of resources and/or network functions. 3. Recursive orchestration in federated virtual environments 3.1. Problem Statement The coordinates creation of a virtual environment with pools of virtual network and non-network functions from heterogeneous, multi- domain and geographically distributed facilities requires appropriate tools for resource and virtual function management and control capable of orchestration and policy control across VNF islands, zones and domains defined above. Elements that belong to this control and orchestration layer operate in a hierarchical way (parent-child) for efficient multi-domain information management and sharing. This is generally referred as Inter-island Orchestration Space [FELIX-D2.1] [FELIX-D2.2]. Once the set of virtual network and non-network functions is determined, reserved and deployed across the different islands, the resulting virtual network environment is ready for being used as a User Space by any tool or application the user wants to deploy in it. In the Inter-island Orchestration Space (see Figure 1), Resource Orchestrators (RO) are responsible for orchestrating end-to-end network services and resources reservations in the whole infrastructure. Moreover, ROs should be able to delegate end-to-end resource and service provisioning in technology-agnostic way. ROs are connected to different types of Resource Managers (RMs), which are in turn used to control and manage different kinds of technological resources. For example, the VNF RM WAN side provides connectivity between L1/L2 network domains at the two ends. This management can be achieved using frame, packet or circuit switching technologies and should support different protocols. On the other hand, the VNF RM (LAN side) manages the network infrastructure composed of SDN-enabled devices, e.g. OpenFlow switches or routers. In short, it can control the user traffic environment by updating flow tables in physical devices. In addition, the Virtual Function pool RM for computing resources is responsible for setting up and configuring computing resources, i.e. Carrozzo & Pentikousis Expires January 7, 2016 [Page 5] Internet-Draft Recursive orchestration July 2015 creating new virtual machine instances, powering on/off instances, network interface card configuration, etc. Authentication and Authorization Infrastructure (AAI) for authenticating and authorizing users, is a cross layer function in the Inter-island Orchestration Space, because it serving as a 'trust anchor' to facilitate authN/authZ procedures in federated facilities. Similarly, Monitoring allows to retrieve, correlate and abstract statistics from the different components of the physical and virtual resource pools and from the user's slice. Figure 1 shows a parent RO coordinating orchestration actions at NFV island level under the responsibility of two child ROs, each orchestrating different types of RMs for the different kinds of virtual network and non-network function pools. +---+ +-------------------------------------------+ +---+ | | | RESOURCE ORCHESTRATOR - RO | | | | |-----------| (parent) |----| | | | +-------------------------------------------+ | | | | | | | | | | | | | | | | +-------------------------------------+ +---------+ | | | M | | RO | | RO | | | | O |--| (child) | | (child) |---| | | N | +-------------------------------------+ +---------+ | A | | I | | | | | | A | | T | | | | | | A | | O | +-----------+ +----------+ +----------+ +-----------+ | | | R | | VF POOL | | VNF POOL | | VNF POOL | | Virtual | | | | I |--| MANAGER | | MANAGER | | MANAGER | | pool |--| | | N | |(computing)| |(LAN side)| |(WAN side)| | manager(s)| | | | G | +-----------+ +----------+ +----------+ +-----------+ | | | | | | | | | | | | +----------+ +----------+ ----------+ +----------+ | | | |--| VF | | VNF | | VNF | | VNF |--| | +---+ +----------+ +----------+ +----------+ +----------+ +---+ Figure 1: Recursive Orchestration architecture model 3.2. Resource Orchestrator RO is the entity that orchestrates the different resources in the Inter-island Orchestration Space. Carrozzo & Pentikousis Expires January 7, 2016 [Page 6] Internet-Draft Recursive orchestration July 2015 There are two different modes in which RO may operate: o Parent o Child For an inter-island federation, RO operates in parent mode and attaches to child ROs, whilst in child mode ROs communicate with RMs. One of RO's main objectives is to forward requests within the infrastructure, either by: o Passing user requests to the appropriate resource management systems (RMs) in the layer below, as with any hierarchical mode. o Proxying requests between other ROs in a recursive mode, depending on the federation policy that is configured for the domain where the RO is located. For this to work it is necessary to ensure similar interfaces for each orchestrator. Key functions of the RO can be summarized as follows. RO manages the different VNF islands and users in terms of their resource and data access policies. It mediates between the user and the technology specific to a Resource Manager (RM) by means of delegation. It is expected that different RMs will handle, for example, technology- dependent aspects in SDN domains (VNF RM LAN side) and transit network domains (VNF RM WAN side), as well non-network resource pools. As part of this mediation, the RO will engage in the creation (provisioning), maintenance, monitoring, and deletion (release) of the used slices. RO also maintains a high-level cross-island topological view, which summarizes the different resources pools available along with their inter-connections. This topology view is initialized and updated by the underlying RMs, thus implementing a distributed hierarchical resource discovery function. It determines which domains and which inter-domain resources should be used to instantiate a given end-to- end service for a particular experimenter's slice. For example, based on a user request for a given type of service to be instantiated in two remote islands, parent RO determines which specific resource domains should be involved. Finally, RO coordinates and ensures that the correct sequence of actions takes place with respect to the operation of the technology- specific RMs. This includes the provisioning of the slice resources as per user's requirements. Carrozzo & Pentikousis Expires January 7, 2016 [Page 7] Internet-Draft Recursive orchestration July 2015 RO also collects and correlates status alarms and warnings on resources, either generated by the resources themselves or the services managing them. This is done on a per-slice basis and proceeds with reporting/notifying the corresponding users. Different deployment models are possible for a Resource Orchestration entity: o hierarchical centralized (see Figure 2.A) o distributed in chain (see Figure 2.B o distributed full-mesh (see Figure 2.C) o hierarchical hybrid (see Figure 3) The hierarchical hybrid model is deemed to guarantee the optimal trade-off between effectiveness of control, federation, trust adjacencies and scalability. +-----------+ | RO | +-----------+ / | \ +------+ +------+ +------+ / | \ | RO |---| RO |---| RO | / | \ +------+ +------+ +------+ +------+ +------+ +------+ | RO | | RO | | RO | +------+ +------+ +------+ (A) (B) +--------------------+ | | +------+ +------+ +------+ +------+ | RO |---| RO |---| RO |---| RO | +------+ +------+ +------+ +------+ | | | | | +--------------------+ | +---------------------------------+ (C) Figure 2: RO deployment models Carrozzo & Pentikousis Expires January 7, 2016 [Page 8] Internet-Draft Recursive orchestration July 2015 +-----------+ +-----------+ | RO +----------------------+ RO | +-----------+ +-----------+ / | \ / | \ / | \ / | \ / | \ / | \ +------+ +------+ +------+ +------+ +------+ +------+ | RO | | RO | | RO | | RO | | RO | | RO | +------+ +------+ +------+ +------+ +------+ +------+ Figure 3: Hybrid RO deployment model 4. Policy-Based Resource Management Aside from the main functions described above, each Resource Manager is also part of the Authentication and Authorization Infrastructure (AAI). AAI provides the necessary mechanisms to authenticate and authorize users, as well as provide accountability. In order to realize these functions, our architecture suggests the implementation of a ClearingHouse (CH) , which establishes the root of a trust chain. This chain can then be used to verify the identity and privileges of all actors in this architecture. By using a certificate-based approach, the architecture has flexibility to easily federate different VNF islands. By installing ClearingHouse certificates, actors can be verified against different ClearingHouses, and thus can utilize a multitude of resources. A ClearingHouse (CH) comprises a set of related services supporting AAA operations. CH serves as a central location to lookup information about members, slices and other available services in the VNF island. There are three groups of CH services: o Registration and management services to lookup for available services in the facility as well as register new members, projects and slice objects. o Authentication and Authorization services to manage the credentials of all entities and enforce predefined policies. o Accountability services to facilitate tracking of all transactions. Carrozzo & Pentikousis Expires January 7, 2016 [Page 9] Internet-Draft Recursive orchestration July 2015 These services are offered by CH with the help of the following functions and authorities. o The Member Authority (MA) is responsible for managing and asserting user attributes. It generates member certificates for identification purposes and credentials to specify the attributes and roles associated with each member. The MA maintains a database of registered members and their associated information including, but not limited to, certificates and credentials, SSH (Secure Shell) and SSL (Secure Sockets Layer) keys as well as the human readable identity information like real name, institute, contact details. o In addition, a Certificate Revocation List (CRL) can also be accessed from MA for use in certificate verification process. o The Slice Authority (SA) creates and manages slice objects and the associated member credentials (called slice credentials). Slice credentials map member roles and privileges on a slice object, i.e., slice credentials authorize user actions at aggregates within a slice context. SA also enables related operations on slice objects like look up, modify, renew, etc. o The Project Service (PS) hosted at SA maintains a list of existing projects and asserts the member roles. o The Service Registry (SR) serves as the primary network contact point as it keeps a record of all available registered services such as SA and MA and offers their URIs. o The Logging Service (LS) realizes accountability by storing the transaction details between user-agents and aggregate managers. The user-agents and ROs can communicate with the CH through XMLRPC calls over a secured connection (SSL). AAI is ultimately responsible for granting access to the resources, and can be further extended through policies, which are a set of rules defined by the administrators to implement an upper-level control on the resource usage (e.g. defining a maximum virtual memory value for a VM resource or a maximum number of flow spaces). 4.1. Certificate-based authN/authZ (C-BAS) Since VNF pools are finite, access to virtual functions and resources should be policed according to set authorization levels throughout the life-cycle of each experiment. Carrozzo & Pentikousis Expires January 7, 2016 [Page 10] Internet-Draft Recursive orchestration July 2015 Access control is also required to ensure that infrastructures remain operational. Although a number of solutions for authentication (authN) and authorization(authZ), such as Kerberos and LDAP, already exist, they have several shortcomings: tight-coupling of authN/authZ mechanisms with the implementation of the architecture; little or no regard for re-usability (i.e., one authN/authZ architecture cannot be reused by a different infrastructure); and no support for a standard access interface between networks and the authN/authZ architecture. C-BAS, certificate-based authN/authZ solution, is designed to serve all these requirements and include i) multiple authoritative source of trust, ii) flexible system of authorization, and iii) synchronization of authN/authZ entities to realize federations. For example, the Registry Service of C-BAS may be exploited to implement load balancing and failover features. In addition, the evolved architecture of C-BAS makes it robust against disruptions and interference from attackers and enables support for various member roles and permissions. C-BAS employs X.509 certificates and SFA styled credentials to realize AAA services. The implementation of C-BAS is publicly available (www.eict.de/c-bas) and is based on eiSoil (github.com/EICT/eiSoil/wiki) thus exploiting its plugin capabilities that enable importing the functionality from one plugin module to another. 4.2. Resource Managers 4.2.1. VNF Pool manager functionality A VNF pool manager is a functional entity in charge of controlling a specific type of VNFs, being the equivalent of the SFA Aggregate Manager (see [SFA]). As such, a VNF pool manager is a Resource Manager within the federation, capable of discovering resources, capabilities ans functions from physical infrastructure, abstracting them before publishing to the supervising RO and eventually capable of managing specific technology-specific configurations and provisioning towards the actual resource layer. Whilst the northbound interface of the Resource Manager is abstract and unified across different technology domains (e.g. based on REST or XMLRPC), the southbound interface is based on the specific interfaces exposed by the different types of resources (e.g. OpenFlow, NETCONF, SNMP, CLI, OVSDB, etc.) Carrozzo & Pentikousis Expires January 7, 2016 [Page 11] Internet-Draft Recursive orchestration July 2015 4.2.2. OpenFlow-based VNF pool manager The VNF pool manager LAN side could be OpenFlow-based and provide the mechanisms to control the network infrastructure inside a domain with SDN-enabled hardware (typically, OpenFlow-enabled switches and routers). The Inter-island Orchestration Space architecture is agnostic of the physical network resources. For a SDN domain based on OpenFlow users can control network behaviour by actively updating the flow tables of the network elements. This update is usually done by a SDN controller, which is configured according the user's requirements. Typically, the controller will respond to an event generated by a network element, such as a flow establishment request, and update the flow tables appropriately. For the network manager, the VNF pool manager LAN side will provide management functionalities for the overall network resources and virtual network functions in the LAN or data center. This element acts as a proxy between the resources and the SDN controller, and grants or denies the forwarding of control messages. The VNF pool manager LAN side provides functions to build a unique flow space for every experimenter so that traffic is isolated and distinguished from that of other slices (e.g. like FlowVisor or OVX do). These functionalities can prevent issues arising when several users wish to use the same physical resources. In detail, a flow space can contain a range of differentiators: source or destination IPs or MAC addresses, TCP or UDP ports, for example. One way to separate the traffic is assigning a VLAN tag to each packet. In this case, the special purpose controller inspects the incoming packet, identifies the VLAN tag and sends it to the corresponding SDN controller. 4.2.3. Stitching Entity VNF pool manager The Stitching Entity VNF pool Manager is among the pool managers WAN side, and is in charge to control the Stitching Entity (SE), a network element providing necessary translation mechanisms for a slice setup on top of the L2 protocol stack performed in order to hide from a user the real complexity of the multi-domain WAN transport network. The main responsibility of the SE is to provide at least one of the following network functions: o QinQ, to encapsulate slice traffic into a transport network Ethernet frames, Carrozzo & Pentikousis Expires January 7, 2016 [Page 12] Internet-Draft Recursive orchestration July 2015 o VLAN translation mechanism to hide from a user the actual VLAN tagging, used by carrier networks while interconnecting two or more VNF islands. The SE RM communicates with an RO, a parent control entity, to i) advertise an internal topology and capabilities of the SE under its control, ii) receive requests, and iii) notify the RO about success and failure events. A single SE RM must relate only to a single RO and must be implemented in each VNF island. A single SE RM is responsible for a single SE or a group of SEs, which belong to a network domain and act as an entry point to the island infrastructure. 4.2.4. Transit network VNF pool manager The main responsibility of the Transit VNF pool manager (TN RM) is to support the FELIX architecture with network connectivity mechanisms within particular domains and between them. In order to deliver the network services, it must be integrated with its southbound interfaces within a particular network domain. Such a domain can use different L1/L2 technologies and may be controlled by a Network Management System (NMS) or by specific interfaces or protocols that are technology-dependent, and unique in each case. The Transit network VNF pool manager must communicate with an RO in order to i) advertise resources under its control, ii) receive requests, and iii) notify the RO about success and failure events. A single TN RM must relate only to a single RO. A single TN RM is responsible for a group of particular network resources, which belong to a network domain and are usually managed by a single entity, i.e. a network administrator or NMS. TN RM usually manages L1/L2 transport networks, which are composed of physical devices using frames/packets or circuit switching technologies and support different protocols, e.g. MPLS/GMPLS. In order to support inter-island connectivity between existing VNF islands, the TN RM also supports the management of VPN set up and tear down procedures. The TN RM southbound interface can be based on Bandwidth on Demand interfaces, like GMPLS UNI or similar approaches. Carrozzo & Pentikousis Expires January 7, 2016 [Page 13] Internet-Draft Recursive orchestration July 2015 4.2.5. RM for virtual computing The function of the Computing Resource pool Manager (C-RM) is to provide a method to assign, set up and configure computing resources. C-RM manages physical computing resources, and also the configuration of its own slicing mechanisms (e.g. common hypervisors or other virtualization stacks) and computer resources as presented to the user (OS images, network interface configuration, and so on). The management of physical computing resources should provide a method for rebooting machines, remote control (of a machine's console), or hard power on/off of a machine experiencing problems, for example using a networked PDU (power distribution unit). Management is typically performed only when problems occur, and when a slice is created, destroyed or modified. Migration of computing resources to other islands may also require reconfiguration. This includes the configuration of network interfaces of the computing resource, and setting the underlying resources (e.g. hypervisor, physical machine), such that those interfaces are bridged onto the correct physical interfaces. In particular, it may be necessary to configure a slicing mechanism in this bridging, in the case where multiple computing resources have to share a single physical interface. This would typically be achieved using a (software-based) SDN solution inside the virtualization platform. Once the SDN solution has been properly set up, it becomes an SDN resource, which is managed by the VNF pool manager LAN side. 5. Positioning w.r.t. existing Orchestration Frameworks 5.1. Openstack orchestration Among cloud orchestration solution, OpenStack is the facto common reference through its Heat module [os-heat]. Openstack Heat implements an orchestration engine to launch multiple composite cloud applications based on templates in the form of text files that can be treated like code. Many existing CloudFormation templates can be launched on OpenStack. Heat provides both an OpenStack-native ReST API and a CloudFormation- compatible Query API. A Heat template describes the infrastructure for a cloud application in a text file. Infrastructure resources that can be described include: servers, floating ips, volumes, security groups, users, etc. Carrozzo & Pentikousis Expires January 7, 2016 [Page 14] Internet-Draft Recursive orchestration July 2015 Templates can also specify the relationships between resources (e.g. this volume is connected to this server). Heat also provides an autoscaling service. Heat primarily manages cloud infrastructure, does not support federation and AAI is bundled in the OpenStack framework. 5.2. OpenMANO OpenMANO is an open source project which implements the reference architecture for Management & Orchestration under standardization at ETSI's NFV ISG (NFV MANO) [openmano]. OpenMANO consists of two main SW components: o NFV VIM (Virtualised Infrastructure Manager) to provide computing and networking capabilities and to deploy virtual machines. o A reference implementation of an NFV-O (Network Functions Virtualisation Orchestrator), which allows creation and deletion of VNF templates, VNF instances, network service templates and network service instances. OpenMANO does not support federation and AAI as of today. 5.3. Other orchestration approaches: federated SDN infrastructures for research experimentation The FELIX project [FELIX] is part of an international research experimentation infrastructure strategy (in Europe under the Future Internet Research Experimentation - FIRE - framework), with a special focus on SDN and Network Service Interface (NSI) developed by the Open Grid Forum. FELIX is implementing federation and integration of different network and computing resources controlled via SDN and NSI in a multi-domain heterogeneous environment across, initially spanning Europe and Japan. FELIX consortium has designed and is implementing an architecture that extends and advances assets previously developed in other Future Internet projects (e.g. OFELIA), by realizing the federation concepts defined in SFA [SFA] with a combination of recursive and hierarchical orchestration, request delegation and inter-domain dependency management. Other research testbeds have been working over the past year on federation of SDN resources. Three of them are particularly relevant on the SDN area: OFELIA, FIBRE and GridARS. The OFELIA project [OFELIA] established a pan-European experimental network facility which enables researchers to experiment with real Carrozzo & Pentikousis Expires January 7, 2016 [Page 15] Internet-Draft Recursive orchestration July 2015 OpenFlow-enabled network equipment and to control and extend the network itself in a precise and dynamic mode. The OFELIA facility uses the OpenFlow protocol (and related tools) to support network virtualization and control of the network environment through secure and standardised interfaces. OFELIA consists of two layers. The physical layer is comprised of the computing resources (servers, processors) and network resources (routers, switches, links, wireless devices and optical components). Resources are managed by the OFELIA Control Framework (OCF). Furthermore, the control framework layer contains components which manage and monitor the applications and devices in the physical layer. Aggregate Managers and Resource Managers are components of this layer, which can be seen as the combination of three components: Expedient is the GUI and allows the connection and federation with different Aggregate Managers via its plugins; Aggregate Managers (AMs) enable experimenters to create both compute and network resources via the VT AM and OF AM respectively; Resource Managers directly interact with the physical layer, provisioning compute resources (OFELIA Xen Agent) or flow rules to establish the topology (FlowVisor). The FIBRE project [FIBRE] federates SDN testbeds distributed across Europe and Brazil. The FIBRE-EU system builds on top of the OFELIA OCF and incorporates several wireless nodes based on commercial Wi-Fi cards and Linux open source drivers. Unlike OFELIA, the FIBRE infrastructure is managed by different types of control and monitoring frameworks (CMFs). FIBRE deployed two top-domain authorities, one in Brazil and one in Europe, to manage and own resources in the respective continents. These inter-connected authorities interoperate to allow the federation of BR and EU testbeds. In Japan, GridARS [GRIDARS] provides a reference implementation of the Open Grid Forum (OGF) Network Services Interfaces Connection Service (NSI-CS) protocol standard. GridARS can coordinate multiple resources (services), such as a network connection, virtual machines and storage spaces, via the NSICS protocol. It provides experimenters a virtual infrastructure, which spans several cloud resources, realised by multiple management domains including commercial solutions. GridARS consists of three main components. 6. IANA Considerations No IANA considerations are applicable. Carrozzo & Pentikousis Expires January 7, 2016 [Page 16] Internet-Draft Recursive orchestration July 2015 7. Security Considerations This document proposes a new architecture for resource and VNF orchestration for the design of which security features are of utmost importance to proceed to operational deployments. Frameworks for Security in SDN are applicable to this document and are discussed in literature, for example, in [SDNSecurity], [SDNSecServ] and [SDNSecOF]. Security considerations regarding specific protocol interfaces are TBD. 8. Acknowledgements This work has been partially supported and funded by the European Commission through the FP7 ICT FELIX project (Federated Testbeds for Large Scale Infrastructure Experiments, grant agreement no. 608638) and the National Institute of Information and Communications Technology (NICT) in Japan. The views expressed here are those of the author only. The European Commission and NICT are not liable for any use that may be made of the information in this document. 9. Contributors Authors would like to acknowledge (in alphabetical order) the following contributors who have provided text, pointers, and ideas for this document: o B. Belter (PSNC, Poland) o C. Bermudo (i2CAT, Spain) o T. Kudoh (Univ. Tokyo/AIST, Japan) o J. Tanaka (KDDI, Japan) o B. Vermeulen(iMinds, Belgium) 10. Informative References [cloudstack] "Apache CloudStack documentation. Cloud Infrastructure Concepts", Available online at http://cloudstack.apache.org/docs/en- US/Apache_CloudStack/4.1.0/html/Admin_Guide/ cloud-infrastructure-concepts.html. [FELIX] "FELIX Project website", http://www.ict-felix.eu. Carrozzo & Pentikousis Expires January 7, 2016 [Page 17] Internet-Draft Recursive orchestration July 2015 [FELIX-D2.1] R. Krzywania, W. Bogacki, B. Belter, K. Pentikousis, T. Rothe, G. Carrozzo, N. Ciulli, C. Bermudo, T. Kudoh, A. Takefusa, J. Tanaka and B. Puype, , "FELIX Deliverable D2.1 - Experiment Use Cases and Requirements", Available online at http://www.ict-felix.eu., September 2013. [FELIX-D2.2] R. Krzywania, W. Bogacki, B. Belter, K. Pentikousis, T. Rothe, M. Broadbent, G. Carrozzo, N. Ciulli, R. Monno, C. Bermudo, A. Vico, C. Fernandez, T. Kudoh, A. Takefusa, J. Tanaka and B. Puype, , "FELIX Deliverable D2.2 - General Architecture and Functional Blocks", Available online at http://www.ict-felix.eu., February 2014. [FIBRE] T. Salmito, L. Ciuffo, I. Machado, M. Salvador, et al, , "FIBRE - An International Testbed for Future Internet Experimentation", 32th Simposio Brasileiro de Redes de Computadores e Sistemas Distribuidos (SBRC'14) , 2014. [GRIDARS] [15] A. Takefusa, H. Nakada, T. Kudoh, Y. Tanaka and S. Sekiguchi, , "GridARS: An Advance Reservation-based Grid Co-allocation Framework for Distributed Computing and Network Resources", Lecture Notes, Computer Science of the Job Scheduling Strategies for Parallel Processing (JSSPP) vol.4942, pp. 152-168, April 2008. [middlebox] A. Greenhalgh, F. Huici, M. Hoerdt, P. Papadimitriou, M. Handley, and L. Mathy, , "Flow Processing and the Rise of Commodity Network Hardware", ACM SIGCOMM Computer Communication Review Volume 39 issue 2, April 2009. [OFELIA] M. Sune, L. Bergesio, H. Woesner, T. Rothe, A. Kopsel, et al., , "Design and implementation of the OFELIA FP7 facility: The European OpenFlow testbed", The International Journal of Computer and Telecommunications Networking , December 2013. [openmano] "OpenMANO", Available online at https://github.com/nfvlabs/openmano/wiki. [openstack] "Scaling Openstack", Available online at http://docs.openstack.org/openstack-ops/content/ scaling.html. Carrozzo & Pentikousis Expires January 7, 2016 [Page 18] Internet-Draft Recursive orchestration July 2015 [os-heat] "OpenStack Orchestration - Heat", Available online at https://wiki.openstack.org/wiki/Heat. [SDNSecOF] Kloti, R., Kotronis, V., and P. Smith, , "OpenFlow: A Security Analysis", 21st IEEE International Conference on Network Protocols (ICNP) pp. 1-6, October 2013. [SDNSecServ] Scott-Hayward, S., O'Callaghan, G., and S. Sezer, , "SDN Security: A Survey", IEEE SDN for Future Networks and Services (SDN4FNS) pp. 1-7, 2013. [SDNSecurity] Kreutz, D., Ramos, F., and P. Verissimo, , "Towards Secure and Dependable Software-Defined Networks", Proceedings of the second ACM SIGCOMM workshop on Hot Topics in Software Defined Networking pp. 55-60, 2013. [SFA] L. Peterson, R. Ricci, A. Falk and J. Chase, , "Slice- based Federation Architecture (SFA) v2.0", , July 2010. Authors' Addresses Gino Carrozzo (Ed.) (editor) Nextworks via Livornese 1027 Pisa 56122 Italy Email: g.carrozzo@nextworks.it Kostas Pentikousis (Ed.) (editor) EICT Torgauer Strasse 12-15 Berlin 10829 Germany Email: k.pentikousis@eict.de Carrozzo & Pentikousis Expires January 7, 2016 [Page 19]