Internet Research Task Force (IRTF) R. Krishnan Internet Draft N. Figueira Category: Experimental Brocade Dilip Krishnaswamy IBM Research D. R. Lopez Telefonica I+D Steven Wright AT&T Tim Hinrichs VMware Ruby Krishnaswamy Orange Expires: August 2015 March 2, 2015 NFVIaaS Architectural Framework for Policy Based Resource Placement and Scheduling draft-krishnan-nfvrg-policy-based-rm-nfviaas-04 Abstract One of the goals of Network Functions Virtualization (NFV) is to offer the NFV infrastructure as a service to other SP customers - this is called NFVIaaS. Virtual Network Function (VNF) deployment in this paradigm will drive support for unique placement policies, given VNF's stringent service level specifications (SLS) required by customer SPs. Additionally, NFV DCs often have capacity, energy and other constraints - thus, optimizing the overall resource usage based on policy is an important part of the overall solution. The purpose of this document is to depict an architectural framework for policy based resource placement and scheduling in the context of NFVIaaS. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Krishnan Expires August 2015 [Page 1] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire in May 2015. Copyright Notice Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Conventions used in this document Table of Contents 1. Introduction...................................................3 2. NFVIaaS Architectural Framework for Policy Based Resource Placement and Scheduling..........................................3 3. System Analysis in a OpenStack Framework.......................5 4. Related Work...................................................9 5. Summary........................................................9 6. Future Work....................................................9 7. IANA Considerations............................................9 8. Security Considerations.......................................10 9. Contributors..................................................10 10. Acknowledgements.............................................10 11. References...................................................10 11.1. Normative References....................................10 11.2. Informative References..................................10 Authors' Addresses...............................................11 Krishnan Expires August 2015 [Page 2] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 1. Introduction One of the goals of NFV [ETSI-NFV-WHITE] is to offer the NFV infrastructure as a service to other SP customers - this is called NFVIaaS [ETSI-NFV-USE-CASES]. In this context, it may be desirable for a Service Provider to run virtual network elements (e.g., virtual routers, virtual firewalls, and etc. - these are called Virtual Network Functions - VNF) as virtual machine instances inside the infrastructure of another Service Provider. In this document, we call the former a customer SP and the latter an NFVIaaS SP. There are many reasons for a customer SP to require the services of an NFVIaaS SP, including: to meet performance requirements (e.g., latency or throughput) in locations where the customer SP does not have physical data center presence, to allow for expanded customer reach, regulatory requirements, and etc. As VNFs are virtual machines, their deployment in such NFVIaaS SPs would share some of the same placement restrictions (i.e., placement policies) as those intended for Cloud Services. However, VNF deployment will drive support for unique placement policies, given VNF's stringent service level specifications (SLS) required/imposed by customer SPs. Additionally, NFV DCs or NFV PoPs [ETSI-NFV-TERM] often have capacity, energy and other constraints - thus, optimizing the overall resource usage based on policy is an important part of the overall solution. The purpose of this document is to depict an architectural framework for policy based resource placement and scheduling in the context of NFVIaaS. 2. NFVIaaS Architectural Framework for Policy Based Resource Placement and Scheduling The policy engine performs policy-based resource placement and scheduling of Virtual Machines (VMs) in support for NFVIaaS. It determines optimized placement and scheduling choices based on the constraints specified in the policy. The NFVIaaS Architectural Framework for Policy Based resource placement and scheduling is based on the NFV policy architectural framework [IRTF-NFV-POLICY- ARCH].This is depicted in Figure 1. In one instantiation of this architecture, the policy engine would interface with the Measurement Collector to periodically retrieve instantaneous per-server CPU utilization, which it would then use to compute a table of per-server average CPU utilization. In an alternative instantiation of this architecture, the measurement Krishnan Expires August 2015 [Page 3] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 collector could itself compute per-server average CPU utilization. The latter approach reduces overhead, since it avoids too frequent pulling of stats from Ceilometer. The policy engine evaluates such policies based on an event trigger or a based on a programmable timer. Other average utilization parameters such as VM CPU utilization, VM Memory utilization, VM disk read IOPS etc. could be used by the policy engine to enforce other types of placement policies. |---------------------------------------| | Policy Engine | |Performs resource placement | |and scheduling function (proactive and | |dynamic policy enforcement) | |--------------------|------------------| | | |-----------|---------------| | Measurement Collector | | VM DB - CPU | | Utilization etc. | | | |---------------------------| Figure 1: NFVIaaS Architecture for Policy Based Resource Placement and Scheduling In an ETSI NFV Architectural Framework [ETSI-NFV-ARCH][NFV-MANO- SPEC], Policy Engine is part of the Orchestrator and the Measurement Collector is part of the Virtual Infrastructure Manager (VIM). Krishnan Expires August 2015 [Page 4] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 3. System Analysis in a OpenStack Framework Consider an NFVIaaS SP that owns a multitude of mini NFV data centers managed by OpenStack [OPENSTACK] where, - The Policy Engine function is performed by OpenStack Congress [OPENSTACK-CONGRESS-POLICY-ENGINE] - The Measurement Collector function is performed by OpenStack Celiometer [OPENSTACK-CELIOMETER-MEASUREMENT] - The Policy Engine has access to the OpenStack Nova database that stores records of mapping of virtual machines to physical servers. Exemplary Mini NFV DC configuration: An exemplary mini NFV DC configuration is depicted below. - There are 210 physical servers in 2U rack server configuration spread over 10 racks. - There are 4 types of physical servers each with a different system configuration and from a particular manufacturer. It is possible that the servers are from the same or different manufacturer. For the purpose of this example, server type 1 is described further. Server type 1 has 32 virtual CPUs and 128GB DRAM from manufacturer x. Assume 55 physical servers of type 1 per mini NFV DC. - There are 2 types of instances large.2 and large.3, which are described in the table below. Each parameter has a minimum guarantee and a maximum usage limit. |--------|----------------- |-------------------|---------------| |Instance|Virtual CPU Units |Memory (GB) |Minimum/Maximum| |Type |Minimum Guarantee |Minimum Guarantee |Physical Server| | |/Maximum Usage |/Maximum Usage |Utilization (%)| |--------|------------------|-------------------|---------------| |large.2 | 0/4 | 0/16 | 0/12.5 | |large.3 | 0/8 | 0/32 | 0/25 | Krishnan Expires August 2015 [Page 5] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 |--------|------------------|-------------------|---------------| Table 1: NFVIaaS Instance Types For the purpose of this example, the Mini NFV DC topology is considered static -- the above topology, including the network interconnection, is available through a simple file-based interface. Policy 1 (an exemplary NFV policy): Policy 1 is an exemplary NFV policy. In a descriptive language, Policy 1 is as follows - "For physical servers of type 1, there can be at most only one active physical server with average overall utilization less than 50%." The goal of this policy is to address the energy efficiency requirements described in the ETSI NFV Virtualization Requirements [ETSI-NFV-REQ]. Policy 1 is an example of reactive enforcement. Policy 2 (another exemplary NFV policy): Policy 2 is necessary to protect NFV servers from failures. In this example we consider failures of physical servers. Policy 2 is as follows - "Not more than one VM of the same HA group must be deployed on the same physical server". Note: There may be conditions (according to current Mini DC usage and policies of type 2 that are currently active) when there may not be any placement solution respecting both policies. It may be better to reformulate Policy 1? For example, "Minimize the number of physical servers with average overall utilization less than 50%" Policy 2 is an example of proactive and reactive enforcement. Various Module Interactions for Policy 1: The various module interactions with respect the architectural framework in Figure 1 for Policy 1 is described below. The policy calls for the identification of servers by type. OpenStack Congress would need to support server type, average CPU utilization, and be able to support additional performance parameters (in the future) to support additional types of placement policies. OpenStack Congress would run the policy periodically or based on events such as deleting/adding VMs etc. Initially, we could use a periodic timer based approach. In case OpenStack Congress Krishnan Expires August 2015 [Page 6] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 detects a violation, it determines optimized placement and scheduling choices so that the policy is not violated. OpenStack Congress could interface with OpenStack Celiometer to periodically retrieve instantaneous per-server CPU utilization, which it would then use to compute a table of per-server average CPU utilization. Alternatively, OpenStack Celiometer could itself compute per-server average CPU utilization which could be used by OpenStack Congress. The proposed module interactions in this NFVIaaS placement policy example are as depicted in the architectural framework in Figure 1. A key goal of Policy 1 above is to ensure that servers are not kept under low utilization, since servers have a non-linear power profile and exhibit relatively higher power wastage at lower utilization. For example, in the active idle state as much as 30% of peak power is consumed. At the physical server level, instantaneous energy consumption can be accurately measured through IPMI standard. At a customer instance level, instantaneous energy consumption can be approximately measured using an overall utilization metric, which is a combination of CPU utilization, memory usage, I/O usage, and network usage. Hence, the policy is written in terms of overall utilization and not power usage. The following example combines Policy 1 and Policy 2. For an exemplary maximum usage scenario, 53 physical servers could be under peak utilization (100%), 1 server (server-a) could be under partial utilization (62.5%) with 2 instances of type large.3 and 1 instance of type large.2 (this instance is referred as large.2.X1), and 1 server (server-b) could be under partial utilization (37.5%) with 3 instances of type large.2. Call these three instances large.2.X2, large.2.Y and large.2.Z One HA-group has been configured and two large.2 instances belong to this HA-group. To enforce Policy 2 large.2.X1 and large2.X2 that belong to the HA-group have been deployed in different physical servers, one in server-a and a second in server-b. When one of the large.3 instances mapped to server-a is deleted from physical server type 1, Policy 1 will be violated, since the overall utilization of server-a falls to 37,5%, since two servers are underutilized (below 50%) OpenStack Congress, on detecting the policy violation, uses various constraint based placement techniques to find the new placement(s) Krishnan Expires August 2015 [Page 7] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 for physical server type 1 to address Policy 1 violation without breaking Policy 2. Constrained based placement will be explored in a convex optimization framework [CONVEX-OPT]; some of the algorithms which would be considered are linear programming [LINEAR-PROGRAM], branch and bound [BRANCH-AND-BOUND], interior point methods, equality constrained minimization, non-linear optimization etc. Various new placement(s) are described below. 1) New placement 1: Move 2 of three instances of large.2 running on server-b to server-a. Overall utilization of server-a - 62,5%. Overall utilization of server-b - 25%. large.2.X2 must not be one of the migrated instances. 2) New placement 2: Move 1 instance of large.3 to server-b. Overall utilization of server-a - 12,5%. Overall utilization of server-b - 62.5%. A third solution consisting of moving 3 large.2 instances to server- a cannot be adopted since this breaks Policy 2. Another policy minimizing the number of migrations could allow choosing between solution (1) and (2). New placements 2 and 3 could be considered optimal, since they achieve maximal bin packing and open up the door for turning off server-a or server-b and maximizing energy efficiency. To detect violations of Policy 1, an example of a classification rule is expressed below in Datalog, the policy language used by OpenStack Congress. Database table exported by the Resource Placement and Scheduler for Policy 1 and Policy 2: The database table exported by the Resource Placement and Scheduler for Policy 1 is below. server_utilization (physical_server, overall_util) - Each database entry has the physical server and the calculated average overall utilization. vm_host_mapping(vm, server) - Each database entry gives the physical server on which VM is deployed. Krishnan Expires August 2015 [Page 8] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 anti-affinity_group(vm, group) - Each entry gives the anti-affinity group to which a VM belongs. Policy 1 (in Datalog [DATALOG] policy language): Policy 1 in a Datalog policy language is as follows. error (physical_server) :- nova [OPENSTACK-NOVA-COMPUTE]: node (physical_server, "type1"), resource placement and scheduler: server_utilization (physical_server, overall_util < 50) Policy 2 (in Datalog policy language): error(vm) :- anti-affinity_group(vm1, grp1), anti-affinity_group(vm2, grp2), grp1 != grp2, nova: vm host mapping(vm1, server-1), nova: vm host mapping(vm2,server-2), server-1 == server-2 4. Related Work A related proof of concept in ETSI NFV on placement and scheduling is [ETSI-NFV-POC-PLACEMENT]. 5. Summary 6. Future Work TBD 7. IANA Considerations This draft does not have any IANA considerations. Krishnan Expires August 2015 [Page 9] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 8. Security Considerations 9. Contributors Hesham ElBakoury Huawei Hesham.ElBakoury@huawei.com 10. Acknowledgements None. 11. References 11.1. Normative References 11.2. Informative References [ETSI-NFV-WHITE] "ETSI NFV White Paper," http://portal.etsi.org/NFV/NFV_White_Paper.pdf [ETSI-NFV-USE-CASES] "ETSI NFV Use Cases," http://www.etsi.org/deliver/etsi_gs/NFV/001_099/001/01.01.01_60/gs_N FV001v010101p.pdf [ETSI-NFV-REQ] "ETSI NFV Virtualization Requirements," http://www.etsi.org/deliver/etsi_gs/NFV/001_099/004/01.01.01_60/gs_N FV004v010101p.pdf [ETSI-NFV-ARCH] "ETSI NFV Architectural Framework," http://www.etsi.org/deliver/etsi_gs/NFV/001_099/002/01.01.01_60/gs_N FV002v010101p.pdf [ETSI-NFV-TERM] "ETSI NFV: Terminology for main concepts in NFV," http://www.etsi.org/deliver/etsi_gs/NFV/001_099/003/01.02.01_60/gs_N FV003v010201p.pdf [OPENSTACK] "OpenStack Open Source Software," https://www.openstack.org/ [OPENSTACK-CONGRESS-POLICY-ENGINE] "A policy as a service open source project in OpenStack," https://wiki.openstack.org/wiki/Congress Krishnan Expires August 2015 [Page 10] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 [OPENSTACK-CELIOMETER-MEASUREMENT] "OpenStack Celiometer," http://docs.openstack.org/developer/ceilometer/measurements.html [OPENSTACK-NOVA-COMPUTE] "OpenStack Nova," https://wiki.openstack.org/wiki/Nova [LINEAR-PROGRAM] Dmitris Alevras and Manfred W. Padberg, "Linear Optimization and Extensions: Problems and Solutions," Universitext, Springer-Verlag, 2001. [BRANCH-AND-BOUND] "Fundamentals of Algorithmics," G. Brassard and P. Bratley. [NFV-MANO-SPEC] "NFV Management and Orchestration Framework Specification," http://docbox.etsi.org/ISG/NFV/Open/Latest_Drafts/NFV-MAN001v061- %20management%20and%20orchestration.pdf [CONVEX-OPT] "Convex Optimization," https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf [ETSI-NFV-POC-PLACEMENT] "ETSI NFV Proof of Concept on Placement and Scheduling," http://nfvwiki.etsi.org/index.php?title=Constraint_based_Placement_a nd_Scheduling_for_NFV/Cloud_Systems [DATALOG] Ceri, S. et al., "What you always wanted to know about Datalog (and never dared to ask)," Knowledge and Data Engineering, IEEE Transactions on (Volume:1 , Issue: 1 ) [IRTF-NFV-POLICY-ARCH] Figueira, N. et al., "Policy Architecture and Framework for NFV and Cloud Services," Authors' Addresses Ram (Ramki) Krishnan Brocade Communications ramk@brocade.com Norival Figueira Brocade Communications nfigueir@Brocade.com Dilip Krishnaswamy IBM Research dilikris@in.ibm.com Krishnan Expires August 2015 [Page 11] Internet-Draft NFVIaaS Policy Resource Placement & Scheduling May 2015 Diego Lopez Telefonica I+D Don Ramon de la Cruz, 82 Madrid, 28006, Spain +34 913 129 041 diego.r.lopez@telefonica.com Steven Wright AT&T sw3588@att.com Tim Hinrichs VMWare thinrichs@vmware.com Ruby Krishnaswamy Orange ruby.krishnaswamy@orange.com Krishnan Expires August 2015 [Page 12]