Internet DRAFT - draft-irtf-nfvrg-policy-based-resource-management

draft-irtf-nfvrg-policy-based-resource-management







nfvrg                                                      R. Szabo, Ed.
Internet-Draft                                                  Ericsson
Intended status: Informational                               S. Lee, Ed.
Expires: May 3, 2017                                                ETRI
                                                             N. Figueira
                                                                 Brocade
                                                        October 30, 2016


                    Policy-Based Resource Management
          draft-irtf-nfvrg-policy-based-resource-management-02

Abstract

   abstract to be defined

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on May 3, 2017.

Copyright Notice

   Copyright (c) 2016 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.




Szabo, et al.              Expires May 3, 2017                  [Page 1]

Internet-Draft               Policy-based RM                October 2016


Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
     1.1.  Scope . . . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
   3.  Definitions . . . . . . . . . . . . . . . . . . . . . . . . .   3
   4.  Requirements  . . . . . . . . . . . . . . . . . . . . . . . .   4
   5.  Architecture Considerations . . . . . . . . . . . . . . . . .   4
     5.1.  MANO Architecture . . . . . . . . . . . . . . . . . . . .   5
     5.2.  Policies in the MANO Architecture . . . . . . . . . . . .   8
     5.3.  Global vs Local Policies  . . . . . . . . . . . . . . . .   9
     5.4.  Hierarchical Policy Framework . . . . . . . . . . . . . .  10
       5.4.1.  Mapping to Hierarchical Resource Orchestration  . . .  12
     5.5.  Policy Pub/Sub Bus  . . . . . . . . . . . . . . . . . . .  13
       5.5.1.  Pub/sub bus in the hierarchical framework . . . . . .  15
     5.6.  Policy Intent Statement versus Subsystem Actions and
           Configurations  . . . . . . . . . . . . . . . . . . . . .  17
     5.7.  Static vs Dynamic vs Autonomic Policies . . . . . . . . .  17
     5.8.  Policy Conflicts and Resolution . . . . . . . . . . . . .  17
     5.9.  Soft vs Hard Policy Constraints . . . . . . . . . . . . .  17
     5.10. Operational Policies for Resource management  . . . . . .  17
       5.10.1.  Operational Policies at NFVO . . . . . . . . . . . .  19
       5.10.2.  Operational Policies at VIM/WIM  . . . . . . . . . .  19
   6.  Policy-Based Resource Management Examples . . . . . . . . . .  20
     6.1.  Policy-Based Multipoint Ethernet Service  . . . . . . . .  20
     6.2.  Policy-Based NFV Placement  . . . . . . . . . . . . . . .  20
     6.3.  Policy-Based VNF-FG Management  . . . . . . . . . . . . .  20
     6.4.  Policy-Based Fault Management . . . . . . . . . . . . . .  22
   7.  Implementation Examples . . . . . . . . . . . . . . . . . . .  28
   8.  Gaps and Open Questions . . . . . . . . . . . . . . . . . . .  28
   9.  Conclusions . . . . . . . . . . . . . . . . . . . . . . . . .  28
     9.1.  Relation to other IETF/IRTF activities  . . . . . . . . .  28
   10. Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  28
   11. Contributors  . . . . . . . . . . . . . . . . . . . . . . . .  28
   12. IANA Considerations . . . . . . . . . . . . . . . . . . . . .  29
   13. Security Considerations . . . . . . . . . . . . . . . . . . .  29
   14. References  . . . . . . . . . . . . . . . . . . . . . . . . .  29
     14.1.  Normative References . . . . . . . . . . . . . . . . . .  29
     14.2.  Informative References . . . . . . . . . . . . . . . . .  29
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  32

1.  Introduction

   NFV "Point of Presence" (PoP) will be likely constrained in compute
   and storage capacity.  Since practically all NFV PoPs are foreseen to
   be distributed, inter-datacenter network capacity is also a
   constraint.  Additionally, energy is also a constraint, both as a
   general concern for NFV operators, and in particular for specific-



Szabo, et al.              Expires May 3, 2017                  [Page 2]

Internet-Draft               Policy-based RM                October 2016


   purpose NFV PoPs such as those in mobile base stations.  This draft
   focuses on the optimized resource management and workload
   distribution based on policy to address such contraints.

1.1.  Scope

   For the first version of the draft, only the research group currently
   adopted drafts (i.e., [I-D.norival-nfvrg-nfv-policy-arch],
   [I-D.irtf-nfvrg-resource-management-service-chain], and
   [I-D.unify-nfvrg-recursive-programming]) are considered as inputs to
   this document.  The initial goal is to summarize these inputs and to
   assess gaps and open questions.

2.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

3.  Definitions

   This document uses the terms of [ETSI-NFV-TERM]:

   o  MANO - Management and Orchestration: Describes the architecture
      framework to manage NFVI and orchestrate the allocation of
      resources needed by the NSs and VNFs.

   o  NF - Network Functions: A functional building block within a
      network infrastructure, which has well-defined external interfaces
      and a well-defined functional behavior.

   o  NFV Framework: The totality of all entities, reference points,
      information models and other constructs defined by the
      specifications published by the ETSI ISG NFV.

   o  NFVI - NFV Infrastructure: The NFV-Infrastructure is the totality
      of all hardware and software components which build up the
      environment in which VNFs are deployed.

   o  NFVI-PoP: A location or point of presence that hosts NFV
      infrastructure

   o  NFVO - Network Function Virtualization Orchestrator: The NFV
      Orchestrator is in charge of the network wide orchestration and
      management of NFV (infrastructure and software) resources, and
      realizing NFV service topology on the NFVI.





Szabo, et al.              Expires May 3, 2017                  [Page 3]

Internet-Draft               Policy-based RM                October 2016


   o  NS - Network service: A composition of network functions and
      defined by its functional and behavioural specification.

   o  VNF - Virtualized Network Function: An implementation of an NF
      that can be deployed on a Network Function Virtualization
      Infrastructure (NFVI).

   o  VNF-FG - VNF Forwarding Graph: A NF forwarding graph where at
      least one node is a VNF.

   Additionally, we use the following terms:

   o  NFP - Network Forwarding Path: The sequence of hardware/software
      switching ports and operations in the NFV network infrastructure
      as configured by management and orchestration that implements a
      logical VNF forwarding graph "link" connecting VNF "node" logical
      interfaces.

   o  Virtual Link: A set of connection points along with the
      connectivity relationship between them and any associated target
      performance metrics (e.g. bandwidth, latency, QoS).  The Virtual
      Link can interconnect two or more entities (VNF components, VNFs,
      or PNFs).

   o  Scaling: Ability to dynamically extend/reduce resources granted to
      the Virtual Network function (VNF) as needed.

   o  NFVIaaS: NFV infrastructure as a service to other SP customers.

   o  SDN: Software Defined Networking.

   o  BSS: Business Support Systems

   o  OSS: Operation Support Systems

   o  DC: Data Center

   o  VM: Virtual machine

4.  Requirements

   tbd

5.  Architecture Considerations







Szabo, et al.              Expires May 3, 2017                  [Page 4]

Internet-Draft               Policy-based RM                October 2016


5.1.  MANO Architecture

   According to the ETSI MANO framework [ETSI-NFV-MANO], an NFVO is
   split into two functions (see Figure 1):

   o  The orchestration of NFVI resources across multiple VIMs,
      fulfilling the Resource Orchestration functions.  The NFVO uses
      the Resource Orchestration functionality to provide services that
      support accessing NFVI resources in an abstracted manner
      independently of any VIMs, as well as governance of VNF instances
      sharing resources of the NFVI infrastructure

   o  The lifecycle management of Network Services, fulfilling the
      network Service Orchestration functions.

   Similarly, a VIM is split into two functions (see Figure 1):

   o  Orchestrating the allocation/upgrade/release/reclamation of NFVI
      resources (including the optimization of such resources usage),
      and

   o  managing the association of the virtualised resources to the
      physical compute, storage, networking resources.




























Szabo, et al.              Expires May 3, 2017                  [Page 5]

Internet-Draft               Policy-based RM                October 2016


                          +-------------------+
                          |NVFO               |
                          |  +--------------+ |
                          |  |NFVO:         | |
                          |  |Service       | |
                          |  |Lifecycle     | |
                          |  |Management    | |
                          |  +------+-------+ |
                          |         |         |
                          |  +------+-------+ |
                          |  |NFVO:         | |
                          |  |Resource      | |
                          |  |Orchestration | |
                          |  +--+---+----+--+ |
                          +-----|---|----|----+
                               /    |     \
                    /---------/     |      \------------\
                   /                |                    \
    +-------------|-----+  +--------|----------+  +------|------------+
    |VIM          |     |  |VIM     |          |  |VIM   |            |
    |  +----------+---+ |  |  +-----+--------+ |  |  +---+----------+ |
    |  |VIM:          | |  |  |VIM:          | |  |  |VIM:          | |
    |  |Orchestration | |  |  |Orchestration | |  |  |Orchestration | |
    |  |&             | |  |  |&             | |  |  |&             | |
    |  |Optimization  | |  |  |Optimization  | |  |  |Optimization  | |
    |  +------+-------+ |  |  +------+-------+ |  |  +------+-------+ |
    |         |         |  |         |         |  |         |         |
    |  +------+-------+ |  |  +------+-------+ |  |  +------+-------+ |
    |  |VIM:          | |  |  |VIM:          | |  |  |VIM:          | |
    |  |Virtualized 2 | |  |  |Virtualized 2 | |  |  |Virtualized 2 | |
    |  |Pys mapping   | |  |  |Pys mapping   | |  |  |Pys mapping   | |
    |  +--------------+ |  |  +--------------+ |  |  +--------------+ |
    +-------------------+  +-------------------+  +-------------------+

   Figure 1: Functional decomposition of the NFVO and the VIM according
                             to the ETSI MANO

   In Figure 2 we show various policies mapped to the MANO architecture
   (see Section 5.2 for more dicussions on policies in the MANO
   architeture):

   o  Tenant Policies: Tenant policies exist whenever a domain offers a
      virtualization service to more than one consumer.  User tenants
      may exists at the northbound of the NFVO.  Additionally, if a VIM
      exposes resource services to more than one NFVO, then each NFVO
      may appear as a tenant (virtualization consumer) at the northbound
      of the VIM.




Szabo, et al.              Expires May 3, 2017                  [Page 6]

Internet-Draft               Policy-based RM                October 2016


   o  Wherever virtualization services are produced or consumed
      corresponding export and import policies may exist.  Export
      policies govern the details of resources, capabilities, costs,
      etc. exposed to consumers.  In turn, consumers (tenants) apply
      import policies to filter, tweak, annotate resources and services
      received from their southbound domains.  An entity may at the same
      time consume and produce virtualization services hence apply both
      import and export policies.

   o  Operational policies support the business logic realized by the
      domain's ownership.  They are often associated with Operations or
      Business Support Systems (OSS or BSS) and frequently determine
      operational objectives like energy optimization, utilization
      targets, offered services, charging models, etc.  Operational
      policies may be split according to different control plane layers,
      for example, i) lifecycle and ii) resource management layers
      within the NFVO.


































Szabo, et al.              Expires May 3, 2017                  [Page 7]

Internet-Draft               Policy-based RM                October 2016


                               T1 T2...Tn
                                |  |   |
                          +-----|--|---|------+
                          |NVFO |  |   |      |            Tenant
                          |  +--+--+---+----+ |         <- Policies
                          |  |NFVO:         | |
           Operational    |  |Service       | |
           Policies->     |  |Lifecycle     | |
                          |  |Management    | |
                          |  +------+-------+ |
                          |         |         |
                          |  +------+-------+ |
                          |  |NFVO:         | |
           Operational    |  |Resource      | |
           Policies->     |  |Orchestration | |       ^
                          |  +--+---+----+--+ |       |Import
       to                 +-----|---|---------+       |Policies
       other NFVO              /    \
           \          +-------+      \
            \        /                \      to NFVO  ^
      +------\------|-----+            \       /      |Export
      |VIM    \     |     |             \     /       |Policies
      |  +-----+----+---+ |     +--------|----|-----+
      |  |VIM:          | |     |VIM     |    |     |     Tenant
      |  |Orchestration | |     |  +-----+----+---+ |  <- Policies
      |  |&             | |     |  |VIM:          | |
      |  |Optimization  | | .   |  |Orchestration | |
      |  +------+-------+ |  .  |  |&             | |  <- Operational
      |         |         |     |  |Optimization  | |     Policies
      |  +------+-------+ |     |  +------+-------+ |
      |  |VIM:          | |     |         |         |
      |  |Virtualized 2 | |     |  +------+-------+ |
      |  |Pys mapping   | |     |  |VIM:          | |  <- Operational
      |  +--------------+ |     |  |Virtualized 2 | |     Policies
      +-------------------+     |  |Pys mapping   | |
                                |  +--------------+ |
                                +-------------------+

               Figure 2: Policies within the MANO framework

5.2.  Policies in the MANO Architecture

   The current industry work in the area of policy for NFV is mostly
   considered in the framework of general cloud services, and typically
   focused on individual subsystems and addressing very specific use
   cases or environments.  For example, [ETSI-NFV-WHITE-PAPER] addresses
   network subsystem policy for network virtualization, [ODL-GB-POLICY]
   and [ODL-NIC-PROJECT] are open source projects in the area of network



Szabo, et al.              Expires May 3, 2017                  [Page 8]

Internet-Draft               Policy-based RM                October 2016


   policy as part of the OpenDaylight [ODL-SDN-CONTROLLER] software
   defined networking (SDN) controller framework, [RFC3060] specifies an
   information model for network policy, [VM-HOSTING-NET-CLUSTER]
   focuses on placement and migration policies for distributed virtual
   computing, [OPENSTACK-CONGRESS] is an open source project proposal in
   OpenStack [OPENSTACK] to address policy for general cloud
   environments.

   A policy framework applicable to the MANO architure must consider NFV
   services from the perspective of overall orchestration requirements
   for services involving multiple subsystems (e.g., Figure 1 and
   Figure 2).

   While this document discusses policy atributes as applicable to the
   MANO architecture, the general topic of policy has already been
   intensively studied and documented on numerous publications over the
   past 10 to 15 years (see [RFC3060], [POLICY-FRAMEWORK-WG], [RFC3670],
   [RFC3198], and [CERI-DATALOG] to name a few).  This document's
   purpose is to discuss and document a policy framework applicable to
   the MANO architecture using known policy concepts and theories to
   address the unique requirements of NFV services including multiple
   PoPs and networks forming hierarchical domain architectures
   [SDN-MULTI-DOMAIN].

   With the above goals, this document analyses "global versus local
   policies" (Section 5.3), a "hierarchical policy framework"
   (Section 5.4) to address the demanding and growing requirements of
   NFV environments, a "policy pub/sub bus in the hierarchical
   framework" (Section 5.5), "policy intent versus subsystem actions"
   (Section 5.6), "static versus dynamic versus autonomic policies"
   (Section 5.7), "policy conflict detection and resolution"
   (Section 5.8), and "soft versus hard policy constraints"
   (Section 5.9), which can be relevant to resource management in
   service chains [RESOURCE-MGMT-SERVICE-CHAIN].

5.3.  Global vs Local Policies

   Some policies may be subsystem specific in scope, while others may
   have broader scope and interact with multiple subsystems.  For
   example, a policy constraining certain customer types (or specific
   customers) to only use certain server types for VNF or Virtual
   Machine (VM) deployment would be within the scope of the compute
   subsystem.  A policy dictating that a given customer type (or
   specific customers) must be given "platinum treatment" could have
   different implications on different subsystems.  As shown in
   Figure 8, that "platinum treatment" could be translated to servers of
   a given performance specification in a compute subsystem and storage
   of a given performance specification in a storage subsystem.



Szabo, et al.              Expires May 3, 2017                  [Page 9]

Internet-Draft               Policy-based RM                October 2016


   Policies with broader scope, or global policies, would be defined
   outside affected subsystems and enforced by a global policy engine
   (Figure 3), while subsystem-specific policies or local policies,
   would be defined and enforced at the local policy engines of the
   respective subsystems.

   Examples of sub-system policies can include thresholds for
   utilization of sub-system resources, affinity/anti-affinity
   constraints with regard to utilization or mapping of sub-system
   resources for specific tasks, network services, or workloads, or
   monitoring constraints regarding under-utilization or over-
   utilization of sub-system resources.

   +----------------------------------------------------------------+
   |        +----------------------------------------------+        |
   |        |             Global Policy Engine             |        |
   |        +----------------------------------------------+        |
   |                                                                |
   |        +----------------------------------------------+        |
   |        |                Global Policies               |        |
   |        +----------------------------------------------+        |
   +----------------------------------------------------------------+
          ^                ^                ^                ^
          |                |                |                |
          V                V                V                V
   +-------------+  +-------------+  +-------------+  +-------------+
   |Compute      |  |Network      |  |Storage      |  |Whatever     |
   |Subsystem    |  |Subsystem    |  |Subsystem    |  |Subsystem    |
   |             |  |             |  |             |  |             |
   |Local Policy |  |Local Policy |  |Local Policy |  |Local Policy |
   |Engine       |  |Engine       |  |Engine       |  |Engine       |
   |             |  |             |  |             |  |             |
   |Local        |  |Local        |  |Local        |  |Local        |
   |Policies:    |  |Policies     |  |Policies     |  |Policies     |
   | P0, P1,     |  | P0, P1,     |  | P0, P1,     |  | P0, P1,     |
   |             |  |             |  |             |  |             |
   +-------------+  +-------------+  +-------------+  +-------------+

               Figure 3: Global versus Local Policy Engines

5.4.  Hierarchical Policy Framework

   So far, we have referenced compute, network, and storage as
   subsystems examples.  However, the following subsystems may also
   support policy engines and subsystem specific policies:

   o  SDN Controllers, e.g., OpenDaylight [ODL-SDN-CONTROLLER].




Szabo, et al.              Expires May 3, 2017                 [Page 10]

Internet-Draft               Policy-based RM                October 2016


   o  OpenStack [OPENSTACK] components such as, Neutron, Cinder, Nova,
      and etc.

   o  Directories, e.g., LDAP, ActiveDirectory, and etc.

   o  Applications in general, e.g., standalone or on top of
      OpenDaylight or OpenStack.

   o  Physical and virtual network elements, e.g., routers, firewalls,
      application delivery controllers (ADCs), and etc.

   o  Energy subsystems, e.g., OpenStack Neat [OPENSTACK-NEAT].

   Therefore, a policy framework may involve a multitude of subsystems.
   Subsystems may include other lower level subsystems, e.g., Neutron
   [OPENSTACK-NEUTRON] would be a lower level subsystem in the OpenStack
   subsystem.  In other words, the policy framework is hierarchical in
   nature, where the policy engine of a subsystem may be viewed as a
   higher level policy engine by lower level subsystems.  In fact, the
   global policy engine in Figure 3 could be the policy engine of a Data
   Center subsystem and multiple Data Center subsystems could be grouped
   in a region containing a region global policy engine.  In addition,
   one could define regions inside regions, hierarchically, as shown in
   Figure 4.

   Metro and wide-area network (WAN) used to interconnect data centers
   would also be independent subsystems with their own policy engines.
























Szabo, et al.              Expires May 3, 2017                 [Page 11]

Internet-Draft               Policy-based RM                October 2016


               To higher level domain
                        ^
            Region 1    |
            Domain      V
            +-------------------+        +-------------------+
            | +---------------+ |        | +---------------+ |
            | |Region 1 Global| |<------>| |WAN 1 Global   | |
            | |Policy Engine  | |        | |Policy Engine  | |
            | +---------------+ |        | +---------------+ |
            |                   |        |                   |
            | +---------------+ |        | +---------------+ |
            | |Whatever       | |        | |Whatever       | |
            | |Subsystems     | |        | |Subsystems     | |
            | |               | |        | |               | |
            | |Local Policy   | |        | |Local Policy   | |
            | |Engines        | |        | |Engines        | |
            | +---------------+ |        | +---------------+ |
            +-------------------+        +-------------------+
                          ^   ^
                          |   |
                          |   +-------------------------+
                          |                             |
            DC 1 Domain   V              DC N Domain    V
            +-------------------+        +-------------------+
            | +---------------+ |        | +---------------+ |
            | |DC 1 Global    | |        | |DC N Global    | |
            | |Policy Engine  | |        | |Policy Engine  | |
            | +---------------+ |        | +---------------+ |
            |                   |        |                   |
            | +---------------+ |        | +---------------+ |
            | |Whatever       | |        | |Whatever       | |
            | |Subsystems     | |        | |Subsystems     | |
            | |               | |        | |               | |
            | |Local Policy   | |        | |Local Policy   | |
            | |Engines        | |        | |Engines        | |
            | +---------------+ |        | +---------------+ |
            +-------------------+        +-------------------+

                 Figure 4: A Hierarchical Policy Framework

5.4.1.  Mapping to Hierarchical Resource Orchestration

   If the MANO framework is extended to multi layer hierarchies
   [I-D.unify-nfvrg-recursive-programming], then a potential mapping of
   the hierarchical policies to the MANO architecture is shown in
   Figure 5





Szabo, et al.              Expires May 3, 2017                 [Page 12]

Internet-Draft               Policy-based RM                October 2016


                                               T1 T2...Tn
                                             +--|--|----|---+
                                             |              |
                                             |Service       |
                                   Domain 4  |  Orchestrator|
     ************************************    +--+-----------+
                              T1 T2...Tn *******|******************
      Tenant Policies ->    +--|--|----|---+    |
                            | NFVO:        |    |
                            | Service      |    |    <-Domain Policies
                            | Lifecycle    |    |
                            | Orchestrator |    |
                            +-------+------+   /
     Domain 3                       |         /
     ********************         +-+---------+--+   <-Tenant Policies
                         *        |NFVO:         |
     ******************   *       |Rersource     |   <-Domain Policies
       T1 T2...Tn      *   *      |Orchestration |
     +--|--|----|---+   *   *     +--+---+-------+
     | NFVO:        |    *   ********|***|************************
     | Service      |     *         /    |
     | Lifecycle    |    /---------/     |           <-Domain Policies
     | Orchestrator |   /   *            |
     +---------+----+  |    *            |
               |       |    *            |
            +--+-------+---+*            |           <-Tenant Policies
            |              |*            |
            |NFVO:         |*            |           <-Domain Policies
            |Rersource     |*            |
            |Orchestration |*            |
            +------+-------+*            |
                  /|\       *   *********|**********
            +------+-------+*   * +------+-------+ * <-Tenant Policies
         +--|a/VIM:        |*   * |VIM:          | *
      +--|b |Resource      |*   * |Resource      | * <-Domain Policies
      |c |  |Orchestrationn|*   * |Orchestration | *
      |  |  +--------------+*   * +--------------+ *
     Domain 1               *   * Domain 2         *
     ************************   *                  *

          Figure 5: Policies in a Hierarchical Orchestration View

5.5.  Policy Pub/Sub Bus

   In [I-D.irtf-nfvrg-nfv-policy-arch] the authors argued for the need
   of policy subsystems to subscribe to policy updates at a higher
   policy level.  A policy publication/subscription (pub/sub) bus would
   be required as shown in Figure 6.



Szabo, et al.              Expires May 3, 2017                 [Page 13]

Internet-Draft               Policy-based RM                October 2016


   +----------------------------------------------------------------+
   |        +----------------------------------------------+        |
   |        |             Global Policy Engine             |        |
   |        +----------------------------------------------+        |
   |                                                                |
   |        +----------------------------------------------+        |
   |        |                Global Policies               |        |
   |        +----------------------------------------------+        |
   +----------------------------------------------------------------+
                                    ^
                                    |
                                    |
   Policy Pub/Sub Bus               V
     --------------------------------------------------------------
          ^                ^                ^                ^
          |                |                |                |
          |                |                |                |
          V                V                V                V
   +-------------+  +-------------+  +-------------+  +-------------+
   |Compute      |  |Network      |  |Storage      |  |Whatever     |
   |Subsystem    |  |Subsystem    |  |Subsystem    |  |Subsystem    |
   |             |  |             |  |             |  |             |
   |Local Policy |  |Local Policy |  |Local Policy |  |Local Policy |
   |Engine       |  |Engine       |  |Engine       |  |Engine       |
   |             |  |             |  |             |  |             |
   |Local        |  |Local        |  |Local        |  |Local        |
   |Policies:    |  |Policies     |  |Policies     |  |Policies     |
   | P0, P1,     |  | P0, P1,     |  | P0, P1,     |  | P0, P1,     |
   |             |  |             |  |             |  |             |
   +-------------+  +-------------+  +-------------+  +-------------+

                      Figure 6: A Policy Pub/Sub Bus

   A higher tier policy engine would communicate policies to lower tier
   policy engines using a policy pub/sub bus.  Conversely, lower tier
   policy engines would communicate their configured policies and
   services to the higher tier policy engine using the same policy pub/
   sub bus.  Such communications require each policy pub/sub bus to have
   a pre-defined/pre-configured policy "name space".  For example, a
   pub/sub bus could define services using the name space "Platinum",
   "Gold", and "Silver".  A policy could then be communicated over that
   pub/sub bus specifying a Silver service requirement.

   In a hierarchical policy framework, a policy engine may use more than
   one policy pub/sub bus, e.g., a policy pub/sub bus named "H" to
   communicate with a higher tier policy engine and a policy pub/sub bus
   named "L" to communicate with lower tier policy engines.  As the name
   spaces of policy pub/sub buses H and L may be different, the policy



Szabo, et al.              Expires May 3, 2017                 [Page 14]

Internet-Draft               Policy-based RM                October 2016


   engine would translate policies defined using the policy pub/sub bus
   H name space to policies defined using the policy pub/sub bus L name
   space, and vice-versa.

5.5.1.  Pub/sub bus in the hierarchical framework

   Figure 7 shows the Pub/sub bus in the hierarchical MANO framework.
   Policy communications would employ a policy pub/sub bus between the
   subsystems' policy engines in the policy hierarchy (see Section 5.4).
   The global NFVO subsystem should have visibility into the policies
   defined locally at each PoP to be able to detect any potential global
   policy conflicts, e.g., a local PoP administrator could add a local
   policy that violates or conflicts with a global policy.  In addition,
   the global NFVO subsystem would benefit from being able to import the
   currently configured services at each PoP.  The global NFVO would use
   such information to monitor global policy conformance and also to
   facilitate detection of policy violations when new global policies
   are created, e.g., a global level administrator is about to add a new
   global policy that, if committed, would make certain already
   configured services a violation of the policy.  The publication of
   subsystem service tables for consumption by a global policy engine is
   a concept used in the Congress [OPENSTACK-CONGRESS] OpenStack
   [OPENSTACK] project.




























Szabo, et al.              Expires May 3, 2017                 [Page 15]

Internet-Draft               Policy-based RM                October 2016


                    Customers
                        |
                        V
                 +--------------+
                 |  Web Portal  |
                 +--------------+
                        ^
                        |
                        V
               +-----------------+        +-----------------+
               | OSS/BSS         |        | Global NFVO     |
               | +-------------+ |        | +-------------+ |
               | |OSS/BSS      | | Policy | |NFVO         | |
               | |Policy Engine|<---------->|Policy Engine| |
               | +-------------+ |        | +-------------+ |
               |                 |        |        ^        |
               |         ...     |        |        | ...    |
               +-----------------+        +--------|--------+
                                                   |
                 Policy (Pub/Sub Bus)              V
                -------------------------------------------
                  ^                  ^                  ^
                  |                  |                  |
          +-------|-------+  +-------|-------+  +-------|-------+
          | PoP A |       |  | PoP B |       |  | PoP C |       |
          |       V       |  |       V       |  |       V       |
          | +-----------+ |  | +-----------+ |  | +-----------+ |
          | |Local NFVO | |  | |Local NFVO | |  | |Local NFVO | |
          | | +-------+ | |  | | +-------+ | |  | | +-------+ | |
          | | |Policy | | |  | | |Policy | | |  | | |Policy | | |
          | | |Engine | | |  | | |Engine | | |  | | |Engine | | |
          | | +-------+ | |  | | +-------+ | |  | | +-------+ | |
          | +-----------+ |  | +-----------+ |  | +-----------+ |
          |       ^       |  |       ^       |  |       ^       |
          |       |       |  |       |       |  |       |       |
          |       V       |  |       V       |  |       V       |
          | +-----------+ |  | +-----------+ |  | +-----------+ |
          | |VIM        | |  | |VIM        | |  | |VIM        | |
          | | +-------+ | |  | | +-------+ | |  | | +-------+ | |
          | | |Policy | | |  | | |Policy | | |  | | |Policy | | |
          | | |Engine | | |  | | |Engine | | |  | | |Engine | | |
          | | +-------+ | |  | | +-------+ | |  | | +-------+ | |
          | +-----------+ |  | +-----------+ |  | +-----------+ |
          |         ...   |  |        ...    |  |         ...   |
          +---------------+  +---------------+  +---------------+

         Figure 7: Pub/sub bus in the hierarchical MANO framework




Szabo, et al.              Expires May 3, 2017                 [Page 16]

Internet-Draft               Policy-based RM                October 2016


5.6.  Policy Intent Statement versus Subsystem Actions and
      Configurations

   Content to be merged

   +----------------------------------------------------------------+
   |   Policy: "a given customer must be given Platinum treatment"  |
   +----------------------------------------------------------------+
          ^                ^                ^                ^
          |                |                |                |
          V                V                V                V
   +-------------+  +-------------+  +-------------+  +-------------+
   |Compute      |  |Network      |  |Storage      |  |Whatever     |
   |Subsystem    |  |Subsystem    |  |Subsystem    |  |Subsystem    |
   |             |  |             |  |             |  |             |
   |Policy       |  |Policy       |  |Policy       |  |Policy       |
   |translation: |  |translation: |  |translation: |  |translation: |
   |             |  |             |  |             |  |             |
   |Install      |  |Give customer|  |Give customer|  | ...         |
   |customer VMs |  |the best QoS,|  |the fastest  |  |             |
   |on servers   |  |which        |  |SSD storage. |  |             |
   |with 3GHz    |  |translates   |  |             |  |             |
   |16-core Xeon |  |here to set  |  |             |  |             |
   |processors,  |  |DHCP to xx,  |  |             |  |             |
   |and etc.     |  |and etc.     |  |             |  |             |
   +-------------+  +-------------+  +-------------+  +-------------+

       Figure 8: Example of Subsystem Translations of Policy Actions

5.7.  Static vs Dynamic vs Autonomic Policies

   Content to be merged

5.8.  Policy Conflicts and Resolution

   Content to be merged

5.9.  Soft vs Hard Policy Constraints

   Content to be merged

5.10.  Operational Policies for Resource management









Szabo, et al.              Expires May 3, 2017                 [Page 17]

Internet-Draft               Policy-based RM                October 2016


                     +-------------------+
                     |NFVO               |  <- AAA Policy
                     |  +--------------+ |
                     |  |NFVO:         | |
                     |  |Service       | |  <- RD Policy
                     |  |Lifecycle     | |
                     |  |Management    | |
                     |  +------+-------+ |
                     |         |         |
                     |  +------+-------+ |
                     |  |NFVO:         | |
                     |  |Resource      | |  <- RS Policy
                     |  |Orchestration | |
                     |  +---+--+----+--+ |
                     +------|--|----|----+
                           /    \    \
                          /      \    \   +-------+
                         /        \    ---+  VNFM |
                        /          \      +---+---+
                       /            \         |
                      /              \        |
                     /                \       |
          +---------+---------+  +-----+------+------+
          |WIM                |  |VIM                |
          |  +--------------+ |  |  +--------------+ |
          |  |WIM:          | |  |  |VIM:          | |
          |  |Orchestration | |  |  |Orchestration | |  <- RA Policy
          |  |&             | |  |  |&             | |
          |  |Optimization  | |  |  |Optimization  | |
          |  +------+-------+ |  |  +------+-------+ |
          |         |         |  |         |         |
          |  +------+-------+ |  |  +------+-------+ |
          |  |WIM:          | |  |  |VIM:          | |
          |  |Virtualized 2 | |  |  |Virtualized 2 | |  <- RE Policy
          |  |Pys mapping   | |  |  |Pys mapping   | |
          |  +--------------+ |  |  +--------------+ |
          +-------------------+  +-------------------+

          Figure 9: Operational policies for resource management

   The use of NFVI resources for multiple network services can be
   optimized in various objectives as defined in the operational
   policies (as described in Section 5.2).

   The operational policies can be split to different layers of NFVO and
   VIM/WIM and they include 1) resource scheduling (RS) policy, resource
   adaptation (RD) policy and authentication, authorization, accounting
   (AAA) policy at NFVO, and 2) resource allocation (RA) policy and



Szabo, et al.              Expires May 3, 2017                 [Page 18]

Internet-Draft               Policy-based RM                October 2016


   resource embedding (RE) policy at VIM/WIM.  They can be mapped to the
   MANO architecture as shown in Figure 9.

5.10.1.  Operational Policies at NFVO

   During NS/VNF lifecycles, states of NFVI/WAN resources or the
   performance of VNF and VL instances may vary in time (e.g., the
   performance degradation due to incorrect placement or incorrect
   forwarding action).  Another concern for such dynamic changes is
   fail-over as a fundamental consideration, i.e., physical resources or
   virtualized resources in NFVI may fail during network services.
   These dynamic changes significantly could affect the overall
   performance for NS.  Therefore, such dynamic changes triggered during
   NS/VNF lifecycles should be coped with for guaranteeing the NS
   performance and the optimized resource usage.  Figure 9 shows that
   NFVO needs to enforce resource adaptation (RD) policy as an
   operational policy at NFVO.  RD policy supports how NFVO adapts the
   allocated NFVI/WAN resources (e.g., VM migration, scaling) by dealing
   with triggered variations.  RD policy engine can detect the changes
   from measurement and diagnosis from VNFM and/or VIM/WIM.

   Figure 9 also shows that NFVO needs to enforce resource scheduling
   (RS) policy.  RS policy determines the locations of VNF and VL
   instances that constitute NS across multiple PoPs and WANs while
   optimally allocating NFVI and WAN resources to the instances.

   In particular, RD and RA policies would consider a business model
   from OSS/BSS which specifies operational (or business) objectives
   (e.g., overall energy consumption and NFVI resource utilization)
   within its domain and with taking account of (on-boarded) network
   service descriptor (NSD) as an NS policy including the virtualization
   aspects of application feature, QoS parameters, affinity, anti-
   affinity rules, and so on.

   On the one hand, for the user authorization, authentication,
   authorization, accounting (AAA) policy may be needed.  Authentication
   policy provides a way of identifying a user while the authorization
   policy determines whether the user has the authority for virtualized
   resources (i.e., NFVI/WAN resources) to receive the network service
   or not.  Accounting policy measures the resources the user consumes
   during the network service.  This can include the amount of system
   time/data, and so on.

5.10.2.  Operational Policies at VIM/WIM

   As shown in Figure 9, RA policy supports how each subsystem (e.g.,
   compute, storage subsystem) in NFVI is allocated depending on the
   placement information from NFVO to further optimize the resource



Szabo, et al.              Expires May 3, 2017                 [Page 19]

Internet-Draft               Policy-based RM                October 2016


   usage.  Moreover, the assigned NFVI resources are embedded (or
   allocated) to physical resources in VIM/WIM depending on states and
   usage of resources by means of resource embedding (RE) policy as
   shown in Figure 9.  In other words, RE policy determines and
   coordinates how the allocated virtual resources are mapped to
   physical resources.  For example, RE policy may be updated when some
   physical resources are failed or a virtualization technique is
   changed.

6.  Policy-Based Resource Management Examples

6.1.  Policy-Based Multipoint Ethernet Service

   Content to be merged

6.2.  Policy-Based NFV Placement

   Content to be merged

6.3.  Policy-Based VNF-FG Management































Szabo, et al.              Expires May 3, 2017                 [Page 20]

Internet-Draft               Policy-based RM                October 2016


                          +-------------------+
                          | NFVO              |
                          | +---------------+ |
                          | |NFVO           | |
                          | |RS Policy      | |
                          | |Engine         | |
                          | +---------------+ |
                          |         ^         |
                          |         |    ...  |
                          +---------|---------+
                                    |
                                    V   Allocation information
             -----------------------------------------------
                ^                   ^                   ^
                |                   |                   |
        +-------|---------+ +-------|---------+ +-------|---------+
        |vPoP A |         | |WAN    |         | |vPoP B |         |
        |       V         | |       V         | |       V         |
        | +-------------+ | | +-------------+ | | +-------------+ |
        | |VIM A        | | | |WIM          | | | |VIM B        | |
        | | +---------+ | | | | +---------+ | | | | +---------+ | |
        | | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | |
        | | |Engine   | | | | | |Engine   | | | | | |Engine   | | |
        | | +---------+ | | | | +---------+ | | | | +---------+ | |
        | +-------------+ | | +-------------+ | | +-------------+ |
        |          ...    | |          ...    | |          ...    |
        |                 | |                 | |                 |
        | +-----+         | |                 | |                 |
        | |VNF-A|         | |                 | |                 |
        | +-----+         | |                 | |                 |
        |    #    +-----+ | |                 | |     +-----+     |
        |    #####|VNF-B|#############################|VNF-C|     |
        |   VL1   +-----+ | |        VL2      | |     +-----+     |
        |                 | |                 | |                 |
        +-----------------+ +-----------------+ +-----------------+

        ### NFP



                 Figure 10: Policy-based VNF-FG Management

   Another subsystem example for the policy framework is VNF-FG.  When
   VNF-FGs of end-to-end network services are realized, NFVI resources
   across multiple NFVI-PoPs and WAN resources that connect among them
   should be allocated to the VNF-FGs.  It depends on the target KPIs of
   individual VNF and VL instances that constitute VNF-FGs.  In
   particular, in case of VNF-FG, chained performances and capabilities



Szabo, et al.              Expires May 3, 2017                 [Page 21]

Internet-Draft               Policy-based RM                October 2016


   of VNF and VL instances need to be considered together with on VL
   instances the inter-connectivity between different NFVI-PoPs.  For
   example, if one of the VNF instances or VL instances along the VNF-FG
   gets overloaded, the end-to-end network service may also get
   affected.  Therefore, while features of such VNF-FG are carefully
   considered, proper operational policies for resource management (see
   Section 5.10) are required.

   As shown in Figure 10, consider a scenario where a user requests a
   VNF-FG composed of "VNF A-VL 1-VNF B-VL 2-VNF C".  For the VNF-FG, an
   RA policy is enforced in which it is designed to avoid over-
   utilization of PoP A and to reduce latency on VL 1.  Therefore, NFVO
   places VNF A, VNF B, and VL 1 on PoP A by consuming its computing and
   network resources to achieve low latency.  On the other hand, VL 2
   and VNF C is allocated to the resources of WAN and PoP B,
   respectively to avoid over-utilization of PoP A.

   On the one hand, dynamic changes such as a VNF failure significantly
   affect on the overall performance of VNF-FG since VNF-FG is a chain
   of VNF and VL instances.  Thus, such dynamic changes should be coped
   with by RD policy for guaranteeing the VNF-FG performance and the
   optimized resource usage.  A fault management for VNF-FG based on
   policy example is shown in Section 6.4.

6.4.  Policy-Based Fault Management


























Szabo, et al.              Expires May 3, 2017                 [Page 22]

Internet-Draft               Policy-based RM                October 2016


                          +-------------------+
                          | NFVO              |
                          | +---------------+ |
                          | |NFVO           | |
                          | |RS Policy      | |
                          | |Engine         | |
                          | +---------------+ |
                          |         ^         |
                          |         |    ...  |
                          +---------|---------+

                                    V
              ----------------------------------------------
                ^                   ^                   ^
                |                   |                   |
        +-------|---------+ +-------|---------+ +-------|---------+
        |vPoP A |         | |vPoP B |         | |vPoP C |         |
        |       V         | |       V         | |       V         |
        | +-------------+ | | +-------------+ | | +-------------+ |
        | |VIM A        | | | |VIM B        | | | |VIM C        | |
        | | +---------+ | | | | +---------+ | | | | +---------+ | |
        | | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | |
        | | |Engine   | | | | | |Engine   | | | | | |Engine   | | |
        | | +---------+ | | | | +---------+ | | | | +---------+ | |
        | +-------------+ | | +-------------+ | | +-------------+ |
        |          ...    | |          ...    | |          ...    |
        |                 | |                 | |                 |
        |    +-----+      | |     +-----+     | |     +-----+     |
        |    |VNF-A|##############|VNF-B|#############|VNF-C|     |
        |    +-----+ VL-1 | |     +-----+ VL-2| |     +-----+     |
        |                 | |        ^        | |                 |
        |                 | |        |        | |                 |
        |                 | |    (failure)    | |                 |
        |                 | |                 | |                 |
        +-----------------+ +-----------------+ +-----------------+

        ### NFP



                  Figure 11: Failure Scenario for VNF-FG










Szabo, et al.              Expires May 3, 2017                 [Page 23]

Internet-Draft               Policy-based RM                October 2016


                 +------------------------------------+
                 |NFVO                                |
                 | +--------------------------------+ |
                 | |       RD policy engine         | |
                 | |Adapts resources to the failed  | |
                 | |elements while guaranteeing     | |
                 | |performance                     | |
                 | +--------------------------------+ |
                 +------------------+-----------------+
                                    |
                                    |
                +-------------------+---------------------+
                |VNFM/VIM/WIM                             |
                | +-------------------------------------+ |
                | |       Diagnosis / Measurement       | |
                | |A failure event                      | |
                | |Throughputs of VNF and VL instances  | |
                | |Topological location...              | |
                | +-------------------------------------+ |
                +-----------------------------------------+


         Figure 12: Architecture for policy-based fault management

   As shown in Figure 11, consider a scenario that a VM related to VNF-B
   (i.e., a VNF-B instance) is failed in the given VNF-FG composed VNF-
   A, VNF-B, VNF-C in order.  Note that the NFVI and WAN resources are
   already allocated to the instances by RS policy.  For service
   continuity, failure of the VNF-B instance needs to be detected based
   on diagnosis function in VIM/VNFM and the failed one needs to be
   replaced with a new instance or to be assigned to the existing
   instance which is available.  The diagnosis and measurement function
   may collect current performance measures and location for instances
   as well as such a failure event.

















Szabo, et al.              Expires May 3, 2017                 [Page 24]

Internet-Draft               Policy-based RM                October 2016


                          +-------------------+
                          | NFVO              |
                          | +---------------+ |
                          | |NFVO           | |
                          | |RD Policy      | |
                          | |Engine         | |
                          | +---------------+ |
                          |         ^         |
                          |         |    ...  |
                          +---------|---------+
                                    |
                                    V
              ----------------------------------------------
                ^                   ^                   ^
                |                   |                   |
        +-------|---------+ +-------|---------+ +-------|---------+
        |vPoP A |         | |vPoP B |         | |vPoP C |         |
        |       V         | |       V         | |       V         |
        | +-------------+ | | +-------------+ | | +-------------+ |
        | |VIM A        | | | |VIM B        | | | |VIM C        | |
        | | +---------+ | | | | +---------+ | | | | +---------+ | |
        | | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | |
        | | |Engine   | | | | | |Engine   | | | | | |Engine   | | |
        | | +---------+ | | | | +---------+ | | | | +---------+ | |
        | +-------------+ | | +-------------+ | | +-------------+ |
        |          ...    | |          ...    | |          ...    |
        |                 | |                 | |                 |
        |    +-----+      | |     +-----+     | |     +-----+     |
        |    |VNF-A|      | |     |VNF-B|     | |     |VNF-C|     |
        |    +-----+      | |     +-----+     | |     +-----+     |
        |       #         | |        ^        | |        #        |
        |       # New     | |        |        | |        #        |
        |       # VL-1    | |    (failure)    |          #        |
        |    +-------+    | |                 | |        #        |
        |    |New    |####################################        |
        |    |VNF-B  |    | |                 | |     New VL-2    |
        |    +-------+    | |                 | |                 |
        |                 | |                 | |                 |
        +-----------------+ +-----------------+ +-----------------+

        ### NFP



                  Figure 13: Re-instantiation for VNF-FG

   In the first case where a VNF instantiation is needed, a new VNF
   instantiation is determined by the RD policy engine in NFVO.  For



Szabo, et al.              Expires May 3, 2017                 [Page 25]

Internet-Draft               Policy-based RM                October 2016


   example, NFVO may avoid replacement of VNF B on NFVI-PoP B owing to
   high possibility of failure.  Therefore, NFVO could instantiate VNF B
   on NFVI-PoP A or NFVI-PoP C with the setup of new connection points
   (CPs) while guaranteeing performance as shown in Figure 13.















































Szabo, et al.              Expires May 3, 2017                 [Page 26]

Internet-Draft               Policy-based RM                October 2016


                          +-------------------+
                          | NFVO              |
                          | +---------------+ |
                          | |NFVO           | |
                          | |RD Policy      | |
                          | |Engine         | |
                          | +---------------+ |
                          |         ^         |
                          |         |    ...  |
                          +---------|---------+
                                    |
                                    V
              ----------------------------------------------
                ^                   ^                   ^
                |                   |                   |
        +-------|---------+ +-------|---------+ +-------|---------+
        |vPoP A |         | |vPoP B |         | |vPoP C |         |
        |       V         | |       V         | |       V         |
        | +-------------+ | | +-------------+ | | +-------------+ |
        | |VIM A        | | | |VIM B        | | | |VIM C        | |
        | | +---------+ | | | | +---------+ | | | | +---------+ | |
        | | |RA Policy| | | | | |RA Policy| | | | | |RA Policy| | |
        | | |Engine   | | | | | |Engine   | | | | | |Engine   | | |
        | | +---------+ | | | | +---------+ | | | | +---------+ | |
        | +-------------+ | | +-------------+ | | +-------------+ |
        |          ...    | |          ...    | |          ...    |
        |                 | |                 | |                 |
        |    +-----+      | |     +-----+     | |     +-----+     |
        |    |VNF-A|      | |     |VNF-B|     | |     |VNF-C|     |
        |    +-----+      | |     +-----+     | |     +-----+     |
        |       #         | |        ^        | |        #        |
        |       #         | |        |        | |        #        |
        |       # New     | |    (failure)    | |        #        |
        |       # VL-1    | |                 | |        # New    |
        |       #         | |     +-------+   | |        # VL-2   |
        |       ##################| VNF-B1|###############        |
        |                 | |     +-------+   | |                 |
        +-----------------+ +-----------------+ +-----------------+

        ### NFP



                 Figure 14: No Re-instantiation for VNF-FG

   In the second case where no VNF instantiation is needed since a
   redundant VNF exists, the available VNF-B instance can used by the
   VNF-FG.  For example, a redundant VNF B instance exists in NFVI-PoP



Szabo, et al.              Expires May 3, 2017                 [Page 27]

Internet-Draft               Policy-based RM                October 2016


   B.  Therefore, NFVO selects the instance and re-constructs two VLs as
   shown in Figure 14, and the corresponding NS can be continued without
   re-instantiation.

7.  Implementation Examples

   tbd

8.  Gaps and Open Questions

   tbd

9.  Conclusions

   tbd

9.1.  Relation to other IETF/IRTF activities

   tbd

10.  Acknowledgements

   The research leading to some of the results described in this
   document has received funding from the European Union Seventh
   Framework Programme (FP7/2007-2013) under grant agreement no. 619609
   - the UNIFY project.  The views expressed here are those of the
   authors only.  The European Commission is not liable for any use that
   may be made of the information in this document.

11.  Contributors

   This document is the result of merging multiple drafts.  This section
   acknowledges those who provided important ideas and text into this
   document.

   o  Z.  Qiang (Ericsson), M.  Kind (DT-AG) from
      [I-D.unify-nfvrg-recursive-programming]

   o  R.  Krishnan (Dell), D.  Lopez (Telefonica) and S.  Wright (AT&T)
      from [I-D.irtf-nfvrg-nfv-policy-arch]

   o  S.  Lee (ETRI), S.  Pack (KU), M-K.  Shin (ETRI) and E.  Paik (KT)
      from [I-D.irtf-nfvrg-resource-management-service-chain]








Szabo, et al.              Expires May 3, 2017                 [Page 28]

Internet-Draft               Policy-based RM                October 2016


12.  IANA Considerations

   tbd

13.  Security Considerations

   tbd

14.  References

14.1.  Normative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <http://www.rfc-editor.org/info/rfc2119>.

   [RFC3060]  Moore, B., Ellesson, E., Strassner, J., and A. Westerinen,
              "Policy Core Information Model -- Version 1
              Specification", RFC 3060, DOI 10.17487/RFC3060, February
              2001, <http://www.rfc-editor.org/info/rfc3060>.

   [RFC3198]  Westerinen, A., Schnizlein, J., Strassner, J., Scherling,
              M., Quinn, B., Herzog, S., Huynh, A., Carlson, M., Perry,
              J., and S. Waldbusser, "Terminology for Policy-Based
              Management", RFC 3198, DOI 10.17487/RFC3198, November
              2001, <http://www.rfc-editor.org/info/rfc3198>.

   [RFC3670]  Moore, B., Durham, D., Strassner, J., Westerinen, A., and
              W. Weiss, "Information Model for Describing Network Device
              QoS Datapath Mechanisms", RFC 3670, DOI 10.17487/RFC3670,
              January 2004, <http://www.rfc-editor.org/info/rfc3670>.

14.2.  Informative References

   [CERI-DATALOG]
              Ceri, S. and others, "What you always wanted to know about
              Datalog (and never dared to ask", IEEE Transactions on
              Knowledge and Data Engineering, (Volume: 1, Issue: 1),
              August 2002.

   [ETSI-NFV-MANO]
              ETSI, "Network Function Virtualization (NFV) Management
              and Orchestration V0.6.3", October 2014.

   [ETSI-NFV-PER001]
              ETSI, "Network Function Virtualization: Performance and
              Portability Best Practices v1.1.1", June 2014.



Szabo, et al.              Expires May 3, 2017                 [Page 29]

Internet-Draft               Policy-based RM                October 2016


   [ETSI-NFV-TERM]
              ETSI, "NFV Terminology for Main Concepts in NFV", October
              2013.

   [ETSI-NFV-WHITE-PAPER]
              ETSI NFV White Paper, "Network Functions Virtualisation,
              An Introduction, Benefits, Enablers, Challenges, & Call
              for Action",
              <http://portal.etsi.org/NFV/NFV_White_Paper.pdf>.

   [I-D.ietf-bmwg-virtual-net]
              Morton, A., "Considerations for Benchmarking Virtual
              Network Functions and Their Infrastructure", draft-ietf-
              bmwg-virtual-net-04 (work in progress), August 2016.

   [I-D.irtf-nfvrg-nfv-policy-arch]
              Figueira, N., Krishnan, R., Lopez, D., Wright, S., and D.
              Krishnaswamy, "Policy Architecture and Framework for NFV
              Infrastructures", draft-irtf-nfvrg-nfv-policy-arch-04
              (work in progress), September 2016.

   [I-D.irtf-nfvrg-resource-management-service-chain]
              Lee, S., Pack, S., Shin, M., Paik, E., and R. Browne,
              "Resource Management in Service Chaining", draft-irtf-
              nfvrg-resource-management-service-chain-03 (work in
              progress), March 2016.

   [I-D.liu-bmwg-virtual-network-benchmark]
              Liu, V., Liu, D., Mandeville, B., Hickman, B., and G.
              Zhang, "Benchmarking Methodology for Virtualization
              Network Performance", draft-liu-bmwg-virtual-network-
              benchmark-00 (work in progress), July 2014.

   [I-D.norival-nfvrg-nfv-policy-arch]
              Figueira, N., Krishnan, R., Lopez, D., and S. Wright,
              "Policy Architecture and Framework for NFV
              Infrastructures", draft-norival-nfvrg-nfv-policy-arch-04
              (work in progress), June 2015.

   [I-D.unify-nfvrg-recursive-programming]
              Szabo, R., Qiang, Z., and M. Kind, "Towards recursive
              virtualization and programming for network and cloud
              resources", draft-unify-nfvrg-recursive-programming-02
              (work in progress), October 2015.







Szabo, et al.              Expires May 3, 2017                 [Page 30]

Internet-Draft               Policy-based RM                October 2016


   [ODL-GB-POLICY]
              "OpenDaylight Group Based Policy",
              <https://wiki.opendaylight.org/view/
              Project_Proposals:Group_Based_Policy_Plugin>.

   [ODL-NIC-PROJECT]
              "OpenDaylight Network Intent Composition Project",
              <https://wiki.opendaylight.org/index.php?title=Network_Int
              ent_Composition:Main#Friday_8AM_Pacific_Time>.

   [ODL-SDN-CONTROLLER]
              "OpenDaylight SDN Controller",
              <http://www.opendaylight.org/>.

   [OPENSTACK]
              "OpenStack", <http://www.openstack.org/>.

   [OPENSTACK-CONGRESS]
              "OpenStack Congress", <https://wiki.openstack.org/wiki/
              Congress>.

   [OPENSTACK-NEAT]
              "OpenStack Neat", <http://openstack-neat.org/>.

   [OPENSTACK-NEUTRON]
              "OpenStack Neutron", <https://wiki.openstack.org/wiki/
              Neutron>.

   [POLICY-FRAMEWORK-WG]
              "Policy Framework Working Group (IETF)",
              <http://www.ietf.org/wg/concluded/policy.html>.

   [RESOURCE-MGMT-SERVICE-CHAIN]
              Lee, S. and others, "Resource Management in Service
              Chaining", <https://datatracker.ietf.org/doc/draft-irtf-
              nfvrg-resource-management-service-chain/>.

   [SDN-MULTI-DOMAIN]
              Figueira, N. and R. Krishnan, "SDN Multi-Domain
              Orchestration and Control: Challenges and Innovative
              Future Directions", IEEE International Conference on
              Computing (ICNC), February 2015.

   [VM-HOSTING-NET-CLUSTER]
              Grit, L. and others, "Virtual Machine Hosting for
              Networked Clusters: Building the Foundations for
              "Autonomic" Orchestration", Virtualization Technology in
              Distributed Computing (VTDC), 2006.



Szabo, et al.              Expires May 3, 2017                 [Page 31]

Internet-Draft               Policy-based RM                October 2016


Authors' Addresses

   Robert Szabo (editor)
   Ericsson
   Konyves Kaman krt. 11
   Budapest, EMEA  1097
   Hungary

   Phone: +36703135738
   Email: robert.szabo@ericsson.com


   Seungik Lee (editor)
   ETRI
   218 Gajeong-ro Yuseung-Gu
   Daejeon  305-700
   Korea

   Phone: +82 42 860 1483
   Email: seungiklee@etri.re.kr


   Norival Figueira
   Brocade

   Email: nfigueir@Brocade.com

























Szabo, et al.              Expires May 3, 2017                 [Page 32]