Internet DRAFT - draft-khs-forces-lfb-subsidiary-management

draft-khs-forces-lfb-subsidiary-management







ForCES WG                                                  B. Khasnabish
Internet-Draft                                              ZTE TX, Inc.
Intended status: Standards Track                           E. Haleplidis
Expires: July 18, 2015                              University of Patras
                                                      J. Hadi Salim, Ed.
                                                       Mojatatu Networks
                                                        January 14, 2015


     IETF ForCES Logical Function Block (LFB) Subsidiary Management
             draft-khs-forces-lfb-subsidiary-management-05

Abstract

   Deployment experience has demonstrated the value of using the
   Forwarding and Control Element Separation (ForCES) architecture to
   manage resources other than packet forwarding.  In that spirit, the
   Forwarding Element Manager (FEM) is modelled by creating a Logical
   Functional Block (LFB) to represent its functionality.  We refer to
   this LFB as the FE Configuration (FEC) LFB.  A Control Element (CE)
   that controls a Forwarding Element's (FE) resources can also manage
   its configuration via the FEC LFB.  This document introduces the FEC
   LFB, an LFB that specifies the configuration parameters of an FE.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at http://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on July 18, 2015.

Copyright Notice

   Copyright (c) 2015 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents



Khasnabish, et al.        Expires July 18, 2015                 [Page 1]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
     1.1.  Requirements Language . . . . . . . . . . . . . . . . . .   5
     1.2.  Definitions . . . . . . . . . . . . . . . . . . . . . . .   5
   2.  Use cases . . . . . . . . . . . . . . . . . . . . . . . . . .   6
     2.1.  FE integration into an NE . . . . . . . . . . . . . . . .   6
     2.2.  CE associations . . . . . . . . . . . . . . . . . . . . .   6
     2.3.  New LFB class installation  . . . . . . . . . . . . . . .   7
   3.  Applicability statement . . . . . . . . . . . . . . . . . . .   7
     3.1.  FE Integrated . . . . . . . . . . . . . . . . . . . . . .   7
     3.2.  Virtual FEs . . . . . . . . . . . . . . . . . . . . . . .   7
     3.3.  FEM . . . . . . . . . . . . . . . . . . . . . . . . . . .   7
   4.  FEC Library . . . . . . . . . . . . . . . . . . . . . . . . .   8
     4.1.  Frame Definitions . . . . . . . . . . . . . . . . . . . .   8
     4.2.  Datatype Definitions  . . . . . . . . . . . . . . . . . .   8
     4.3.  Metadata Definitions  . . . . . . . . . . . . . . . . . .   8
     4.4.  FEC . . . . . . . . . . . . . . . . . . . . . . . . . . .   8
       4.4.1.  Data Handling . . . . . . . . . . . . . . . . . . . .   9
       4.4.2.  Components  . . . . . . . . . . . . . . . . . . . . .   9
       4.4.3.  Capabilities  . . . . . . . . . . . . . . . . . . . .   9
       4.4.4.  Events  . . . . . . . . . . . . . . . . . . . . . . .   9
   5.  XML for FEC LFB . . . . . . . . . . . . . . . . . . . . . . .   9
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .  13
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  13
     7.1.  LFB Class Names and LFB Class Identifiers . . . . . . . .  13
   8.  Acknowledgments . . . . . . . . . . . . . . . . . . . . . . .  14
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  14
     9.1.  Normative References  . . . . . . . . . . . . . . . . . .  14
     9.2.  Informative References  . . . . . . . . . . . . . . . . .  14
   Appendix A.  Appendix . . . . . . . . . . . . . . . . . . . . . .  15
     A.1.  Use of Virtualized ForCES Elements  . . . . . . . . . . .  15
       A.1.1.  Use of Virtualized CEs  . . . . . . . . . . . . . . .  15
       A.1.2.  Use of Virtualized FEs  . . . . . . . . . . . . . . .  15
       A.1.3.  Generic Lifecycle of Physical/Virtual Elements  . . .  15
     A.2.  Potential Scenarios . . . . . . . . . . . . . . . . . . .  16
       A.2.1.  Recovery from FE failure  . . . . . . . . . . . . . .  16
       A.2.2.  Recovery from CE failure  . . . . . . . . . . . . . .  18
       A.2.3.  Load Balancing  . . . . . . . . . . . . . . . . . . .  19
       A.2.4.  Orchestration . . . . . . . . . . . . . . . . . . . .  19



Khasnabish, et al.        Expires July 18, 2015                 [Page 2]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


       A.2.5.  Generic LFB Lifecycle Management  . . . . . . . . . .  19
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  19

1.  Introduction

   Deployment experience has demonstrated the value of using the
   Forwarding and Control Element Separation (ForCES) architecture to
   manage resources other than packet forwarding.  In that spirit, the
   Forwarding Element Manager (FEM) is modelled by creating a Logical
   Functional Block (LFB) to represent its functionality.  We refer to
   this LFB as the FE Configuration (FEC) LFB.  A Control Element (CE)
   that controls a Forwarding Element's (FE) resources can also manage
   its configuration via the FEC LFB.  This document introduces the FEC
   LFB, an LFB that specifies the configuration parameters of an FE.

   On a running FE, a CE application may update an FE's runtime
   configuration via the FEC LFB.


































Khasnabish, et al.        Expires July 18, 2015                 [Page 3]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


                              ForCES Network Element
                             +-------------------------------------+
                             |         +---------------------+     |
                             |         | Control Application |     |
                             |         +--+--------------+---+     |
                             |            |              |         |
                             |            |              |         |
      --------------   Fc    | -----------+--+      +-----+------+ |
      | CE Manager |---------+-|     CE 1    |------|    CE 2    | |
      --------------         | |             |  Fr  |            | |
            |                | +-+---------+-+      +------------+ |
            | Fl             |   |         | Fp/Ff     /           |
            |                |   |         +--------+ /            |
            |                |   |Fp/Ff             |/             |
            |                |   |                  |              |
            |                |   |        Fp/Ff    /|----+         |
            |                |   |       /--------/      |         |
      --------------     Ff  | ---+----------      --------------  |
      | FE Manager |---------+-|     FE 1   |  Fi  |     FE 2   |  |
      --------------         | |            |------|            |  |
                             | --------------      --------------  |
                             |   |  |  |  |          |  |  |  |    |
                             ----+--+--+--+----------+--+--+--+-----
                                 |  |  |  |          |  |  |  |
                                 |  |  |  |          |  |  |  |
                                   Fi/f                   Fi/f

          Fp: CE-FE interface
          Fc: Interface between the CE Manager and a CE
          Ff: Interface between the FE Manager and an FE
          Fl: Interface between the CE Manager and the FE Manager
          Fi/f: FE external interface

                  Figure 1: ForCES Architectural Diagram

   Figure 1 shows a control application manipulating, at runtime, FE
   config via the FEC LFB control.  The above illustration is derived
   from Figure 1 in [RFC3746] with modifications showing the messaging
   for Ff (FEM to FE interface) going via the standard Fp plane.  This
   is merely to demonstrate that the messaging is happening via the
   traditional Fp interface to the FEM/FEC; it does not however suggest
   moving away from the Ff interface.

   The FEC LFB describes the configuration parameters of an FE, namely,
   the FEID, the FE IP address(es), the CEs it should be associated
   with, as well as the LFBs that it supports.





Khasnabish, et al.        Expires July 18, 2015                 [Page 4]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


   This document assumes that FEs are already booted.  The FE's
   configuration can then be updated at runtime via the FEC LFB for
   runtime config purposes.  This document does not specify or
   standardize the FEM-FE (Ff) interface as depicted in [RFC3746].  This
   document describes a mechanism with which a CE can instruct the FEC
   for FE management using ForCES.

   In the case where we have a pool of unused packet processing
   resources that can be utilized as FEs, the FEC can be utilized to
   instruct the FE resource to join the Network Element cluster.  The
   initiation would involve control of the creation, configuration, and
   resource assignment of FEs so as to be part of an NE.  Appendix A
   describes how a resource pool of virtual machines could be initiated
   as basic CEs or FEs via an orchestration system and subsequently
   initiated into being part of an FE via the FEM.  Again, it should be
   emphasized that the pools of FEs and CEs are already booted up by
   some resource owner, e.g. an FEM or an Element Management System
   (EMS).  Subsidiary management provides the LFB library necessary, to
   manage and configure at runtime, these FEs to disconnect from the
   "resource pool CE" and join one or more CEs in the running NE.

   This work item makes no assumption of whether FE resources are
   physical or virtual.  In fact, the LFB library provided here is
   applicable to both.  Thus it can also be useful in addressing control
   of virtual FEs where individual FEM Managers can be addressed to
   control the creation, configuration, and resource assignment of such
   virtual FEs within a physical FE.

1.1.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

1.2.  Definitions

   This document follows the terminology defined by [RFC3654],
   [RFC3746], [RFC5810] and [RFC5812].  In particular, the reader is
   expected to be familiar with the following terms:

   o  Logical Functional Block (LFB)

   o  Forwarding Element (FE)

   o  Control Element (CE)

   o  ForCES Network Element (NE)




Khasnabish, et al.        Expires July 18, 2015                 [Page 5]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


   o  FE Manager (FEM)

   o  CE Manager

   o  ForCES Protocol

   o  ForCES Protocol Layer (ForCES PL)

   o  ForCES Protocol Transport Mapping Layer (ForCES TML)

2.  Use cases

   In this section we present sample use cases to illustrate the need
   and usefulness of the FEC LFB.

   All use cases assume that an FEs and CEs have already been
   bootstrapped and instantiated but have not joined the NE.  These FEs
   and CEs belong to a pool of untapped resources.

2.1.  FE integration into an NE

   The CE may request, for reasons such as performance, redundancy,
   load-sharing or new functionality request, to incorporate a new FE in
   the NE.  The CE would be required to specify the following
   parameters.  Firstly the FEID in order for the new FE to be uniquely
   identified within the NE.  Second the FE IP address to bind to, IPv4
   or IPv6.  Thirdly the LFBs to be instantiated within the FE, by means
   of providing the LFB class, LFB version and LFB name.  Finally the
   CEs that the new FE should associate within the NE as soon as it is
   integrated within the NE.  This includes the CE IDs as well as their
   corresponding CE IP addresses.

2.2.  CE associations

   A CE may request for redundancy reasons that an FE to be associated
   to another CE as a backup at runtime.  To achieve this goal, the
   master CE specifies the CEID of the new backup CE (to be uniquely
   identified within the NE) and the CE's IP address (IPv4 or IPv6, or
   IP addresses should the CE support multiple interfaces).

   The CE will configure all FEC LFBs to the FEs within the NE of the CE
   ID and CE IP addresses in order for the FEs to perform the necessary
   actions ordered by the CE and described by [RFC7121].

   the master CE, e.g. detecting a malfunctioning CE, could remove a
   backup CE from the FE.





Khasnabish, et al.        Expires July 18, 2015                 [Page 6]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


2.3.  New LFB class installation

   A CE can learn via the capability of FEC LFB whether an FE is capable
   of loading new LFB classes.  Provided that the FE supports new LFB
   class loading, the CE can request a new LFB to be installed and
   supported by the FE.

   To load an LFB class on an FE, the CE will have to provide the LFB
   class and the LFB class version.  There are optional fields which may
   be need to be described, depending on the implementation (out of
   scope for this document).  Example:

      location of the LFB Class to be installed and/or mechanism to
      download it.  The exact detail of the location semantics is
      implementation specific and out of scope of this document.

      Parameters needed by the LFB class module to allow for its
      initialization

3.  Applicability statement

   Examples of FEC usage are the following, but not limiting, three
   usage scenarios.  These scenarios are not implementation details, but
   rather depict how the FEC class can be used to achieve the intended
   subsidiary mechanism for manipulating the configuration of FEs.

3.1.  FE Integrated

   Only one instance of the FEC class can exist and is directly related
   to the FE.  The configuration parameters pertain to the parent FE.

3.2.  Virtual FEs

   In the case of the FE software that has hierarchical virtual FEs,
   multiple instances of the FEC class can exist, one per each virtual
   FE.

3.3.  FEM

   The third scenario, pertains to FEC LFB implementation as FE Manager
   paremeters.  In such a case, the FE configurations are locally and
   logically centralized by the FE manager.  The FE manager may hold
   multiple instances of the FEC class in the FEM, one per each FE.
   Using the ForCES protocol a CE, through the Fp interface, or a CE
   Manager via the Fl interface, will instruct the FEM to change the
   configuration of the FEs.  The FEM may hold more information
   pertaining the NE, such as the topology and chaining of FEs which the




Khasnabish, et al.        Expires July 18, 2015                 [Page 7]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


   CE would require to alter, along with the FE changes.  In such a case
   the Ff interface is out of scope.

4.  FEC Library

4.1.  Frame Definitions

   This LFB does not define any frames

4.2.  Datatype Definitions

   This library defines the following datatypes.

   +----------+-----------------------------------------+--------------+
   | DataType | Type                                    | Synopsis     |
   | Name     |                                         |              |
   +----------+-----------------------------------------+--------------+
   | IPs      | A Struct of 2 components. IPv4          | A struct     |
   |          | (byte[4]) and IPv6 (byte[16])           | that defines |
   |          | addresses.                              | an IPv4 and  |
   |          |                                         | an IPv6      |
   |          |                                         | address      |
   | LFBDefs  | A Struct that contains three            | A struct     |
   |          | components. The LFB Class ID (uint32),  | that defines |
   |          | the LFB version (string) and optional   | basic LFB    |
   |          | the LFB name (string) and the location  | definitions  |
   |          | of the LFB where it will be retrieved   |              |
   |          | from.                                   |              |
   | CEParams | A Struct that contains two components.  | A struct     |
   |          | A CE's ID (uint32) and the CE's IPs     | that defines |
   |          | (array of IPs)                          | CE           |
   |          |                                         | parameters.  |
   +----------+-----------------------------------------+--------------+

                              FEM Data Types

4.3.  Metadata Definitions

   This LFB does not define any metadata definition

4.4.  FEC

   The FE Configuration LFB is an LFB that standardizes configuration of
   the FE parameters.







Khasnabish, et al.        Expires July 18, 2015                 [Page 8]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


4.4.1.  Data Handling

   The FEC LFB does not handle any packets.  It's function is to provide
   the configuration parameters to the CE to be updated at runtime.

4.4.2.  Components

   This LFB has four components specified.  The FEID, a uint32 component
   that defines the ID of the FE.  The FEIP, a table of IP address, and
   each row is a struct of an IPv4 and an IPv6 address.  The LFB
   Parameters, a table of LFBs, each row a struct of LFB Class ID, LFB
   Version and optional LFB name and location.  Finally the CEs, a table
   of CE parameters, each row a struct of a CE ID and a table of CE IPs.

4.4.3.  Capabilities

   This capability specifies whether this FE supports dynamic loading of
   new LFBs.

4.4.4.  Events

   This LFB has five events specified.  These events notify the CE
   whether the FEID has been changed, an entry of the FEIP table has
   been created or changed and an entry of the CE information added or
   deleted.  The event reports are the respective data that has been
   modified.

5.  XML for FEC LFB

<?xml version="1.0" encoding="UTF-8"?>
<LFBLibrary xmlns="urn:ietf:params:xml:ns:forces:lfbmodel:1.0"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" provides="FEC">
  <dataTypeDefs>
    <dataTypeDef>
      <name>IPs</name>
      <synopsis>IP definition</synopsis>
      <struct>
        <component componentID="1">
          <name>FEIPv4</name>
          <synopsis>The FEs IPv4</synopsis>
          <typeRef>byte[4]</typeRef>
        </component>
        <component componentID="2">
          <name>FEIPv6</name>
          <synopsis>The FEs IPv6</synopsis>
          <typeRef>byte[16]</typeRef>
        </component>
      </struct>



Khasnabish, et al.        Expires July 18, 2015                 [Page 9]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


    </dataTypeDef>
    <dataTypeDef>
      <name>LFBDefs</name>
      <synopsis>LFB parameters inside the FE</synopsis>
      <struct>
        <component componentID="1">
          <name>LFBClassID</name>
          <synopsis>The LFB Class ID</synopsis>
          <typeRef>uint32</typeRef>
        </component>
        <component componentID="2">
          <name>LFBVersion</name>
          <synopsis>The Version of the LFB</synopsis>
          <typeRef>string</typeRef>
        </component>
        <component componentID="3">
          <name>LFBName</name>
          <synopsis>The name of the LFB</synopsis>
          <optional/>
          <typeRef>string</typeRef>
        </component>
        <component componentID="4">
          <name>LFBLocation</name>
          <synopsis>The location of the LFB to be retrieved
           from</synopsis>
          <optional/>
          <typeRef>string</typeRef>
        </component>
      </struct>
    </dataTypeDef>
    <dataTypeDef>
      <name>CEParams</name>
      <synopsis>CE parameters</synopsis>
      <struct>
        <component componentID="1">
          <name>CEID</name>
          <synopsis>The CE ID</synopsis>
          <typeRef>uint32</typeRef>
        </component>
        <component componentID="2">
          <name>CEIP</name>
          <synopsis>The CEIP</synopsis>
          <array>
            <typeRef>IPs</typeRef>
          </array>
        </component>
      </struct>
    </dataTypeDef>



Khasnabish, et al.        Expires July 18, 2015                [Page 10]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


  </dataTypeDefs>
  <LFBClassDefs>
    <LFBClassDef LFBClassID="21">
      <name>FEC</name>
      <synopsis>Forwarding Element Configuration</synopsis>
      <version>1.0</version>
      <components>
        <component componentID="1" access="read-write">
          <name>FEID</name>
          <synopsis>The FEID</synopsis>
          <typeRef>uint32</typeRef>
        </component>
        <component componentID="2" access="read-write">
          <name>FEIP</name>
          <synopsis>The FE's IP</synopsis>
          <array>
            <typeRef>IPs</typeRef>
          </array>
        </component>
        <component componentID="3" access="read-write">
          <name>LFBparameters</name>
          <synopsis>The LFBs in this FE</synopsis>
          <array>
            <typeRef>LFBDefs</typeRef>
          </array>
        </component>
        <component componentID="4" access="read-write">
          <name>CEs</name>
          <synopsis>The CEs that should be associated with this
                  FE</synopsis>
          <array>
            <typeRef>CEParams</typeRef>
          </array>
        </component>
      </components>
      <capabilities>
        <capability componentID="10">
          <name>DynamicLFBLoading</name>
          <synopsis>This capability specifies whether this FE supports
           dynamic loading of new LFBs</synopsis>
          <typeRef>boolean</typeRef>
        </capability>
      </capabilities>
      <events baseID="20">
        <event eventID="1">
          <name>IDChanged</name>
          <synopsis>The FE ID has been changed</synopsis>
          <eventTarget>



Khasnabish, et al.        Expires July 18, 2015                [Page 11]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


            <eventField>FEID</eventField>
          </eventTarget>
          <eventChanged/>
          <eventReports>
            <eventReport>
              <eventField>FEID</eventField>
            </eventReport>
          </eventReports>
        </event>
        <event eventID="2">
          <name>FEIPChanged</name>
          <synopsis>An IP of the FE has been changed</synopsis>
          <eventTarget>
            <eventField>FEIP</eventField>
          </eventTarget>
          <eventCreated/>
          <eventReports>
            <eventReport>
              <eventField>FEIP</eventField>
              <eventSubscript>_FEIPsrowid_</eventSubscript>
            </eventReport>
          </eventReports>
        </event>
        <event eventID="3">
          <name>FEIPCreated</name>
          <synopsis>An FEIP has been deleted</synopsis>
          <eventTarget>
            <eventField>FEIP</eventField>
          </eventTarget>
          <eventDeleted/>
          <eventReports>
            <eventReport>
              <eventField>FEIP</eventField>
              <eventSubscript>_FEIPsrowid_</eventSubscript>
            </eventReport>
          </eventReports>
        </event>
        <event eventID="4">
          <name>CEAdded</name>
          <synopsis>An CE has been added</synopsis>
          <eventTarget>
            <eventField>CEs</eventField>
          </eventTarget>
          <eventCreated/>
          <eventReports>
            <eventReport>
              <eventField>CEs</eventField>
            </eventReport>



Khasnabish, et al.        Expires July 18, 2015                [Page 12]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


          </eventReports>
        </event>
        <event eventID="5">
          <name>CEDeleted</name>
          <synopsis>An CE has been added</synopsis>
          <eventTarget>
            <eventField>CEs</eventField>
          </eventTarget>
          <eventDeleted/>
          <eventReports>
            <eventReport>
              <eventField>CEs</eventField>
            </eventReport>
          </eventReports>
        </event>
      </events>
    </LFBClassDef>
  </LFBClassDefs>
</LFBLibrary>

                       Figure 2: FEM XML LFB library

6.  Security Considerations

   Security considerations for ForCES LFB subsidiary management will be
   added in a future version of this daft.

7.  IANA Considerations

7.1.  LFB Class Names and LFB Class Identifiers

   LFB classes defined by this document belong to LFBs defined by
   Standards Track RFCs.  According to IANA, the registration procedure
   is Standards Action for the range 0 to 65535 and First Come First
   Served with any publicly available specification for over 65535.
   This specification includes the following LFB class names and LFB
   class identifiers:














Khasnabish, et al.        Expires July 18, 2015                [Page 13]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


   +------------+---------+---------+----------------------+-----------+
   | LFB Class  |   LFB   |   LFB   |     Description      | Reference |
   | Identifier |  Class  | Version |                      |           |
   |            |   Name  |         |                      |           |
   +------------+---------+---------+----------------------+-----------+
   |     21     |   FEC   |   1.0   |    An FEC LFB to     |    This   |
   |            |         |         | standardize creation |  document |
   |            |         |         |  of ForCES Network   |           |
   |            |         |         |       Elements       |           |
   +------------+---------+---------+----------------------+-----------+

     Logical Functional Block (LFB) Class Names and Class Identifiers

8.  Acknowledgments

   The authors would like to thank DJ, Joel, ChuanhuangLi, and many
   others for their discussions and support.

9.  References

9.1.  Normative References

   [RFC5810]  Doria, A., Hadi Salim, J., Haas, R., Khosravi, H., Wang,
              W., Dong, L., Gopal, R., and J. Halpern, "Forwarding and
              Control Element Separation (ForCES) Protocol
              Specification", RFC 5810, March 2010.

   [RFC5812]  Halpern, J. and J. Hadi Salim, "Forwarding and Control
              Element Separation (ForCES) Forwarding Element Model", RFC
              5812, March 2010.

   [RFC7121]  Ogawa, K., Wang, W., Haleplidis, E., and J. Hadi Salim,
              "High Availability within a Forwarding and Control Element
              Separation (ForCES) Network Element", RFC 7121, February
              2014.

9.2.  Informative References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119, March 1997.

   [RFC3654]  Khosravi, H. and T. Anderson, "Requirements for Separation
              of IP Control and Forwarding", RFC 3654, November 2003.

   [RFC3746]  Yang, L., Dantu, R., Anderson, T., and R. Gopal,
              "Forwarding and Control Element Separation (ForCES)
              Framework", RFC 3746, April 2004.




Khasnabish, et al.        Expires July 18, 2015                [Page 14]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


Appendix A.  Appendix

A.1.  Use of Virtualized ForCES Elements

   Virtualization of ForCES Elements allows efficient, scalable, and
   robust utilization of network control and transmission resources.
   Virtualization has been discussed (and deployed) widely in the
   Computing Industry (e.g., server) in the context of efficient
   utilization of server resources.

   As mentioned before, the currently existing techniques and solutions
   may be either slow or not directly applicable to ForCES LFB
   subsidiary management.

A.1.1.  Use of Virtualized CEs

   In this section we discuss the use of virtualized ForCES control
   elements (CEs).  The resulting operating entities in virtualized
   environment are Virtual CEs of VCEs.  The CE Visor (CEV) has the
   visibility to all of the VCEs in a domain, and can assign one of the
   VCEs as primary Master-VCE and another as secondary Master-VCE.  CEV
   can dynamically manage the role of primary and secondary master-VCEs
   from a pool of VCEs.

A.1.2.  Use of Virtualized FEs

   In this section we discuss the use of virtualized ForCES forwarding
   elements (FEs).  The resulting operating entities in virtualized
   environment are Virtual FEs of VFEs.  The FE Visor (FEV) has the
   visibility to all of the VFEs in a domain, and can assign one of the
   VFEs as primary Master-VFE and another as secondary Master-VFE.  FEV
   can dynamically manage the role of primary and secondary master-VFEs
   from a pool of VFEs.

A.1.3.  Generic Lifecycle of Physical/Virtual Elements

   The generic lifecycle of physical/virtual elements including NEs,
   FEs, VNEs, VCEs, VFEs, etc. consists of the following FOUR states:

   o  (a)Instantiation -- This refers to instantiation of CEs and FEs.

   o  (b) Association -- This refers to associating FEs to the CEs

   o  (c) Activation -- This refers to activation of CEs and FEs for
      normal operation.  This state may include monitoring as well with
      an objective to satisfy both scaling and reliability requirements.





Khasnabish, et al.        Expires July 18, 2015                [Page 15]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


   o  (d) Release -- This refers to releasing resources (both physical
      and virtual elements) to the pool of available (that is un-
      assigned) elements, and reporting this to the appropriate (CE or
      FE) manager.  It may be required to cleanse the physical/virtual
      elements before releasing in order to prevent harvesting of data/
      information by the the next user of the CEs/FEs.  The details of
      the cleansing operation is out of scope of this draft.

   Figure 1 shows physical/virtual elements states and their transition.


           o--------------o                  o--------------o
           |              |                  |              |
           |Instantiation +----------------->| Association  |
           |              |                  |              |
           o--------------o                  o------+-------o
                  ^                                 |
                  |                                 |
                  |                                 |
                  |                                 |
                  |                                 |
           o------+-------o                  o------V-------o
           |              |                  |              |
           |    Release   |<-----------------+  Activation  |
           |              |                  |              |
           o--------------o                  o--------------o

      Figure 1: Physical/Virtual Elements States and their Transition


A.2.  Potential Scenarios

   In this section we discuss a few potential scenarios that can utilize
   ForCES LFB subsidiary management for efficient and robust operation
   of networks without using excessive additional resources.

A.2.1.  Recovery from FE failure

   In this section we discuss how virtualization of FEs can be used for
   efficient recovery from FE failure(s).  An FE can initially boot
   using a default Association and Configuration.  The Association and
   Configuration can be updated at runtime via an FE-Visor or FEC LFB
   for runtime configuration purposes.  This can be achieved, for
   example, by adding a new CE and its associated IP address.  A CE can
   initially boot using a default Configuration and State(s).  The
   Configuration and State(s) can be updated at runtime via a CE-Visor
   or a similar CE Configuration (CEC) LFB to satisfy the runtime
   requirements.



Khasnabish, et al.        Expires July 18, 2015                [Page 16]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


                       .--------------.
            [Apps/     |              |
             Service]--|Orchestration |
                       |              |
                       .--------------.
                         |       |                           .--------.
         .-----------.   |       |                           |        |
         |           |---|       |---------------------------|        |
         |Controller |                                       |        |
         |           |                                       |        |
         .-|-----|---.                                       |Hyper-  |
           |     |                                           |Visor   |
           4     2                                           |        |
           |     |                                           |        |
           |     CEy       CEw    ....      CE?              |        |
           |     |  \      /\                                |        |
           |     |   \----/--\-------------------------------|----|   |
           |     |       /    \                              |  .-|-. |
           |    FEM-----/      \-----------------------------|--|   | |
           .-->(FEz)<-----------------3----------------------|--|FEx| |
                                                             |  .---. |
                                                    (1)----->|        |
                                                             .--------.


    Figure 2: Sequence of Events in FEM for Recovery from FE Failure


   Note: 1.(Hyervisor) Boots up FEx, and connects to CEy and CEw, 2.
   Boot a VM of Type FE 3.  FEx Boots FEz, and Connects to CEy,
   4.Connect to CEw

   As described in Figure 2, the following is a sequence of events in
   FEM (an example).

   o  Step-1: Hypervisor boots up with FEx that connects to CEy and CEw.

   o  Step-2: The Controller (attached to CEy) instructs FEx to boot an
      FE-type VM.

   o  Step-3: FEx boots FEz and instructs it to connect to CEy

   o  Step-4: The Controller (attached to FEz) instructs FEz to also
      connect to CEw.  This is essentially the "Association" part of
      Association and Configuration, as discussed earlier.






Khasnabish, et al.        Expires July 18, 2015                [Page 17]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


   o  Step-5: The Controller (attached to FEz) instructs FEz to increase
      its Syslog debug level.  This is essentially the "Configuration"
      part of Association and Configuration, as discussed earlier.

   Note that the 4th (FEM part of the charter) and 5th steps are what we
   would like to achieve here.  In addition, the FEVM may not need to be
   aware which Virtual FEs are in one Virtual NE, it only needs know of
   the information about a Virtual FE in the physical FE.  CE Manager
   may need to have visibility to all Virtual NEs.  The component "NE"
   of the LFB may be considered as Virtual NE as well.

A.2.2.  Recovery from CE failure

   In this section we discuss how virtualization of CEs can be used for
   efficient recovery from CE failure(s).

   A CE can initially boot using a default Association, State, and
   Configuration.  The Association and Configuration can be updated at
   runtime via a CE-Visor or FEC-LFB for runtime configuration purposes,
   for example, by adding a new CE and its associated IP address.

   An FE can initially boot using a default Configuration, Association,
   with a CE, and State.  The Configuration, Association can be updated
   at runtime via a FE-Visor or FEC LFB to satisfy runtime requirements.
   The sequence of events, an example, can be as follows.

   o  Step-1: The CEx is Active with CEy as its Standby with Standby-
      Active or Active-Active setup.

   o  Step-2: The CEx controls FEy and FEw with both FEy and FEw having
      Standby control links to CEy, with Standby-Active or Active-Active
      setup.  Note that CEx and CEy are controlled, assigned, by CE-
      visor, and may have a common, virtual, IP address.

   o  Step-3: The Controller is fully aware of the status of all of the
      CEs, physical and virtual; When CEx fails, its states are fully
      transferred (may already be synced) to CEy.

   o  Step-4: The Standby links from CEy to FEy and FEw become fully
      active, and the control, of FEy and FEw, is fully transferred from
      CEx and CEy.

   o  Step-5: A graceful-smooth failover of CEx to CEy is now
      successfully complete, and SysLog debug level for CEy is
      increased..

   As discussed earlier, the last two steps are concerned with
   Subsidiary management.  Although we discuss the recovery method by



Khasnabish, et al.        Expires July 18, 2015                [Page 18]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


   using virtualization of CEs, the role of FEVM in the recovery process
   will be described further later.

A.2.3.  Load Balancing

   In this section we discuss efficient load balancing of both CE and FE
   in virtualized environment.

A.2.4.  Orchestration

   In this section we discuss efficient Orchestration of both CE and FE
   in virtualized multi-admin-domain environment.

A.2.5.  Generic LFB Lifecycle Management

   In this section we discuss generic lifecycle management of
   subsidiaries of LFBs in virtualized environment(s).  The typical
   management activities in the life of FE/CE are discussed in the
   following sub-sections.

A.2.5.1.  Booting a CE/FE

   When an entity needs to boot a CE/FE, if this is a VM, some
   orchestration would scheme/plan to do this.  In case of ForCES, we
   have a control App that boots a CE or an FE via a management FE.  So
   here we have a management plane details that is described either in
   FEM or other LFB.

A.2.5.2.  Bootstrapping the Configuration

   The FE, e.g., the VM which has just been booted, as described in the
   previous sub-section, needs initial bootstrap configuration (e.g.,
   what CEs to connect to etc).  This clearly falls in the FEC LFB
   domain.

A.2.5.3.  Runtime Management

   At runtime of the FE, for example, the management could introduce a
   new CE for the FE to associate with; it may also be for an FE to
   dissociate from a CE, and so on.

Authors' Addresses









Khasnabish, et al.        Expires July 18, 2015                [Page 19]

Internet-Draft      ForCES LFB Subsidiary Management        January 2015


   Bhumip Khasnabish
   ZTE TX, Inc.
   55 Madison Avenue, Suite 160
   Morristown, New Jersey  07960
   USA

   Phone: +001-781-752-8003
   Email: vumip1@gmail.com, bhumip.khasnabish@ztetx.com
   URI:   http://tinyurl.com/bhumip/


   Evangelos Haleplidis
   University of Patras
   Department of Electrical and Computer Engineering
   Patras  26500
   Greece

   Email: ehalep@ece.upatras.gr


   Jamal Hadi Salim (editor)
   Mojatatu Networks
   Suite 400, 303 Moodie Dr.
   Ottawa, Ontario  K2H 9R4
   Canada

   Email: hadi@mojatatu.com
























Khasnabish, et al.        Expires July 18, 2015                [Page 20]