Internet DRAFT - draft-geoffhart-cmlp

draft-geoffhart-cmlp



draft-geoffhart-cmlp-00                                       Revision 0

Internet Draft   					      Geoff Hart
Category:  Standards Track                  Western Governors University
                                                               July 2014


  Change Management Logging Protocol as a Means for Increasing System 
                              Availability

                                ABSTRACT

CMLP is proposed as a tool System and Network Administrators can use to
rapidly identify maintenance work performed on local and remote
networked hosts including routers, switches, servers, and firewalls as
well as other networked devices.  The primary drivers for creation
of CMLP are improving network availability and decreasing restoral time
through rapid identifiation of change activity in the enterprise network
environment.


			   Status of this Memo

This Internet Draft expires January 31, 2015.

Internet-Drafts are draft documents valid for a maximum of six months
     and may be updated, replaced, or obsoleted by other documents at
     any time.  It is inappropriate to use Internet-Drafts as
     reference material or to cite them other than as "work in
     progress."

Internet-Drafts are working documents of the Internet Engineering Task
     Force (IETF), its areas, and its working groups.  Note that other
     groups may also distribute working documents as Internet-Drafts.

The list of current Internet-Drafts can be accessed at
     http://www.ietf.org/1id-abstracts.html

The list of Internet-Draft Shadow Directories can be accessed at
     http://www.ietf.org/shadow.html


                           About This Document

IETF Trust Legal Provisions of 28-dec-2009, Section 6.a:
     This Internet-Draft is submitted in full conformance with the provisions
     of BCP 78 and BCP 79.

IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 2:
     Copyright (c) 2014 IETF Trust and the persons identified as the document
     authors.  All rights reserved.

IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 3:
     This document is subject to BCP 78 and the IETF Trust's Legal Provisions
     Relating to IETF Documents (http://trustee.ietf.org/license-info)
     in effect on the date of publication of this document.  Please
  
Hart                          Experimental                     [Page 1]

INTERNET DRAFT    Change Management Logging Protocol          July 2014

     review these documents carefully, as they describe your rights
     and restrictions with respect to this document.  Code Components
     extracted from this document must include Simplified BSD License
     text as described in Section 4.e of the Trust Legal Provisions
     and are provided without warranty as described in the Simplified
     BSD License.


1.  Introduction

Corporations heavily rely on the availability, integrity, and
confidentiality of information and information systems in this day and
age.  Network outages cost companies large sums of money, both in
technical maintenance costs as well as monies spent to repair corporate
image in the marketplace.  Change Management Logging Protocol (CMLP) is
proposed as a frontline solution to troubleshooting information system
downtime or degradation due to system maintenance, software upgrades and
other change work applied in the enterprise network.  CMLP finds its key
benefit as rapid identification of maintenance work performed and
reducing restoral time.

2.  Background

One of the greatest disturbances the enterprise network environment can 
face is system downtime.  This becomes even more apparent when the 
system downtime finds its root cause in maintenance, including but not
limited to software or hardware upgrades.  System administrators, IT 
management, and other network operations team members are engaged to 
investigate system outages affecting customers.  System administrators 
need to be able to rapidly and accurately determine if and what change 
work was performed and what systems were updated.  This becomes even 
more evident when the system downtime occurs sometime after the change 
work is performed.  The system administrators (SA's) engaged to
troubleshoot the adverse network or system condition most often were not
involved in the change work that caused the adverse condition.  As
opposed to spending valuable time searching for the change work
performed, SA's can rapidly check their systems to identify any and all
change work performed during a given period of time.  This serves to
increase network availability and reduce restoral time since SA's will
not have to search through email notifications, calendar invites, and
query Change Management (CM) systems to identify work performed.  

3.  Change Management Logging Protocol (CMLP)


CMLP provides three modes of operations as follows:

    - Basic
    - Enhanced
    - Strict

CMLP provides a means for local and remote tracking of CM work performed
on a given networked element.  The networked element prompts the network
or SA for the Change Management Request****(CMR) number.  The CMR number
is recorded locally on the networked element and can be queried as

Hart                          Experimental                     [Page 2]

INTERNET DRAFT    Change Management Logging Protocol          July 2014

desired at a later time.  When the network or system administrator
requests access to an information system resource for the purpose of
making a configuration change, they are prompted to enter the CMR
number.  The information stored regarding changes made to the networked
element can be queried and reviewed at a later time as part of routine
administration tasks or troubleshooting work.   

3.1.  CMLP Parameters

CMLP can be configured to use the following parameters.

 - Mode
 - Domain
 - Multicast notifications
 - CM Server
 - CM Server downtime action 
   - Fail-allow
   - Fail-prohibit
 - CM Server Poll Interval
 - Multicast IP address
       
3.2.  CMLP Database Format

CMLP information is stored in a database on the networked element.  The
format specified is as follows:

  - CMR identification number
  - CMR Status 
  - CM Domain
  - CM Node:  Local or remote 
  - CMR Time Stamp Start 
  - CMR Time Stamp End 
  - CMR Time Stamp Work Completed 

It should be noted that the CM Node type local is designated for the
networked element storing the local database.  Remote indicates that a
multicast notification was received by the networked element and the
information was entered into the local database for later reference.  
   
3.3.   Modes of CMLP Operation

CMLP operates in one of three modes: Basic, Enhanced, or Strict.

3.3.1.   CMLP Basic Mode 

CMLP Basic mode when configured on the networked element enables simple
tracking of CM work.  The networked element is configured to track the
CMR number and stores it locally.  Additional information recorded
includes the time that entry into configuration mode was authorized.
Basic mode only allows configuration of domain.  Other parameters are
not configured.  Networked elements configured for Basic mode do not
multicast CMLP notifications.  No CM server is configured in basic mode.  

NOTE: If CM server is to be configured,then the use of Enhanced or
Strict mode is required.  As expected, the CM Server Poll Interval is

Hart                          Experimental                     [Page 3]

INTERNET DRAFT    Change Management Logging Protocol          July 2014

also not configured.  

3.3.2.  CMLP Enhanced Mode

CMLP Enhanced mode when configured on the networked element enables an
advanced feature set for tracking of CM activity.  The networked element
is configured to track the CMR number and stores it locally.  In
Enhanced mode the networked element is configured with domain, multicast
notifications, CM servers, CM server downtime action, and CM server
polling interval.  Enhanced mode requires the configuration of at least
one CM server.  Multicast notifications can be disabled if desired.
Multicast notifications require the Multicast IP address to be
configured.  

3.3.3.  CMLP Strict Mode

CMLP Strict mode when configured on the networked element requires
configuration of at least two unique CM server IP addresses.  In Strict
mode, CM Server Downtime Action is set to fail-prohibit and the SA
performing the maintenance work would have to follow CM server downtime
procedures.  In terms of the maintenance work at hand, this is
recommended to include a second attempt at entering configuration mode.
If further attempts fail, a health check would be performed on the
networked element and troubleshooting connectivity should begin.  If
the issue is found to be on the CM servers themselves, the SA needs to
follow a pre-determined procedure for proceeding with the work.  If it
is determined work should continue, the SA would have to execute the
break-fix sequence to bypass CMLP prompt in order to enter
configuration mode.  The Server Poll Interval is configured and
should be set such that network devices and paths do not become
over-utilized.  Multicast notifications require the Multicast IP
address to be configured.  Multicast notifications can be disabled
if desired.  

3.4.  Connection to Change Management Server

As mentioned when CMLP is configured on a networked element using either
Enhanced or Strict modes, a connection to one or more central Change
Management servers is configured.  CMLP requests from the networked
element where change work is to be performed can be received and a
validation response can be sent back.  In Enhanced Mode the response
indicates a valid or invalid CMR identification number has been provided
by the SA.  In Strict mode the response from the CM server to the
networked element indicates whether the work to be performed has been
approved.  In Strict mode, if the networked element receives a reply
back from CM server indicating that a valid CMR number was received but
the CM is not approved or the work is not approved for the current time,
then the SA is not authorized to enter configuration mode to make
changes.  At that point the SA would then have to make necessary 
corrections to CMR and initiate another request to enter configuration
mode.  

3.5.   CMLP Domains

The CMLP domain parameter is used to control the flow of CMLP within the

Hart                          Experimental                     [Page 4]

INTERNET DRAFT    Change Management Logging Protocol          July 2014

enterprise network architecture and across network boundaries.  As
mentioned the domain is a required parameter in Enhanced and Strict
modes of operation.  The domain is used to control responses to requests
for CMLP data stored on the networked element.  The host sending the
poll request for CMLP data from the networked element must provide the
proper domain name to be authorized and authenticated.  The domain
parameter also serves as a means of controlling CMLP multicasts.  
 
Domains prevent large amounts of unnecessary CMLP traffic to hosts that
do not need to receive information. This can be due to any number of
reasons.  A group of routers, for example, does not necessarily need to
know about routine server upgrades on a database serverfarm.  

3.6.  Multicast Notifications

CMLP can be configured to send multicast notifications out to other
systems on the enterprise network.  The notifications are sent to a
configured multicast address.  The CMLP multicast notification includes
the CMR number as well as the logged time that the change work was
performed.  In Enhanced and Strict modes the time stamps are based on
the time stamps listed for the CMR on the CM server (or servers).
Multicast notifications can be stored as SAs determine necessary on
receiving hosts.  For example, after CM work is completed on Router A,
Router A can send a notice to other networked elements indicating that
CM work has been performed and completed successfully.  Other networked
elements receiving the multicast notification can include switches,
servers, workstations, and other routers.  Following the example, if
Router B receives the multicast notification and logs the notification,
an SA at a later time can query the local CMLP database on Router B for
information about CM work performed on other networked elements such as
Router A.  

3.6.1.  Multicast Notifications - IP Address Assignment

As of July 7, 2014, the multicast IP address 224.0.0.116 is unassigned
and could be used in an enterprise network architecture for the purpose
of multicasting CMLP notifications.

3.7.  CMLP Database Record Updates

CMLP nodes configured to use either Enhanced or Strict mode will send
CMLP Database Record Update Requests to the CM servers.  The CMLP
Database Record Update Request will be sent to the first available
CM server IP address configured in the networked element.  The request
will serve to secure the current status of the CMR.  If the SA has

completed the maintenance work and has closed the CMR in the CM server
application, the CMR will be marked as Completed.  The time stamp for
the actual completion time of the work will be logged in the CMR at the
CM server.  The CM server will respond to the CMLP Database Record
Update Request with a CMLP Database Record Update Response.  The
response from the CM server contains the CMR number, the status of CMR
as listed in the CM server, and the time stamp the work was marked as
completed.  If the CMR is not marked as completed at the CM server then
the CMLP Database Record Update Response does not contain a time stamp

Hart                          Experimental                     [Page 5]

INTERNET DRAFT    Change Management Logging Protocol          July 2014

for work completion.  

4.  CMLP Sample Call Flows

The following subsections highlight a handful of sample "call flows" for
the various tasks requiring the use of CMLP.  It is not exhaustive
rather it highlights the general operation of CMLP during CM activity
and troubleshooting processes.  

4.1.  Call Flow - Basic Mode - SA Performing Change Work

Given the following configuration parameters:
     - Mode: Basic
     - Domain:  <configured>
 
SA user requests access to configuration mode of router to perform
change work
  |
  V
Router prompts SA user for CMR identification number
  |
  V
SA enters CMR number
  |
  V 
Router stores CMR number and is authorized to enter configuration mode

4.2.  Call Flow - Enhanced Mode - SA Performing Change Work
  - Mode: Enhanced
  - Domain:  <configured, required for Enhanced mode>
  - CM server:  <one or more properly configured>
  - CM server downtime action:  <configured, fail-allow or
    fail-prohibit>
  - CM server polling interval

SA requests access to configuration mode of router to perform change
work
  |
  V
Router prompts SA user for CMR number
  |
  V
Router sends CMLP authorization request to CM server
  |
  V
CM server validates that the indicated CMR has been approved
  |
  V
CM server sends back "authorization granted" if the CMR has been
approved
  | 
  V
Networked element receives "authorization granted" message from CM
server and grants SA access to configuration mode
  |

Hart                          Experimental                     [Page 6]

INTERNET DRAFT    Change Management Logging Protocol          July 2014

  V
The networked element logs the CMLP data for to the CMLP local database

4.3.  Call Flow - Strict Mode - SA Performing Change Work
  - Mode: Strict
  - Domain: <configured, required for Strict mode>
  - CM server: <one or more properly configured>
  - CM server polling interval

SA requests access to configuration mode of router to perform change
work
  |
  V
Router prompts SA user for CMR number
  |
  V
SA sends CMLP authorization request to CM server
  |
  V
CM server validates that the indicated CMR has been approved to be
performed at that time and that the networked element sending the
"authorization request" is listed on the CMR.  
  |
  V
CM server sends back "authorization granted" if the CMR has been
approved and other criteria match
  | 
  V
Networked element receives "authorization granted" message from CM
server and grants SA access to configuration mode
  |
  V
The networked element logs the CMLP data to the CMLP local database

4.4.  Call Flow - Local Query for CMLP

SA executes command in networked element querying local CMLP database
  | 
  V
Networked element returns CMLP data stored locally that matches the
query criteria

4.5.  Call Flow - Remote Query for CMLP - 

User executes a command on remote host querying CMLP compliant networked
element for CMLP data
  | 
  V
Request is transmitted to CMLP compliant networked element
  | 
  V
CMLP compliant network element receives request
  |
  V
CMLP compliant network element returns information if domain name in

Hart                          Experimental                     [Page 7]

INTERNET DRAFT    Change Management Logging Protocol          July 2014

request matches.  Otherwise Invalid Request response is sent back to
requesting host.
 
4.6.  Call Flow - Enhanced or Strict Mode - Multicast Notification 

SA completes change work and marks CMR as successfully completed in CM
server
  |
  V
Networked element polls CM Server to update local CMLP Database record
for the given CM
  |
  V
CM Server sends CMLP data back to networked element
  |
  V
Since CM Server indicates that the CMR is completed, the networked
element updates the CMLP Database Record to indicate the work is
completed and the SA considers the work successful.  Time stamp for the
actual work completion time is updated in the CMLP Database Record.  

5.  Benefits of CMLP

CMLP allows SA's to rapidly identify whether an adverse condition in a
server or network device is due to maintenance.  If the adverse
condition is due to maintenance the SA is able to determine very quickly
the CMR number.  The SA is then able to locate instructions for
performing restoral or rollback work. This is all done without having to
perform the time consuming tasks of locating and reviewing maintenance
notifications or querying the CM server manually.  Rapid identification
of CM work performed serves to reduce restoral time thus increasing
availability.  

CMLP also improves accountability since SA's will be required to enter a
CMR number when requesting access into configuration mode.  This ensures
that configuration changes are not made "on the fly," thus supporting
best practices for CM activities.

6.  Risks of CMLP Implementation

As with the implementation of any new protocol, there exists some risks
which network professionals and organizational management need to be
aware of.  The key risks identified to date are listed in the following
subsections.

6.1.  Man-In-The-Middle Attack/Snooping

It is possible that CMLP traffic could be sniffing passively or
actively.  The packets containing CMLP could be intercepted by network
attackers.  The data contained in the CMLP packets is not overly
sensitive and is primarily used to identify what CM work has been
performed.  

The data-by-obscurity concept could also be implemented to protect the
information as it is transmitted across the network.  When the

Hart                          Experimental                     [Page 8]

INTERNET DRAFT    Change Management Logging Protocol          July 2014

data-by-obscurity technique is implemented the network attacker is not
able to determine what it is they have captured.  The data is not easily
interpreted and the attacker would most likely move onto the next
option.  

ACLs can be used to control flow of CMLP information across the network
as they travel from source to destination.  

6.2.  System Downtime on CM Server

If one or more of the configured CM servers experiences downtime or is
undergoing maintenance, CMLP requests will fail in Enhanced and Strict
modes.  The fail-allow setting when cconfigured would allow the SA to
enter configuration mode when the CMR number is provided.  However, if
fail-prohibit setting is configured, the networked element will not be
able to authorize entry to the SA to configuration mode.  The SA will
either need to re-schedule the maintenance work or enter configuration
mode using a break-fix work around.

6.3.  Break Fix

During an outage situation where rapid recovery is required, the CMLP
request and authorization process will need to be circumvented.  The
networked element must be able to allow access to configuration mode for
the purpose of break-fix/system recovery.  As mentioned, the
over-arching ideal of CMLP is to improve network availability and
reducing network restoral time while minimizing impedence to the
restoral process.  The method for entering configuration mode for the
purpose of break-fix should be well-known and relatively simple to
achieve.  That being said, best practices for securing networked
elements should be adhered to such that the attack surface for the
network and its elements are not compromised.  

The organization or vendor implementation of CMLP could specify separate
ticketing formats to address CM work and break-fix scenarios.  Given the
CM work format (Example: "CMR004321"), the networked element identifies
the request to enter configuration mode as an attempt to start
maintenance work.  Given the break-fix format (Example: "TT00123456"),
the networked element identifies the request to enter configuration
mode as an attempt to perform break-fix work.  In the break-fix
scenario, the networked element could log the entry into configuration
mode as purposed for break-fix.  

6.4.  CMLP Traffic Floods 

An enterprise network environment where several hosts, routers,
switches, CM servers and other devices are sending CMLP packets, it is
possible that the CMLP traffic could overwhelm network devices or links.
CMLP traffic flows could be limited, as mentioned, through the use of
ACLs and rate limiting.  

7.  CMLP Compliance

The following items must be followed for a networked element to be CMLP
compliant:

Hart                          Experimental                     [Page 9]

INTERNET DRAFT    Change Management Logging Protocol          July 2014

  - Enhanced and Strict modes require connection to one or more CMLP
    servers.  Configuration of multiple CM servers is recommended to
    address issues of redundancy.  Strict mode requires at least two CM
    server IP addresses to be configured.  This assumes either IPv4 or
    IPv6 addressing is used for CM servers.  
  - To ensure proper logging and adherence to common best practices, the
    networked element should be configured with connections to one or
    more NTP servers.
  - To ensure proper operation of CMLP throughout the enterprise
    network, unique hostnames need to be implemented across the network.
    No two networked elements within the enterprise network should
    utilize the same hostname.
 
8.  Standardization Considerations

CMLP can be configured to send multicast notifications.  In order for
CMLP to function in an enterprise network environment where multiple
platforms are used, CMLP will require a multicast IP address assignment.
If IPv6 addressing is used in the enterprise network, an IPv6 multicast
address would be required.  This will require assignment of an IPv6
multicast address, similar to IPv4.

With the exception of CMLP Multicast Notifications, CMLP is best suited
to use TCP for sending CMLP packets across the network.  UDP is
recommended for the multicast notifications, particularly since no
response to the notification is necessary or desired.  

As CMLP develops further, the ports designated for use by CMLP capable
networked elements should be standardized to support cross-platform
implementations.  

9.  Assumptions 

The following assumptions are made
  -  When multiple CM server IP addresses are configured, the CM servers
     are assumed to be redundant and contain identical information.
     Multiple CM servers are configured to address CM server downtime      
     for a server at a given IP address.  
  - Organizations choosing to implement CMLP-capable networked devices
     have existing CM policies and procedures.  The policies and
     procedures include management oversight to the CM process.    
  -  Organizations providing network connectivity to clients, internal
     or external, look to provide best-in-class customer experience and
     consider high network availability a key priority.


10.  Glossary and Abbreviations

ACL - Access Control Lists

CM - Change Management 

CMR - A Change Management Request is the request submitted for review
and approval by network management or senior management for
implementation team/s to make changes to specified network elements.

Hart                          Experimental                    [Page 10]

INTERNET DRAFT    Change Management Logging Protocol          July 2014

CMLP - Change Management Logging Protocol

Networked Element - Any information system element connected to the
    network.  This includes but is not limited to routers, switches,
    firewalls, and servers.  

NTP - Network Time Protocol  

SA - System Administrator.  


       ---- This Internet Draft expires January 31, 2015  ----

Hart                          Experimental                    [Page 11]