draft-geoffhart-cmlp-00 Revision 0 Internet Draft Geoff Hart Category: Standards Track Western Governors University July 2014 Change Management Logging Protocol as a Means for Increasing System Availability ABSTRACT CMLP is proposed as a tool System and Network Administrators can use to rapidly identify maintenance work performed on local and remote networked hosts including routers, switches, servers, and firewalls as well as other networked devices. The primary drivers for creation of CMLP are improving network availability and decreasing restoral time through rapid identifiation of change activity in the enterprise network environment. Status of this Memo This Internet Draft expires January 31, 2015. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html About This Document IETF Trust Legal Provisions of 28-dec-2009, Section 6.a: This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 2: Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved. IETF Trust Legal Provisions of 28-dec-2009, Section 6.b(i), paragraph 3: This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please Hart Experimental [Page 1] INTERNET DRAFT Change Management Logging Protocol July 2014 review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. 1. Introduction Corporations heavily rely on the availability, integrity, and confidentiality of information and information systems in this day and age. Network outages cost companies large sums of money, both in technical maintenance costs as well as monies spent to repair corporate image in the marketplace. Change Management Logging Protocol (CMLP) is proposed as a frontline solution to troubleshooting information system downtime or degradation due to system maintenance, software upgrades and other change work applied in the enterprise network. CMLP finds its key benefit as rapid identification of maintenance work performed and reducing restoral time. 2. Background One of the greatest disturbances the enterprise network environment can face is system downtime. This becomes even more apparent when the system downtime finds its root cause in maintenance, including but not limited to software or hardware upgrades. System administrators, IT management, and other network operations team members are engaged to investigate system outages affecting customers. System administrators need to be able to rapidly and accurately determine if and what change work was performed and what systems were updated. This becomes even more evident when the system downtime occurs sometime after the change work is performed. The system administrators (SA's) engaged to troubleshoot the adverse network or system condition most often were not involved in the change work that caused the adverse condition. As opposed to spending valuable time searching for the change work performed, SA's can rapidly check their systems to identify any and all change work performed during a given period of time. This serves to increase network availability and reduce restoral time since SA's will not have to search through email notifications, calendar invites, and query Change Management (CM) systems to identify work performed. 3. Change Management Logging Protocol (CMLP) CMLP provides three modes of operations as follows: - Basic - Enhanced - Strict CMLP provides a means for local and remote tracking of CM work performed on a given networked element. The networked element prompts the network or SA for the Change Management Request****(CMR) number. The CMR number is recorded locally on the networked element and can be queried as Hart Experimental [Page 2] INTERNET DRAFT Change Management Logging Protocol July 2014 desired at a later time. When the network or system administrator requests access to an information system resource for the purpose of making a configuration change, they are prompted to enter the CMR number. The information stored regarding changes made to the networked element can be queried and reviewed at a later time as part of routine administration tasks or troubleshooting work. 3.1. CMLP Parameters CMLP can be configured to use the following parameters. - Mode - Domain - Multicast notifications - CM Server - CM Server downtime action - Fail-allow - Fail-prohibit - CM Server Poll Interval - Multicast IP address 3.2. CMLP Database Format CMLP information is stored in a database on the networked element. The format specified is as follows: - CMR identification number - CMR Status - CM Domain - CM Node: Local or remote - CMR Time Stamp Start - CMR Time Stamp End - CMR Time Stamp Work Completed It should be noted that the CM Node type local is designated for the networked element storing the local database. Remote indicates that a multicast notification was received by the networked element and the information was entered into the local database for later reference. 3.3. Modes of CMLP Operation CMLP operates in one of three modes: Basic, Enhanced, or Strict. 3.3.1. CMLP Basic Mode CMLP Basic mode when configured on the networked element enables simple tracking of CM work. The networked element is configured to track the CMR number and stores it locally. Additional information recorded includes the time that entry into configuration mode was authorized. Basic mode only allows configuration of domain. Other parameters are not configured. Networked elements configured for Basic mode do not multicast CMLP notifications. No CM server is configured in basic mode. NOTE: If CM server is to be configured,then the use of Enhanced or Strict mode is required. As expected, the CM Server Poll Interval is Hart Experimental [Page 3] INTERNET DRAFT Change Management Logging Protocol July 2014 also not configured. 3.3.2. CMLP Enhanced Mode CMLP Enhanced mode when configured on the networked element enables an advanced feature set for tracking of CM activity. The networked element is configured to track the CMR number and stores it locally. In Enhanced mode the networked element is configured with domain, multicast notifications, CM servers, CM server downtime action, and CM server polling interval. Enhanced mode requires the configuration of at least one CM server. Multicast notifications can be disabled if desired. Multicast notifications require the Multicast IP address to be configured. 3.3.3. CMLP Strict Mode CMLP Strict mode when configured on the networked element requires configuration of at least two unique CM server IP addresses. In Strict mode, CM Server Downtime Action is set to fail-prohibit and the SA performing the maintenance work would have to follow CM server downtime procedures. In terms of the maintenance work at hand, this is recommended to include a second attempt at entering configuration mode. If further attempts fail, a health check would be performed on the networked element and troubleshooting connectivity should begin. If the issue is found to be on the CM servers themselves, the SA needs to follow a pre-determined procedure for proceeding with the work. If it is determined work should continue, the SA would have to execute the break-fix sequence to bypass CMLP prompt in order to enter configuration mode. The Server Poll Interval is configured and should be set such that network devices and paths do not become over-utilized. Multicast notifications require the Multicast IP address to be configured. Multicast notifications can be disabled if desired. 3.4. Connection to Change Management Server As mentioned when CMLP is configured on a networked element using either Enhanced or Strict modes, a connection to one or more central Change Management servers is configured. CMLP requests from the networked element where change work is to be performed can be received and a validation response can be sent back. In Enhanced Mode the response indicates a valid or invalid CMR identification number has been provided by the SA. In Strict mode the response from the CM server to the networked element indicates whether the work to be performed has been approved. In Strict mode, if the networked element receives a reply back from CM server indicating that a valid CMR number was received but the CM is not approved or the work is not approved for the current time, then the SA is not authorized to enter configuration mode to make changes. At that point the SA would then have to make necessary corrections to CMR and initiate another request to enter configuration mode. 3.5. CMLP Domains The CMLP domain parameter is used to control the flow of CMLP within the Hart Experimental [Page 4] INTERNET DRAFT Change Management Logging Protocol July 2014 enterprise network architecture and across network boundaries. As mentioned the domain is a required parameter in Enhanced and Strict modes of operation. The domain is used to control responses to requests for CMLP data stored on the networked element. The host sending the poll request for CMLP data from the networked element must provide the proper domain name to be authorized and authenticated. The domain parameter also serves as a means of controlling CMLP multicasts. Domains prevent large amounts of unnecessary CMLP traffic to hosts that do not need to receive information. This can be due to any number of reasons. A group of routers, for example, does not necessarily need to know about routine server upgrades on a database serverfarm. 3.6. Multicast Notifications CMLP can be configured to send multicast notifications out to other systems on the enterprise network. The notifications are sent to a configured multicast address. The CMLP multicast notification includes the CMR number as well as the logged time that the change work was performed. In Enhanced and Strict modes the time stamps are based on the time stamps listed for the CMR on the CM server (or servers). Multicast notifications can be stored as SAs determine necessary on receiving hosts. For example, after CM work is completed on Router A, Router A can send a notice to other networked elements indicating that CM work has been performed and completed successfully. Other networked elements receiving the multicast notification can include switches, servers, workstations, and other routers. Following the example, if Router B receives the multicast notification and logs the notification, an SA at a later time can query the local CMLP database on Router B for information about CM work performed on other networked elements such as Router A. 3.6.1. Multicast Notifications - IP Address Assignment As of July 7, 2014, the multicast IP address 224.0.0.116 is unassigned and could be used in an enterprise network architecture for the purpose of multicasting CMLP notifications. 3.7. CMLP Database Record Updates CMLP nodes configured to use either Enhanced or Strict mode will send CMLP Database Record Update Requests to the CM servers. The CMLP Database Record Update Request will be sent to the first available CM server IP address configured in the networked element. The request will serve to secure the current status of the CMR. If the SA has completed the maintenance work and has closed the CMR in the CM server application, the CMR will be marked as Completed. The time stamp for the actual completion time of the work will be logged in the CMR at the CM server. The CM server will respond to the CMLP Database Record Update Request with a CMLP Database Record Update Response. The response from the CM server contains the CMR number, the status of CMR as listed in the CM server, and the time stamp the work was marked as completed. If the CMR is not marked as completed at the CM server then the CMLP Database Record Update Response does not contain a time stamp Hart Experimental [Page 5] INTERNET DRAFT Change Management Logging Protocol July 2014 for work completion. 4. CMLP Sample Call Flows The following subsections highlight a handful of sample "call flows" for the various tasks requiring the use of CMLP. It is not exhaustive rather it highlights the general operation of CMLP during CM activity and troubleshooting processes. 4.1. Call Flow - Basic Mode - SA Performing Change Work Given the following configuration parameters: - Mode: Basic - Domain: SA user requests access to configuration mode of router to perform change work | V Router prompts SA user for CMR identification number | V SA enters CMR number | V Router stores CMR number and is authorized to enter configuration mode 4.2. Call Flow - Enhanced Mode - SA Performing Change Work - Mode: Enhanced - Domain: - CM server: - CM server downtime action: - CM server polling interval SA requests access to configuration mode of router to perform change work | V Router prompts SA user for CMR number | V Router sends CMLP authorization request to CM server | V CM server validates that the indicated CMR has been approved | V CM server sends back "authorization granted" if the CMR has been approved | V Networked element receives "authorization granted" message from CM server and grants SA access to configuration mode | Hart Experimental [Page 6] INTERNET DRAFT Change Management Logging Protocol July 2014 V The networked element logs the CMLP data for to the CMLP local database 4.3. Call Flow - Strict Mode - SA Performing Change Work - Mode: Strict - Domain: - CM server: - CM server polling interval SA requests access to configuration mode of router to perform change work | V Router prompts SA user for CMR number | V SA sends CMLP authorization request to CM server | V CM server validates that the indicated CMR has been approved to be performed at that time and that the networked element sending the "authorization request" is listed on the CMR. | V CM server sends back "authorization granted" if the CMR has been approved and other criteria match | V Networked element receives "authorization granted" message from CM server and grants SA access to configuration mode | V The networked element logs the CMLP data to the CMLP local database 4.4. Call Flow - Local Query for CMLP SA executes command in networked element querying local CMLP database | V Networked element returns CMLP data stored locally that matches the query criteria 4.5. Call Flow - Remote Query for CMLP - User executes a command on remote host querying CMLP compliant networked element for CMLP data | V Request is transmitted to CMLP compliant networked element | V CMLP compliant network element receives request | V CMLP compliant network element returns information if domain name in Hart Experimental [Page 7] INTERNET DRAFT Change Management Logging Protocol July 2014 request matches. Otherwise Invalid Request response is sent back to requesting host. 4.6. Call Flow - Enhanced or Strict Mode - Multicast Notification SA completes change work and marks CMR as successfully completed in CM server | V Networked element polls CM Server to update local CMLP Database record for the given CM | V CM Server sends CMLP data back to networked element | V Since CM Server indicates that the CMR is completed, the networked element updates the CMLP Database Record to indicate the work is completed and the SA considers the work successful. Time stamp for the actual work completion time is updated in the CMLP Database Record. 5. Benefits of CMLP CMLP allows SA's to rapidly identify whether an adverse condition in a server or network device is due to maintenance. If the adverse condition is due to maintenance the SA is able to determine very quickly the CMR number. The SA is then able to locate instructions for performing restoral or rollback work. This is all done without having to perform the time consuming tasks of locating and reviewing maintenance notifications or querying the CM server manually. Rapid identification of CM work performed serves to reduce restoral time thus increasing availability. CMLP also improves accountability since SA's will be required to enter a CMR number when requesting access into configuration mode. This ensures that configuration changes are not made "on the fly," thus supporting best practices for CM activities. 6. Risks of CMLP Implementation As with the implementation of any new protocol, there exists some risks which network professionals and organizational management need to be aware of. The key risks identified to date are listed in the following subsections. 6.1. Man-In-The-Middle Attack/Snooping It is possible that CMLP traffic could be sniffing passively or actively. The packets containing CMLP could be intercepted by network attackers. The data contained in the CMLP packets is not overly sensitive and is primarily used to identify what CM work has been performed. The data-by-obscurity concept could also be implemented to protect the information as it is transmitted across the network. When the Hart Experimental [Page 8] INTERNET DRAFT Change Management Logging Protocol July 2014 data-by-obscurity technique is implemented the network attacker is not able to determine what it is they have captured. The data is not easily interpreted and the attacker would most likely move onto the next option. ACLs can be used to control flow of CMLP information across the network as they travel from source to destination. 6.2. System Downtime on CM Server If one or more of the configured CM servers experiences downtime or is undergoing maintenance, CMLP requests will fail in Enhanced and Strict modes. The fail-allow setting when cconfigured would allow the SA to enter configuration mode when the CMR number is provided. However, if fail-prohibit setting is configured, the networked element will not be able to authorize entry to the SA to configuration mode. The SA will either need to re-schedule the maintenance work or enter configuration mode using a break-fix work around. 6.3. Break Fix During an outage situation where rapid recovery is required, the CMLP request and authorization process will need to be circumvented. The networked element must be able to allow access to configuration mode for the purpose of break-fix/system recovery. As mentioned, the over-arching ideal of CMLP is to improve network availability and reducing network restoral time while minimizing impedence to the restoral process. The method for entering configuration mode for the purpose of break-fix should be well-known and relatively simple to achieve. That being said, best practices for securing networked elements should be adhered to such that the attack surface for the network and its elements are not compromised. The organization or vendor implementation of CMLP could specify separate ticketing formats to address CM work and break-fix scenarios. Given the CM work format (Example: "CMR004321"), the networked element identifies the request to enter configuration mode as an attempt to start maintenance work. Given the break-fix format (Example: "TT00123456"), the networked element identifies the request to enter configuration mode as an attempt to perform break-fix work. In the break-fix scenario, the networked element could log the entry into configuration mode as purposed for break-fix. 6.4. CMLP Traffic Floods An enterprise network environment where several hosts, routers, switches, CM servers and other devices are sending CMLP packets, it is possible that the CMLP traffic could overwhelm network devices or links. CMLP traffic flows could be limited, as mentioned, through the use of ACLs and rate limiting. 7. CMLP Compliance The following items must be followed for a networked element to be CMLP compliant: Hart Experimental [Page 9] INTERNET DRAFT Change Management Logging Protocol July 2014 - Enhanced and Strict modes require connection to one or more CMLP servers. Configuration of multiple CM servers is recommended to address issues of redundancy. Strict mode requires at least two CM server IP addresses to be configured. This assumes either IPv4 or IPv6 addressing is used for CM servers. - To ensure proper logging and adherence to common best practices, the networked element should be configured with connections to one or more NTP servers. - To ensure proper operation of CMLP throughout the enterprise network, unique hostnames need to be implemented across the network. No two networked elements within the enterprise network should utilize the same hostname. 8. Standardization Considerations CMLP can be configured to send multicast notifications. In order for CMLP to function in an enterprise network environment where multiple platforms are used, CMLP will require a multicast IP address assignment. If IPv6 addressing is used in the enterprise network, an IPv6 multicast address would be required. This will require assignment of an IPv6 multicast address, similar to IPv4. With the exception of CMLP Multicast Notifications, CMLP is best suited to use TCP for sending CMLP packets across the network. UDP is recommended for the multicast notifications, particularly since no response to the notification is necessary or desired. As CMLP develops further, the ports designated for use by CMLP capable networked elements should be standardized to support cross-platform implementations. 9. Assumptions The following assumptions are made - When multiple CM server IP addresses are configured, the CM servers are assumed to be redundant and contain identical information. Multiple CM servers are configured to address CM server downtime for a server at a given IP address. - Organizations choosing to implement CMLP-capable networked devices have existing CM policies and procedures. The policies and procedures include management oversight to the CM process. - Organizations providing network connectivity to clients, internal or external, look to provide best-in-class customer experience and consider high network availability a key priority. 10. Glossary and Abbreviations ACL - Access Control Lists CM - Change Management CMR - A Change Management Request is the request submitted for review and approval by network management or senior management for implementation team/s to make changes to specified network elements. Hart Experimental [Page 10] INTERNET DRAFT Change Management Logging Protocol July 2014 CMLP - Change Management Logging Protocol Networked Element - Any information system element connected to the network. This includes but is not limited to routers, switches, firewalls, and servers. NTP - Network Time Protocol SA - System Administrator. ---- This Internet Draft expires January 31, 2015 ---- Hart Experimental [Page 11]