| detnet | P. Wetterwald |
| Internet-Draft | Cisco |
| Intended status: Informational | J. Raymond |
| Expires: April 30, 2015 | Hydro-Quebec |
| October 27, 2014 |
Deterministic Networking Uitilities requirements
draft-wetterwald-detnet-utilities-reqs-01
This paper documents the needs in Smart Grid industry to establish multi-hop paths for characterized flows with deterministic properties .
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on April 30, 2015.
Copyright (c) 2014 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
[I-D.finn-detnet-problem-statement] defines the characteristics of a deterministic flow as a data communication flow with a bounded latency, extraordinarily low frame loss, and a very narrow jitter. This document intends to define the utility requirements for deterministic networking.
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].
Evolution of Utility Telecom Networks
The business and technology trends that are sweeping the utility industry will drastically transform the utility business from the way it has been for many decades. At the core of many of these changes is a drive to modernize the electrical grid with an integrated telecommunications infrastructure. However, interoperability, concerns, legacy networks, disparate tools, and stringent security requirements all add complexity to grid transformation. Given the range and diversity of the requirement that should be addressed by the next generation telecommunications infrastructure utilities need to adopt a holistic architectural approach to integrate the electrical grid with digital telecommunication across the entire power delivery chain.
Many utilities still rely on complex environments formed of multiple application-specific, proprietary networks. Information is siloed between operational areas. This prevents utility operations from realizing the operational efficiency benefits, visibility, and functional integration of operational information across grid applications and data networks. The key to modernizing grid telecommunications is to provide a common, adaptable, multi-service network infrastructure for the entire utility organization. Such a network serves as the platform for current capabilities while enabling future expansion of the network to accommodate new applications and services.
To meet this diverse set of requirements, both today and in the future, the next generation utility telecommunnication network will be based on open-standards-based IP architecture. An end-to-end IP architecture takes advantage of nearly three decades of IP technology development, facilitating interoperability across disparate networks and devices, as it has been already demonstrated in many mission-critical and highly secure networks.
IEC and different National Committees have mandated a specific adhoc group (AHG8) to define the migration strategy to IPv6 for all the IEC TC57 power automation standards. IPv6 is seen as the obvious future telecommunication technology for the Smart Grid. The Adhoc Group will discose, to the IEC coordination group, their conclusions at the end of 2014.
It is imperative that utilities participate in standards development bodies to influence the development of future solutions and to benefit from shared experiences of other utilities and vendors.
These general telecommunication requirements are over and above the specific requirements of the use cases that have been addressed so far. These include both current and future telecommunication related requirements that should be factored into the network architecture and design.
Throughout the world, utilities are increasingly planning for a future based on smart grid applications requiring advanced telecommunications systems. Many of these applications utilize packet connectivity for communicating information and control signals across the utility's Wide Area Network (WAN), made possible by technologies such as multiprotocol label switching (MPLS). The data that traverses the utility WAN includes:
WANs support this wide variety of traffic to and from substations, the transmission and distribution grid, generation sites, between control centers, and between work locations and data centers. To maintain this rapidly expanding set of applications, many utilities are taking steps to evolve present time-division multiplexing (TDM)based and frame relay infrastructures to packet systems. Packet-based networks are designed to provide greater functionalities and higher levels of service for applications, while continuing to deliver reliability and deterministic (real-time) traffic support.
Among the numerous applications and use cases that a utility deploys today, many rely on high availability and deterministic behaviour of the telecommunications networks. Protection use cases and generation control are the most demanding and can't rely on a best effort approach.
Protection means not only the protection of the human operator but also the protection of the electric equipments and the preservation of the stability and frequency of the grid. If a default occurs on the transmission or the distribution of the electricity, important damages could occured to the human operator but also to very costly electrical equipments and perturb the grid leading to blackouts. The time and reliability requirements are very strong to avoid dramatic impacts to the electrical infrastructure.
The key criteria for measuring Teleprotection performance are command transmission time, dependability and security. These criteria are defined by the IEC standard 60834 as follows:
Additional key elements that may impact Teleprotection performance include bandwidth rate of the Teleprotection system and its resiliency or failure recovery capacity. Transmission time, bandwidth utilization and resiliency are directly linked to the telecommunications equipment and the connections that are used to transfer the commands between relays.
Delay requirements for utility networks may vary depending upon a number of parameters, such as the specific protection equipment used. Most power line equipment can tolerate short circuits or faults for up to approximately five power cycles before sustaining irreversible damage or affecting other segments in the network. This translates to total fault clearance time of 100ms. As a safety precaution, however, actual operation time of protection systems is limited to 70- 80 percent of this period, including fault recognition time, command transmission time and line breaker switching time. Some system components, such as large electromechanical switches, require particularly long time to operate and take up the majority of the total clearance time, leaving only a 10ms window for the telecommunications part of the protection scheme, independent of the distance to travel. Given the sensitivity of the issue, new networks impose requirements that are even more stringent: IEC standard 61850 limits the transfer time for protection messages to 1/4 - 1/2 cycle or 4 - 8ms (for 60Hz lines) for the most critical messages
In addition to minimal transmission delay, a differential protection telecommunication channel must be synchronous, i.e., experiencing symmetrical channel delay in transmit and receive paths. This requires special attention in jitter-prone packet networks. While optimally Teleprotection systems should support zero asymmetric delay, typical relays can tolerate discrepancies of up to 750us.
The main tools available for lowering delay variation below this threshold are:
The following table captures the main network requirements (this is based on IEC 61850 standard)
| Teleprotection Requirement | Attribute |
|---|---|
| One way maximum delay | 4-10 ms |
| Asymetric delay required | Yes |
| Maximum jitter | less than 250 us |
| Topology | Point to point, point to Multi-point |
| Availability | 99.9999 |
| precise timing required | Yes |
| Recovery time on node failure | less than 50ms - hitless |
| performance management | Yes, Mandatory |
| Redundancy | Yes |
| Packet loss | 0.1% to 1% |
Inter-tripping is the controlled tripping of a circuit breaker to complete the isolation of a circuit or piece of apparatus in concert with the tripping of other circuit breakers. The main use of such schemes is to ensure that protection at both ends of a faulted circuit will operate to isolate the equipment concerned. Inter-tripping schemes use signaling to convey a trip command to remote circuit breakers to isolate circuits.
| Inter-Trip protection Requirement | Attribute |
|---|---|
| One way maximum delay | 5 ms |
| Asymetric delay required | No |
| Maximum jitter | Not critical |
| Topology | Point to point, point to Multi-point |
| Bandwidth | 64 Kbps |
| Availability | 99.9999 |
| precise timing required | Yes |
| Recovery time on node failure | less than 50ms - hitless |
| performance management | Yes, Mandatory |
| Redundancy | Yes |
| Packet loss | 0.1% |
Current differential protection is commonly used for line protection, and is typical for protecting parallel circuits. A main advantage for differential protection is that, compared to overcurrent protection, it allows only the faulted circuit to be de-energized in case of a fault. At both end of the lines, the current is measured by the differential relays, and based on Kirchhoff's law, both relays will trip the circuit breaker if the current going into the line does not equal the current going out of the line. This type of protection scheme assumes some form of communication being present between the relays at both end of the line, to allow both relays to compare measured current values. A fault in line 1 will cause overcurrent to be flowing in both lines, but because the current in line 2 is a through following current, this current is measured equal at both ends of the line, therefore the differential relays on line 2 will not trip line 2. Line 1 will be tripped, as the relays will not measure the same currents at both ends of the line. Line differential protection schemes assume a very low telecommunications delay between both relays, often as low as 5ms. Moreover, as those systems are often not time-synchronized, they also assume symmetric telecommunications paths with constant delay, which allows comparing current measurement values taken at the exact same time.
| Current Differential protection Requirement | Attribute |
|---|---|
| One way maximum delay | 5 ms |
| Asymetric delay Required | Yes |
| Maximum jitter | less than 250 us |
| Topology | Point to point, point to Multi-point |
| Bandwidth | 64 Kbps |
| Availability | 99.9999 |
| precise timing required | Yes |
| Recovery time on node failure | less than 50ms - hitless |
| performance management | Yes, Mandatory |
| Redundancy | Yes |
| Packet loss | 0.1% |
Distance (Impedance Relay) protection scheme is based on voltage and current measurements. A fault on a circuit will generally create a sag in the voltage level. If the ratio of voltage to current measured at the protection relay terminals, which equates to an impedance element, falls within a set threshold the circuit breaker will operate. The operating characteristics of this protection are based on the line characteristics. This means that when a fault appears on the line, the impedance setting in the relay is compared to the apparent impedance of the line from the relay terminals to the fault. If the relay setting is determined to be below the apparent impedance it is determined that the fault is within the zone of protection. When the transmission line length is under a minimum length, distance protection becomes more difficult to coordinate. In these instances the best choice of protection is current differential protection.
| Distance protection Requirement | Attribute |
|---|---|
| One way maximum delay | 5 ms |
| Asymetric delay Required | No |
| Maximum jitter | Not critical |
| Topology | Point to point, point to Multi-point |
| Bandwidth | 64 Kbps |
| Availability | 99.9999 |
| precise timing required | Yes |
| Recovery time on node failure | less than 50ms - hitless |
| performance management | Yes, Mandatory |
| Redundancy | Yes |
| Packet loss | 0.1% |
This use case describes the exchange of Sampled Value and/or GOOSE message between IEDs in two substations for protection and tripping coordination. The two IEDs are in a master-slave mode.
The CT/VT in one substation sends the sampled analog voltage or current value to the Merging Unit (MU) over hard wire. The merging unit sends the time-synchronized 61850-9-2 sampled values to the slave IED. The slave IED forwards the information to the Master IED in the other substation. The master IED makes the determination (for example based on sampled value differentials) to send a trip command to the originating IED. Once the slave IED/Relay receives the GOOSE trip for breaker tripping, it opens the breaker. It then sends a confirmation message back to the master. All data exchanges between IEDs are either through Sampled Value and/or GOOSE messages.
| Inter-Substation protection Requirement | Attribute |
|---|---|
| One way maximum delay | 5 ms |
| Asymetric delay Required | No |
| Maximum jitter | Not critical |
| Topology | Point to point, point to Multi-point |
| Bandwidth | 64 Kbps |
| Availability | 99.9999 |
| precise timing required | Yes |
| Recovery time on node failure | less than 50ms - hitless |
| performance management | Yes, Mandatory |
| Redundancy | Yes |
| Packet loss | 1% |
This use case describes the data flow from the CT/VT to the IEDs in the substation via the merging unit (MU). The CT/VT in the substation send the sampled value (analog voltage or current) to the Merging Unit (MU) over hard wire. The merging unit sends the time-synchronized 61850-9-2 sampled values to the IEDs in the substation in GOOSE message format. The GPS Master Clock can send 1PPS or IRIG-B format to MU through serial port, or IEEE 1588 protocol via network. Process bus communication using 61850 simplifies connectivity within the substation and removes the requirement for multiple serial connections and removes the slow serial bus architectures that are typically used. This also ensures increased flexibility and increased speed with the use of multicast messaging between multiple devices.
| Intra-Substation protection Requirement | Attribute |
|---|---|
| One way maximum delay | 5 ms |
| Asymetric delay Required | No |
| Maximum jitter | Not critical |
| Topology | Point to point, point to Multi-point |
| Bandwidth | 64 Kbps |
| Availability | 99.9999 |
| precise timing required | Yes |
| Recovery time on Node failure | less than 50ms - hitless |
| performance management | Yes, Mandatory |
| Redundancy | Yes - No |
| Packet loss | 0.1% |
The application of synchrophasor measurement data from Phasor Measurement Units (PMU) to Wide Area Monitoring and Control Systems promises to provide important new capabilities for improving system stability. Access to PMU data enables more timely situational awareness over larger portions of the grid than what has been possible historically with normal SCADA data. Handling the volume and real-time nature of synchrophasor data presents unique challenges for existing application architectures. Wide Area management System (WAMS) makes it possible for the condition of the bulk power system to be observed and understood in real-time so that protective, preventative, or corrective action can be taken. Because of the very high sampling rate of measurements and the strict requirement for time synchronization of the samples, WAMS has stringent telecommunication requirements in an IP network that are captured in the following table:
| WAMS Requirement | Attribute |
|---|---|
| One way maximum delay | 50 ms |
| Asymetric delay Required | No |
| Maximum jitter | Not critical |
| Topology | Point to point, point to Multi-point, Multi-point to Multi-point |
| Bandwidth | 100 Kbps |
| Availability | 99.9999 |
| precise timing required | Yes |
| Recovery time on Node failure | less than 50ms - hitless |
| performance management | Yes, Mandatory |
| Redundancy | Yes |
| Packet loss | 1% |
As the name implies, Fault Location, Isolation, and Service Restoration (FLISR) refers to the ability to automatically locate the fault, isolate the fault, and restore service in the distribution network. It is a self-healing feature whose purpose is to minimize the impact of faults by serving portions of the loads on the affected circuit by switching to other circuits. It reduces the number of customers that experience a sustained power outage by reconfiguring distribution circuits. This will likely be the first wide spread application of distributed intelligence in the grid. Secondary substations can be connected to multiple primary substations. Normally, static power switch statuses (open/closed) in the network dictate the power flow to secondary substations. Reconfiguring the network in the event of a fault is typically done manually on site to operate switchgear to energize/de-energize alternate paths. Automating the operation of substation switchgear allows the utility to have a more dynamic network where the flow of power can be altered under fault conditions but also during times of peak load. It allows the utility to shift peak loads around the network. Or, to be more precise, alters the configuration of the network to move loads between different primary substations. The FLISR capability can be enabled in two modes:
There are 3 distinct sub-functions that are performed:
1. Fault Location Identification
This sub-function is initiated by SCADA inputs, such as lockouts, fault indications/location, and, also, by input from the Outage Management System (OMS), and in the future by inputs from fault-predicting devices. It determines the specific protective device, which has cleared the sustained fault, identifies the de-energized sections, and estimates the probable location of the actual or the expected fault. It distinguishes faults cleared by controllable protective devices from those cleared by fuses, and identifies momentary outages and inrush/cold load pick-up currents. This step is also referred to as Fault Detection Classification and Location (FDCL). This step helps to expedite the restoration of faulted sections through fast fault location identification and improved diagnostic information available for crew dispatch. Also provides visualization of fault information to design and implement a switching plan to isolate the fault.
2. Fault Type Determination
I. Indicates faults cleared by controllable protective devices by distinguishing between:
a. Faults cleared by fuses
b. Momentary outages
c. Inrush/cold load current
II. Determines the faulted sections based on SCADA fault indications and protection lockout signals
III. Increases the accuracy of the fault location estimation based on SCADA fault current measurements and real-time fault analysis
3. Fault Isolation and Service Restoration
Once the location and type of the fault has been pinpointed the systems will attempt to isolate the fault and restore the non-faulted section of the network. This can have three modes of operation:
I. Closed-loop mode : This is initiated by the Fault location sub-function. It generates a switching order (i.e., sequence of switching) for the remotely controlled switching devices to isolate the faulted section, and restore service to the non-faulted sections. The switching order is automatically executed via SCADA.
II. Advisory mode : This is initiated by the Fault location sub-function. It generates a switching order for remotely and manually controlled switching devices to isolate the faulted section, and restore service to the non-faulted sections. The switching order is presented to operator for approval and execution
III. Study mode : the operator initiates this function. It analyzes a saved case modified by the operator, and generates a switching order under the operating conditions specified by the operator.
With the increasing volume of data that are collected through fault sensors, utilities will use Big Data query and analysis tools to study outage information to anticipate and prevent outages by detecting failure patterns and their correlation with asset age, type, load profiles, time of day, weather conditions, and other conditions to discover conditions that lead to faults and take the necessary preventive and corrective measures.
| FLISR Requirement | Attribute |
|---|---|
| One way maximum delay | 80 ms |
| Asymetric delay Required | No |
| Maximum jitter | 40 ms |
| Topology | Point to point, point to Multi-point, Multi-point to Multi-point |
| Bandwidth | 64 Kbps |
| Availability | 99.9999 |
| precise timing required | Yes |
| Recovery time on Node failure | Depends on customer impact |
| performance management | Yes, Mandatory |
| Redundancy | Yes |
| Packet loss | 0.1% |
The system frequency should be maintained within a very narrow band. Deviations from the acceptable frequency range are detected and forwarded to the Load Frequency Control (LFC) system so that required up or down generation increase / decrease pulses can be sent to the power plants for frequency regulation. The trend in system frequency is a measure of mismatch between demand and generation, and is a necessary parameter for load control in interconnected systems.
Automatic generation control (AGC) is a system for adjusting the power output of generators at different power plants, in response to changes in the load. Since a power grid requires that generation and load closely balance moment by moment, frequent adjustments to the output of generators are necessary. The balance can be judged by measuring the system frequency; if it is increasing, more power is being generated than used, and all machines in the system are accelerating. If the system frequency is decreasing, more demand is on the system than the instantaneous generation can provide, and all generators are slowing down.
Where the grid has tie lines to adjacent control areas, automatic generation control helps maintain the power interchanges over the tie lines at the scheduled levels. The AGC takes into account various parameters including the most economical units to adjust, the coordination of thermal, hydroelectric, and other generation types, and even constraints related to the stability of the system and capacity of interconnections to other power grids.
For the purpose of AGC we use static frequency measurements and averaging methods are used to get a more precise measure of system frequency in steady-state conditions.
During disturbances, more real-time dynamic measurements of system frequency are taken using PMUs, especially when different areas of the system exhibit different frequencies. But that is outside the scope of this use case.
| FCAG Requirement | Attribute |
|---|---|
| One way maximum delay | 500 ms |
| Asymetric delay Required | No |
| Maximum jitter | Not critical |
| Topology | Point to point |
| Bandwidth | 20 Kbps |
| Availability | 99.999 |
| precise timing required | Yes |
| Recovery time on Node failure | N/A |
| performance management | Yes, Mandatory |
| Redundancy | Yes |
| Packet loss | 1% |
Utilities often have very large private telecommunications networks. It covers an entire territory / country. The main purpose of the network, until now, has been to support transmission network monitoring, control, and automation, remote control of generation sites, and providing FCAPS services from centralized network operation centers.
Going forward, one network will support operation and maintenance of electrical networks (generation, transmission, and distribution), voice and data services for ten of thousands of employees and for exchange with neighboring interconnections, and administrative services. To meet those requirements, utility may deploy several physical networks leveraging different technologies across the country: an optical network and a microwave network for instance. Each protection and automatism system between two points has two telecommunication circuits, one on each network. Path diversity between two substations is key. Regardless of the event type (hurricane, ice storm, etc.), one path shall stay available so the SPS can still operate.
In the optical network, signals are transmitted over more than tens of thousands of circuits using fiber optic links, microwave and telephone cables. This network is the nervous system of the utility's power transmission operations. The optical network represents ten of thousands of km of cable deployed along the power lines.
Due to vast distances between transmission substations (as far as 280km apart), the fiber signal is amplified to reach a distance of 280 km without attenuation.
Some utilities do not use GPS clocks in generation substations. One of the main reasons is that some of the generation plants are 30 to 50 meters deep under ground and the GPS signal can be weak and unreliable. Instead, atomic clocks are used. Clocks are synchronized amongst each other. Rubidium clocks provide clock and 1ms timestamps for IRIG-B. Some companies plan to transition to the Precision Time Protocol (IEEE 1588), distributing the synchronization signal over the IP/MPLS network.
The Precision Time Protocol (PTP) is defined in IEEE standard 1588. PTP is applicable to distributed systems consisting of one or more nodes, communicating over a network. Nodes are modeled as containing a real-time clock that may be used by applications within the node for various purposes such as generating time-stamps for data or ordering events managed by the node. The protocol provides a mechanism for synchronizing the clocks of participating nodes to a high degree of accuracy and precision.
PTP operates based on the following assumptions :
A time-stamp event is generated at the time of transmission and reception of any event message. The time-stamp event occurs when the message's timestamp point crosses the boundary between the node and the network.
This memo includes no request to IANA.
Grid monitoring and control devices are already targets for cyber attacks and legacy telecommunication protocols have many intrinsic network related vulnerabilities. DNP3, Modbus, PROFIBUS/PROFINET, and other protocols are designed around a common paradigm of request and respond. Each protocol is designed for a master device such as an HMI system to send commands to subordinate slave devices to retrieve data (reading inputs) or control (writing to outputs). Because many of these protocols lack authentication, encryption, or other basic security measures, they are prone to network-based attacks, allowing a malicious actor or attacker to utilize the request-and-respond system as a mechanism for command-and-control like functionality. Specific security concerns common to most industrial control, including utility telecommunication protocols include the following:
These inherent vulnerabilities, along with increasing connectivity between IT an OT networks, make network-based attacks very feasible. Simple injection of malicious protocol commands provides control over the target process. Altering legitimate protocol traffic can also alter information about a process and disrupt the legitimate controls that are in place over that process. A man- in-the-middle attack could provide both control over a process and misrepresentation of data back to operator consoles.
Although advanced telecommunication networks can assist in transforming the energy industry, playing a critical role in maintaining high levels of reliability, performance, and manageability, they also introduce the need for an integrated security infrastructure. Many of the technologies being deployed to support smart grid projects such as smart meters and sensors can increase the vulnerability of the grid to attack. Top security concerns for utilities migrating to an intelligent smart grid telecommunications platform center on the following trends:
This development of a diverse set of networks to support the integration of microgrids, open-access energy competition, and the use of network-controlled devices is driving the need for a converged security infrastructure for all participants in the smart grid, including utilities, energy service providers, large commercial and industrial, as well as residential customers. Securing the assets of electric power delivery systems, from the control center to the substation, to the feeders and down to customer meters, requires an end-to-end security infrastructure that protects the myriad of telecommunication assets used to operate, monitor, and control power flow and measurement. Cyber security refers to all the security issues in automation and telecommunications that affect any functions related to the operation of the electric power systems. Specifically, it involves the concepts of:
When designing and deploying new smart grid devices and telecommunication systems, it's imperative to understand the various impacts of these new components under a variety of attack situations on the power grid. Consequences of a cyber attack on the grid telecommunications network can be catastrophic. This is why security for smart grid is not just an ad hoc feature or product, it's a complete framework integrating both physical and Cyber security requirements and covering the entire smart grid networks from generation to distribution. Security has therefore become one of the main foundations of the utility telecom network architecture and must be considered at every layer with a defense-in-depth approach. Migrating to IP based protocols is key to address these challenges for two reasons:
1. IP enables a rich set of features and capabilities to enhance the security posture
2. IP is based on open standards, which allows interoperability between different vendors and products, driving down the costs associated with implementing security solutions in OT networks.
Securing OT telecommunication over packet-switched IP networks follow the same principles that are foundational for securing the IT infrastructure, i.e., consideration must be given to enforcing electronic access control for both person-to-machine and machine-to-machine communications, and providing the appropriate levels of data privacy, device and platform integrity, and threat detection and mitigation.
Faramarz Maghsoodlou, Ph. D. IoT Connected Industries and Energy Practice Cisco
Pascal Thubert, CTAO Cisco
| [RFC2119] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. |
| [min_ref] | authSurName, authInitials., "Minimal Reference", 2006. |
| [I-D.finn-detnet-problem-statement] | Finn, N. and P. Thubert, "Deterministic Networking Problem Statement", Internet-Draft draft-finn-detnet-problem-statement-01, October 2014. |