INTERNET-DRAFT S. Fenter Intended Status: Informational Enterprise Data Center Operators Expires: September 6, 2018 March 5, 2018 Why Enterprises Need Out-of-Band TLS Decryption draft-fenter-tls-decryption-00 Abstract Some enterprises are heavily TLS encrypted within their own enterprise network boundaries. Many of these enterprises are also utilizing out-of-band TLS decryption in order to inspect their own traffic for purposes of troubleshooting, network security monitoring, and for other kinds of monitoring. These monitoring functions are mission critical, and cannot just be done without when TLS 1.3 (draft-ietf-tls-tls13-26) is released or when the RSA key exchange is someday deprecated from TLS 1.2 (RFC5246). This draft will outline the use cases for out-of-band TLS decryption, as well as alternative suggestions for monitoring and troubleshooting and the limitations of those alternatives. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/1id-abstracts.html The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html Fenter Expires September 6, 2018 [Page 1] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 Copyright and License Notice Copyright (c) 2018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Fenter Expires September 6, 2018 [Page 2] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5 2. Out-of-Band Decryption Use Cases for Diagnostics and Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . 6 2.1 Application Performance Monitoring . . . . . . . . . . . . 6 2.2 Network Diagnostics and Troubleshooting . . . . . . . . . . 6 2.2.1 Network Packet Analysis . . . . . . . . . . . . . . . . 6 2.2.2 Packet Analysis with Source Address Translation . . . . 7 2.2.3 TCP Pipelining - Session Multiplexing . . . . . . . . . 7 2.2.4 TLS Encrypted MQ to the mainframe . . . . . . . . . . . 8 2.2.5 Application Layer Data . . . . . . . . . . . . . . . . 8 2.2.6 Customer Experience Monitoring . . . . . . . . . . . . 9 2.2.7 VoIP Analysis . . . . . . . . . . . . . . . . . . . . . 9 2.3 Out-of-Band Decryption Use Cases for Network Security Monitoring . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3.1 Layer 7 DDoS Attacks . . . . . . . . . . . . . . . . . 10 2.3.2 Fraud Monitoring . . . . . . . . . . . . . . . . . . . 10 2.3.3 Intrusion Detection System . . . . . . . . . . . . . . 11 2.3.4 Threat Detection and Incident Response . . . . . . . . 11 2.3.5 Regulatory Examples . . . . . . . . . . . . . . . . . . 11 2.3.5.1 PCI (Payment Card Industry) . . . . . . . . . . . . 11 2.3.5.1.1 PCI and TLS Encryption . . . . . . . . . . . . . 11 2.3.5.1.2 Intrusion Detection . . . . . . . . . . . . . . 12 2.3.5.1.3 TLS 1.2 and PCI . . . . . . . . . . . . . . . . 12 2.3.5.2 e-CFR (Electronic Code of Federal Regulations) . . . 12 2.3.5.2.1 Insider Abuse . . . . . . . . . . . . . . . . . 12 2.3.5.3 Regulatory Requirements - Summary . . . . . . . . . 13 3. Alternative Solutions Offered and Their Limitations . . . . . 13 3.1 Inline/MITM Decryption . . . . . . . . . . . . . . . . . . 13 3.2 Using TCP or UDP Extensions to Supply Extra Information . . 14 3.3 Using IP and TCP Headers for Monitoring and Troubleshooting . . . . . . . . . . . . . . . . . . . . . . 15 3.4 TLS 1.2 . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.5 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.6 Troubleshooting at the Endpoint . . . . . . . . . . . . . . 16 3.7 Security Monitoring at the Endpoint . . . . . . . . . . . . 16 3.8 Encrypted Traffic Inspection . . . . . . . . . . . . . . . 17 3.9 IPsec instead of TLS . . . . . . . . . . . . . . . . . . . 17 4. An Examination of Arguments Against All Network Decryption . . 18 4.1 Technical Arguments . . . . . . . . . . . . . . . . . . . . 18 4.1.1 "I work for a large company and we don't have to decrypt packets." . . . . . . . . . . . . . . . . . . . 18 4.2 Privacy Arguments . . . . . . . . . . . . . . . . . . . . . 18 4.2.1 "It's a violation of personal privacy to decrypt TLS traffic anywhere except at the TLS endpoint." . . . . . 18 4.2.2 "Pervasive Monitoring is an Attack" . . . . . . . . . . 19 5. Possible TLS 1.3 Decryption Solutions . . . . . . . . . . . 19 Fenter Expires September 6, 2018 [Page 3] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 5.1 Static Diffie Hellman . . . . . . . . . . . . . . . . . . . 19 5.2 TLS 1.3 Option for Negotiation of Visibility in the Datacenter . . . . . . . . . . . . . . . . . . . . . . . . 19 5.3 Solution Summary . . . . . . . . . . . . . . . . . . . . . 20 6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 20 7. Security Considerations . . . . . . . . . . . . . . . . . . . 20 8. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 20 9. References . . . . . . . . . . . . . . . . . . . . . . . . . . 21 9.1 Normative References . . . . . . . . . . . . . . . . . . . 21 9.2 Informative References . . . . . . . . . . . . . . . . . . 21 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 21 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 21 Fenter Expires September 6, 2018 [Page 4] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 1. Introduction Most enterprise networks originally transmitted packet data in the clear inside their internal networks. Many still do today. When certain enterprises started TLS encrypting their internal networks to protect against insider threat and/or for regulatory compliance reasons, they always had the option of using RSA key exchanges and using static RSA private keys for a small, privileged group to decrypt and inspect their traffic out-of-band. Out-of-band decryption provides ubiquitous packet payload visibility inside the enterprise that cannot be replaced by inline/MITM decryption solutions. Today there are enterprises with extensive packet broker networks who are doing out-of-band TLS decryption to feed network sniffers, intrusion detection devices, fraud detection, malware detection, application performance monitoring tools, customer experience monitoring tools, and other solutions. The capability to do out-of-band decryption has been available for twenty years, and for the first time in history it will be gone with the move to TLS1.3 [TLS13]. A large body of tools has grown up over the last twenty years that is dependent on out-of-band decryption. These tools are performing mission critical functions for enterprises, and the loss of out-of-band decryption will create major operational problems for TLS encrypted enterprises if TLS 1.3 is implemented as-is inside the enterprise. Ubiquitous packet capture and decryption are required for enterprise troubleshooting, and without this capability there will be high severity outages that cannot be solved in an acceptable time frame. The outcome will be the same as extended Denial of Service attacks on enterprises worldwide. Without an out-of-band decryption solution, enterprises are left with the unattractive option of inline/MITM decryption at the data center edge and running traffic with legacy protocols or in the clear throughout the data center if they need packet payload visibility. This opens certain enterprises up to significant regulatory and insider threat problems. There are reasons why other forms of troubleshooting and monitoring do not functionally replace the visibility lost from losing out-of-band TLS decryption. These alternative suggestions are discussed in the sections below. TLS 1.2 [RFC5246] is not a long term option for enterprises. The RSA key exchange is gradually being removed by vendors as a TLS 1.2 option. For example, mobile devices have been seen to send TLS 1.2 Client Hello's with no RSA key exchange options. There is also the risk that new vulnerabilities and weaknesses will be discovered with TLS 1.2 and/or RSA that will accelerate its removal by other vendors. Fenter Expires September 6, 2018 [Page 5] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 2. Out-of-Band Decryption Use Cases for Diagnostics and Troubleshooting 2.1 Application Performance Monitoring Network-based Application Performance Monitoring requires TLS decryption in order to track application response time by user and by URL, which is the information that the application owners and the lines of business need. The end user IP address is obscured by TLS termination and by source address translation, both on the Internet and within the data center. The identity of the user can only be determined by looking at the payload of the TLS packet. URL identification allows the application support team to do granular, code level troubleshooting and performance monitoring at multiple tiers of an application. 2.2 Network Diagnostics and Troubleshooting 2.2.1 Network Packet Analysis The key to effective network packet troubleshooting is the ability to follow a transaction through multiple tiers of an application in order to isolate the fault domain. The extensive use of both encryption and source address translation in some enterprises has made it difficult, or even impossible to follow a transaction without the ability to decrypt TLS and examine the payload of the packet. When the payload is available, the packet analyst can find unique identifiers (userids, session ids, etc.) as well as the URL the end user is accessing in order to identify the correct TCP conversation. The packet payload allows analysts to trace the slow or failing transaction above and below firewalls, above and below load balancers, above and below switches, etc., in order to isolate the fault domain. This kind of analysis is not possible from the endpoint alone. There are firewalls and load balancers that do not terminate TLS, and there can be a significant number of infrastructure devices in between the TLS endpoints, any one of which can be the cause of a problem. This above/below analysis across all intermediate infrastructure devices often provides the only insights into where the root problem is introduced. It is noteworthy that adding more inline/MITM decryption solutions to a multi-tiered application environment would increase the need for above/below analysis to rule in or out the inline decryption solution itself as the cause of a problem. The increased complexity of the environment would lead to more failures and longer problem resolution times. Packet payload visibility also allows an analyst to match up a Fenter Expires September 6, 2018 [Page 6] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 network packet trace with log entries in an application log that give an indication of the problem that occurred. This kind of application level packet tracing could be between bare metal servers, or between two VM's on the same hypervisor. Inline/MITM TLS decryption solutions do not scale for this kind of troubleshooting. As an example, because a problem could hit anywhere, an inline/MITM solution would be needed between every possible pair of VM's in a virtual environment, which could scale into the thousands. This large number of inline/MITM solutions would also create its own problems due to the complexity added to the environment. 2.2.2 Packet Analysis with Source Address Translation Content Delivery Networks on the Internet have multiple TLS termination points, and the user's source (client) IP is lost. When tracing a data center's Internet connection, it is not possible to even find the correct TCP conversation for an end user that is having a problem without TLS decryption. The packets must be decrypted and the appropriate HTTP header fields examined in order to find the end user IP address. Within the data center, the source IP for inbound TLS can again be changed multiple times. If a load balancer does not terminate TLS it will NAT the source IP so that return packets will find their way back to the load balancer, and to the correct load balancer in a pair. Alternatively, the load balancer may terminate TLS, and start an independent TLS session to the next layer below. Again, the source IP becomes the load balancer's IP address, and return packets will find their way back to the correct load balancer. Reverse proxies, web servers, app servers, and middleware servers can all terminate TLS and start independent TLS calls to lower layers, each time altering the source (client side) IP of packets, and calling a completely different URL. User sessions are also often sprayed randomly by load balancers to all these devices, and the network troubleshooter is left with no option except to trace all packets at a particular layer, decrypt them all, and look at the payload to find a user session. Servers and infrastructure devices typically don't have the horsepower to trace and decrypt all packets like this, but an out-of-band packet broker/sniffer infrastructure is designed to handle this load, and also provides a centralized location for managing and securing capture files. 2.2.3 TCP Pipelining - Session Multiplexing When TCP Pipelining/Session Multiplexing is used, multiple end user sessions share the same TCP connection. For the network troubleshooter, even if he/she could find the correct encrypted TCP Fenter Expires September 6, 2018 [Page 7] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 connection for an end user, there is no way to tell which packet belongs to which end user without decryption. 2.2.4 TLS Encrypted MQ to the mainframe MQ requests to the mainframe are farmed out to multiple processing nodes, and the MQ response comes back asynchronously from any one of those processing nodes. There is no way to find the response to a particular request without looking at identifiers in the payload of the packet. This requires TLS decryption. MQ requests can also pass through many infrastructure devices (i.e. load balancers, firewalls, etc.) before reaching the mainframe. Inline/MITM decryption solutions are not scalable for this environment, because visibility could be needed anywhere along the path. 2.2.5 Application Layer Data A decrypted TLS packet contains a wealth of critical troubleshooting information for HTTP (e.g. HTTP requests and return codes) as well as for a number of other protocols. Without this level of information, network troubleshooters are blind, unless the problem is some kind of a basic network problem. Even in the case of network problems, the application level detail is sometimes critical for isolating problems, for example, in the case of an intermittent network slowdown or failure. When looking through millions of packets, transactional/application level detail can help the analyst zero in on the correct location in the trace where a network problem is occurring. It is not enough, though, to look only at the HTTP headers. Applications have been known, for example, to return an HTTP 200 OK, yet contain an error message in the payload of the HTTP response. This can only be seen in the decrypted application layer payload of the packet. Applications also use XML or JSON structures in the payload of the packet to store interesting information like user ids and session IDs. Oftentimes this is the level of information that the application support teams possess ("I sent out a request with this session ID and didn't get a response."). The application team doesn't, however, know where their request went wrong among the many layers of infrastructure, network connections, etc., that their request passes through. The network packet troubleshooters are able to follow the transaction through the many layers of infrastructure if they are able to access the packet payload and find a matching identifier. Fenter Expires September 6, 2018 [Page 8] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 2.2.6 Customer Experience Monitoring Enterprises involved in online commerce have a business need to monitor customer behavior on their web sites. This monitoring requires TLS decryption of the full packet payload, as any location on the web page can be of interest to the user and to those monitoring user behavior. 2.2.7 VoIP Analysis When attempting to monitor and/or troubleshoot user experience within voice and video communications, the ability to understand the signaling (session setup and teardown) is absolutely critical. Session Initiation Protocol (SIP) is among the most commonly used control protocols in the VoIP environment, and increasingly it is being encrypted with TLS. SIP request and response codes, when visible, make it possible for even a novice user to understand the basics of what is being requested and what type of response is being provided (Request: INVITE, Response: 200 OK). The detail available in SIP messages enable a level of analysis difficult to mirror using any other method. The SIP/RTP stream must be decrypted in order to allow analysis of these setup messages. In fact, it is not possible to even find the UDP connections to analyze without decryption of the SIP header, because the IP/port pair is found in an SDP of the SIP signaling packets. Phone endpoints are typically not designed for detailed troubleshooting. Many handsets do not have the ability to output SIP signaling information. Endpoints are also not completely trustworthy in a troubleshooting scenario, and network analysis is needed to verify what is happening on the wire. VoIP calls can also be affected by network conditions. Tracing may be done in different locations to identify the effect the network is having on the VOIP call. An inline/MITM solution doesn't scale for this use case. Session Border Controllers can trace SIP signaling, but tracing is often too resource intensive to run on these devices, as they are not designed to handle the extra load. This means that VOIP analysts need out of band packet capture and decryption solutions which are designed for this purpose. Call quality on the audio RTP stream can be monitored with network based tools, if TLS on the audio stream can be decrypted. This gives the VoIP analyst a view of the problem that they can't get from the endpoints. The audio stream is peer-to-peer communication necessitating many visibility points. Again, an inline/MITM Fenter Expires September 6, 2018 [Page 9] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 decryption strategy doesn't scale. 2.3 Out-of-Band Decryption Use Cases for Network Security Monitoring 2.3.1 Layer 7 DDoS Attacks Layer 7 DDoS attacks can involve multiple IPs and source-ports generating traffic very similar to that of genuine users. The only way to identify layer 7 attack traffic is via inspecting fields in the packet payload, which are invisible until the packet is decrypted. Internet based DDoS protection services are not perfect. If they are tuned too tightly they block some legitimate production traffic. If they are tuned too loosely, some attack traffic gets through. In reality, during a DDoS attack some attack traffic usually gets through, and enterprises have to be armed to protect themselves. One of the tools they need is the ability to decrypt Internet TLS traffic so they can block layer 7 DDoS attacks. This decryption could be done by an inline/MITM solution, although there is a possibility that an inline decryption solution could be overwhelmed by a volumetric DDoS attack or by an attack targeting session state, becoming a point of failure. An out-of-band or transparent TLS decryption solution does not carry this risk of being overwhelmed and blocking all legitimate traffic. 2.3.2 Fraud Monitoring Fraud monitoring is the monitoring and detection of suspicious activities within, through, or perpetrated against a company. It must be reported to regulatory agencies as required by applicable laws and regulations. Examples of fraud are unauthorized account access and identify theft. Fraud monitoring is a mission critical function for financial institutions, and there are network-based tools performing this function with decrypted TLS packets. If fraud monitoring is down, then it is a severity one problem for critical applications. Fraud monitoring looks at network packets in many locations. An inline/MITM solution in this environment does not scale. One of the major fraud monitoring applications consists of an array of servers, including a database, all talking to each other via TLS. Application errors for this fraud monitoring app need to be analyzed just like any other application, including network packet analysis, and TLS decryption is needed in order to match up log errors with network packets on the wire. Fenter Expires September 6, 2018 [Page 10] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 2.3.3 Intrusion Detection System IDS inspection looks for known and custom malware signatures, potential attack patterns, and known observables associated to Indicators of Compromise in the payload of TLS packets when decryption is available. IDS inspection for inbound and/or internal TLS sometimes depends on out-of-band TLS decryption, and its effectiveness is severely impacted if decryption is not available. IDS inspection is often a regulatory requirement, for example, for cardholder data environments, of which an enterprise may have many. An inline/MITM TLS decryption solution has scalability problems in this kind of environment. 2.3.4 Threat Detection and Incident Response IDS Alerts - Threat Detection teams receive IDS alerts and will analyze decrypted network packet traces in order to verify if the alert was valid or was a false alarm. SQL Injection Attacks - This particular alert also needs manual analysis of packet traces in order to identify if the attack was successful, and if so, what data was returned. Endpoint Monitoring Alerts - These alerts often need to be verified with decrypted packet traces, including identification of the source of the attack on the endpoint. Endpoints are less trustworthy than network monitoring tools, and network monitoring is also needed as a backstop for any failures of monitoring on the endpoint. Manual Hunting - Not all attacks are caught by automated monitoring. Threat Detection teams will do manual hunting for known vulnerabilities with decrypted packet traces. Ubiquitous packet payload visibility can be provided by out-of-band decryption for inbound or internal TLS sessions. Traffic sources for malware can be anywhere within the enterprise or external to the enterprise. An inline/MITM decryption solution doesn't scale. 2.3.5 Regulatory Examples 2.3.5.1 PCI (Payment Card Industry) 2.3.5.1.1 PCI and TLS Encryption The PCI Security Standards Council strongly recommends segmenting the cardholder data environment in order to protect the cardholder systems as well as to limit the scope of PCI assessment (PCI Fenter Expires September 6, 2018 [Page 11] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 Information Supplement: Guidance for PCI DSS Scoping and Network Segmentation, p. 6). PCI has a concept of "connected to" systems, meaning that any system communicating with a system in the cardholder data environment is drawn into PCI requirements. As a practical reality, large enterprises could not get a PCI assessment completed without segmenting the cardholder data environment with tools like firewalling and encryption of data in transit. This creates the need for TLS decryption in order for those enterprises to do troubleshooting, network security monitoring, and other packet-based analysis in which the clear-text payload must be available. 2.3.5.1.2 Intrusion Detection The PCI DSS (Data Security Standard) requires IDS/IPS inspection at the perimeter of the cardholder data environment, as well as at critical points in the cardholder data environment (PCI DSS section 11.4). If an organization's monitoring and data loss prevention strategy includes payload inspection, TLS encrypted traffic in these environments must be decrypted. PCI applications have grown up over time, and there may be many cardholder data environments in a data center. Inline/MITM decryption solutions are not scalable for this environment due to cost, introduced latency, and production risk from the more complicated, inline/MITM environment. 2.3.5.1.3 TLS 1.2 and PCI When significant vulnerabilities were found in SSL and early TLS in late 2014 (including POODLE), it took the PCI Security Standards Council less than a year to require a migration plan away from these SSL/TLS versions (PCI Information Supplement: Migrating from SSL and Early TLS). Enterprises are at risk that vulnerabilities could be found in TLS 1.2 or in the RSA key exchange, and that PCI will require upgrade to TLS 1.3. There is no guarantee that TLS 1.2 will be available many years into the future. 2.3.5.2 e-CFR (Electronic Code of Federal Regulations) 2.3.5.2.1 Insider Abuse The United States e-CFR, Title 12, Chapter 1, Part 21 requires that national banks and federal branches and agencies of foreign banks monitor and report on insider abuse. This monitoring looks for criminal behavior like employee fraud, and is a mission critical and legally required function for financial institutions operating in the United States. If potentially illegal or fraudulent activity is detected, a Suspicious Activity Report must be filed with FinCEN (the Financial Crimes Enforcement Network of the US Department of the Treasury). Fenter Expires September 6, 2018 [Page 12] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 One of the tools for accomplishing this monitoring is a network based tool that uses out-of-band TLS decryption to inspect network packets and analyze user activity. The user endpoint cannot be trusted for this monitoring, as it is controlled by the user being suspected of fraudulent or illegal activity. The loss of out-of-band decryption would be crippling to this monitoring. There are many monitoring points for this tool, and inline/MITM decryption is not a scalable option. Also, these kinds of tools have no need to be inline, and forcing the solution inline adds unnecessary complexity and production risk into a mission critical environment. 2.3.5.3 Regulatory Requirements - Summary This section is by no means an exhaustive discussion of all regulatory requirements for all verticals in all countries worldwide. However, it does illustrate the kinds of issues that can come up when a long standing "feature" is eliminated from a network security protocol. 3. Alternative Solutions Offered and Their Limitations 3.1 Inline/MITM Decryption This is a valid sounding option but presents scalability problems for large, diversified enterprises. TLS decryption for security monitoring is needed in more locations than just everything in and out of the Internet; for example, network security monitoring is done on Business to Business connections, the branch/MPLS head-end, the mainframe, cardholder data environments, wireless controllers, DNS servers, etc. Network troubleshooters need to be able to take traces and decrypt them anywhere in the enterprise network. This can be hundreds or even thousands of locations, depending on the particular problem that hits, and can include the data center, branches, the virtual environment, and public or private clouds. It's not scalable to try and put an inline/MITM decryption solution between any two servers in the enterprise who may be talking to each other via TLS. This could include VM's talking to each other on the same hypervisor or containers talking to each other on the same VM. Cost - Bypass taps and TLS decryption appliances are expensive, and the cost adds up when adequate resiliency and failover is architected. The cost for implementing an inline/MITM decryption strategy can quickly escalate into millions of dollars. Latency - TLS decryption appliances add 1-3 ms of latency per packet Fenter Expires September 6, 2018 [Page 13] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 in a hardware appliance. Virtual decryption appliances may take a larger latency hit. When multiple inline decryption locations are implemented, the latency becomes prohibitive. Production Risk - Bypass taps and TLS decryption appliances are complicated devices that can and do fail. When an inline/MITM TLS decryption solution fails, production traffic is brought down. An inline/MITM solution is far more complex than putting in passive network taps, which are very simple and almost never fail. Out-of-band TLS decryption is a better design for much of the enterprise. It provides for ubiquitous packet payload visibility with lower cost, no latency, and almost non-existent production risk. Obtaining packets in the virtual and cloud environments is more complex, but an out-of-band TLS decryption solution is still more scalable than inline/MITM TLS decryption everywhere in the virtual or cloud environment. 3.2 Using TCP or UDP Extensions to Supply Extra Information This is also an argument that sounds good on the surface and looks like it helps preserve privacy. However, there are a number of reasons why this idea doesn't work in an enterprise. There can be session identifiers in the payload of packets that are needed in order to match up packets (whose source IP may be changing) inside and outside of an infrastructure device, and also to match up application layer requests with application layer responses (the responses don't always ride on the same TCP conversation as the request). These session identifiers are unique to each application, and it is not possible to anticipate, among thousands of applications, which fields in the payload are going to be important for a particular problem. An individual enterprise can have thousands of unique applications. The next enterprise down the road can have thousands more applications of their own, all unique and different from the first enterprise. Software bugs can leave telltale signs in the payload of a packet. These telltale signs are critical pieces of information for troubleshooting difficult problems. It is not possible to anticipate, among thousands of applications, which parts of the payload are going to be important for finding a software bug. Indicators of Compromise can exist anywhere within the payload of a packet. It is not possible to anticipate for every new attack which part of the payload will be important for threat detection and incident analysis. Fenter Expires September 6, 2018 [Page 14] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 Fields that can be used to block layer 7 DDoS attacks can be anywhere within the payload of the packet. It's not possible to determine which field to block on for any particular DDoS attack until the full payload is decrypted and examined. Customer Experience Monitoring requires full packet payload. A click anywhere on a web page can be of interest to those doing the monitoring. 3.3 Using IP and TCP Headers for Monitoring and Troubleshooting This approach has all the same problems as those outlined in section 3.2 above. 3.4 TLS 1.2 No enterprise wants to run an older, less secure version of a protocol for the long term. The RSA key exchange is already in the process of being deprecated from TLS 1.2 in some environments. Examples of this have already been seen in the mobile device environment. Suggestions have been made on the TLS email list that we need to deprecate the RSA key exchange from TLS 1.2. All we need is for another major RSA vulnerability be found, and this sentiment will gain traction. 3.5 Logging There are many enterprise outages and slowdowns where there is either no log message on the offending device, or there is a log message that indicates a problem but no clue as to the fault domain or the root cause. Infrastructure devices do not understand layer 7 and so are unable to log meaningful information about a layer 7 transaction that had a problem, even if that particular infrastructure device was the cause of the problem. In many cases, network packet analysis with TLS decryption is required in order to identify the fault domain and/or get to the root cause. It is not possible for a code developer to anticipate every possible problem that is going to occur and put a log message in just the right place. Also, the very nature of a software bug is that the developer doesn't know it's there, so there is not going to be any log message when a bug hits. It is not feasible to go through millions of lines of code in an enterprise environment and "improve" the logging on each device. Fenter Expires September 6, 2018 [Page 15] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 Between infrastructure and security devices, and application code, this would involve getting hundreds or thousands of vendors to invest and cooperate with this idea. Vendors would have no idea what to log other than what is currently being logged in order to try and catch the "next" problem. Adding log messages after a problem hits is like playing enterprise "Whack-a-mole". The next problem to hit is invariably something completely different. 3.6 Troubleshooting at the Endpoint The shortcomings of endpoint logging are covered in section 3.5 above. Endpoints in a typical enterprise don't have the robustness to run a full packet capture of all packets, decrypt them all, and keep the trace running all the time. This kind of trace is necessary because analysts don't know which web or app server, for example, is going to be hit for a particular user session, and they don't know when an intermittent problem is going to hit. Instead, enterprises have built up robust, out-of-band packet sniffing devices with TLS decryption capability fed by passive network taps and/or passive mirror ports. Endpoint analysis also misses the crucial troubleshooting function of isolating the fault domain of a problem among many infrastructure devices between the TLS endpoints. 3.7 Security Monitoring at the Endpoint Network security monitoring is done by a number of purpose built network devices such as IDS/IPS and security analysis solutions. Network based fraud detection applications can include multiple servers and databases that all communicate with each other. It's not feasible to put all this functionality into an endpoint and have its normal workload unaffected. Endpoints can be overwhelmed by too much security monitoring and their performance impacted. Networks can also be overwhelmed by extensive security reporting from endpoints. As a result, endpoint monitoring is often scaled back to a level that the endpoint and its network connection can handle. Endpoints cannot be completely trusted for network security monitoring. Malware can delete logs and turn off future logging. It's also not always possible to secure data stored on an endpoint, for example, if the endpoint is a laptop and the user packs it up and Fenter Expires September 6, 2018 [Page 16] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 walks out of the enterprise. The great variety of endpoint types also makes it difficult to implement a consistent monitoring strategy using endpoints alone. Network security monitoring is an important complement to endpoint security monitoring, and is part of a "defense in depth" strategy. 3.8 Encrypted Traffic Inspection This technology, while interesting and applicable in some situations, does not fully satisfy the requirements of enterprise traffic inspection. From an application performance and availability perspective, encrypted traffic inspection will not figure out severity one slowdowns or outages, or any other level of problem that may hit an enterprise. Large enterprises have thousands of unique applications that all behave differently at layer 7, and any one of these applications may need layer 7 analysis when a problem hits. This factor of troubleshooting alone is enough to make encrypted traffic inspection an unacceptable, or at least incomplete, solution for enterprise encryption problems. Encrypted traffic inspection does not address fraud detection for either internal or external fraud, both of which look at decrypted TLS packets. Encrypted packet inspection does not address Application Performance Monitoring, Customer Experience Monitoring, or the use of decrypted packets for regulatory compliance monitoring. From a security perspective, encrypted traffic inspection is not going to detect every zero day attack. The parameters it is looking for in the TLS handshake can be varied by new malware. Encrypted traffic inspection, for some methodologies, may be less effective under TLS 1.3 when the handshake is encrypted. Encrypted traffic inspection doesn't take into account the manual, deep packet inspection done by threat detection teams in order to analyze malware alerts, track down their source, and to identify if an attack succeeded or failed. 3.9 IPsec instead of TLS The enterprise rollout of internal TLS has been a multi-year project. Enterprises can't just flip a switch and start running IPsec. Moving to IPsec would likely be a multi-year and expensive project. There is extensive manual configuration that would need to be done. Fenter Expires September 6, 2018 [Page 17] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 There are a number of infrastructure services that don't support IPsec today. For example, IPsec is not supported today by load balancers for data center load balancing services. IPsec is also not supported by Internet proxies for outbound Internet services. A number of new IETF protocols are tied to TLS, including HTTP/2, DPRIVE, and QUIC. Enterprises need a TLS decryption solution in order to support these protocols. 4. An Examination of Arguments Against All Network Decryption 4.1 Technical Arguments 4.1.1 "I work for a large company and we don't have to decrypt packets." Large Internet companies being put forward as examples of promoting encryption are not necessarily encrypted through their entire private enterprise environment as some financial and health care institutions are. There are varying levels of data center depth and complexity between enterprises. Some enterprises have a flatter data center structure, depending on the kinds of services they offer. Other enterprises have many layers to their applications, with multiple layers of infrastructure like firewalls and load balancers, and many layers of middleware, authentication, fraud detection, mainframe, etc. There are also legacy applications that add complexity to the infrastructure, and add to the requirement for decrypted packet payload analysis. Network packet decryption, if it is happening, is likely not visible to all employees within an enterprise. Typically, only select groups within an organization utilize or are aware of this level of detail. As more enterprises in different verticals add TLS decryption inside their data centers, they are going to realize that they also have a need for out-of-band TLS decryption. 4.2 Privacy Arguments 4.2.1 "It's a violation of personal privacy to decrypt TLS traffic anywhere except at the TLS endpoint." Enterprises have many legitimate business reasons for inspecting their own data, and the IETF should provide them with well studied and standardized options that meet these critical business needs. Fenter Expires September 6, 2018 [Page 18] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 All TLS data is already being decrypted multiple times in the enterprise data center, and customer data is already available to certain employees of that enterprise. Network packet decryption is just one more decryption of its own data by the same enterprise. 4.2.2 "Pervasive Monitoring is an Attack" Pervasive monitoring inside the enterprise has many legitimate use cases, including troubleshooting and network security monitoring. As an example, some applications are too complicated to troubleshoot at the network packet level without advanced preparation of the packet level monitoring, meaning that it takes too much time during a critical outage to get traces set up in all the right places. The answer to this is pervasive monitoring of that application or that environment so that network packet traces, including TLS decryption, are ready when a problem hits. RFC 7258 [RFC7258] should be modified to account for the many enterprise use cases where pervasive monitoring is not an attack. 5. Possible TLS 1.3 Decryption Solutions 5.1 Static Diffie Hellman Static Diffie Hellman [draft-green] as described in draft-green-tls- static-dh-in-tls13 meets the enterprise need in a manner similar to running RSA key exchanges and using static RSA private keys. Enterprises would be obligated to protect their static keys as they are today in the RSA environment. Draft-green requires no changes to the TLS client, and no changes to the TLS 1.3 spec. It has no impact on the CPU load of the TLS server. Enterprises have the option of rotating their static Diffie-Hellman private keys as often as they see fit. 5.2 TLS 1.3 Option for Negotiation of Visibility in the Datacenter TLS 1.3 Option for Negotiation of Visibility in the Datacenter [draft-rhrd] as a solution has some alternative features in comparison to draft-green [draft-green]. It eliminates static Diffie-Hellman private keys from the TLS server as in the case of draft-green. The key manager would only write static private keys from the SSWrapDH1 key pair to the decryption appliances in the protected enterprise network. Draft-rhrd provides for client opt-in and visibility on the wire that traffic payload inspection may be happening. It also allows for decryption in the case of session reuse, which solves a large problem for enterprise monitoring and troubleshooting. It will have some impact on the CPU load of the TLS server. Fenter Expires September 6, 2018 [Page 19] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 5.3 Solution Summary For both of these solutions, a standard is needed so that all the related systems will interoperate. Draft-green [draft-green] would need to be implemented by multiple TLS server vendors, multiple decryption appliance vendors, and multiple key management solutions. Draft-rhrd [draft-rhrd] would require all of these in addition to implementation by TLS clients. For the above reasons, custom code is a highly unattractive, and possibly unworkable solution. 6. Conclusion Out-of-band TLS decryption is used by a number of enterprise tools for mission critical functions, and it supplies ubiquitous packet payload visibility that can't be replaced by other methods. Endpoint analysis is limited by its lack of robustness for analytic activities, by the fact that it can't be completely trusted, and by its blindness to issues in the intervening infrastructure. Inline/MITM decryption adds cost, latency, and production risk at every point it is implemented, and it doesn't scale to meet the requirements of the use cases presented above. 7. Security Considerations There are security tradeoffs that enterprises should be allowed to decide. On the one side are the benefits of Forward Secrecy inside the enterprise. On the other side are the benefits of ubiquitous packet payload visibility inside the enterprise. Enterprises are most qualified to make this business decision for themselves, and the TLS Working Group should provide them options rather than making the decision for them. Enterprises choosing to do out-of-band decryption need to continue to implement whatever security controls are appropriate for protection of this decryption environment, including protection of keys, controlling access to the decrypted data, etc. 8. IANA Considerations There are no IANA considerations. Fenter Expires September 6, 2018 [Page 20] INTERNET DRAFT draft-fenter-tls-decryption-00 March 5, 2018 9. References 9.1 Normative References [draft-green] Green, M., Droms, R., Housley, R., Turner, P., and S. Fenter, "Data Center use of Static Diffie-Hellman in TLS 1.3", draft- green-tls-static-dh-in-tls13-01 (work in progress), July 2017. [draft-rhrd] Housley, R. and Droms, R., "TLS 1.3 Option for Negotiation of Visibility in the Datacenter", draft-rhrd-tls-tls13- visibility-01 (work in progress), March 2018. [RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer Security (TLS) Protocol Version 1.2", RFC 5246, DOI 10.17487/RFC5246, August 2008, . [RFC7258] Farrell, S. and Tschofenig, H., "Pervasive Monitoring Is an Attack", BCP 188, RFC 7258, DOI 10.17487/RFC7258, May 2014, . [TLS13] Rescorla, E., "The Transport Layer Security (TLS) Protocol Version 1.3", draft-ietf-tls-tls13-26 (work in progress), March 2018. 9.2 Informative References Acknowledgments Nalini Elkins, Mike Ackermann, Darin Pettis, and Russ Housley contributed through discussion to the development of this document. Authors' Addresses Steve Fenter Enterprise Data Center Operators, Inc. 36A Upper Circle Carmel Valley, CA 93924 EMail: info@e-dco.com Fenter Expires September 6, 2018 [Page 21]