HyBi Working Group T. Yoshino Internet-Draft Google, Inc. Intended status: Standards Track April 15, 2015 Expires: October 17, 2015 Compression Extensions for WebSocket draft-ietf-hybi-permessage-compression-21 Abstract This document defines a framework for creating WebSocket extensions that add compression functionality to the WebSocket Protocol. An extension based on this framework compresses the payload data portion of WebSocket data messages on a per-message basis using parameters negotiated during the opening handshake. This framework provides a general method for applying a compression algorithm to the contents of WebSocket messages. Each compression algorithm has to be defined in a document defining the extension by specifying parameter negotiation and payload transformation algorithm in detail. This document also specifies one specific compression extension using the DEFLATE algorithm. Please send feedback to the hybi@ietf.org mailing list. Status of this Memo This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on October 17, 2015. Copyright Notice Copyright (c) 2015 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Yoshino Expires October 17, 2015 [Page 1] Internet-Draft Compression Extensions for WebSocket April 2015 Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Conformance Requirements and Terminology . . . . . . . . . . . 4 3. Complementary Terminology . . . . . . . . . . . . . . . . . . 5 4. WebSocket Per-message Compression Extension . . . . . . . . . 6 5. Extension Negotiation . . . . . . . . . . . . . . . . . . . . 7 5.1. General Negotiation Flow . . . . . . . . . . . . . . . . . 9 5.2. Negotiation Examples . . . . . . . . . . . . . . . . . . . 10 6. Framing . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 6.1. Compression . . . . . . . . . . . . . . . . . . . . . . . 12 6.2. Decompression . . . . . . . . . . . . . . . . . . . . . . 13 7. Intermediaries . . . . . . . . . . . . . . . . . . . . . . . . 15 8. permessage-deflate extension . . . . . . . . . . . . . . . . . 16 8.1. Method Parameters . . . . . . . . . . . . . . . . . . . . 17 8.1.1. Context Takeover Control . . . . . . . . . . . . . . . 17 8.1.2. Limiting the LZ77 sliding window size . . . . . . . . 19 8.1.3. Examples . . . . . . . . . . . . . . . . . . . . . . . 21 8.2. Message Payload Transformation . . . . . . . . . . . . . . 22 8.2.1. Compression . . . . . . . . . . . . . . . . . . . . . 22 8.2.2. Decompression . . . . . . . . . . . . . . . . . . . . 23 8.2.3. Examples . . . . . . . . . . . . . . . . . . . . . . . 24 8.3. Implementation Notes . . . . . . . . . . . . . . . . . . . 28 8.4. Intermediaries . . . . . . . . . . . . . . . . . . . . . . 28 9. Security Considerations . . . . . . . . . . . . . . . . . . . 29 10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 30 10.1. Registration of the "permessage-deflate" WebSocket Extension Name . . . . . . . . . . . . . . . . . . . . . . 30 10.2. Registration of the "Per-message Compressed" WebSocket Framing Header Bit . . . . . . . . . . . . . . . . . . . . 30 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 31 12. References . . . . . . . . . . . . . . . . . . . . . . . . . . 32 12.1. Normative References . . . . . . . . . . . . . . . . . . . 32 12.2. Informative References . . . . . . . . . . . . . . . . . . 32 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 33 Yoshino Expires October 17, 2015 [Page 2] Internet-Draft Compression Extensions for WebSocket April 2015 1. Introduction This document specifies a framework for adding compression functionality to the WebSocket Protocol [RFC6455]. The framework specifies how to define WebSocket Per-message Compression Extensions (PMCEs) for a compression algorithm based on the extension concept of the WebSocket Protocol specified in Section 9 of [RFC6455]. A WebSocket client and a peer WebSocket server negotiate the use of a PMCE and determine the parameters required to configure the compression algorithm during the WebSocket opening handshake. The client and server can then exchange data messages whose frames contain compressed data in the payload data portion. This framework only specifies a general method for applying a compression algorithm to the contents of WebSocket messages. Each individual PMCE has to be specified in a document describing in detail how to negotiate the configuration parameters for the specific compression algorithm used by that PMCE and how to transform (compress and decompress) data in the payload data portion. A WebSocket client may offer multiple PMCEs during the WebSocket opening handshake. A peer WebSocket server receiving the offer may choose to accept the preferred PMCE or decline all of them. PMCEs use the RSV1 bit of the WebSocket frame header to indicate whether a message is compressed or not, so that an endpoint can choose not to compress messages with incompressible contents. This document also specifies one specific PMCE based on the DEFLATE [RFC1951] algorithm. The DEFLATE algorithm is widely available on various platforms and its overhead is small. The extension name of this PMCE is "permessage-deflate". To align the end of compressed data to an octet boundary, this extension uses the algorithm described in Section 2.1 of [RFC1979]. Endpoints can take over the LZ77 sliding window [LZ77] used to build frames for previous messages to achieve a better compression ratio. For resource-limited devices, this extension provides parameters to limit memory usage for compression context. Yoshino Expires October 17, 2015 [Page 3] Internet-Draft Compression Extensions for WebSocket April 2015 2. Conformance Requirements and Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Requirements phrased in the imperative as part of algorithms (such as "strip any leading space characters" or "return false and abort these steps") are to be interpreted with the meaning of the key word ("MUST", "SHOULD", "MAY", etc.) used in introducing the algorithm. Conformance requirements phrased as algorithms or specific steps can be implemented in any manner, so long as the end result is equivalent. In particular, the algorithms defined in this specification are intended to be easy to understand and are not intended to be performant. This document references the procedure to _Fail the WebSocket Connection_. This procedure is defined in Section 7.1.7 of [RFC6455]. This document references the event that _The WebSocket Connection is Established_ and the event that _A WebSocket Message Has Been Received_. These events are defined in Section 4.1 and Section 6.2, respectively, of [RFC6455]. This document uses the Augmented Backus-Naur Form (ABNF) notation of [RFC5234]. The DIGIT (decimal 0-9) rule is included by reference, as defined in the Appendix B.1 of [RFC5234]. Yoshino Expires October 17, 2015 [Page 4] Internet-Draft Compression Extensions for WebSocket April 2015 3. Complementary Terminology This document defines some terms about WebSocket and WebSocket Extension mechanisms that are underspecified or not defined at all in [RFC6455]. This terminology is effective only in this document and any other documents explicitly referring to this section. "A data message" means a message consisting of Data Frames as defined in Section 5.6 of [RFC6455]. "A message payload (or payload of a message)" means the concatenation of the payload data portion of all Data Frames (see Section 6.2 of [RFC6455]) representing a single message. "An extension in use next to extension X" means the extension listed next to X in the "Sec-WebSocket-Extensions" header in the server's opening handshake as defined in Section 9.1 of [RFC6455]. Such an extension is applied to outgoing data from the application right after X on the sender side, but applied right before X to incoming data from the underlying transport. "An extension in use preceding extension X" means the extension listed right before X in the "Sec-WebSocket-Extensions" header in the server's opening handshake. Such an extension is applied to outgoing data from the application right before X on the sender side, but applied right after X to incoming data from the underlying transport. "An extension negotiation offer" means each element in the "Sec-WebSocket-Extensions" header in the client's opening handshake. "An extension negotiation response" means each element in the "Sec-WebSocket-Extensions" header in the server's opening handshake. "A corresponding extension negotiation response for an extension negotiation offer" means an extension negotiation response that a server sends back to the peer client containing the same extension name as the offer and meeting the requirements represented by the offer. "Accepting an extension negotiation offer" means including a corresponding extension negotiation response for the offer in the "Sec-WebSocket-Extensions" header in the server's opening handshake. "Declining an extension negotiation offer" means not including a corresponding extension negotiation response for the offer in the "Sec-WebSocket-Extensions" header in the server's opening handshake. Yoshino Expires October 17, 2015 [Page 5] Internet-Draft Compression Extensions for WebSocket April 2015 4. WebSocket Per-message Compression Extension WebSocket Per-message Compression Extensions (PMCEs) are extensions to the WebSocket Protocol enabling compression functionality. PMCEs are built based on the extension concept of the WebSocket Protocol specified in Section 9 of [RFC6455]. PMCEs are individually defined for each compression algorithm to be implemented, and are registered in the WebSocket Extension Name Registry created in Section 11.4 of [RFC6455]. Each PMCE referring to this framework MUST define the following: o The extension name of the PMCE and any applicable extension parameters that MUST be included in the "Sec-WebSocket-Extensions" header during the extension negotiation offer/response. o How to interpret the extension parameters exchanged during the opening handshake. o How to transform the payload of a message. One PMCE extension is defined in Section 8 of this document and is registered in Section 10. Other PMCEs may be defined in the future in other documents. Section 5 describes the basic extension negotiation process. Section 6 describes how to apply the compression algorithm with negotiated parameters to the contents of WebSocket messages. Yoshino Expires October 17, 2015 [Page 6] Internet-Draft Compression Extensions for WebSocket April 2015 5. Extension Negotiation To offer use of a PMCE, a client MUST include the extension name of the PMCE in the "Sec-WebSocket-Extensions" header field of its opening handshake of the WebSocket connection. Extension parameters are used to specify the PMCE offer in detail. For example, a client lists its preferred configuration parameter values for the compression algorithm of the PMCE. A client may also offer multiple PMCE choices to the server by including multiple elements in the "Sec-WebSocket-Extensions" header, one for each PMCE offered. This set of elements MAY include multiple PMCEs with the same extension name to offer the possibility to use the same algorithm with different configuration parameters. The order of elements is important as it specifies the client's preference. An element preceding another element has higher preference. It is recommended that a server accepts PMCEs with higher preference if the server supports them. A PMCE negotiation offer provides requests and/or hints to the server. A request in a PMCE negotiation offer indicates constraints on the server's behavior that must be satisfied if the server accepts the offer. For example, suppose that a server sends data compressed with the DEFLATE algorithm to a client. The server must keep the original bytes of data that it recently compressed and sent to the client. The client must keep the result of decompressing the bytes of data that it recently received from the server. The amount of bytes of data kept is called the LZ77 window size. The LZ77 window size of the client must not be less than the LZ77 window size of the server. In a PMCE negotiation offer, the client MUST inform the server of its LZ77 window size so that the server uses an LZ77 window size that is not greater than the LZ77 window size of the client. This restriction on the LZ77 window size is an example of a request in a PMCE negotiation offer. A hint in a PMCE negotiation offer provides information about the client's behavior that the server may either safely ignore or refer to when the server decides its behavior. For example, suppose that a client sends data compressed with the DEFLATE algorithm to a server. The client must keep the original bytes of data that it recently compressed and sent to the server. The server must keep the result of decompressing the bytes of data that it recently received from the client. The LZ77 window size of the server must not be less than the LZ77 window size of the client. In a PMCE negotiation offer, the client may inform the maximum LZ77 window size the client can afford so that the server can choose to use an LZ77 window size that is not greater than the maximum size of the client. This information is an Yoshino Expires October 17, 2015 [Page 7] Internet-Draft Compression Extensions for WebSocket April 2015 example of a hint in a PMCE negotiation offer. It's waste of memory to use an LZ77 window size greater than the LZ77 window size the client actually uses. Using the hint, the server can avoid the waste of memory. Since the hint itself doesn't specify the constraints on the endpoints, the server must use the "agreed parameters" (defined below) to explicitly ask the client not to use an LZ77 window size greater than the LZ77 window size of the server. To accept the use of an offered PMCE, a server MUST include the extension name of the PMCE in the "Sec-WebSocket-Extensions" header field of its opening handshake of the WebSocket connection. Extension parameters represent the detailed configuration parameters of the PMCE to use. These extension parameters and their values are called "agreed parameters". The element MUST represent a PMCE that is fully supported by the server. The contents of the element don't need to be exactly the same as those of the received extension negotiation offers. For example, suppose that a server received a PMCE extension negotiation offer with an extension parameter "X" indicating that the client can enable an optional feature named X. The server may accept the PMCE offer with an element without the extension parameter "X" meaning that the server chose not to enable the feature X. In this case, the offer contains the extension parameter "X" but the "agreed parameters" don't contain the extension parameter "X". "Agreed parameters" MUST represent how the requests and hints in the client's extension negotiation offer have been handled in addition to the server's requests and hints on the client's behavior, so that the client can configure its behavior without identifying exactly which PMCE extension negotiation offer has been accepted. For example, if a client sends an extension negotiation offer that includes a parameter "enable_compression" and another without this parameter, the server accepts the former and informs the client by sending back an element that includes parameter(s) acknowledging "enable_compression". The name of the acknowledging parameter doesn't need to be exactly the same as the offer. For example, two parameters "enable_strong_compression" and "enable_weak_compression" may be defined as acknowledging parameters for "enable_compression". Compression features can be applied differently for each direction. For such features, the acknowledging parameter and the parameter in the reverse direction must be chosen to distinguish them. For example, in order to make parameters distinguishable, a "server_" prefix can be added to parameters affecting data sent from a server and a "client_" prefix can be added to parameters affecting data sent from a client. Yoshino Expires October 17, 2015 [Page 8] Internet-Draft Compression Extensions for WebSocket April 2015 A server MUST NOT accept a PMCE extension negotiation offer together with another extension if the PMCE will conflict with the extension on their use of the RSV1 bit. A client that received a response accepting a PMCE extension negotiation offer together with such an extension MUST _Fail the WebSocket Connection_. A server MUST NOT accept a PMCE extension negotiation offer together with another extension if the PMCE will be applied to the output of the extension and any of the following conditions applies to the extension: o The extension requires the boundary of frames to be preserved between the output from the extension at the sender and the input to the extension at the receiver. o The extension uses the "Extension data" field or any of the reserved bits on the WebSocket header as a per-frame attribute. A client that receives a response accepting a PMCE extension negotiation offer together with such an extension MUST _Fail the WebSocket Connection_. A server declining all offered PMCEs MUST not include any element with PMCE names. If a server responds with no PMCE element in the "Sec-WebSocket-Extensions" header, both endpoints proceed without Per-message Compression once _the WebSocket Connection is established_. If a server gives an invalid response, such as accepting a PMCE that the client did not offer, the client MUST _Fail the WebSocket Connection_. If a server responds with a valid PMCE element in the "Sec-WebSocket-Extensions" header and _the WebSocket Connection is established_, both endpoints MUST use the algorithm described in Section 6 and the message payload transformation (compressing and decompressing) procedure of the PMCE configured with the "agreed parameters" returned by the server to exchange messages. 5.1. General Negotiation Flow This section describes a general negotiation flow. How to handle parameters in detail must be specified in the document specifying the PMCE. A client makes an offer including parameters identifying the following: Yoshino Expires October 17, 2015 [Page 9] Internet-Draft Compression Extensions for WebSocket April 2015 o Hints about how the client is planning to compress data o Requests about how the server compresses data o Limitations concerning the client's compression functionality The peer server makes a determination of its behavior based on these parameters. If the server can and wants to proceed with this PMCE enabled, the server responds to the client with parameters identifying the following: o Requests about how the client compresses data o How the server will compress data Based on these parameters received from the server, the client determines its behavior and if it can and wants to proceed with this PMCE enabled. Otherwise, the client starts closing handshake with close code 1010. 5.2. Negotiation Examples The following are example values for the "Sec-WebSocket-Extensions" header offering PMCEs. permessage-foo and permessage-bar in the examples are hypothetical extension names of PMCEs for compression algorithm foo and bar. o Offer the permessage-foo. permessage-foo o Offer the permessage-foo with a parameter x with a value of 10. permessage-foo; x=10 The value MAY be quoted. permessage-foo; x="10" o Offer the permessage-foo as first choice and the permessage-bar as a fallback plan. permessage-foo, permessage-bar o Offer the permessage-foo with a parameter use_y which enables a feature y as first choice, and the permessage-foo without the use_y parameter as a fallback plan. Yoshino Expires October 17, 2015 [Page 10] Internet-Draft Compression Extensions for WebSocket April 2015 permessage-foo; use_y, permessage-foo Yoshino Expires October 17, 2015 [Page 11] Internet-Draft Compression Extensions for WebSocket April 2015 6. Framing PMCEs operate only on data messages. This document allocates the RSV1 bit of the WebSocket header for PMCEs, and calls the bit the "Per-message Compressed" bit. On a WebSocket connection where a PMCE is in use, this bit indicates whether a message is compressed or not. A message with the "Per-message Compressed" bit set on the first fragment of the message is called a "compressed message". Frames of a compressed message have compressed data in the payload data portion. An endpoint receiving a compressed message decompresses the concatenation of the compressed data of the frames of the message by following the decompression procedure specified by the PMCE in use. The endpoint uses the bytes corresponding to the application data portion in this decompressed data for the _A WebSocket Message Has Been Received_ event instead of the received data as-is. A message with the "Per-message Compressed" bit unset on the first fragment of the message is called an "uncompressed message". Frames of an uncompressed message have uncompressed original data as-is in the payload data portion. An endpoint receiving an uncompressed message uses the concatenation of the application data portion of the frames of the message as-is for the _A WebSocket Message Has Been Received_ event. 6.1. Compression An endpoint MUST use the following algorithm to send a message in the form of a compressed message. 1. Compress the message payload of the original message by following the compression procedure of the PMCE. The original message may be input from the application layer or output of another WebSocket extension depending on which extensions were negotiated. 2. If this PMCE is the last extension to process of outgoing messages, build frame(s) by using the compressed data instead of the original data for the message payload, and set the "Per-message Compressed" bit of the first frame, then send the frame(s) as described in Section 6.1 of RFC6455. Otherwise, pass the transformed message payload and modified header values including the "Per-message Compressed" bit value set to 1 to the extension next to the PMCE. If the extension expects frames for input, build a frame for the message and pass it. Yoshino Expires October 17, 2015 [Page 12] Internet-Draft Compression Extensions for WebSocket April 2015 An endpoint MUST use the following algorithm to send a message in the form of an uncompressed message. If this PMCE is the last extension to process of outgoing messages, build frame(s) by using the original data for the payload data portion as-is and unset the "Per-message Compressed" bit of the first frame, then send the frame(s) as described in Section 6.1 of RFC6455. Otherwise, pass the message payload and header values to the extension next to the PMCE as-is. If the extension expects frames for input, build a frame for the message and pass it. An endpoint MUST NOT set the "Per-message Compressed" bit of control frames and non-first fragments of a data message. An endpoint receiving such a frame MUST _Fail the WebSocket Connection_. PMCEs do not change the opcode field. The opcode of the first frame of a compressed message indicates the opcode of the original message. The payload data portion in frames generated by a PMCE is not subject to the constraints for the original data type. For example, the concatenation of the output data corresponding to the application data portion of frames of a compressed text message is not required to be valid UTF-8. At the receiver, the payload data portion after decompression is subject to the constraints for the original data type again. 6.2. Decompression An endpoint MUST use the following algorithm to receive a message in the form of a compressed message. 1. Concatenate the payload data portion of the received frames of the compressed message. The received frames may be direct input from the underlying transport or output of another WebSocket extension depending on which extensions were negotiated. 2. Decompress the concatenation by following the decompression procedure of the PMCE. 3. If this is the last extension to process incoming messages, deliver the _A WebSocket Message Has Been Received_ event to the application layer with the decompressed message payload and header values including the "Per-message Compressed" bit unset to 0. Otherwise, pass the decompressed message payload and header values including the "Per-message Compressed" bit unset to 0 to the extension preceding the PMCE. If the extension expects frames for input, build a frame for the message and pass it. An endpoint MUST use the following algorithm to receive a message in Yoshino Expires October 17, 2015 [Page 13] Internet-Draft Compression Extensions for WebSocket April 2015 the form of an uncompressed message. If this PMCE is the last extension to process incoming messages, deliver the _A WebSocket Message Has Been Received_ event to the application layer with the received message payload and header values as-is. Otherwise, pass the message payload and header values to the extension preceding the PMCE as-is. If the extension expects frames for input, build a frame for the message and pass it. Yoshino Expires October 17, 2015 [Page 14] Internet-Draft Compression Extensions for WebSocket April 2015 7. Intermediaries When an intermediary proxies a WebSocket connection, the intermediary MAY add, change or remove Per-message Compression of proxied messages if the intermediary meets all of the following requirements: o The intermediary understands the PMCE. o The intermediary can read all data of the proxied WebSocket connection including the opening handshake request, opening handshake response, and messages. o The intermediary can alter the proxied data before forwarding them in accordance with to the constraints of the new combination of extensions. For example, if Per-message Compression is removed from messages, the corresponding element in the "Sec-WebSocket-Extensions" header in the opening handshake response which enabled the Per-message Compression must also be removed. Otherwise, the intermediary MUST NOT add, change or remove Per- message Compression of proxied messages. Yoshino Expires October 17, 2015 [Page 15] Internet-Draft Compression Extensions for WebSocket April 2015 8. permessage-deflate extension This section defines a specific PMCE called "permessage-deflate". It compresses the payload of a message using the DEFLATE algorithm [RFC1951] and uses the byte boundary alignment method introduced in [RFC1979]. This section uses the term "byte" with the same meaning as RFC1951, i.e. 8 bits stored or transmitted as a unit (same as an octet). The registered extension name for this extension is "permessage-deflate". Four extension parameters are defined for "permessage-deflate" to help endpoints manage per-connection resource usage. o "server_no_context_takeover" o "client_no_context_takeover" o "server_max_window_bits" o "client_max_window_bits" These parameters enable two methods (no_context_takeover and max_window_bits) of constraining memory usage that may be applied independently to either direction of WebSocket traffic. The extension parameters with the "client_" prefix are used by the client to configure its compressor and by the server to configure its decompressor. The extension parameters with the "server_" prefix are used by the server to configure its compressor and by the client to configure its decompressor. All four parameters are defined for both a client's extension negotiation offer and a server's extension negotiation response. A server MUST decline an extension negotiation offer for this extension if any of the following conditions are met: o The negotiation offer contains an extension parameter not defined for use in an offer. o The negotiation offer contains an extension parameter with an invalid value. o The negotiation offer contains multiple extension parameters with the same name. Yoshino Expires October 17, 2015 [Page 16] Internet-Draft Compression Extensions for WebSocket April 2015 o The server doesn't support the offered configuration. A client MUST _Fail the WebSocket Connection_ if the peer server accepted an extension negotiation offer for this extension with an extension negotiation response meeting any of the following conditions: o The negotiation response contains an extension parameter not defined for use in a response. o The negotiation response contains an extension parameter with an invalid value. o The negotiation response contains multiple extension parameters with the same name. o The client does not support the configuration that the response represents. The term "LZ77 sliding window" used in this section means the buffer used by the DEFLATE algorithm to store recently processed input. The DEFLATE compression algorithm searches the buffer for a match with the following input. The term "use context take over" used in this section means that the same LZ77 sliding window used by the endpoint to build frames of the previous sent message is reused to build frames of the next message to be sent. 8.1. Method Parameters 8.1.1. Context Takeover Control 8.1.1.1. server_no_context_takeover A client MAY include the "server_no_context_takeover" extension parameter in an extension negotiation offer. This extension parameter has no value. By including this extension parameter in an extension negotiation offer, a client prevents the peer server from using context take over. If the peer server doesn't use context take over, the client doesn't need to reserve memory to retain the LZ77 sliding window between messages. Absence of this extension parameter in an extension negotiation offer indicates that the client can decompress a message which the server built using context take over. A server accepts an extension negotiation offer that includes the Yoshino Expires October 17, 2015 [Page 17] Internet-Draft Compression Extensions for WebSocket April 2015 "server_no_context_takeover" extension parameter by including the "server_no_context_takeover" extension parameter in the corresponding extension negotiation response to send back to the client. The "server_no_context_takeover" extension parameter in an extension negotiation response has no value. It is RECOMMENDED that a server supports the "server_no_context_takeover" extension parameter in an extension negotiation offer. A server MAY include the "server_no_context_takeover" extension parameter in an extension negotiation response even if the extension negotiation offer being accepted by the extension negotiation response didn't include the "server_no_context_takeover" extension parameter. 8.1.1.2. client_no_context_takeover A client MAY include the "client_no_context_takeover" extension parameter in an extension negotiation offer. This extension parameter has no value. By including this extension parameter in an extension negotiation offer, a client informs the peer server of a hint that even if the server doesn't include the "client_no_context_takeover" extension parameter in the corresponding extension negotiation response to the offer, the client is not going to use context take over. A server MAY include the "client_no_context_takeover" extension parameter in an extension negotiation response. If the received extension negotiation offer includes the "client_no_context_takeover" extension parameter, the server may either ignore the parameter or use the parameter to avoid taking over the LZ77 sliding window unnecessarily by including the "client_no_context_takeover" extension parameter in the corresponding extension negotiation response to the offer. The "client_no_context_takeover" extension parameter in an extension negotiation response has no value. By including the "client_no_context_takeover" extension parameter in an extension negotiation response, a server prevents the peer client from using context take over. This reduces the amount of memory that the server has to reserve for the connection. Absence of this extension parameter in an extension negotiation response indicates that the server can decompress messages built by the client using context take over. A client MUST support the "client_no_context_takeover" extension parameter in an extension negotiation response. Yoshino Expires October 17, 2015 [Page 18] Internet-Draft Compression Extensions for WebSocket April 2015 8.1.2. Limiting the LZ77 sliding window size 8.1.2.1. server_max_window_bits A client MAY include the "server_max_window_bits" extension parameter in an extension negotiation offer. This parameter has a decimal integer value without leading zeroes between 8 to 15 inclusive indicating the base-2 logarithm of the LZ77 sliding window size and MUST conform to the ABNF below. server_max_window_bits = 1*DIGIT By including this parameter in an extension negotiation offer, a client limits the LZ77 sliding window size that the server will use to compress messages. If the peer server uses a small LZ77 sliding window to compress messages, the client can reduce the memory needed for the LZ77 sliding window. A server declines an extension negotiation offer with this parameter if the server doesn't support it. Absence of this parameter in an extension negotiation offer indicates that the client can receive messages compressed using an LZ77 sliding window of up to 32,768 bytes. A server accepts an extension negotiation offer with this parameter by including the "server_max_window_bits" extension parameter in the extension negotiation response to send back to the client with the same or smaller value as the offer. The "server_max_window_bits" extension parameter in an extension negotiation response has a decimal integer value without leading zeroes between 8 to 15 inclusive indicating the base-2 logarithm of the LZ77 sliding window size and MUST conform to the ABNF below. server_max_window_bits = 1*DIGIT A server MAY include the "server_max_window_bits" extension parameter in an extension negotiation response even if the extension negotiation offer being accepted by the response didn't include the "server_max_window_bits" extension parameter. 8.1.2.2. client_max_window_bits A client MAY include the "client_max_window_bits" extension parameter in an extension negotiation offer. This parameter has no value or a decimal integer value without leading zeroes between 8 to 15 inclusive indicating the base-2 logarithm of the LZ77 sliding window size. If a value is specified for this parameter, the value MUST Yoshino Expires October 17, 2015 [Page 19] Internet-Draft Compression Extensions for WebSocket April 2015 conform to the ABNF below. client_max_window_bits = 1*DIGIT By including this parameter in an offer, a client informs the peer server that the client supports the "client_max_window_bits" extension parameter in an extension negotiation response, and optionally a hint by attaching a value to the parameter. If the "client_max_window_bits" extension parameter in an extension negotiation offer has a value, the parameter also informs the peer server of a hint that even if the server doesn't include the "client_max_window_bits" extension parameter in the corresponding extension negotiation response with a value greater than the one in the extension negotiation offer or the server doesn't include the extension parameter at all, the client is not going to use an LZ77 sliding window size greater than the size specified by the value in the extension negotiation offer to compress messages. If a received extension negotiation offer has the "client_max_window_bits" extension parameter, the server MAY include the "client_max_window_bits" extension parameter in the corresponding extension negotiation response to the offer. If the "client_max_window_bits" extension parameter in a received extension negotiation offer has a value, the server may either ignore this value or use this value to avoid allocating an unnecessarily big LZ77 sliding window by including the "client_max_window_bits" extension parameter in the corresponding extension negotiation response to the offer with a value equal to or smaller than the received value. The "client_max_window_bits" extension parameter in an extension negotiation response has a decimal integer value without leading zeroes between 8 to 15 inclusive indicating the base-2 logarithm of the LZ77 sliding window size and MUST conform to the ABNF below. client_max_window_bits = 1*DIGIT By including this extension parameter in an extension negotiation response, a server limits the LZ77 sliding window size that the client uses to compress messages. This reduces the amount of memory for the decompression context that the server has to reserve for the connection. If a received extension negotiation offer doesn't have the "client_max_window_bits" extension parameter, the corresponding extension negotiation response to the offer MUST NOT include the "client_max_window_bits" extension parameter. Absence of this extension parameter in an extension negotiation response indicates that the server can receive messages compressed Yoshino Expires October 17, 2015 [Page 20] Internet-Draft Compression Extensions for WebSocket April 2015 using an LZ77 sliding window of up to 32,768 bytes. 8.1.3. Examples The simplest "Sec-WebSocket-Extensions" header in a client's opening handshake to offer use of the permessage-deflate extension looks like this: Sec-WebSocket-Extensions: permessage-deflate Since the "client_max_window_bits" extension parameter is not included in this extension negotiation offer, the server must not accept the offer with an extension negotiation response that includes the "client_max_window_bits" extension parameter. The simplest "Sec-WebSocket-Extensions" header in a server's opening handshake to accept use of the permessage-deflate extension is the same: Sec-WebSocket-Extensions: permessage-deflate The following extension negotiation offer sent by a client is asking the server to use an LZ77 sliding window with a size of 1,024 bytes or less and declaring that the client supports the "client_max_window_bits" extension parameter in an extension negotiation response. Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits; server_max_window_bits=10 This extension negotiation offer might be rejected by the server because the server doesn't support the "server_max_window_bits" extension parameter in an extension negotiation offer. This is fine if the client cannot receive messages compressed using a larger sliding window size, but if the client just prefers using a small window but wants to fall back to the "permessage-deflate" without the "server_max_window_bits" extension parameter, the client can make an offer with the fallback option like this: Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits; server_max_window_bits=10, permessage-deflate; client_max_window_bits The server can accept permessage-deflate by picking any supported one from the listed offers. To accept the first option, for example, the server may send back a response as follows: Yoshino Expires October 17, 2015 [Page 21] Internet-Draft Compression Extensions for WebSocket April 2015 Sec-WebSocket-Extensions: permessage-deflate; server_max_window_bits=10 To accept the second option, for example, the server may send back a response as follows: Sec-WebSocket-Extensions: permessage-deflate 8.2. Message Payload Transformation 8.2.1. Compression An endpoint uses the following algorithm to compress a message. 1. Compress all the octets of the payload of the message using DEFLATE. 2. If the resulting data does not end with an empty DEFLATE block with no compression (the "BTYPE" bits are set to 00), append an empty DEFLATE block with no compression to the tail end. 3. Remove 4 octets (that are 0x00 0x00 0xff 0xff) from the tail end. After this step, the last octet of the compressed data contains (possibly part of) the DEFLATE header bits with the "BTYPE" bits set to 00. When using DEFLATE in the first step above: o An endpoint MAY use multiple DEFLATE blocks to compress one message. o An endpoint MAY use DEFLATE blocks of any type. o An endpoint MAY use both DEFLATE blocks with the "BFINAL" bit set to 0 and DEFLATE blocks with the "BFINAL" bit set to 1. o When any DEFLATE block with the "BFINAL" bit set to 1 doesn't end at a byte boundary, an endpoint MUST add minimal padding bits of 0 to make it end at a byte boundary. The next DEFLATE block follows the padded data if any. An endpoint fragments a compressed message by splitting the result of running this algorithm. Even when only part of the payload is available, a fragment can be built by compressing the available data and choosing the block type appropriately so that the end of the resulting compressed data is aligned at a byte boundary. Note that for non-final fragments, the removal of 0x00 0x00 0xff 0xff must not be done. Yoshino Expires October 17, 2015 [Page 22] Internet-Draft Compression Extensions for WebSocket April 2015 An endpoint MUST NOT use an LZ77 sliding window longer than 32,768 bytes to compress messages to send. If the "agreed parameters" contain the "client_no_context_takeover" extension parameter, the client MUST start compressing each new message with an empty LZ77 sliding window. Otherwise, the client MAY take over the LZ77 sliding window used to build the last compressed message. Note that even if the client has included the "client_no_context_takeover" extension parameter in its offer, the client MAY take over the LZ77 sliding window used to build the last compressed message if the "agreed parameters" don't contain the "client_no_context_takeover" extension parameter. The client-to- server "client_no_context_takeover" extension parameter is just a hint for the server to build an extension negotiation response. If the "agreed parameters" contain the "server_no_context_takeover" extension parameter, the server MUST start compressing each new message with an empty LZ77 sliding window. Otherwise, the server MAY take over the LZ77 sliding window used to build the last compressed message. If the "agreed parameters" contain the "client_max_window_bits" extension parameter with a value of w, the client MUST NOT use an LZ77 sliding window longer than the w-th power of 2 bytes to compress messages to send. Note that even if the client has included in its offer the "client_max_window_bits" extension parameter with a value smaller than one in the "agreed parameters", the client MAY use an LZ77 sliding window with any size to compress messages to send as long as the size conforms to the "agreed parameters". The client-to- server "client_max_window_bits" extension parameter is just a hint for the server to build an extension negotiation response. If the "agreed parameters" contain the "server_max_window_bits" extension parameter with a value of w, the server MUST NOT use an LZ77 sliding window longer than the w-th power of 2 bytes to compress messages to send. 8.2.2. Decompression An endpoint uses the following algorithm to decompress a message. 1. Append 4 octets of 0x00 0x00 0xff 0xff to the tail end of the payload of the message. 2. Decompress the resulting data using DEFLATE. If the "agreed parameters" contain the "server_no_context_takeover" extension parameter, the client MAY decompress each new message with Yoshino Expires October 17, 2015 [Page 23] Internet-Draft Compression Extensions for WebSocket April 2015 an empty LZ77 sliding window. Otherwise, the client MUST decompress each new message using the LZ77 sliding window used to process the last compressed message. If the "agreed parameters" contain the "client_no_context_takeover" extension parameter, the server MAY decompress each new message with an empty LZ77 sliding window. Otherwise, the server MUST decompress each new message using the LZ77 sliding window used to process the last compressed message. Note that even if the client has included the "client_no_context_takeover" extension parameter in its offer, the server MUST decompress each new message using the LZ77 sliding window used to process the last compressed message if the "agreed parameters" don't contain the "client_no_context_takeover" extension parameter. The client-to-server "client_no_context_takeover" extension parameter is just a hint for the server to build an extension negotiation response. If the "agreed parameters" contain the "server_max_window_bits" extension parameter with a value of w, the client MAY reduce the size of its LZ77 sliding window to decompress received messages down to the w-th power of 2 bytes. Otherwise, the client MUST use a 32,768 byte LZ77 sliding window to decompress received messages. If the "agreed parameters" contain the "client_max_window_bits" extension parameter with a value of w, the server MAY reduce the size of its LZ77 sliding window to decompress received messages down to the w-th power of 2 bytes. Otherwise, the server MUST use a 32,768 byte LZ77 sliding window to decompress received messages. Note that even if the client has included in its offer the "client_max_window_bits" extension parameter with a value smaller than one in the "agreed parameters", the client MUST use an LZ77 sliding window of a size that conforms the "agreed parameters" to compress messages to send. The client-to-server "client_max_window_bits" extension parameter is just a hint for the server to build an extension negotiation response. 8.2.3. Examples This section introduces examples of how the permessage-deflate extension transforms messages. 8.2.3.1. A message compressed using 1 compressed DEFLATE block Suppose that an endpoint sends a text message "Hello". If the endpoint uses 1 compressed DEFLATE block (compressed with fixed Huffman code and the "BFINAL" bit not set) to compress the message, the endpoint obtains the compressed data to use for the message payload as follows. Yoshino Expires October 17, 2015 [Page 24] Internet-Draft Compression Extensions for WebSocket April 2015 The endpoint compresses "Hello" into 1 compressed DEFLATE block and flushes the resulting data into a byte array using an empty DEFLATE block with no compression: 0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00 0x00 0x00 0xff 0xff By stripping 0x00 0x00 0xff 0xff from the tail end, the endpoint gets the data to use for the message payload: 0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00 Suppose that the endpoint sends this compressed message without fragmentation. The endpoint builds one frame by putting the whole compressed data in the payload data portion of the frame: 0xc1 0x07 0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00 The first 2 octets (0xc1 0x07) are the WebSocket frame header (FIN=1, RSV1=1, RSV2=0, RSV3=0, opcode=text, MASK=0, Payload length=7). The following figure shows what value is set in each field of the WebSocket frame header. 0 1 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 +-+-+-+-+-------+-+-------------+ |F|R|R|R| opcode|M| Payload len | |I|S|S|S| |A| | |N|V|V|V| |S| | | |1|2|3| |K| | +-+-+-+-+-------+-+-------------+ |1|1|0|0| 1 |0| 7 | +-+-+-+-+-------+-+-------------+ Suppose that the endpoint sends the compressed message with fragmentation. The endpoint splits the compressed data into fragments and builds frames for each fragment. For example, if the fragments are 3 and 4 octet, the first frame is: 0x41 0x03 0xf2 0x48 0xcd and the second frame is: 0x80 0x04 0xc9 0xc9 0x07 0x00 Note that the RSV1 bit is set only on the first frame. Yoshino Expires October 17, 2015 [Page 25] Internet-Draft Compression Extensions for WebSocket April 2015 8.2.3.2. Sharing LZ77 Sliding Window Suppose that a client has sent a message "Hello" as a compressed message and will send the same message "Hello" again as a compressed message. 0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00 The above is the payload of the first message that the client has sent. If the "agreed parameters" contain the "client_no_context_takeover" extension parameter, the client compresses the payload of the next message into the same bytes (if the client uses the same "BTYPE" value and "BFINAL" value). So, the payload of the second message will be: 0xf2 0x48 0xcd 0xc9 0xc9 0x07 0x00 If the "agreed parameters" did not contain the "client_no_context_takeover" extension parameter, the client can compress the payload of the next message into less bytes by referencing the history in the LZ77 sliding window. So, the payload of the second message will be: 0xf2 0x00 0x11 0x00 0x00 So, 2 bytes are saved in total. Note that even if some uncompressed messages (with the RSV1 bit unset) are inserted between the two "Hello" messages, they don't affect the LZ77 sliding window. 8.2.3.3. Using a DEFLATE Block with No Compression 0xc1 0x0b 0x00 0x05 0x00 0xfa 0xff 0x48 0x65 0x6c 0x6c 0x6f 0x00 This is a frame constituting a text message "Hello" built using a DEFLATE block with no compression. The first 2 octets (0xc1 0x0b) are the WebSocket frame header (FIN=1, RSV1=1, RSV2=0, RSV3=0, opcode=text, MASK=0, Payload length=7). Note that the RSV1 bit is set for this message (only on the first fragment if the message is fragmented) because the RSV1 bit is set when DEFLATE is applied to the message, including the case when only DEFLATE blocks with no compression are used. The 3rd to 13th octets consist the payload data containing "Hello" compressed using a DEFLATE block with no compression. Yoshino Expires October 17, 2015 [Page 26] Internet-Draft Compression Extensions for WebSocket April 2015 8.2.3.4. Using a DEFLATE Block with BFINAL Set to 1 On platforms on which the flush method using an empty DEFLATE block with no compression is not available, implementors can choose to flush data using DEFLATE blocks with "BFINAL" set to 1. 0xf3 0x48 0xcd 0xc9 0xc9 0x07 0x00 0x00 This is the payload of a message containing "Hello" compressed using a DEFLATE block with "BFINAL" set to 1. The first 7 octets constitute a DEFLATE block with "BFINAL" set to 1 and "BTYPE" set to 01 containing "Hello". The last 1 octet (0x00) contains the header bits with "BFINAL" set to 0 and "BTYPE" set to 00, and 5 padding bits of 0. This octet is necessary to allow the payload to be decompressed in the same manner as messages flushed using DEFLATE blocks with BFINAL unset. 8.2.3.5. Two DEFLATE Blocks in 1 Message Two or more DEFLATE blocks may be used in 1 message. 0xf2 0x48 0x05 0x00 0x00 0x00 0xff 0xff 0xca 0xc9 0xc9 0x07 0x00 The first 3 octets (0xf2 0x48 0x05) and the least significant two bits of the 4th octet (0x00) constitute one DEFLATE block with "BFINAL" set to 0 and "BTYPE" set to 01 containing "He". The rest of the 4th octet contains the header bits with "BFINAL" set to 0 and "BTYPE" set to 00, and the 3 padding bits of 0. Together with the following 4 octets (0x00 0x00 0xff 0xff), the header bits constitute an empty DEFLATE block with no compression. A DEFLATE block containing "llo" follows the empty DEFLATE block. 8.2.3.6. Generating an Empty Fragment Manually Suppose that an endpoint is sending data of unknown size. The endpoint may encounter the end of data signal from the data source when its buffer for uncompressed data is empty. In such a case, the endpoint just needs to send the last fragment with FIN bit set to 1 and payload set to DEFLATE block(s) which contains 0 bytes of data. If the compression library being used doesn't generate any data when its buffer is empty, an empty uncompressed DEFLATE block can be built manually and used for this purpose as follows: 0x00 The only octet 0x00 contains the header bits with "BFINAL" set to 0 and "BTYPE" set to 00, and 5 padding bits of 0. Yoshino Expires October 17, 2015 [Page 27] Internet-Draft Compression Extensions for WebSocket April 2015 8.3. Implementation Notes On most common software development platforms, the DEFLATE compression library provides a method for aligning compressed data to byte boundaries using an empty DEFLATE block with no compression. For example, Zlib [Zlib] does this when "Z_SYNC_FLUSH" is passed to the deflate function. Some platforms may provide only methods to output and process compressed data with a ZLIB header and an Adler-32 checksum. On such platforms, developers need to write stub code to remove and complement them manually. To obtain a useful compression ratio, an LZ77 sliding window size of 1,024 or more is RECOMMENDED. If a side disallows context takeover, its endpoint can easily figure out whether a certain message will be shorter if compressed or not. Otherwise, it's not easy to know whether future messages will benefit from having a certain message compressed. Implementors may employ some heuristics to determine this. 8.4. Intermediaries When an intermediary forwards a message, the intermediary MAY change the compression of messages provided that the resulting sequence of messages conforms to the constraints based on the "agreed parameters". For example, an intermediary may decompress a received message, unset the "Per-message Compressed" bit and forward it to the other peer. Since such a compression change may affect the LZ77 sliding window, the intermediary may need to parse and transform the following messages, too. Yoshino Expires October 17, 2015 [Page 28] Internet-Draft Compression Extensions for WebSocket April 2015 9. Security Considerations There is a known exploit when history-based compression is combined with a secure transport [CRIME]. Implementors should pay attention to this point when integrating this extension with other extensions or protocols. Yoshino Expires October 17, 2015 [Page 29] Internet-Draft Compression Extensions for WebSocket April 2015 10. IANA Considerations 10.1. Registration of the "permessage-deflate" WebSocket Extension Name This section describes a WebSocket extension name registration in the WebSocket Extension Name Registry [RFC6455]. Extension Identifier permessage-deflate Extension Common Name WebSocket Per-message Deflate Extension Definition This document. Known Incompatible Extensions None The "permessage-deflate" extension name is used in the "Sec-WebSocket-Extensions" header in the WebSocket opening handshake to negotiate use of the permessage-deflate extension. 10.2. Registration of the "Per-message Compressed" WebSocket Framing Header Bit This section describes a WebSocket framing header bit registration in the WebSocket Framing Header Bits Registry [RFC6455]. Header Bit RSV1 Common Name Per-message Compressed Meaning The message is compressed or not. Reference Section 6 of this document. The "Per-message Compressed" framing header bit is used on the first fragment of data messages to indicate whether the payload of the message is compressed by the PMCE or not. Yoshino Expires October 17, 2015 [Page 30] Internet-Draft Compression Extensions for WebSocket April 2015 11. Acknowledgements Special thanks to Patrick McManus who wrote up the initial specification of a DEFLATE-based compression extension for the WebSocket Protocol to which I referred to write this specification. Thank you to the following people who participated in discussions on the HyBi WG and contributed ideas and/or provided detailed reviews (the list is likely to be incomplete): Adam Rice, Alexander Philippou, Alexey Melnikov, Arman Djusupov, Bjoern Hoehrmann, Brian McKelvey, Dario Crivelli, Greg Wilkins, Inaki Baz Castillo, Jamie Lokier, Joakim Erdfelt, John A. Tamplin, Julian Reschke, Kenichi Ishibashi, Mark Nottingham, Peter Thorson, Roberto Peon, Salvatore Loreto, Simone Bordet, Tobias Oberstein and Yutaka Hirano. Note that people listed above didn't necessarily endorse the end result of this work. Yoshino Expires October 17, 2015 [Page 31] Internet-Draft Compression Extensions for WebSocket April 2015 12. References 12.1. Normative References [RFC1951] Deutsch, P., "DEFLATE Compressed Data Format Specification version 1.3", RFC 1951, May 1996. [RFC5234] Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, January 2008. [RFC6455] Fette, I. and A. Melnikov, "The WebSocket Protocol", RFC 6455, December 2011. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. [LZ77] Ziv, J. and A. Lempel, "A Universal Algorithm for Sequential Data Compression", IEEE Transactions on Information Theory, Vol. 23, No. 3, pp. 337-343. 12.2. Informative References [RFC1979] Woods, J., "PPP Deflate Protocol", RFC 1979, August 1996. [Zlib] Gailly, J. and M. Adler, "Zlib", . [CRIME] Rizzo, J. and T. Duong, "The CRIME attack", Ekoparty 2012, September 2012. Yoshino Expires October 17, 2015 [Page 32] Internet-Draft Compression Extensions for WebSocket April 2015 Author's Address Takeshi Yoshino Google, Inc. Email: tyoshino@google.com Yoshino Expires October 17, 2015 [Page 33]