Network Working Group A. Rousskov Internet-Draft The Measurement Factory Updates: 3507 (if approved) October 10, 2016 Intended status: Informational Expires: April 13, 2017 ICAP Trailers draft-rousskov-icap-trailers-01 Abstract This document defines an ICAP trailer feature which allows ICAP agents to reliably send message metadata after the message body. The ICAP trailer is independent from the HTTP trailer that might also be encapsulated in an ICAP message. ICAP changes defined here are backward compatible and address a long-standing ICAP specification errata entry. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on April 13, 2017. Copyright Notice Copyright (c) 2016 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of Rousskov Expires April 13, 2017 [Page 1] Internet-Draft ICAP Trailers October 2016 the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1. Motivation and Design Choices . . . . . . . . . . . . . . . . 2 2. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3. Overall Operation . . . . . . . . . . . . . . . . . . . . . . 4 4. Notations . . . . . . . . . . . . . . . . . . . . . . . . . . 5 5. Extended Use of the Allow Header Field . . . . . . . . . . . 5 6. Message Syntax . . . . . . . . . . . . . . . . . . . . . . . 6 7. Trailer Field Syntax . . . . . . . . . . . . . . . . . . . . 6 8. Client Requirements . . . . . . . . . . . . . . . . . . . . . 7 9. Server Requirements . . . . . . . . . . . . . . . . . . . . . 8 10. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 9 11. Security Considerations . . . . . . . . . . . . . . . . . . . 11 12. Normative References . . . . . . . . . . . . . . . . . . . . 12 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 12 1. Motivation and Design Choices ICAP [RFC3507] specification says that the Trailer header field is "defined in ICAP the same as [...] in HTTP". Unfortunately, that phrase alone is not enough for trailer-related interoperability in the ICAP context because of the following conflicting interpretations, requirements, and needs: o Both ICAP and HTTP message headers might contain a Trailer field. o HTTP messages might contain HTTP trailers (that ICAP servers could be interested in receiving or even sending). An HTTP trailer can be present with or without an HTTP Trailer header field. o ICAP agents need to distinguish an HTTP trailer from an ICAP trailer. o HTTP uses Chunked Transfer Coding [RFC7230] to transmit trailers. The chunked coding is applied to the entire HTTP message body. This choice places an HTTP trailer inside an HTTP message body. o It is possible to interpret the ICAP specification as placing the ICAP trailer either inside the HTTP message body or inside the ICAP message body. o ICAP does not chunk-encode ICAP message bodies. Instead, ICAP message bodies contain a combination of zero, one, or two HTTP headers possibly followed by a chunked-encoded HTTP message body. Rousskov Expires April 13, 2017 [Page 2] Internet-Draft ICAP Trailers October 2016 o Chunked coding does not support multiple trailers: Chunked HTTP messages always contain exactly one (but possibly empty) trailer part. o HTTP effectively restricts trailers usage to messages with bodies, presumably because, without a body, the information in the trailer can usually be placed in the message header. In ICAP context, it is not obvious whether trailers ought to be restricted to messages with HTTP bodies (embedded in ICAP bodies) or to messages with ICAP bodies (that might only contain HTTP headers and no HTTP body). These problems led to a ban on ICAP trailers [Errata]. Several designs were considered for introducing proper ICAP trailers support: 1. Extend chunked coding to support multiple trailers (one for HTTP and one for ICAP). This option was rejected because many ICAP agents use existing HTTP-focused libraries to parse embedded HTTP bodies. As anecdotal evidence related to the ICAP-only "ieof" chunk extension support shows, it would be difficult to extend those libraries to handle a complicated ICAP-only extension. Also, this design would make it difficult to send an ICAP trailer when processing large HTTP messages without bodies. 2. Embed ICAP trailer fields inside the chunked HTTP message body trailer, using an ICAP-specific field name prefix (e.g., "ICAP-"). This option was rejected because it would either allow malicious HTTP messages to inject ICAP trailers or require ICAP clients to hide conflicting HTTP trailer fields from the ICAP server. This design also badly violates layering boundaries by mixing HTTP- and ICAP-level information in the same protocol structure. 3. Extend Encapsulated header with a "trailer" token. This option was rejected because the Encapsulated header describes embedded HTTP message parts and an ICAP trailer is not a part of any HTTP message. In other words, ICAP trailers do not get encapsulated. 4. Clarify the ICAP Trailer semantics (and transfer mechanism) without introducing any new trailer support-negotiation mechanism. This option was rejected because trailers affect message framing and many existing ICAP agent implementations cannot parse any form of trailers. 5. Add a new trailer support-negotiation mechanism (e.g., "Allow: trailers") and a new trailer presence-signaling mechanism (e.g., Rousskov Expires April 13, 2017 [Page 3] Internet-Draft ICAP Trailers October 2016 Trailer2) while leaving the poorly defined Trailer header semantics as is. This option was narrowly rejected because a new trailer support-negotiation mechanism alone was deemed sufficient to resolve conflicts between this specification and any reasonable existing implementation of the poorly defined Trailer semantics. 6. Add a new trailer support-negotiation mechanism and only clarify Trailer header semantics (and transfer mechanism) upon successful negotiation, while reusing the well-known Trailer header field name as the trailer presence-signaling mechanism (requiring successful support negotiation). This specification documents this design. 2. Use Cases Trailers allow an ICAP agent to transmit metadata after the message body. Such delayed transmission is useful when the same information was not available at the start of the message transmission. For example: o A client uses an ICAP trailer to relay the current HTTP/1.1 connection or HTTP/2 stream status after transmitting a large HTTP message. A server uses that information to optimize message analysis (e.g., skip or abort analysis of HTTP requests sent by already disconnected HTTP clients). o A server uses an ICAP trailer to relay audit information about viruses present at the end of a large HTTP message. o A server uses an ICAP trailer to relay prefetching information about HTML parts referenced from a large HTTP message. 3. Overall Operation This section informally describes the overall feature operation. This description is deliberately imprecise and cannot be used to build compliant implementations. The following sections contain actual protocol requirements. To announce feature support, ICAP agents exchange "Allow: trailers" settings during an OPTIONS transaction. To send a trailer at the end of a particular REQMOD or RESPMOD transaction, the agent first sends both "Allow: trailers" and "Trailer" header fields. The Trailer field lists header field names expected in the message trailer section. After sending the entire ICAP message body, the agent sends the trailer section (a.k.a. "trailer"). The trailer section is syntactically equivalent to the ICAP message header section. The Rousskov Expires April 13, 2017 [Page 4] Internet-Draft ICAP Trailers October 2016 trailer section does not have to contain any of the promised fields and might even have no fields at all. A trailer makes sense only in an ICAP message with a body. However, a trailer could be sent if the ICAP message body encapsulates just HTTP headers. If a trailer is sent, its bytes are always the last bytes sent during the entire ICAP transaction. Thus, a client never sends a trailer at the end of Preview unless it sent the "ieof" chunk extension as well. Similarly, a server never a sends trailer with a 100 (Continue) control message. 4. Notations The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119]. Conformance criteria and error handling considerations are defined in Section 2.5 of [RFC7230]. This specification uses the Augmented Backus-Naur Form (ABNF) notation of [RFC5234] with a list extension defined in Section 7 of [RFC7230]. All syntax rules not explicitly defined in this specification (e.g., header-field and CRLF) are defined in (or included by reference from) [RFC7230]. The "Allow/X" notation is defined in Section 5. 5. Extended Use of the Allow Header Field The use of the Allow header field defined in sections 4.6 and 4.10.2 of [RFC3507] is extended while preserving its original syntax and general semantics (i.e., a comma-separated list of tokens, each identifying an ICAP feature supported by the sending agent): Allow = 1#token [RFC3507] defines Allow usage in OPTIONS requests and responses. This specification extends Allow usage to other messages and adds "trailers" to the list of possible Allow value tokens. An agent MUST treat multiple Allow header fields as one Allow field with a comma-separated list of individual field value tokens, concatenated in the order of their appearance in the ICAP header. An ICAP agent MUST ignore Allow header tokens it does not understand. This document does not specify the significance of Allow tokens order and impact of repeated tokens. Rousskov Expires April 13, 2017 [Page 5] Internet-Draft ICAP Trailers October 2016 An agent MAY send an Allow header in any message. Such header contains a list of ICAP features supported by the sending agent. For example, a client could send an "Allow: trailers" in a REQMOD request and receive an "Allow: 204, trailers, 206" header field in response. The exact meaning of each Allow token in a context of its message is defined by the corresponding feature specification. By sending Allow, the agent indicates compliance with each listed feature's specification but does not necessarily commit to offer or use any of the listed features during future transactions. To keep this specification succinct, we introduce "Allow/X" notation to mean an ICAP Allow header field which value contains an "X" token, possibly among other tokens. For example, "Allow/trailers" stands for an Allow header field with a "trailers" token and possibly other tokens. This notation covers multi-field Allow headers as well because they are equivalent to a combined single-field Allow header. The same approach to extending Allow header usage was successfully applied to the ICAP 206 extension (XXX: reference our expired ICAP icap-ext-partial-content draft). This specification is compatible with the ICAP 206 extension. 6. Message Syntax An ICAP message with a trailer is a concatenation of a regular ICAP message and an trailer section. The trailer section syntax is identical to the ICAP header syntax: ICAP-message-without-trailer = ICAP-message-with-trailer = ICAP-message-without-trailer trailer-section trailer-section = *( trailer-field CRLF ) CRLF trailer-field = header-field Note that any trailer, even a trailer without fields, ends with CRLF. That terminating sequence is essential for proper message framing on persistent ICAP connections. A sender MUST NOT generate a trailer section that contains a field necessary for message framing (e.g., Encapsulated, Preview, and Trailer), routing (e.g., Host), or authentication. 7. Trailer Field Syntax The ICAP Trailer header value syntax is identical to the HTTP Trailer syntax [RFC7230]: Rousskov Expires April 13, 2017 [Page 6] Internet-Draft ICAP Trailers October 2016 Trailer = 1#field-name A Trailer header field sender SHOULD enumerate the names of all expected trailer fields. This a priori knowledge of trailer fields might help the recipient with trailer processing (e.g., certain message annotation actions could be delayed in anticipation of the trailer section). However, enumerating all expected trailer fields can be impractical or even impossible in some environments. A Trailer sender MAY send a trailer section with a set of field names that differs from the set of field names listed in the Trailer header field. This specification does not place any restriction on the order of field names in the Trailer header field. Senders SHOULD NOT generate duplicate names for the Trailer header field. 8. Client Requirements A client compliant with this specification SHOULD send Allow/trailers in each OPTIONS request. A non-authenticating server cannot be expected to mark an Allow/trailers-sending client specially, but this support announcement requirement is meant to minimize interoperability problems associated with servers sending Allow/ trailers in OPTIONS responses. Some servers that do not support trailers might not be able to ignore Allow/trailers in OPTIONS requests; therefore, a client SHOULD offer a configuration option or other means of disabling sending Allow/trailers in OPTIONS requests. An ICAP service sending Allow/trailers in OPTIONS response is called a trailers-supporting service. That service designation, maintained by the client, starts upon receiving the OPTIONS service response carrying Allow/trailers and lasts until OPTIONS expiration or a new OPTIONS response from that service. A client compliant with this specification SHOULD send Allow/trailers in each request to a trailers-supporting service. Doing so allows the service to respond with a trailer (and is also necessary for sending a client trailer, as detailed further below). A client receiving both a Trailer header field and Allow/trailers in the response MUST expect a trailer section in that response. In all other cases, a client MUST use the usual trailer-free ICAP response syntax. A client receiving a Trailer header field without Allow/ trailers in a response MAY treat the response as syntactically malformed and, regardless of this response treatment, MUST NOT reuse the connection for any other messages (including pending pipelined requests, if any). Rousskov Expires April 13, 2017 [Page 7] Internet-Draft ICAP Trailers October 2016 A client MAY send a trailer in any request that satisfies all of these conditions: 1. the request is sent to a trailers-supporting service; 2. the request has a body. A client MUST NOT send a trailer in any other request. To send a trailer, the client MUST send Allow/trailers and a Trailer header field in the same request. A client MUST NOT send a combination of those two header fields without sending a trailer. 9. Server Requirements A server compliant with this specification SHOULD send Allow/trailers in each successful response to an OPTIONS request carrying Allow/ trailers. Although ICAP/1.0 [RFC3507] allows a list of features in the Allow header, some ICAP clients might not be able to handle an Allow header other than "Allow: 204"; therefore, a server SHOULD NOT send Allow/trailers in a response to a request without Allow/ trailers. A server compliant with this specification MAY send Allow/trailers in a response without a trailer to a request with Allow/trailers. The client receiving a no-trailer REQMOD or RESPMOD response with Allow/ trailers ought to ignore Allow/trailers. Nevertheless, the server is allowed to respond with Allow/trailers in this context because doing so might simplify server implementation and configuration. A server receiving both a Trailer header field and Allow/trailers in the request MUST expect a trailer in that request. In all other cases, a server MUST use the usual trailer-free ICAP request syntax. A server receiving a Trailer header field without Allow/trailers in a request MAY treat this request as syntactically malformed and, regardless of the request treatment, MUST NOT reuse the connection for any future requests. A previously received pipelined request is not a "future request", even if the server has not finished responding to it yet. A server MAY send a trailer in any response that satisfies all of these conditions: 1. the response is for a request containing Allow/trailers; 2. the response has a body. A server MUST NOT send a trailer in any other response. Rousskov Expires April 13, 2017 [Page 8] Internet-Draft ICAP Trailers October 2016 To send a trailer, the server MUST send Allow/trailers and a Trailer header field in the same response. A server MUST NOT send a combination of those two header fields without sending a trailer. 10. Examples The following examples illustrate trailer exchanges between ICAP agents compliant with this specification. To clarify message and message part boundaries, all CRLF sequences after major message parts are shown as "\r\n" lines, while boundaries between ICAP requests and responses are signified by empty lines. CRLF sequences at the end of other lines are implied. Unimportant low-level details such as irrelevant HTTP and ICAP header fields or Encapsulated offsets are shown as "...". Figure 1 shows an OPTIONS request and response when both the client and the server are compliant with this specification. The client also supports the ICAP 206 extension, but the server does not. OPTIONS icap://example.net/sample-service ICAP/1.0 ... Allow: 204, trailers, 206 \r\n ICAP/1.0 200 OK ... Allow: 204 Allow: trailers \r\n Figure 1: OPTIONS handshake The OPTIONS response in Figure 1 contains two Allow headers to illustrate one of several possible implementations. A compliant server can also send a single Allow header with a list of values, just like the client does in the above example. Figure 2 is a RESPMOD request with an ICAP trailer that the client can send after receiving an OPTIONS response with Allow/trailers shown in Figure 1. Note that the Trailer field value ("TBD") does not match the actual field names in the trailer. Rousskov Expires April 13, 2017 [Page 9] Internet-Draft ICAP Trailers October 2016 RESPMOD icap://example.net/sample-service ICAP/1.0 ... Allow: 204, trailers Trailer: TBD Encapsulated: req-hdr=0, res-hdr=..., res-body=... \r\n GET /origin-resource HTTP/1.1 ... \r\n HTTP/1.1 200 OK ... Content-Length: 24 \r\n 18 Origin server sent this. 0 \r\n X-Client-Log-Lineno: 15612570 X-Client-Status: disconnected (at 1470262108) \r\n Figure 2: Request Trailer Figure 3 is a RESPMOD response with both HTTP and ICAP trailers present. The ICAP trailer contains X-Threat-Found and Connection fields. The ICAP trailer Connection field overwrites the Connection field in the ICAP header. To respond with this ICAP trailer, the server ought to receive Allow/trailers in the corresponding ICAP request (e.g., like the request shown in in Figure 2). Rousskov Expires April 13, 2017 [Page 10] Internet-Draft ICAP Trailers October 2016 ICAP/1.0 200 OK ... Connection: keep-alive Encapsulated: res-hdr=0, res-body=... Allow: trailers Trailer: X-ICAP-Log-Lineno, X-Threat-Found \r\n HTTP/1.1 200 OK ... Content-Length: 22 Trailer: X-Content-Checksum \r\n 16 Origin server sent \0. 0 X-Content-Checksum: sha1-short=183caa016 \r\n X-Threat-Found: Type=0; Resolution=1; Threat=backslash Connection: close \r\n Figure 3: Two Trailers TODO: Figure out how to keep figure descriptions and figures on the same page without inserting empty space into HTML rendering of the draft. 11. Security Considerations Proper trailer support reduces old ICAP [RFC3507] security concerns because implementations unaware of trailer complexities are arguably more likely to misbehave when receiving HTTP or ICAP trailers. Sending request headers with multiple Allow tokens could crash poor- quality ICAP servers unaware of this specification. Trailer support negotiation rules partially mitigate that risk by restricting unaware implementations exposure; such implementations are exposed only during OPTIONS exchanges. Since OPTIONS transaction has to precede any HTTP message processing, and since virtually all ICAP client- server relationships are stable, most poor-quality implementations would be detected early and reliably. Rousskov Expires April 13, 2017 [Page 11] Internet-Draft ICAP Trailers October 2016 A naive implementation of trailer interpretation logic might update an already "frozen" or "committed" (at header parsing time) state of the ICAP transaction or connection, resulting in crashes and other problems. For example, such an implementation could panic after discovering a Connection trailer field value that contradicts the Connection header field value that has already been received and processed at the beginning of the same transaction. 12. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ RFC2119, March 1997, . [RFC3507] Elson, J. and A. Cerpa, "Internet Content Adaptation Protocol (ICAP)", RFC 3507, DOI 10.17487/RFC3507, April 2003, . [RFC5234] Crocker, D., Ed. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", STD 68, RFC 5234, DOI 10.17487/ RFC5234, January 2008, . [RFC7230] Fielding, R., Ed. and J. Reschke, Ed., "Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing", RFC 7230, DOI 10.17487/RFC7230, June 2014, . [Errata] Various Authors, , "RFC 3507 Errata", . Author's Address Alex Rousskov The Measurement Factory Email: rousskov@measurement-factory.com URI: http://www.measurement-factory.com/ Rousskov Expires April 13, 2017 [Page 12]