Internet Engineering Task Force G. Bertrand, Ed. Internet-Draft I. Oprescu, Ed. Intended status: Informational France Telecom - Orange Expires: November 28, 2013 F. Le Faucheur, Ed. Cisco Systems R. Peterkofsky Skytide, Inc. May 27, 2013 CDNI Logging Interface draft-ietf-cdni-logging-02 Abstract This memo specifies the Logging interface between a downstream CDN (dCDN) and an upstream CDN (uCDN) that are interconnected as per the CDN Interconnection (CDNI) framework. First, it describes a reference model for CDNI logging. Then, it specifies the CDNI Logging File format and the actual protocol for exchange of CDNI Logging Files. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on November 28, 2013. Copyright Notice Copyright (c) 2013 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents Bertrand, et al. Expires November 28, 2013 [Page 1] Internet-Draft CDNI Logging May 2013 carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 2. CDNI Logging Reference Model . . . . . . . . . . . . . . . . 5 2.1. CDNI Logging interactions . . . . . . . . . . . . . . . . 5 2.2. Overall Logging Chain . . . . . . . . . . . . . . . . . . 8 2.2.1. Logging Generation and During-Generation Aggregation 9 2.2.2. Logging Collection . . . . . . . . . . . . . . . . . 10 2.2.3. Logging Filtering . . . . . . . . . . . . . . . . . . 10 2.2.4. Logging Rectification and Post-Generation Aggregation 11 2.2.5. Log-Consuming Applications . . . . . . . . . . . . . 12 2.2.5.1. Maintenance/Debugging . . . . . . . . . . . . . . 12 2.2.5.2. Accounting . . . . . . . . . . . . . . . . . . . 12 2.2.5.3. Analytics and Reporting . . . . . . . . . . . . . 13 2.2.5.4. Security . . . . . . . . . . . . . . . . . . . . 13 2.2.5.5. Legal Logging Duties . . . . . . . . . . . . . . 13 2.2.5.6. Notions common to multiple Log Consuming Applications . . . . . . . . . . . . . . . . . . 13 3. CDNI Logging File Format . . . . . . . . . . . . . . . . . . 15 3.1. CDNI Logging File Directives . . . . . . . . . . . . . . 16 3.2. Logging Records . . . . . . . . . . . . . . . . . . . . . 19 3.2.1. HTTP Request Logging Record . . . . . . . . . . . . . 20 3.2.2. CDNI Logging File Example . . . . . . . . . . . . . . 26 3.3. Fields and Directives Formats . . . . . . . . . . . . . . 27 4. CDNI Logging File Exchange Protocol . . . . . . . . . . . . . 27 4.1. CDNI Logging Feed . . . . . . . . . . . . . . . . . . . . 28 4.2. CDNI Logging File Pull . . . . . . . . . . . . . . . . . 28 5. Open Issues . . . . . . . . . . . . . . . . . . . . . . . . . 29 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 31 Bertrand, et al. Expires November 28, 2013 [Page 2] Internet-Draft CDNI Logging May 2013 7. Security Considerations . . . . . . . . . . . . . . . . . . . 31 7.1. Authentication, Confidentiality, Integrity Protection . . 31 7.2. Non Repudiation . . . . . . . . . . . . . . . . . . . . . 32 7.3. Privacy . . . . . . . . . . . . . . . . . . . . . . . . . 32 8. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 32 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 33 9.1. Normative References . . . . . . . . . . . . . . . . . . 33 9.2. Informative References . . . . . . . . . . . . . . . . . 33 Appendix A. Requirements . . . . . . . . . . . . . . . . . . . . 34 A.1. Compliance with cdni-requirements . . . . . . . . . . . . 34 A.2. Additional Requirements . . . . . . . . . . . . . . . . . 34 A.2.1. Timeliness . . . . . . . . . . . . . . . . . . . . . 34 A.2.2. Reliability . . . . . . . . . . . . . . . . . . . . . 35 A.2.3. Security . . . . . . . . . . . . . . . . . . . . . . 35 A.2.4. Scalability . . . . . . . . . . . . . . . . . . . . . 35 A.2.5. Consistency between CDNI Logging and CDN Logging . . 35 A.2.6. Dispatching/Filtering . . . . . . . . . . . . . . . . 35 Appendix B. Analysis of candidate protocols for Logging Transport . . . . . . . . . . . . . . . . . . . . . 36 B.1. Syslog . . . . . . . . . . . . . . . . . . . . . . . . . 36 B.2. XMPP . . . . . . . . . . . . . . . . . . . . . . . . . . 36 B.3. SNMP . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 36 1. Introduction This memo specifies the Logging interface between a downstream CDN (dCDN) and an upstream CDN (uCDN). First, it describes a reference model for CDNI logging. Then, it specifies the CDNI Logging File format and the actual protocol for exchange of CDNI Logging Files. The reader should be familiar with the following documents: o CDNI problem statement [RFC6707] and framework [I-D.ietf-cdni-framework] identify a Logging interface, o Section 8 of [I-D.ietf-cdni-requirements] specifies a set of requirements for Logging, o [RFC6770] outlines real world use-cases for interconnecting CDNs. These use cases require the exchange of Logging information between the dCDN and the uCDN. As stated in [RFC6707], "the CDNI Logging interface enables details of logs or events to be exchanged between interconnected CDNs". The present document describes: Bertrand, et al. Expires November 28, 2013 [Page 3] Internet-Draft CDNI Logging May 2013 o The CDNI Logging reference model (Section 2), o The CDNI Logging File format (Section 3), o The CDNI Logging File Exchange protocol (Section 4). 1.1. Terminology In this document, the first letter of each CDNI-specific term is capitalized. We adopt the terminology described in [RFC6707] and [I-D.ietf-cdni-framework], and extend it with the additional terms defined below. For clarity, we use the word "Log" only for referring to internal CDN logs and we use the word "Logging" for any inter-CDN information exchange and processing operations related to CDNI Logging interface. Log and Logging formats may be different. CDN Logging information: logging information generated and collected within a CDN CDNI Logging information: logging information exchanged across CDNs using the CDNI Logging Interface Logging information: logging information generated and collected within a CDN or obtained from another CDN using the CDNI Logging Interface CDNI Logging Field: an atomic element of information that can be included in a CDNI Logging Record. The time an event/task started, the IP address of an End user to whom content was delivered, and the URI of the content delivered are examples of CDNI Logging Fields. CDNI Logging Record: an information record providing information about a specific event. This comprises a collection of CDNI Logging Fields. CDNI Logging File: a file containing CDNI Logging Records, as well as additional information facilitating the processing of the CDNI Logging Records. CDN Reporting: the process of providing the relevant information that will be used to create a formatted content delivery report provided to the CSP in deferred time. Such information typically includes aggregated data that can cover a large period of time (e.g., from hours to several months). Uses of Reporting include the collection of charging data related to CDN services and the computation of Key Performance Indicators (KPIs). Bertrand, et al. Expires November 28, 2013 [Page 4] Internet-Draft CDNI Logging May 2013 CDN Monitoring: the process of providing content delivery information in real-time. Monitoring typically includes data in real time to provide visibility of the deliveries in progress, for service operation purposes. It presents a view of the global health of the services as well as information on usage and performance, for network services supervision and operation management. In particular, monitoring data can be used to generate alarms. 2. CDNI Logging Reference Model 2.1. CDNI Logging interactions The CDNI logging reference model between a given uCDN and a given dCDN involves the following interactions: o customization by the uCDN of the CDNI logging information to be provided by the dCDN to the uCDN (e.g. control of which logging fields are to be communicated to the uCDN for a given task performed by the dCDN, control of which types of events are to be logged). The dCDN takes into account this CDNI logging customization information to determine what logging information to provide to the uCDN, but it may, or may not, take into account this CDNI logging customization information to influence what CDN logging information is to be generated and collected within the dCDN (e.g. even if the uCDN requests a restricted subset of the logging information, the dCDN may elect to generate a broader set of logging information). The mechanism to support the customisation by the uCDN of CDNI Logging information is outside the scope of this document and left for further study. We note that the CDNI Control interface or the CDNI Metadata interface appear as candidate interfaces on which to potentially build such a customisation mechanism in the future. Before such a mechanism is available, the uCDN and dCDN are expected to agree off-line on what CDNI logging information is to be provide by dCDN to UCDN and rely on management plane actions to configure the CDNI Logging functions to generate (respectively, expect) in dCDN (respectively, in uCDN). o generation and collection by the dCDN of logging information related to the completion of any task performed by the dCDN on behalf of the uCDN (e.g., delivery of the content to an end user) or related to events happening in the dCDN that are relevant to the uCDN (e.g., failures or unavailability in dCDN). This takes place within the dCDN and does not directly involve CDNI interfaces. o communication by the dCDN to the uCDN of the logging information collected by the dCDN relevant to the uCDN. This is supported by Bertrand, et al. Expires November 28, 2013 [Page 5] Internet-Draft CDNI Logging May 2013 the CDNI Logging interface and in the scope of the present document. For example, the uCDN may use this logging information to charge the CSP, to perform analytics and monitoring for operational reasons, to provide analytics and monitoring views on its content delivery to the CSP or to perform trouble-shooting. o customization by the dCDN of the logging to be performed by the uCDN on behalf of the dCDN. The mechanism to support the customisation by the dCDN of CDNI Logging information is outside the scope of this document and left for further study. o generation and collection by the uCDN of logging information related to the completion of any task performed by the uCDN on behalf of the dCDN (e.g., serving of content by uCDN to dCDN for acquisition purposes by dCDN) or related to events happening in the uCDN that are relevant to the dCDN. This takes place within the uCDN and does not directly involve CDNI interfaces. o communication by the uCDN to the dCDN of the logging information collected by the uCDN relevant to the dCDN. For example, the dCDN might potentially benefit form this information for security auditing or content acquisition troubleshooting. This is outside the scope of this document and left for further study. Figure 1 provides an example of CDNI Logging interactions (focusing only on the interactions that are in the scope of this document) in a particular scenario where 4 CDNs are involved in the delivery of content from a given CSP: the uCDN has a CDNI interconnection with dCDN-1 and dCDN-2. In turn, dCDN2 has a CDNI interconnection with dCDN3. In this example, uCDN, dCDN-1, dCDN-2 and dCDN-3 all participate in the delivery of content for the CSP. In this example, the CDNI Logging interface enables the uCDN to obtain logging information from all the dCDNs involved in the delivery. In the example, uCDN uses the Logging data: o to analyze the performance of the delivery operated by the dCDNs and to adjust its operations (e.g., request routing) as appropriate, o to provide reporting (non real-time) and monitoring (real-time) information to CSP. For instance, uCDN merges Logging data, extracts relevant KPIs, and presents a formatted report to the CSP, in addition to a bill for the content delivered by uCDN itself or by its dCDNs on his behalf. uCDN may also provide Logging data as raw log files to the CSP, so that the CSP can use its own logging analysis tools. Bertrand, et al. Expires November 28, 2013 [Page 6] Internet-Draft CDNI Logging May 2013 +-----+ | CSP | +-----+ ^ Reporting and monitoring data * Billing ,--*--. Logging ,-' `-. Data =>( uCDN )<= Logging // `-. _,-' \\ Data || `-'-'-' || ,-----. ,-----. ,-' `-. ,-' `-. ( dCDN-1 ) ( dCDN-2 )<== Logging `-. ,-' `-. _,-' \\ Data `--'--' `--'-' || ,-----. ,' `-. ( dCDN-3 ) `. ,-' `--'--' ===> CDNI Logging Interface ***> outside the scope of CDNI Figure 1: Interactions in CDNI Logging Reference Model A dCDN (e.g., dCDN-2) integrates the relevant logging information obtained from its dCDNs (e.g., dCDN-3) in the logging information that it provides to the uCDN, so that the uCDN ultimately obtains all logging information relevant to a CSP for which it acts as the authoritative CDN. Note that the format of Logging information that a CDN provides over the CDNI interface might be different from the one that the CDN uses internally. In this case, the CDN needs to reformat the Logging information before it provides this information to the other CDN over the CDNI Logging interface. Similarly, a CDN might reformat the Logging data that it receives over the CDNI Logging interface before injecting it into its log-consuming applications or before providing some of this logging information to the CSP. Such reformatting operations introduce latency in the logging distribution chain and introduce a processing burden. Therefore, there are benefits in specifying CDNI Logging format that are suitable for use inside CDNs and also are close to the CDN Log formats commonly used in CDNs today. Bertrand, et al. Expires November 28, 2013 [Page 7] Internet-Draft CDNI Logging May 2013 2.2. Overall Logging Chain This section discusses the overall logging chain within and across CDNs to clarify how CDN Logging information is expected to fit in this overall chain. Figure 2 illustrates the overall logging chain within the dCDN, across CDNs using the CDNI Logging interface and within the uCDN. Note that the logging chain illustrated in the Figure is obviously only indicative and varies depending on the specific environments. For example, there may be more or less instantiations of each entity (i.e., there may be 4 Log consuming applications in a given CDN). As another example, there may be one instance of Rectification process per Log Consuming Application instead of a shared one. Bertrand, et al. Expires November 28, 2013 [Page 8] Internet-Draft CDNI Logging May 2013 Log Consuming Log Consuming App App /\ /\ | | Rectification-------- /\ | Filtering /\ | Collection uCDN /\ /\ | | | Generation | CDNI Logging --------------------------------------------- exchange /\ Log Consuming Log Consuming | App App | /\ /\ | | | Rectification Rectification--------- /\ /\ | | Filtering /\ | Collection dCDN /\ /\ | | Generation Generation Figure 2: CDNI Logging in the overall Logging Chain The following subsections describe each of the processes potentially involved in the logging chain of Figure 2. 2.2.1. Logging Generation and During-Generation Aggregation CDNs typically generate logging information for all significant task completions, events, and failures. Logs are typically generated by many devices in the CDN including the surrogates, the request routing system, and the control system. The amount of Logging information generated can be huge. Therefore, during contract negotiations, interconnected CDNs often agree on a Bertrand, et al. Expires November 28, 2013 [Page 9] Internet-Draft CDNI Logging May 2013 Logging retention duration, and optionally, on a maximum size of the Logging data that the dCDN must keep. If this size is exceeded, the dCDN must alert the uCDN but may not keep more Logs for the considered time period. In addition, CDNs may aggregate logs and transmit only summaries for some categories of operations instead of the full Logging data. Note that such aggregation leads to an information loss, which may be problematic for some usages of Logging (e.g., debugging). [I-D.brandenburg-cdni-has] discusses logging for HTTP Adaptive Streaming (HAS). In accordance with the recommendations articulated there, it is expected that a surrogate will generate separate logging information for delivery of each chunk of HAS content. This ensures that separate logging information can then be provided to interconnected CDNs over the CDNI Logging interface. Still in line with the recommendations of [I-D.brandenburg-cdni-has], the logging information for per-chunck delivery may include some information (a Content Collection IDentifier and a Session IDentifier) intended to facilitate subsequent post-generation aggregation of per-chunk logs into per-session logs. Note that a CDN may also elect to generate aggregate per-session logs when performing HAS delivery, but this needs to be in addition to, and not instead of, the per-chunk delivery logs. We note that this may be revisited in future versions of this document. Note that in the case of non real-time logging, the trigger of the transmission or generation of the logging file appears to be a synchronous process from a protocol standpoint. The implementation algorithm can choose to enforce a maximum size for the logging file beyond which the transmission is automatically triggered (and thus allow for an asynchronous transmission process). 2.2.2. Logging Collection This is the process that continuously collects logs generated by the log-generating entities within a CDN. In a CDNI environment, in addition to collecting logging information from log-generating entities within the local CDN, the Collection process also collects logging information provided by another CDN, or other CDNs, through the CDNI Logging interface. This is illustrated in Figure 2 where we see that the Collection process of the uCDN collects logging information from log-generating entities within the uCDN as well as logging information coming through CDNI Logging exchange with the dCDN through the CDNI Logging interface. 2.2.3. Logging Filtering Bertrand, et al. Expires November 28, 2013 [Page 10] Internet-Draft CDNI Logging May 2013 A CDN may require to only present different subset of the whole logging information collected to various log-consuming applications. This is achieved by the Filtering process. In particular, the Filtering process can also filter the right subset of information that needs to be provided to a given interconnected CDN. For example, the filtering process in the dCDN can be used to ensure that only the logging information related to tasks performed on behalf of a given uCDN are made available to that uCDN (thereby filtering all the logging information related to deliveries by the dCDN of content for its own CSPs). Similarly, the Filtering process may filter or partially mask some fields, for example, to protect End Users' privacy when communicating CDNI Logging information to another CDN. Filtering of logging information prior to communication of this information to other CDNs via the CDNI Logging interface requires that the downstream CDN can recognize the set of log records that relate to each interconnected CDN. The CDN will also filter some internal scope information such as information related to its internal alarms (security, failures, load, etc). In some use cases described in [RFC6770], the interconnected CDNs do not want to disclose details on their internal topology. The filtering process can then also filter confidential data on the dCDNs' topology (number of servers, location, etc.). In particular, information about the requests served by every Surrogate may be confidential. Therefore, the Logging information must be protected so that data such as Surrogates' hostnames is not disclosed to the uCDN. In the "Inter-Affiliates Interconnection" use case, this information may be disclosed to the uCDN because both the dCDN and the uCDN are operated by entities of the same group. 2.2.4. Logging Rectification and Post-Generation Aggregation If Logging is generated periodically, it is important that the sessions that start in one Logging period and end in another are correctly reported. If they are reported in the starting period, then the Logging of this period will be available only after the end of the session, which delays the Logging generation. A Logging rectification/update mechanism could be useful to reach a good trade-off between the Logging generation delay and the Logging accuracy. Depending on the selected Logging protocol(s), such mechanism may be invaluable for real time Logging, which must be provided rapidly and cannot wait for the end of operations in progress. Bertrand, et al. Expires November 28, 2013 [Page 11] Internet-Draft CDNI Logging May 2013 In the presence of HAS, some log-consuming applications can benefit from aggregate per-session logs. For example, for analytics, per- session logs allow display of session-related trends which are much more meaningful for some types of analysis than chunk-related trends. In the case where the log-generating entities have generated during- generation aggregate logs, those can be used by the applications. In the case where aggregate logs have not been generated, the Rectification process can be extended with a Post-Generation Aggregation process that generates per-session logs from the per- chunk logs, possibly leveraging the information included in the per- chunk logs for that purpose (Content Collection IDentifier and a Session IDentifier). However, in accordance with [I-D.brandenburg-cdni-has], this document does not define exchange of such aggregate logs on the CDNI Logging interface. We note that this may be revisited in future versions of this document. 2.2.5. Log-Consuming Applications 2.2.5.1. Maintenance/Debugging Logging is useful to permit the detection (and limit the risk) of content delivery failures. In particular, Logging facilitates the resolution of configuration issues. To detect faults, Logging must enable the reporting of any CDN operation success and failure, such as request redirection, content acquisition, etc. The uCDN can summarize such information into KPIs. For instance, Logging format should allow the computation of the number of times during a given epoch that content delivery related to a specific service succeeds/fails. Logging enables the CDN providers to identify and troubleshoot performance degradations. In particular, Logging enables the communication of traffic data (e.g., the amount of traffic that has been forwarded by a dCDN on behalf of an uCDN over a given period of time), which is particularly useful for CDN and network planning operations. 2.2.5.2. Accounting Logging is essential for accounting, to permit inter-CDN billing and CSP billing by uCDNs. For instance, Logging information provided by dCDNs enables the uCDN to compute the total amount of traffic delivered by every dCDN for a particular Content Provider, as well as, the associated bandwidth usage (e.g., peak, 95th percentile), and the maximum number of simultaneous sessions over a given period of time. Bertrand, et al. Expires November 28, 2013 [Page 12] Internet-Draft CDNI Logging May 2013 2.2.5.3. Analytics and Reporting The goal of analytics is to gather any relevant information to track audience, analyze user behavior, and monitor the performance and quality of content delivery. For instance, Logging enables the CDN providers to report on content consumption (e.g., delivered sessions per content) in a specific geographic area. The goal of reporting is to gather any relevant information to monitor the performance and quality of content delivery and allow detection of delivery issues. For instance, reporting could track the average delivery throughput experienced by End-Users in a given region for a specific CSP or content set over a period of time. 2.2.5.4. Security The goal of security is to prevent and monitor unauthorized access, misuse, modification, and denial of access of a service. A set of information is logged for security purposes. In particular, a record of access to content is usually collected to permit the CSP to detect infringements of content delivery policies and other abnormal End User behaviors. 2.2.5.5. Legal Logging Duties Depending on the country considered, the CDNs may have to retain specific Logging information during a legal retention period, to comply with judicial requisitions. 2.2.5.6. Notions common to multiple Log Consuming Applications 2.2.5.6.1. Logging Information Views Within a given log-consuming application, different views may be provided to different users depending on privacy, business, and scalability constraints. For example, an analytics tool run by the uCDN can provide one view to an uCDN operator that exploits all the logging information available to the uCDN, while the tool may provide a different view to each CSP exploiting only the logging information related to the content of the given CSP. As another example, maintenance and debugging tools may provide different views to different CDN operators, based on their operational role. Bertrand, et al. Expires November 28, 2013 [Page 13] Internet-Draft CDNI Logging May 2013 2.2.5.6.2. Key Performance Indicators (KPIs) This section presents, for explanatory purposes, a non-exhaustive list of Key Performance Indicators (KPIs) that can be extracted/ produced from logs. Multiple log-consuming applications, such as analytics, monitoring, and maintenance applications, often compute and track such KPIs. In a CDNI environment, depending on the situation, these KPIs may be computed by the uCDN or by the dCDN. But it is usually the uCDN that computes KPIs, because uCDN and dCDN may have different definitions of the KPIs and the computation of some KPIs requires a vision of all the deliveries performed by the uCDN and all its dCDNs. Here is a list of important examples of KPIs: o Number of delivery requests received from End-Users in a given region for each piece of content, during a given period of time (e.g., hour/day/week/month) o Percentage of delivery successes/failures among the aforementioned requests o Number of failures listed by failure type (e.g., HTTP error code) for requests received from End Users in a given region and for each piece of content, during a given period of time (e.g., hour/ day/week/month) o Number and cause of premature delivery termination for End Users in a given region and for each piece of content, during a given period of time (e.g., hour/day/week/month) o Maximum and mean number of simultaneous sessions established by End Users in a given region, for a given Content Provider, and during a given period of time (e.g., hour/day/week/month) o Volume of traffic delivered for sessions established by End Users in a given region, for a given Content Provider, and during a given period of time (e.g., hour/day/week/month) o Maximum, mean, and minimum delivery throughput for sessions established by End Users in a given region, for a given Content Provider, and during a given period of time (e.g., hour/day/week/ month) Bertrand, et al. Expires November 28, 2013 [Page 14] Internet-Draft CDNI Logging May 2013 o Cache-hit and byte-hit ratios for requests received from End Users in a given region for each piece of content, during a given period of time (e.g., hour/day/week/month) o Top 10 of the most popularly requested content (during a given day /week/month), o Terminal type (mobile, PC, STB, if this information can be acquired from the browser type header, for example). Additional KPIs can be computed from other sources of information than the Logging, for instance, data collected by a content portal or by specific client-side application programming interfaces. Such KPIs are out of scope for the present memo. The KPIs used depend strongly on the considered log-consuming application -- the CDN operator may be interested in different metrics than the CSP is. In particular, CDN operators are often interested in delivery and acquisition performance KPIs, information related to Surrogates' performance, caching information to evaluate the cache-hit ratio, information about the delivered file size to compute the volume of content delivered during peak hour, etc. Some of the KPIs, for instance those providing an instantaneous vision of the active sessions for a given CSP's content, are useful essentially if they are provided in real-time. By contrast, some other KPIs, such as the one averaged on a long period of time, can be provided in non-real time. 3. CDNI Logging File Format As defined in Section 1.1 a CDNI logging field is as an atomic logging information element and a CDNI Logging Record is a collection of CDNI Logging Fields containing all logging information corresponding to a single logging event. This document defines a third level of structure, the CDNI Logging File, that is a collection of CDNI Logging Records. This structure is illustrated in Figure 3. The CDNI Logging File structure and encoding is specified in the present section. +------------------------------------------------------+ |CDNI Logging File | | | | +--------------------------------------------------+ | | |CDNI Logging Record | | | | +-------------+ +-------------+ +-------------+ | | | | |CDNI Logging | |CDNI Logging | |CDNI Logging | | | | | | Field | | Field | | Field | | | Bertrand, et al. Expires November 28, 2013 [Page 15] Internet-Draft CDNI Logging May 2013 | | +-------------+ +-------------+ +-------------+ | | | +--------------------------------------------------+ | | | | +--------------------------------------------------+ | | |CDNI Logging Record | | | | +-------------+ +-------------+ +-------------+ | | | | |CDNI Logging | |CDNI Logging | |CDNI Logging | | | | | | Field | | Field | | Field | | | | | +-------------+ +-------------+ +-------------+ | | | +--------------------------------------------------+ | | | | +--------------------------------------------------+ | | |CDNI Logging Record | | | | +-------------+ +-------------+ +-------------+ | | | | |CDNI Logging | |CDNI Logging | |CDNI Logging | | | | | | Field | | Field | | Field | | | | | +-------------+ +-------------+ +-------------+ | | | +--------------------------------------------------+ | +------------------------------------------------------+ Figure 3: Structure of Logging Files The CDNI Logging File format is inspired from the W3C Extended Log File Format [ELF]. However, it is fully specified by the present document. Where the present document differs from the W3C Extended Log File Format, an implementation of CDNI Logging MUST comply with the present document. A CDNI Logging File MUST contain a sequence of lines containing US- ASCII characters [CHAR_SET] terminated by either the sequence LF or CRLF. A CDNI Logging implementation consuming CDNI Logging Files MUST accept lines terminated by either LF or CRLF. Each line of a CDNI Logging File MUST contain either a directive or a CDNI Logging Record. Directives record information about the CDNI Logging process itself. Lines containing directives MUST begin with the "#" character. Directives are specified in Section 3.1. Logging Records provide actual details of the logged event. Logging Records are specified in Section 3.2. 3.1. CDNI Logging File Directives Bertrand, et al. Expires November 28, 2013 [Page 16] Internet-Draft CDNI Logging May 2013 An implementation of the CDNI Logging interface MUST support the following directives (formats specified in the form <...> are specified in Section 3.3): o Version: * format: . * semantic: indicates the version of the CDNI Logging File format. The value MUST be "1.0" for the version specified in the present document. * occurrence: there MUST be one and only one instance of this directive. It MUST be the first line of the CDNI Logging file. o UUID: * format: * semantic: this is Universally Unique IDentifier for the CDNI Logging File as specified in [RFC4122]. * occurrence: there MUST be one and only one instance of this directive. o Origin: * format: * semantic: this identifies the entity transmitting the CDNI Logging File (e.g. the host in a dCDN supporting the CDNI Logging interface) or the entity responsible for transmitting the CDNI Logging File (e.g. the dCDN). * occurrence: there MUST be zero or one instance of this directive. This directive MAY be included by the implementation transmitting the CDNI Logging file. When included by the transmitting side, it MUST be validated or over-written by the receiving side. When, it is not included by the transmitting side, it MAY be added locally by the receiving side. [Editor's Note if we include a non-repudiation mechanism: discuss the fact that this will provide incentive to dCDN to not cheat , as it can be detected] o Record-Type: * format: Bertrand, et al. Expires November 28, 2013 [Page 17] Internet-Draft CDNI Logging May 2013 * semantic: indicates the type of the CDNI Logging Records that follow this directive, until another Record-Type directive (or the end of the CDNI Logging File). "cdni_http_request_v1" MUST be indicated in the Record-Type directive for CDNI Logging records corresponding to HTTP request (e.g. a HTTP delivery request) as specified in Section 3.2.1. * occurrence: there MUST be at least one instance of this directive. The first instance of this directive MUST precede a Fields directive and precede any CDNI Logging Record. o Fields: * format: [ ], where the allowed list of are specified for each Record-Type in Section 3.2. * semantic: this lists the names of all the fields for which a value is to appear in the CDNI Logging Records that are after this directive. The names of the fields, as well as their possible occurrences, are specified for each type of CDNI Logging Records in Section 3.2. The field names listed in this directive MUST be separated by a whitespace (" "). * occurrence: there MUST be at least one instance of this directive per Record-Type directive. The first instance of this directive for a given Record-Type MUST precede any CDNI Logging Record for this Record-Type. o Integrity-Hash: * format: * semantic: This directive permits the detection of a corrupted CDNI Logging File. This can be useful, for instance, if a problem occurs on the filesystem of the dCDN Logging system and leads to a truncation of a logging file. The Integrity-Hash value is computed, and included in this directive by the entity that transmits the CDNI Logging File, by applying the MD5 ([RFC1321]) cryptographic hash function on the CDNI Logging File, including all the directives and logging records, up to the Intergrity-Hash directive itself, excluding the Integrity- Hash directive itself and, when present, also excluding the Non-Repudiation-Hash directive. The Integrity-Hash value is represented as a US-ASCII encoded hexadecimal number, 32 digits long (representing a 128 bit hash value). The entity receiving the CDNI Logging File also computes in a similar way the MD5 hash on the received CDNI Logging File and compares this hash to the value of the Integrity-Hash directive. If the two Bertrand, et al. Expires November 28, 2013 [Page 18] Internet-Draft CDNI Logging May 2013 values are equal, then the received CDNI Logging File MUST be considered non-corrupted. If the two values are different, the received CDNI Logging File MUST be considered corrupted. The behavior of the entity that received a corrupted CDNI Logging File is outside the scope of this specification; we note that the entity MAY attempt to pull again the same CDNI Logging file from the transmitting entity. * occurrence: there MUST be one and only one instance of this directive. This field MUST be the last line of the CDNI Logging File when the Non-Repudiation-Hash is absent, and MUST be the one before last line of the CDNI Logging File when the Non-Repudiation-Hash is present. o Non-Repudiation-Hash: * format: * semantic: This hash field permits the non-repudiation of the CDNI Logging File by the entity that transmitted the CDNI Logging File. [Editor's Note: I need help for specifying the appropriate hash - ie hash must be signed with private-key of entity transmitting the CDNI Logging File] * occurrence: there MAY be one and only one instance of this directive. When present, this directive MUST be the last line of the CDNI Logging File. 3.2. Logging Records A CDNI Logging Record consists of a sequence of CDNI Logging Fields relating to that single CDNI Logging Record. CDNI Logging Fields MUST be separated by the "horizontal tabulation (TAB)" character. Some CDNI Logging field names use a prefix scheme similar to the one used in W3C Extended Log File Format [ELF] to facilitate readability. The semantics of the prefix in the present document is: o c: refers to the User Agent that issues the request (corresponds to the "client" of W3C Extended Log Format) o s: refers to the dCDN Surrogate that serves the request (corresponds to the "server" of W3C Extended Log Format) o cs: refers to communication from the dCDN Surrogate towards the User-Agent Bertrand, et al. Expires November 28, 2013 [Page 19] Internet-Draft CDNI Logging May 2013 o sc: refers to communication from the User-Agent towards the dCDN Surrogate [Editor's Note: see discussion with Rob about adding definition for "r"] An implementation of the CDNI Logging interface as per the present specification MUST support the CDNI HTTP Delivery Records as specified in Section 3.2.1. [Editor's Note": other types of delivery records will be listed here if we specify other types for this version eg Request Routing]. The formats listed in this section in the form <...> are specified in Section 3.3). 3.2.1. HTTP Request Logging Record The HTTP Request Logging Record contains the following CDNI Logging Fields, listed by their field name: o date: * format: * semantic: the date at which the processing of request started on the Surrogate. * occurrence: there MUST be one and only one instance of this field. o time: * format: