Problems and Possible Solutions for Advancing Metrics on the Standards Track
draft-morton-ippm-advance-metrics-00

Status of this Memo

This Internet-Draft is submitted to IETF in full conformance with the provisions of BCP 78 and BCP 79. This document may contain material from IETF Documents or IETF Contributions published or made publicly available before November 10, 2008. The person(s) controlling the copyright in some of this material may not have granted the IETF Trust the right to allow modifications of such material outside the IETF Standards Process. Without obtaining an adequate license from the person(s) controlling the copyright in such materials, this document may not be modified outside the IETF Standards Process, and derivative works of it may not be created outside the IETF Standards Process, except to format it for publication as an RFC or to translate it into languages other than English.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”

The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.

The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.

This Internet-Draft will expire on January 7, 2010.

Copyright Notice

Copyright (c) 2009 IETF Trust and the persons identified as the document authors. All rights reserved.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents in effect on the date of publication of this document (http://trustee.ietf.org/license-info). Please review these documents carefully, as they describe your rights and restrictions with respect to this document.

Abstract

This memo identifies some issues with the process of progressing performance metric RFCs along the standards track. This memo takes the position that the metric definitions themselves should be the primary focus, rather than the implementations of metrics. This appears to allow some simplification of the task at hand and subsequently leads to solutions for the issues raised.

Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.) [RFC2119].

Table of Contents

1.  Introduction
2.  Issues with comparing implementations
    2.1.  Implementation variability
    2.2.  Deciding on the statistical methods
    2.3.  Assumption of non-interoperable implementations
    2.4.  Determining whether Lab testing can serve the process
    2.5.  Achieving "identical" network conditions
    2.6.  IETF is not in the Certification Business
3.  A Definition-centric metric advancement process
4.  Examples of checking metric definitions in the Lab
    4.1.  One-way Delay, Loss threshold, RFC 2679
    4.2.  One-way Delay, First-bit to Last bit, RFC 2679
    4.3.  One-way Delay, RFC 2679
    4.4.  Error Calibration, RFC 2679
5.  Security Considerations
6.  IANA Considerations
7.  Acknowledgements
8.  Normative References
§  Author's Address

1.  Introduction

IPPM has been considering how to advance their metrics along the standards track since 2001, with the initial publication of Bradner/Paxson/Mankin's memo [ref to work in progress]. The original proposal was to compare the results of implementations of the metrics, because the usual procedures for advancing protocols did not appear to apply. It was found to be difficult to achieve consensus on exactly how to compare implementations, since there were many legitimate sources of variation that would emerge in the results despite the best attempts to keep the network measured equal for both.

This author takes the position that the metric definitions themselves should be the primary focus, rather than the implementations of metrics. The advancement process should produce confidence that the metric definitions and supporting material are clearly worded and unambiguous, OR, it should identify ways in which the metric definitions should be revised to achieve clarity.

The process should also permit identification of options that were not implemented, so that they can be removed from the advancing specification (this is an aspect more typical of protocol advancement along the standards track).

This memo first identifies some issues with the current approach of comparing implementations (primarily on live network paths), and then looks at the metric RFC advancement process in a different way to see how some of the issues might be resolved.

This memo's purpose is to make some constructive observations on the approach as the author perceives it to be (at the moment). It was prepared to help progress discussions on the topic of metric advancement, both through e-mail and at the upcoming IPPM meeting at IETF-75 in Stockholm.

2.  Issues with comparing implementations

A few topics have been, and continue to be issues for comparing implementations. This section lists some of these issues, and simplifying solutions when they seem possible under the current approach.

2.1.  Implementation variability

The metric RFCs generally allow quite a bit of flexibility for implementators to design metrics that meet their particular needs. However, if two implementations cannot use identical options, it adds complexity to the comparison because the statistical analysis must allow for the differences.

A solution would be to mandate that implementations used in the advancement process use identical metric parameters and options.

2.2.  Deciding on the statistical methods

As mentioned above, allowing for implementation variability made this a complex process.

Another complexity is the variability in metric statistics allowed in the RFCs: min, max, percentiles, ratios, etc. Comparisons vary by the statistic.

This problem is being worked, but perhaps the problem statement will need simplification before it can be solved.

2.3.  Assumption of non-interoperable implementations

The original stdmetrics memo was written before OWAMP [RFC4656] (Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. Zekauskas, “A One-way Active Measurement Protocol (OWAMP),” September 2006.) and TWAMP [RFC5357] (Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. Babiarz, “A Two-Way Active Measurement Protocol (TWAMP),” October 2008.) were proposed, and correctly assumed that measurement implementations could not talk to each other, as this was the common case at the time. Only the results of measurements could be compared.

With the approval of OWAMP and TWAMP, and the emergence of many implementations, this facility could be exploited to produce:

• testing opportunities with less coordination required to transport and install different implementations side-by-side.
• development of test vectors in OWAMP-Test or TWAMP-Test packet formats.

2.4.  Determining whether Lab testing can serve the process

Since the measurement systems developed for metrics in the RFC 2330 framework are intended for use on live networks, it was taken that the comparison would also involve live network measurements. Lab measurements were not considered the primary basis for comparison, if they were to be used at all.

A solution to this question may stem from deciding exactly what the RFC advancement process will entail.

This memo posits that several key aspects of the metric definition implementations cannot be conveniently examined in field measurement scenarios, and that lab measurements could be a reasonable basis for a purely metric-definition-centric advancement process (as described in section 3).

2.5.  Achieving "identical" network conditions

The basis for this concern is the amount of parallelism that is common in devices/networks today. Parallel resources are applied to increase capacity, and hashing functions (on addresses and ports) determine which set of resources all the packets in a flow will traverse. For one analysis of the situation, see the Benchmarking WG's Hash and Stuffing [RFC4814] (Newman, D. and T. Player, “Hash and Stuffing: Overlooked Factors in Network Device Benchmarking,” March 2007.).

Two measurement devices on the same sub-net will have different addresses, and their packet streams are likely to be assigned to different network resources where delay, loss, and other impairments can differ for legitimate reasons. Testing in the lab may not remove the parallel resources, but it can provide some time stability that's never assured in live network testing.

One possible solution proposed in Geib's work-in-progress is to encapsulate all measurement streams in an IP tunnel, specifically and IPsec tunnel. This needs further investigation for feasibility and potential new issues raised by encaulation/de-encapsulation processes.

2.6.  IETF is not in the Certification Business

We're not.

The solution seems to be carefully wording the process of metric advancement so that it is clear that the metric definitions are the focus throughout, and not the implementations.

3.  A Definition-centric metric advancement process

The process described in this section takes as a first principle that the metric definitions, embodied in the text of the RFCs, are the objects that require evaluation and possible revision in order to advance to the next step on the standards track.

IF two implementations do not measure an equivalent singleton, or sample, or produce the an equivalent statistic,

AND sources of measurement error do not adequately explain the lack of agreement,

THEN the details of each implementation should be audited along with the exact definition text, to determine if there is a lack of clarity that has caused the implementations to vary in a way that affects the correspondence of the results.

IF there was a lack of clarity or multiple legitimate interpretations of the definition text,

THEN the text should be modified and the resulting memo proposed for consensus and advancement along the standards track.

The figure below illustrates this process:

     ,---.
    /     \
   ( Start )
    \     /    Implementations
     `-+-'        +-------+
       |         /|   1   `.
   +---+----+   / +-------+ `.-----------+      ,-------.
   |  RFC   |  /             |Check for  |    ,' was RFC `.  YES
   |        | /              |Equivalence.....  clause x   -------+
   |        |/    +-------+  |under      |    `. clear?  ,'       |
   | Metric \.....|   2   ....relevant   |      `---+---'    +----+---+
   | Metric |\    +-------+  |identical  |       No |        |Advance |
   | Metric | \              |network    |      +---+---.----+spec    |
   |  ...   |  \             |conditions |      |Modify |    +----+---+
   |        |   \ +-------+  |           |      |Spec   |         |
   +--------+    \|   n   |.'+-----------+      +-------+      +--+-+
                  +-------+                                    |DONE|
                                                               +----+

4.  Examples of checking metric definitions in the Lab

This section describes some quick examples of lab tests with devices and/or simulators to create relevant conditions to test whether the metric definitions were interpreted consistently by implementors.

For these tests, a stream of 100 (?) packets SHALL be sent from Source to Destination in each implementation.

These examples do not entirely avoid the problem of declaring equivalence with a statistical test, but the lab conditions should simplify the problem by removing as much variability as possible.

Note that there are only five instances of the requirement term "MUST" in [RFC2679] (Almes, G., Kalidindi, S., and M. Zekauskas, “A One-way Delay Metric for IPPM,” September 1999.) outside of the boilerplate and [RFC2119] (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.) reference.

4.1.  One-way Delay, Loss threshold, RFC 2679

This test determines if implementations use the same configured maximum waiting time delay from one measurement to another under different delay conditions, and correctly declare packets arriving in excess of the waiting time threshold as lost.

See Section 3.5 of [RFC2679] (Almes, G., Kalidindi, S., and M. Zekauskas, “A One-way Delay Metric for IPPM,” September 1999.), 3rd bullet point and also Section 3.8.2 of [RFC2679] (Almes, G., Kalidindi, S., and M. Zekauskas, “A One-way Delay Metric for IPPM,” September 1999.).

1. configure a path with 1 sec one-way constant delay
2. measure one-way delay with 2 or more implementations, using identical waiting time thresholds for loss set at 2 seconds
3. configure the path with 3 sec one-way delay
4. repeat measurements
5. observe that the increase measured in step 4 caused all packets to be declared lost, and that all packets that arrive successfully in step 2 are assigned a valid one-way delay.

4.2.  One-way Delay, First-bit to Last bit, RFC 2679

This test determines if implementations register the same relative increase in delay from one measurement to another under different delay conditions. This test tends to cancel the sources of error which may be present in an implementation.

See Section 3.7.2 of [RFC2679] (Almes, G., Kalidindi, S., and M. Zekauskas, “A One-way Delay Metric for IPPM,” September 1999.), and Section 10.2 of [RFC2330] (Paxson, V., Almes, G., Mahdavi, J., and M. Mathis, “Framework for IP Performance Metrics,” May 1998.).

1. configure a path with X ms one-way constant delay, and ideally including a low-speed link
2. measure one-way delay with 2 or more implementations, using identical options and equal size small packets (e.g., 100 octet IP payload)
3. maintain the same path with X ms one-way delay
4. measure one-way delay with 2 or more implementations, using identical options and equal size large packets (e.g., 1500 octet IP payload)
5. observe that the increase measured in steps 2 and 4 is equivalent to the increase in ms expected due to the larger serialization time for each implementation. Most of the measurement errors in each system should cancel, if they are stationary.

4.3.  One-way Delay, RFC 2679

This test determines if implementations register the same relative increase in delay from one measurement to another under different delay conditions. This test tends to cancel the sources of error which may be present in an implementation.

This test is intended to evaluate measurments in sections 3 and 4 of [RFC2679] (Almes, G., Kalidindi, S., and M. Zekauskas, “A One-way Delay Metric for IPPM,” September 1999.).

1. configure a path with X ms one-way constant delay
2. measure one-way delay with 2 or more implementations, using identical options
3. configure the path with X+Y ms one-way delay
4. repeat measurements
5. observe that the increase measured in steps 2 and 4 is ~Y ms for each implementation. Most of the measurement errors in each system should cancel, if they are stationary.

4.4.  Error Calibration, RFC 2679

This is a simple check to determine if an implementation reports the error calibration as required in Section 4.8 of [RFC2679] (Almes, G., Kalidindi, S., and M. Zekauskas, “A One-way Delay Metric for IPPM,” September 1999.). Note that the context (Type-P) must also be reported.

5.  Security Considerations

There are no security issues raised by discussing the topic of metric RFC advancement along the standards track.

The security considerations that apply to any active measurement of live networks are relevant here as well. See [RFC4656] (Shalunov, S., Teitelbaum, B., Karp, A., Boote, J., and M. Zekauskas, “A One-way Active Measurement Protocol (OWAMP),” September 2006.) and [RFC5357] (Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J. Babiarz, “A Two-Way Active Measurement Protocol (TWAMP),” October 2008.).

6.  IANA Considerations

This memo makes no requests of IANA, and hopes that IANA will leave it alone, as well.

7.  Acknowledgements

The author would like to thank anyone who reads this memo for helpful review and comments.

8. Normative References

Author's Address