Network Working Group Mike O'Dell Internet-Draft UUNET Technologies Harald T. Alvestrand Maxware Bert Wijnen IBM T. J. Watson Research Scott Bradner Harvard University August 1998 Advancement of MIB specifications on the IETF Standards Track 1. Status of this Memo This document is an Internet-Draft. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet- Drafts as reference material or to cite them other than as "work in progress." To view the entire list of current Internet-Drafts, please check the "1id-abstracts.txt" listing contained in the Internet-Drafts Shadow Directories on ftp.is.co.za (Africa), ftp.nordu.net (Northern Europe), ftp.nis.garr.it (Southern Europe), munnari.oz.au (Pacific Rim), ftp.ietf.org (US East Coast), or ftp.isi.edu (US West Coast). A revised version of this draft document will be submitted to the RFC editor as a BCP (Best Current Practice) documenting an IESG procedure for the Internet Community. Discussion and suggestions for improvement are requested. This document will expire before January, 1999. Distribution of this draft is unlimited. 2. Abstract The Internet Standards Process [1] requires that all IETF Standards Track specifications must have "multiple, independent, and interoperable implementations" before they can be advanced beyond Proposed Standard status. This document specifies the process which the IESG will use to determine if a MIB document meets these requirements. It also discusses the rationale for this process. O'Dell/Alvestrand/Wijnen/Bradner [Page 1] Internet-Draft MIB Advancement August 1998 3. The Nature of the Problem The Internet Standards Process [1] requires that for a IETF specification to advance beyond the Proposed Standard level, at least two genetically unrelated implementations must be shown to interoperate correctly with all features and options. There are two distinct reasons for this requirement. The first reason is to verify that the text of the specification is adequately clear and accurate. This is demonstrated by showing that multiple implementation efforts have used the specification to achieved interoperable implementations. The second reason is to discourage excessive options and "feature creep". This is accomplished by requiring interoperable implementation of all features, including options. If an option is not included in at least two different interoperable implementations, it is safe to assume that it has not been deemed useful and must be removed before the specification can advance. In the case of a protocol specification which specifies the "bits on the wire" exchanged by executing state machines, the notion of "interoperability" is reasonably intuitive - the implementations must successfully "talk to each other", exchanging "bits on the wire", while exercising all features and options. In the case of an SNMP Management Information Base (MIB) specification, exactly what constitutes "interoperation" is less obvious. This document specifies how the IESG has decided to judge "MIB interoperability" in the context of the IETF Standards Process. There are a number of plausible interpretations of MIB interoperability, many of which have merit but which have very different costs and difficulties in realization. The aim is to ensure that the dual goals of specification clarity and feature evaluation have been met using an interpretation of the concept of MIB interoperability that strikes a balance between testing complexity and practicality. 4. On The Nature of MIBs Compared to "state machine" protocols which focus on procedural specifications, a MIB is much more data oriented. To over- generalize, in a typical MIB the collection of data type and instance specifications outnumbers inter-object procedural or causal semantics by a significant amount. O'Dell/Alvestrand/Wijnen/Bradner [Page 2] Internet-Draft MIB Advancement August 1998 A central issue is that a MIB does not stand alone; it forms the access interface to the instrumentation underneath it. Without the instrumentation, a MIB has form but no values. Coupled with the large number of objects even in a simple MIB, a MIB tends to have more of the look and feel of an API or a dictionary than a state machine protocol. It is important to distinguish between assessing the interoperability of applications which may use or interact with MIBs, and the MIBs themselves. It is fairly obvious that "black-box testing" can be applied to such applications and that the approach enjoys a certain maturity in the software engineering arts. A MIB, on the other hand is not readily amenable to black box test plans. 5. Discussion and Recommended Process In order to meet their obligations under the IETF Standards Process the Operations and Management Area Directors and the IESG must be convinced that each MIB specification advanced to Draft Standard or Internet Standard status is clearly written, that there are the required multiple interoperable implementations, and that all options have been implemented. There are multiple ways to achieve this goal. Appendix A lists some testing approaches that could be used when attempting to document multiple implementations. The Full Coverage or Stimulus-Response approaches are very through, and would increase confidence that the requirement has been met, if applied. However, experience in real-world software engineering makes it clear that such confidence comes at an extremely high price; even with the most exhaustive testing, it is often not clear what precisely has been demonstrated by such testing. We believe that both of those standards of evidence are materially beyond what can be reasonably accomplished in an operational sense, and achieving the requisite semantic specifications are even more unlikely. Therefore, the Operations and Management Area and the IESG have adopted a more pragmatic approach to determining the suitability of a MIB specification for advancement on the standards track beyond Proposed Standard status. Each MIB specification suggested for advancement must have one or more advocates who can make a convincing argument that the MIB specification meets the multiple implementation and feature support requirements of the IETF Standards Process. The specific way to make the argument is left to the advocate, but will normally include reports that basic object comparison testing has been done. Thus any recommendation for the advancement of a MIB specification must be accompanied by an implementation report, as is the case with O'Dell/Alvestrand/Wijnen/Bradner [Page 3] Internet-Draft MIB Advancement August 1998 all requests for the advancement of IETF specifications. The implementation report must include the reasons why the IESG should believe that there are multiple implementations of the MIB in question and that the all of the MIB objects in the specification to be advanced are supported in more than one implementation. But note that the prime concern of the IESG will be that the underlying reasons for the interoperable implementations are met, i.e. that the text of the specification is clear and unambiguous, and that features of the specification which have not garnered support have been removed. The implementation report will be placed on the IETF web page along with the other pre-advancement implementation reports and will be specifically referred to in the IETF Last-Call. As with all such implementation reports, the determination of adequacy is made by the Area Director(s) of the relevant IETF Area. This determination of adequacy can be challenged during the Last-Call period. 6. Security Considerations Some may view this policy as possibly leading to a reduction in the level of confidence people can have in MIBs but the O&M Area Directors and the IESG feel that it will adequately ensure a reasonable evaluation of the level of clarity of MIB specifications and to ensure that unused options can be identified and removed before the advancement of a specification. Good, clearly written MIBs can be of great assistance in the management of the Internet and other networks and thus assist in the reduction of some types of security threats. 8. References [RFC2026] "The Internet Standards Process -- Revision 3", Bradner, October 1996 9. Author's Addresses Michael D. O'Dell UUNET Technologies, Inc. 3060 Williams Drive Fairfax, VA 22031 Email: mo@uu.net Phone: +1-703-206-5890 O'Dell/Alvestrand/Wijnen/Bradner [Page 4] Internet-Draft MIB Advancement August 1998 Harald T. Alvestrand Maxware Pirsenteret N-7005 Trondheim, Norway EMail: Harald.Alvestrand@maxware.no Phone: +47-73-54-57-94 Bert Wijnen IBM T. J. Watson Research Schagen 33 3461 GL Linschoten Netherlands EMail: wijnen@vnet.ibm.com phone: +31-348-432-794 Scott Bradner Harvard University 1350 Mass. Ave. Cambridge MA 02138 Email: sob@harvard.edu Phone: +1-617-495-3864 Appendix A A. Some Testing Alternatives The IESG debated a number of interoperability and testing models in formulating this specification. The following list is not an exhaustive enumeration of the alternatives, but it does capture the major plausible models which were examined in the course of the discussion. A.1 Basic Object Comparison Assume the requisite two genetically unrelated implementations of the MIB in an SNMP agent and an SNMP management station which can do a "MIB Dump" (extract the complete set of MIB object types and values from the agent implementation). Extract a MIB Dump from each implementation and compare the two dumps to verify that both provide the complete set of mandatory and optional objects and that the individual objects are of the correct types. A.2 Stimulus/Response Testing O'Dell/Alvestrand/Wijnen/Bradner [Page 5] Internet-Draft MIB Advancement August 1998 Proceed as in A.1, but in addition, comprehensively exercise the two (network) elements containing the agent implementations to verify that all the MIB objects reflect plausible values in operational conditions. An even stricter interpretation would require that the MIB objects in the two network elements track identically given the identical stimulus. While this would test "read-only" or "monitoring" information obtained from the underlying instrumentation, it is important to observe that such instrumentation is actually an *application* which uses the MIB and is not part of the MIB itself. A.3 Full Coverage Testing This model extends the notion of Stimulus/Response Testing to its logical extreme. The MIB is viewed as an API and the software engineering notion of full coverage testing is applied to a MIB. This involves exercising all paths through the causal semantics and verifying that all objects change state correctly in all cases. Again, note that much more than the MIB definition is being exercised and evaluated. O'Dell/Alvestrand/Wijnen/Bradner [Page 6]