Network Working Group J. Klensin Internet-Draft S. Dawkins Expires: September 16, 2004 March 18, 2004 A model for IETF Process Experiments draft-klensin-process-july14-01.txt Status of this Memo By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3667. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http:// www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on September 16, 2004. Copyright Notice Copyright (C) The Internet Society (2004). All Rights Reserved. Abstract In the last two years, the IETF has initiated a number of interrelated efforts to improve or fine-tune its standards process and its internal procedures using the procedures intended for development of protocol specifications. None of these efforts has had an observable impact on the quality or timeliness of IETF outputs, and, based on the proposed charter milestones now under discussion, approval to try to improve things is still between six and eighteen months away. This document proposes a radically different approach to the system of making changes to IETF process, one that relies heavily on a "propose and carry out an experiment, evaluate the experiment, and then establish permanent procedures Klensin & Dawkins Expires September 16, 2004 [Page 1] Internet-Draft Process Experiments March 2004 based on operational experience" model rather than the ones that have been attempted previously. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Not an Accident, Not a Coincidence . . . . . . . . . . . . . . 5 3. Evidence We Are Headed in the Wrong Direction . . . . . . . . 6 4. Proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5. Implications for the Present Paralysis . . . . . . . . . . . . 11 6. Security Considerations . . . . . . . . . . . . . . . . . . . 12 7. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 13 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 13 A. References . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Intellectual Property and Copyright Statements . . . . . . . . 15 Klensin & Dawkins Expires September 16, 2004 [Page 2] Internet-Draft Process Experiments March 2004 1. Introduction Starting with IETF 54 in Yokohama in July 2002, the IETF has been spending significant time and energy on analysis of actual and perceived problems that keep us from producing high quality standards on a timely and efficient basis. Ironically, these reform activities have been impeded by some of the same difficulties that impact the standards process itself, e.g., o Seemingly interminable discussion about charter details that derails a focus on the work to be done and the problems to be solved. o Discussion that doesn't converge, because it relies on logic and persuasion, instead of experiments and experience. o Proposals for "improvements" based on intuition, not experience. A significant percentage of the postings about how the IETF standards process should work are coming from participants who have never authored an RFC. A large percentage of the MPOWR postings about what WG chairs should do is coming from people who have never had the experience of being WG chairs. An overwhelming percentage of the postings on several mailing lists about offloading the IESG is coming from people who are not present or former IESG members and have no other direct information about how the IESG functions internally. o There has been uncertainty about the level of IESG enthusiasm for some of the proposals and doubts about whether they were worth the investment to develop if the IESG was not enthusiastic. In addition, these activities have shared two unfortunate characteristics with previous major process initiatives in the IETF: o There is an inevitable time conflict -- whether in following and contributing to mailing lists, developing and evaluating documents and proposals, or scheduling face to face meetings -- between process efforts and actual IETF technical/ engineering work. Especially when process efforts drag out for many months, there is a tendency for people whose primary commitment is to the development of technical specifications to go back to doing that, leaving process efforts to those who are strongly interested in process issues and, to some measure, less inclined or able to do the technical work. This creates a situation in which the process groups and efforts may be significantly unrepresentative of the range of IETF participants who are materially concerned with their outcomes. The meeting time conflict portion of the problem is the reason why the then co-Chairs of POISSON tried to avoid face to Klensin & Dawkins Expires September 16, 2004 [Page 3] Internet-Draft Process Experiments March 2004 face meetings entirely: a situation in which IETF participants needed to choose between attending standards-oriented WGs and a process-oriented one would ultimately benefit no one. o With protocol standards, the IETF understands, as a community, the dangers of standardization in the absence of implementation, deployment, and operational experience. As a result, we have traditionally had Experimental RFCs, a standards track with multiple maturity levels, and even implementations from Internet Drafts that could then be evaluated as definition and standardization progressed. Except, possibly, when the IESG has initiated process changes on its own initiative without going through a community-wide review first, we have had no parallel for procedures. We try to develop them (usually in Working Groups), instantiate them in BCPs, and then put them into effect. This appears to be unwise. It is certainly inefficient if efficiency is measured by the length of time between achievement of some level of community consensus and putting a procedure into effect. This document is specifically addressed to that set of problems with proposed modifications to procedures, especially the last one. In reading the balance of this document, it is worth noting that the difference between a procedural document and a protocol standard is that the former comes with a body (the IESG), that is responsible both for its adoption and for its ongoing and dynamic interpretation. For protocol specifications, the specification is required to be good enough to enable interoperable independent implementations (without a lot of shared oral tradition, especially beyond the first maturity level). That will typically require getting more details nailed down, and nailed down fairly precisely. That difference implies that we can be a lot more relaxed about fine details of procedural documents, especially experimental ones, at least as long as we can trust the IESG to document their interpretations so that those can be incorporated into a final document should the "experiment" succeed. Klensin & Dawkins Expires September 16, 2004 [Page 4] Internet-Draft Process Experiments March 2004 2. Not an Accident, Not a Coincidence To be very clear - the problems encountered using procedures intended for protocol specification development to fix procedural problems are systemic, and have existed since the POISED working group in the early 1990s. These problems will not go away. We recognize the efforts that have been made to make this work by a number of contributors, including current IESG members. At this time, the longest-serving Area Director the IETF has had is shepherding NEWTRK, and both ICAR and MPOWR are being shepherded by current Area Directors. The quality of these contributors argues strongly that it's not possible to use the existing working group process to accomplish significant process improvements in the IETF and that, if it is possible, it is probably not efficient. Klensin & Dawkins Expires September 16, 2004 [Page 5] Internet-Draft Process Experiments March 2004 3. Evidence We Are Headed in the Wrong Direction [[Note in draft: This section, and some other parts of the document, are very specific to observations about events in the year or two preceding its writing. If the document evolves in the direction of an RFC, the sections will, at least, need to be recast to put the remarks in historical context.]] The authors observe that, in the year prior to the creation of the initial version of this draft, o The Senior Internet Reviewer effort fizzled out, largely due to a "chicken and egg problem". It became clear that it was not possible to get an adequate number of volunteers and generate high-quality reviews unless there was a strong commitment to using the results, but it also became clear that there couldn't be a strong commitment to integrating the SIR reviews into the specification review process unless there were adequate volunteers and it had been demonstrated that reviews of sufficient quality were being produced. o Proposals to create new process WGs [NEWTRK] and [ICAR] to try to devise and define solutions to particular problems became bogged down for extended periods in discussions of charter details, discussions that did nothing to either address the problems or to quickly generate alternatives for solving them. Five proposals to modify the existing standards track were under active discussion on [SOLUTIONS] before IETF 58, where they were presented at the NEWTRK BoF [NEWTRK58], but since IETF 58 the NEWTRK focus for an entire IETF meeting cycle has been on charter discussion, not on proposing changes that might improve document quality or timeliness. None of the existing proposals have been discussed on the mailing list or updated, no new proposals have been discussed on the mailing list or posted, and two-thirds of the NEWTRK postings have discussed the charter and planning for the upcoming IETF meeting. o Other proposals for ways to address procedural issues, or to schedule BOFs to discuss some of them, have deteriorated into arguments (or whining) about boundaries, definitions, priorities, etc. o We have not been successful in modularizing these efforts - taking SOLUTIONS, NEWTRK, and ICAR mailing lists as one example, only one active participant has stayed exclusively on one list. Anyone who cares about overall IETF process improvement is forced to subscribe to an increasingly large set of semi-connected mailing lists (SOLUTIONS, NEWTRK, ICAR, MPOWR, PROTO, EDU,...). We believe Klensin & Dawkins Expires September 16, 2004 [Page 6] Internet-Draft Process Experiments March 2004 there is already a "backlash" of people who have contributed previously, but don't think continued participation is worth the time and effort it takes to keep track of all the semi-independent proposals. o Finally, and most damaging - the draft charters for all of these efforts feature schedules in which the first milestone that produces a possible improvement is six to eighteen months away. If the problems involved are important enough to need solving -- keeping in mind that we have already been at the problem definition and evaluation process for eighteen months -- another six to eighteen months before we can start to do anything is far too long - and this assumes no slippages from the milestones as proposed. If they are not that important, then we should stop wasting time on them. In some cases, these lengthy discussions about how to organize work (rather than about the work itself) may have been completely appropriate. But they are not an efficient way to improve document quality or timeliness. It cannot be overstated that no one, regardless of depth or breadth of experience, really "knows" the effect that these proposals will have on the IETF's ability to produce quality specifications in a timely manner. After all of the discussion is complete, we still will not know with certainty whether a proposal is a good idea, and we will still need experience to know whether we have improved anything at all. Klensin & Dawkins Expires September 16, 2004 [Page 7] Internet-Draft Process Experiments March 2004 4. Proposal Since the 1992 changes, the IESG has adopted a number of procedural changes on its own initiative and documented them informally, utilizing the wide discretion implicitly granted them by RFC 2026. This document proposes to regularize and formalize that mechanism as a means of moving forward with procedural changes that might prove valuable. We note that, if the procedures the IESG has adopted (and procedural exceptions it has made) over the last decade are legitimate, then the IESG has the authority to institute the changes proposed here by bootstrapping the proposed process. The mechanisms outlined here are not intended to be exclusive: they add to the IESG's range of tools for dealing with process issues on an ongoing basis, rather that replacing those tools with a single "magic bullet". The choice as to whether to use the procedure outlined in this document, if it is adopted, or other mechanisms available to the IESG and the community --present or future-- remains in the IESG's hands. If the IESG does not exercise that discretion wisely, this document provides no additional remedies. Some have read the current procedures as giving the IESG all of the capabilities outlined in this document, i.e., that it changes almost nothing. If this is true, this document only encourages the IESG to use this type of mechanism more frequently rather than less streamlined ones, and to more explicitly document what it is doing and what decisions it is making. We propose to permit (and encourage) the IESG to adopt and institute "process experiments" using the following procedure: 1. An I-D is written that describes what the proposed new or altered procedure is about and how it works. A statement of what problem it is expected to solve would be desirable, but is not a requirement (the intent is to keep the firm requirements for such an experiment as lightweight as possible). Similarly, specific experimental or evaluation criteria are very desirable, but not required -- for some of the process changes we anticipate, having the IESG reach a conclusion at the end of the sunset period that the community generally believes things to be better (or worse) will be both adequate and sufficient. The I-D must state an explicit "sunset" timeout, typically not to exceed one year after adoption. 2. If the IESG believes the proposal is plausible and plausibly useful, a four week IETF Last Call is initiated. The IESG can institute whatever procedures it wishes to make that determination and to avoid denial of service attacks from large Klensin & Dawkins Expires September 16, 2004 [Page 8] Internet-Draft Process Experiments March 2004 numbers of spurious or unimportant proposals. In particular, they might institute a procedure requiring some number of endorsements, or endorsements of a particular type, before the IESG considers the draft. The IESG is, however, expected to understand that procedures or review processes that act as a mechanism for significant delays do not fall within the intent of this proposal. 3. At the conclusion of the Last Call, the IESG reevaluates the plausibility and appropriateness of the proposal. If they conclude that the proposed experiment is appropriate, a second I-D is generated (either by the IESG or by the original authors with IESG advice) that cleans up any definitional issues exposed in the Last Call and that explicitly identifies and responds to issues raised during that Last Call. 4. The document and experiment are then announced, the experiment is begun, and the document is forwarded for RFC publication. The IESG could, of course, reach a "bad idea" conclusion at any stage in this process and abandon the experiment. It might recommend publication of the experimental document, with a discussion of why it was a bad idea, but is not required to do so. The list above is deliberately agnostic about where the I-Ds come from: a WG, design team, individual contribution, editing group, or other mechanism, could be used in the first and/or third steps, but no specific mechanisms are required and the IESG is explicitly permitted to generate such proposals internally. In each case, the IESG's making of the decisions to go forward (or not) with a procedural experiment, or the steps leading up to one, is expected to reflect their judgment of the existence of rough consensus in the community. That judgment may be appealed using the usual procedures, but the IESG and the community are reminded that an experimental attempt to try something for a fixed period is typically a better engineering approach than extended philosophical discussion without any experience to back it up. Nothing above is to be construed as a requirement that any given process experiment be attempted IETF-wide. A proposal for such an experiment may specify selected areas, selected working groups, working groups meeting some specific criteria (such as those created after a particular time or more than a specified number of years old), or be specific in other ways. At or before the end of the "sunset" timeout, the IESG would either revise (or cause to be revised) the document into a BCP RFC or the procedure would expire and, presumably, not be tried again unless Klensin & Dawkins Expires September 16, 2004 [Page 9] Internet-Draft Process Experiments March 2004 something changed radically. A document describing why the experiment had succeeded or failed would be desirable but could not, realistically, be a requirement. If the procedure went to BCP, the BCP would reflect what we would call "operational experience" in the real world. Klensin & Dawkins Expires September 16, 2004 [Page 10] Internet-Draft Process Experiments March 2004 5. Implications for the Present Paralysis On the basis of this model, if the IESG believes that ICAR is probably a useful idea and that the community is at least tentatively in favor of trying it, then the first steps are not community review and IESG approval of a charter for a WG to study the issue. Instead, the first step is a proposal to do something and IESG consensus that the "something" is worth trying. A year later, if it seems to have helped and not caused harm, it gets cast in concrete. If not, we discard it and move on. Similar comments apply to other proposed WGs (or BOFs) concerned with process, such as MPOWR: if the IESG is convinced that something is needed, and that there is community backing for that "something", then it should be documented and deployed experimentally, with formal procedural changes, if any, occurring only after experience had been accumulated, observed, and evaluated. This model can, and in the opinion of the authors, probably should, be applied to changes in the standards process as well. Specifically, we tune the standards process in place, experimentally, largely by changes in what IESG _does_, rather than the language used to describe things. If those actions produce favorable results or directions, we _then_ go off and write up the details, rather than trying to make changes based on speculation about what might or might not work better (much less spending months on debates about the charter for considering those changes). Klensin & Dawkins Expires September 16, 2004 [Page 11] Internet-Draft Process Experiments March 2004 6. Security Considerations This document specifies a mechanism for evolving IETF procedures. It does not raise or consider any protocol-specific security issues. In considering experimental changes to procedures, the IESG should, of course, exercise due caution that such changes not reduce the quality of security review and consideration for protocols or, at least, that the process experiment proposals contain early detection and correction mechanisms should quality deterioration occur. Klensin & Dawkins Expires September 16, 2004 [Page 12] Internet-Draft Process Experiments March 2004 7. Acknowledgements The first revision of this document benefited significantly from suggestions and comments from Avri Doria, Margaret Wasserman, and Harald Alvestrand, and from discussions with the General Area Directorate and at its open meeting during IETF 59. Authors' Addresses John C Klensin 1770 Massachusetts Ave, #322 Cambridge, MA 02140 USA Phone: +1 617 491 5735 EMail: john-ietf@jck.com Spencer Dawkins 1547 Rivercrest Blvd. Allen, TX 75002 USA Phone: +1 469 330 3616 EMail: spencer@mcsr-labs.org Klensin & Dawkins Expires September 16, 2004 [Page 13] Internet-Draft Process Experiments March 2004 Appendix A. References [EDU]: Edu-Discuss Mailing List, https://www1.ietf.org/mail-archive/ working-groups/edu-discuss/current/threads.html [ICAR]: Improved Cross-Area Review, https://www1.ietf.org/mailman/ listinfo/icar [MPOWR]: Management Positions -- Oversight, Work and Results, https:/ /www1.ietf.org/mailman/listinfo/mpowr [NEWTRK]: New IETF Standards Track mailing list, http:// darkwing.uoregon.edu/~llynch/newtrk/threads.html [NEWTRK58]: New IETF Standards Track BoF at IETF 58, https:// www1.ietf.org/proceedings/03nov/130.htm [SOLUTION]: Solutions Mailing List, http://eikenes.alvestrand.no/ mailman/listinfo/solutions Klensin & Dawkins Expires September 16, 2004 [Page 14] Internet-Draft Process Experiments March 2004 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the IETF's procedures with respect to rights in IETF Documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2004). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Klensin & Dawkins Expires September 16, 2004 [Page 15]