Network Working Group J. Klensin Internet-Draft S. Dawkins Expires: August 8, 2004 February 08, 2004 A model for IETF Process Experiments draft-klensin-process-july14-00.txt Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http:// www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on August 7, 2004. Copyright Notice Copyright (C) The Internet Society (2004). All Rights Reserved. Abstract In the last two years, the IETF has initiated a number of interrelated efforts to improve or fine-tune its standards process and its internal procedures using the procedures intended for development of protocol specifications. None of these efforts has had an observable impact on the quality or timeliness of IETF outputs, and, based on the proposed charter milestones now under discussion, approval to try to improve things is still between six and eighteen months away. This document proposes a radically different approach to the system of making changes to IETF process, one that relies heavily on a "propose and carry out an experiment, evaluate the experiment, and then establish permanent procedures based on operational experience" model rather than the ones that have been attempted previously. Klensin & Dawkins Expires August 8, 2004 [Page 1] Internet-Draft Process Experiments February 2004 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Not an Accident, Not a Coincidence . . . . . . . . . . . . . . 5 3. Evidence We Are Headed in the Wrong Direction . . . . . . . . 6 4. Proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 5. Implications for the Present Paralysis . . . . . . . . . . . . 10 6. Security Considerations . . . . . . . . . . . . . . . . . . . 11 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 11 A. References . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Intellectual Property and Copyright Statements . . . . . . . . 13 Klensin & Dawkins Expires August 8, 2004 [Page 2] Internet-Draft Process Experiments February 2004 1. Introduction Starting with IETF 54 in Yokohama in July 2002, the IETF has been spending significant time and energy on analysis of actual and perceived problems that keep us from producing high quality standards on a timely and efficient basis. Ironically, these reform activities have been impeded by some of the same difficulties that impact the standards process itself, e.g., o Seemingly interminable discussion about charter details that derails a focus on the work to be done and the problems to be solved. o Discussion that doesn't converge, because it relies on logic and persuasion, instead of experiments and experience. o Proposals for "improvements" based on intuition, not experience. A significant percentage of the postings about how the IETF standards process should work are coming from participants who have never authored an RFC. A large percentage of the MPOWR postings about what WG chairs should do is coming from people who have never had the experience of being WG chairs. An overwhelming percentage of the postings on several mailing lists about offloading the IESG is coming from people who are not present or former IESG members and have no other direct information about how the IESG functions internally. o There has been uncertainty about the level of IESG enthusiasm for some of the proposals and doubts about whether they were worth the investment to develop if the IESG was not enthusiastic. In addition, these activities have shared two unfortunate characteristics with previous major process initiatives in the IETF: o There is an inevitable time conflict -- whether in following and contributing to mailing lists, developing and evaluating documents and proposals, or scheduling face to face meetings -- between process efforts and actual IETF technical/ engineering work. Especially when process efforts drag out for many months, there is a tendency for people whose primary commitment is to the development of technical specifications to go back to doing that, leaving process efforts to those who are strongly interested in process issues and, to some measure, less inclined or able to do the technical work. This creates a situation in which the process groups and efforts may be significantly unrepresentative of the range of IETF participants who are materially concerned with their outcomes. The meeting time conflict portion of the problem is the reason why the then co-Chairs of POISSON tried to avoid face to Klensin & Dawkins Expires August 8, 2004 [Page 3] Internet-Draft Process Experiments February 2004 face meetings entirely: a situation in which IETF participants needed to choose between attending standards-oriented WGs and a process-oriented one would ultimately benefit no one. o With protocol standards, the IETF understands, as a community, the dangers of standardization in the absence of implementation, deployment, and operational experience. As a result, we have traditionally had Experimental RFCs, a standards track with multiple maturity levels, and even implementations from Internet Drafts that could then be evaluated as definition and standardization progressed. Except, possibly, when the IESG has initiated process changes on its own initiative without going through a community-wide review first, we have had no parallel for procedures. We try to develop them (usually in Working Groups), instantiate them in BCPs, and then put them into effect. This appears to be unwise. It is certainly inefficient if efficiency is measured by the length of time between achievement of some level of community consensus and putting a procedure into effect. This document is specifically addressed to that set of problems with proposed modifications to procedures, especially the last one. Klensin & Dawkins Expires August 8, 2004 [Page 4] Internet-Draft Process Experiments February 2004 2. Not an Accident, Not a Coincidence To be very clear - the problems encountered using procedures intended for protocol specification development to fix procedural problems are systemic, and have existed since the POISED working group in the early 1990s. These problems will not go away. We recognize the efforts that have been made to make this work by a number of contributors, including current IESG members. At this time, the longest-serving Area Director the IETF has had is shepherding NEWTRK, and both ICAR and MPOWR are being shepherded by current Area Directors. The quality of these contributors argues strongly that it's just not possible to use the existing working group process to accomplish significant process improvements in the IETF. Klensin & Dawkins Expires August 8, 2004 [Page 5] Internet-Draft Process Experiments February 2004 3. Evidence We Are Headed in the Wrong Direction [[Note in draft: This section, and some other parts of the document, are very specific to observations about events in the year or two preceding its writing. If the document evolves in the direction of an RFC, the sections will, at least, need to be recast to put the remarks in historical context.]] The authors observe that, in the year prior to the creation of this draft, o The Senior Internet Reviewer effort fizzled out, largely due to a "chicken and egg problem". It became clear that it was not possible to get an adequate number of volunteers and generate high-quality reviews unless there was a strong commitment to using the results, but it also became clear that there couldn't be a strong commitment to integrating the SIR reviews into the specification review process unless there were adequate volunteers and it had been demonstrated that reviews of sufficient quality were being produced. o Proposals to create new process WGs [NEWTRK] and [ICAR] to try to devise and define solutions to particular problems became bogged down for extended periods in discussions of charter details, discussions that did nothing to either address the problems or to quickly generate alternatives for solving them. Five proposals to modify the existing standards track were under active discussion on [SOLUTIONS] before IETF 58, where they were presented at the NEWTRK BoF [NEWTRK58], but since IETF 58 the NEWTRK focus for an entire IETF meeting cycle has been on charter discussion, not on proposing changes that might improve document quality or timeliness. None of the existing proposals have been discussed on the mailing list or updated, no new proposals have been discussed on the mailing list or posted, and two-thirds of the NEWTRK postings have discussed the charter and planning for the upcoming IETF meeting. o Other proposals for ways to address procedural issues, or to schedule BOFs to discuss some of them, have deteriorated into arguments (or whining) about boundaries, definitions, priorities, etc. o We have not been successful in modularizing these efforts - taking SOLUTIONS, NEWTRK, and ICAR mailing lists as one example, only one active participant has stayed exclusively on one list. Anyone who cares about overall IETF process improvement is forced to subscribe to an increasingly large set of semi-connected mailing lists (SOLUTIONS, NEWTRK, ICAR, MPOWR, PROTO, EDU,...). We believe Klensin & Dawkins Expires August 8, 2004 [Page 6] Internet-Draft Process Experiments February 2004 there is already a "backlash" of people who have contributed previously, but don't think continued participation is worth the time and effort it takes to keep track of all the semi-independent proposals. o Finally, and most damaging - the draft charters for all of these efforts feature schedules in which the first milestone that produces a possible improvement is six to eighteen months away. If the problems involved are important enough to need solving -- keeping in mind that we have already been at the problem definition and evaluation process for eighteen months -- another six to eighteen months before we can start to do anything is far too long - and this assumes no slippages from the milestones as proposed. If they are not that important, then we should stop wasting time on them. In some cases, these lengthy discussions about how to organize work (rather than about the work itself) may have been completely appropriate. But they are not an efficient way to improve document quality or timeliness. It cannot be overstated that no one, regardless of depth or breadth of experience, really "knows" the effect that these proposals will have on the IETF's ability to produce quality specifications in a timely manner. After all of the discussion is complete, we still will not know with certainty whether a proposal is a good idea, and we will still need experience to know whether we have improved anything at all. Klensin & Dawkins Expires August 8, 2004 [Page 7] Internet-Draft Process Experiments February 2004 4. Proposal Since the 1992 changes, the IESG has adopted a number of procedural changes on its own initiative and documented them informally, utilizing their wide discretion implicit in RFC 2026. This document proposes to regularize and formalize that mechanism as a means of moving forward with procedural changes that might prove valuable. We note that, if the procedures the IESG has adopted (and procedural exceptions it has made) over the last decade are legitimate, then the IESG has the authority to institute the changes proposed here by bootstrapping the proposed process. We propose to permit (and encourage) the IESG to adopt and institute "process experiments" using the following procedure: 1. An I-D is written that describes what the proposed new or altered procedure is about and how it works. A statement of what problem it is expected to solve would be desirable, but is not a requirement (the intent is to keep the firm requirements for such an experiment as lightweight as possible). The I-D must state an explicit "sunset" timeout, typically not to exceed one year after adoption. 2. If the IESG believes the proposal is plausible and plausibly useful, a four week IETF Last Call is initiated. 3. At the conclusion of the Last Call, the IESG reevaluates the plausibility and appropriateness of the proposal. If they conclude that the proposed experiment is appropriate, a second I-D is generated (either by the IESG or by the original authors with IESG advice) that cleans up any definitional issues exposed in the Last Call and that explicitly identifies and responds to issues raised during that Last Call. 4. The document and experiment are then announced, the experiment is begun, and the document is forwarded for RFC publication. The IESG could, of course, reach a "bad idea" conclusion at any stage in this process and abandon the experiment. It might recommend publication of the experimental document, with a discussion of why it was a bad idea, but is not required to do so. The list above is deliberately agnostic about where the I-Ds come from: a WG, design team, individual contribution, editing group, or other mechanism, could be used in the first and/or third steps, but no specific mechanisms are required and the IESG is explicitly permitted to generate such proposals internally. In each case, the IESG's making of the decisions to go forward (or Klensin & Dawkins Expires August 8, 2004 [Page 8] Internet-Draft Process Experiments February 2004 not) with a procedural experiment, or the steps leading up to one, is expected to reflect their judgment of the existence of rough consensus in the community. That judgment may be appealed using the usual procedures, but the IESG and the community are reminded that an experimental attempt to try something for a fixed period is typically a better engineering approach than extended philosophical discussion without any experience to back it up. Nothing above is to be construed as a requirement that any given process experiment be attempted IETF-wide. A proposal for such an experiment may specify selected areas, selected working groups, working groups meeting some specific criteria (such as those created after a particular time or more than a specified number of years old), or be specific in other ways. At or before the end of the "sunset" timeout, the IESG would either revise (or cause to be revised) the document into a BCP RFC or the procedure would expire and, presumably, not be tried again unless something changed radically. A document describing why the experiment had succeeded or failed would be desirable but could not, realistically, be a requirement. If the procedure went to BCP, the BCP would reflect what we would call "operational experience" in the real world. Klensin & Dawkins Expires August 8, 2004 [Page 9] Internet-Draft Process Experiments February 2004 5. Implications for the Present Paralysis On the basis of this model, if the IESG believes that ICAR is probably a useful idea and that the community is at least tentatively in favor of trying it, then the first steps are not community review and IESG approval of a charter for a WG to study the issue. Instead, the first step is a proposal to do something and IESG consensus that the "something" is worth trying. A year later, if it seems to have helped and not caused harm, it gets cast in concrete. If not, we discard it and move on. Similar comments apply to other proposed WGs (or BOFs) concerned with process, such as MPOWR: if the IESG is convinced that something is needed, and that there is community backing for that "something", then it should be documented and deployed experimentally, with formal procedural changes, if any, occurring only after experience had been accumulated, observed, and evaluated. This model can, and in the opinion of the authors, probably should, be applied to changes in the standards process as well. Specifically, we tune the standards process in place, experimentally, largely by changes in what IESG _does_, rather than the language used to describe things. If those actions produce favorable results or directions, we _then_ go off and write up the details, rather than trying to make changes based on speculation about what might or might not work better (much less spending months on debates about the charter for considering those changes). Klensin & Dawkins Expires August 8, 2004 [Page 10] Internet-Draft Process Experiments February 2004 6. Security Considerations This document specifies a mechanism for evolving IETF procedures. It does not raise or consider any protocol-specific security issues. In considering experimental changes to procedures, the IESG should, of course, exercise due caution that such changes not reduce the quality of security review and consideration for protocols or, at least, that the process experiment proposals contain early detection and correction mechanisms should quality deterioration occur. Authors' Addresses John C Klensin 1770 Massachusetts Ave, #322 Cambridge, MA 02140 USA Phone: +1 617 491 5735 EMail: john-ietf@jck.com Spencer Dawkins 1547 Rivercrest Blvd. Allen, TX 75002 USA Phone: +1 469 330 3616 EMail: spencer@mcsr-labs.org Klensin & Dawkins Expires August 8, 2004 [Page 11] Internet-Draft Process Experiments February 2004 Appendix A. References [EDU]: Edu-Discuss Mailing List, https://www1.ietf.org/mail-archive/ working-groups/edu-discuss/current/threads.html [ICAR]: Improved Cross-Area Review, https://www1.ietf.org/mailman/ listinfo/icar [MPOWR]: Management Positions -- Oversight, Work and Results, https:/ /www1.ietf.org/mailman/listinfo/mpowr [NEWTRK]: New IETF Standards Track mailing list, http:// darkwing.uoregon.edu/~llynch/newtrk/threads.html [NEWTRK58]: New IETF Standards Track BoF at IETF 58, https:// www1.ietf.org/proceedings/03nov/130.htm [SOLUTION]: Solutions Mailing List, http://eikenes.alvestrand.no/ mailman/listinfo/solutions Klensin & Dawkins Expires August 8, 2004 [Page 12] Internet-Draft Process Experiments February 2004 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on the IETF's procedures with respect to rights in standards-track and standards-related documentation can be found in BCP-11. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementors or users of this specification can be obtained from the IETF Secretariat. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this standard. Please address the information to the IETF Executive Director. Full Copyright Statement Copyright (C) The Internet Society (2004). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assignees. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION Klensin & Dawkins Expires August 8, 2004 [Page 13] Internet-Draft Process Experiments February 2004 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Klensin & Dawkins Expires August 8, 2004 [Page 14]