ICAR D. Partain Internet-Draft Ericsson Expires: November 22, 2004 May 24, 2004 ICAR Proposed EArly Review (PEAR) draft-partain-icar-pear-00.txt Status of this Memo By submitting this Internet-Draft, I certify that any applicable patent or other IPR claims of which I am aware have been disclosed, and any of which I become aware will be disclosed, in accordance with RFC 3668. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http:// www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on November 22, 2004. Copyright Notice Copyright (C) The Internet Society (2004). All Rights Reserved. Abstract This memo proposes a lightweight and incremental approach to improving early cross-area review in the IETF. This proposal has two specific goals. Firstly, the current perception is that the IETF standardization process is bogged down and takes far too long. One factor in this slowness appears to be the workload placed upon the IESG when documents are brought to them. Given that one role of the IESG has been to provide cross-area review, offloading some portion of that work may increase the speed of the process. Secondly, cross-area review done early in the process may be a good tool for identifying significant problems with a Working Group's efforts long before the documents reach the IESG. As such, the early cross-area Partain Expires November 22, 2004 [Page 1] Internet-Draft ICAR Proposed EArly Review (PEAR) May 2004 review can improve the quality of the output from the Working Group by sifting the wheat from the chaff. Table of Contents 1. What is cross-area review? . . . . . . . . . . . . . . . . . 3 2. The proposal for Improved Cross-Area Review . . . . . . . . 4 2.1 ADs select Area's review team leaders(s) . . . . . . . . . 4 2.2 Review volunteers . . . . . . . . . . . . . . . . . . . . 4 2.3 Who may be a reviewer? . . . . . . . . . . . . . . . . . . 4 2.4 Role of the review process . . . . . . . . . . . . . . . . 4 2.5 WG Chair(s) determine that the time is ripe . . . . . . . 5 2.6 WG Chair(s) request early review . . . . . . . . . . . . . 5 2.7 Reviewers chosen . . . . . . . . . . . . . . . . . . . . . 5 2.8 The review team does its thing . . . . . . . . . . . . . . 6 2.9 Using the results of the review . . . . . . . . . . . . . 6 2.10 Report results to AD . . . . . . . . . . . . . . . . . . 6 3. Implementation . . . . . . . . . . . . . . . . . . . . . . . 8 4. Significant open issues . . . . . . . . . . . . . . . . . . 9 5. Security Considerations . . . . . . . . . . . . . . . . . . 10 Author's Address . . . . . . . . . . . . . . . . . . . . . . 10 Intellectual Property and Copyright Statements . . . . . . . 11 Partain Expires November 22, 2004 [Page 2] Internet-Draft ICAR Proposed EArly Review (PEAR) May 2004 1. What is cross-area review? Working groups tend to work somewhat in isolation. While by no means always the case, they tend not to get very much input from outside the working group (or, at best, their Area) until their work is presented to the IESG. At that point, the ADs are expected to review the work to provide a "sanity check" from the AD's area's perspective. The author would like to see this "sanity check" take place MUCH earlier than this. The idea of "review early, review often" has the potential of improving both the quality of documents and the speed of the standardization process. "Cross-area review" simply means that reviewers with competence outside of the WG's particular work area read and provide feedback on the document, with particular emphasis on how the document(s) interact with their area of competence. For simplicity, at least at the outset we define the "competence areas" as being equivalent to the IETF's Areas. This proposal specifically avoids giving "power" or "authority" to the reviewers. Instead, it attempts to fit into the IETF's consensus-based decision model. The author considers the consensus-based model to be a fundamental part of what makes the IETF unique and strongly desires to maintain that model. Partain Expires November 22, 2004 [Page 3] Internet-Draft ICAR Proposed EArly Review (PEAR) May 2004 2. The proposal for Improved Cross-Area Review This section outlines the suggested steps that need to be taken prior to introduction of these reviews, the steps taken for initiating a review, and how the output of the review should be handled. 2.1 ADs select Area's review team leaders(s) There must be a set of people who are responsible for coordinating the review work. However, it is important that the introduction of early cross-area review does NOT result in additional administrative work to be done by the IESG. This proposal suggests that the AD has a group of people (perhaps a group of one initially) who are responsible for leading the early reviews for the Area's working groups. This "review leader" can then build early review teams from a pool of volunteers. This proposal suggests that the ADs choose a set of team leaders. Obviously, there are other ways in which this could be handled (e.g., with the nomcom), but that will introduce additional bureaucracy that does not appear to be necessary. 2.2 Review volunteers Each Area in the IETF will have a set of volunteers willing to participate in early reviews. Those willing to help with reviews will submit their names to the AD or the AD may ask specific individuals. The group needs to be sufficiently large such that the work does not overburden the volunteers. 2.3 Who may be a reviewer? In a volunteer organization like the IETF, it seems dangerous to put too many constraints on who may be a volunteer. As such, this list should initially include all who wish to volunteer (see, however, the open issues in Section 4). Over time, the ambition is that the volunteer's level of expertise will become clear and thus make it easier to know which volunteers are appropriate in a particular review. 2.4 Role of the review process The review teams are expected to provide technical feedback based upon their technical area of expertise. As far as the working group is concerned, the review is not binding Partain Expires November 22, 2004 [Page 4] Internet-Draft ICAR Proposed EArly Review (PEAR) May 2004 in any way. The working group can choose to ignore the results of the review if they wish to, but the reasons for doing so must be clearly documented. See Section 2.10 below for further discussion. As far as the IESG is concerned, the review is a tool to provide valuable input to their decision-making process. The review does not obviate the need for the IESG to review documents themselves ("late" review) but may help improve the speed with which documents can move through the IESG. 2.5 WG Chair(s) determine that the time is ripe The WG chair(s) determine when a work item has progressed sufficiently to warrant a review. This must be the pervue of the chair(s). Of course, document editors can request that the chair(s) make the request. There is nothing preventing a WG from requesting review multiple times during a document's life cycle. In fact, this should be encouraged if there are enough people to provide quality reviews. 2.6 WG Chair(s) request early review The WG chair(s) submits a request for early review to the AD. This request must include: 1. a list of IETF areas that the chair(s) consider particularly important that they be included in the review. This list should be considered "advisory". That is, reviewers from other areas can, of course, participate. However, if at all possible, at least the areas requested by the chair(s) should be included in the review. 2. the date by which the review should be complete. This helps the reviewers plan and gives the WG some idea of when the work will be done. 3. a list of documents to be reviewed. 2.7 Reviewers chosen The AD asks a review leader to select a group of reviewers from the pool of volunteers for the relevant areas. The review leader selects the reviewers in such a way as to balance the workload to the degree possible and to ensure that a reasonable "cross-area" review can be accomplished. It is important that this be done without the influence of the AD so that reviews are not simply seen as a tool that the IESG (or specific AD) uses to push a particular "agenda". If the review leader cannot put together a review team with Partain Expires November 22, 2004 [Page 5] Internet-Draft ICAR Proposed EArly Review (PEAR) May 2004 volunteers from all requested areas, s/he will inform the AD and relevant WG chair(s) of the makeup of the review team s/he was able to put together and ask for a go/no-go with that review team. The AD and chair(s) will decide whether the review should proceed or not. 2.8 The review team does its thing The review process itself is not formalized. The review team is expected to coordinate the review internally and to provide a coordinated response. The response should take the form of specific recommendations with respect to the work. These recommendations might be, for example, to change approach significantly, to change particular details, to start again, etc. Without specific recommendations, it will be difficult for the working group to determine an appropriate response to the review. Note that it is extremely important that the review is clearly documented and archived for future reference. 2.9 Using the results of the review Upon receipt of the results of the review, the WG should carefully consider the review team's recommendations. At this point, the WG can take one of two approaches in considering each recommendation: 1. If the WG agrees with the recommendation, it will so note and begin taking appropriate action to follow the recommendation. 2. If the WG disagrees with the recommendation, it will document the reason(s) for doing so. This will then be discussed with the review team (or the person who made that specific recommendation) and the WG will attempt to reach rough consensus with the review team. If, in the end, the reviewer and the WG cannot agree, the reasons will be documented. To reiterate: it is extremely important that the review team's results are clearly documented and archived, and that the WG's response to those results is also clearly documented and archived. 2.10 Report results to AD When the review is complete and the review team and WG have concluded their discussion, the results will be reported to the ADs responsible for the WG. This report should be as simple as possible and should include (there's almost certain to be more): 1. The issue raised by the reviewer Partain Expires November 22, 2004 [Page 6] Internet-Draft ICAR Proposed EArly Review (PEAR) May 2004 2. The recommendation of the reviewer 3. The WG's response 4. Any relevant discussion for future reference The purpose of this historical record is to help the IESG in its deliberation. If a review team's recommendations have been dismissed by a WG, there must be enough information in the record to allow the IESG to determine the merits of the decision to disregard the recommendation. Partain Expires November 22, 2004 [Page 7] Internet-Draft ICAR Proposed EArly Review (PEAR) May 2004 3. Implementation It is important to understand that the author sees a stepwise introduction of early cross-area review as the only way to introduce such a change in the way the IETF works. It seems appropriate to run an "early review experiment" in a small set of WGs in one or a small number of IETF Areas before trying to introduce this on a wider scale. This allows us to (1) verify that the results are worth the pain and (2) to find bugs in the system. Introduction of early cross-area review does not need to imply a large set of new complicated processes. Rather, a small interested subset of ADs could choose to subject their area to cross-area review experimentation using specific working groups as guinea pigs. If these experiments prove to be successful, other ADs can join the ICAR parade. If, instead, we attempt to introduce such a process across the entire IETF, it is likely to become bogged down in endless administrative and process discussions. Let interested WGs prove or disprove the value of early cross-area review as well as providing input into how the process for early review can be improved. Partain Expires November 22, 2004 [Page 8] Internet-Draft ICAR Proposed EArly Review (PEAR) May 2004 4. Significant open issues 1. How does one measure the sucess/failure of ICAR experiments? 2. Can the review process be run without any IESG involvement in order to cement its independent nature? That is, can the ADs also be removed from (the admittedly small) administrative role envisioned in this document? 3. Where do we find the volunteers? It remains unclear where the pool of volunteers for this effort are. In order for this effort to be successful, a group of highly-qualified individuals (and their employers) must be willing to set aside a significant chunk of their time for doing reviews. The process of finding these people is not yet clear. 4. There may need to be some clearly defined procedures for handling adding/removal of review leaders, but I am anxious to avoid adding any "process" if it is not truly needed. 5. How do we ensure that the reviewers are competent? For the process to be of any value, high-quality reviews are essential. Perhaps there need to be mechanisms by which reviewers are removed from the volunteer pool? Should this be the decision of the two ADs in concert with the review leader(s) (i.e., if they agree, the reviewer is removed)? This would by no means preclude their providing input, but simply that they could not do so in the name of "official" cross-area review. Should the AD be able to add/remove people without the list becoming a hand-picked set of "favorites"? 6. Choosing reviewers seems simple at first glance but is administratively an issue. 7. Do there need to be formal ways for the review team to provide feedback to the working group? I am inclined to say no at this point. 8. How should reviews be archived? Partain Expires November 22, 2004 [Page 9] Internet-Draft ICAR Proposed EArly Review (PEAR) May 2004 5. Security Considerations None. Author's Address David Partain Ericsson AB P.O. Box 1248 SE-581 12 Linkoping, Sweden EMail: david.partain@ericsson.com Partain Expires November 22, 2004 [Page 10] Internet-Draft ICAR Proposed EArly Review (PEAR) May 2004 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the IETF's procedures with respect to rights in IETF Documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Disclaimer of Validity This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Copyright Statement Copyright (C) The Internet Society (2004). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. Acknowledgment Funding for the RFC Editor function is currently provided by the Internet Society. Partain Expires November 22, 2004 [Page 11]