Network Working Group S. Krishnan Internet-Draft Ericsson Intended status: Informational P. Eronen Expires: November 26, 2008 Nokia E. Gray Ericsson May 25, 2008 Guidelines to authors and reviewers regarding the IETF review process draft-krishnan-review-process-00 Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on November 26, 2008. Abstract This document describes the IETF review process and provides guidelines to draft authors and reviewers on how to effectively participate in it. Krishnan, et al. Expires November 26, 2008 [Page 1] Internet-Draft IETF Review Guidelines May 2008 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Sources of reviews . . . . . . . . . . . . . . . . . . . . . . 3 3. Guidelines for authors . . . . . . . . . . . . . . . . . . . . 3 3.1. Who responds to reviews . . . . . . . . . . . . . . . . . 3 3.2. Before reading the review . . . . . . . . . . . . . . . . 4 3.3. Responding to reviews . . . . . . . . . . . . . . . . . . 4 3.3.1. Timeframe for response . . . . . . . . . . . . . . . . 4 3.3.2. Contents of a response . . . . . . . . . . . . . . . . 5 3.4. Keeping track of the issues . . . . . . . . . . . . . . . 6 3.5. New version of the draft . . . . . . . . . . . . . . . . . 6 3.6. Acknowledging the reviewer . . . . . . . . . . . . . . . . 7 4. Guidelines for reviewers . . . . . . . . . . . . . . . . . . . 7 4.1. Level of review . . . . . . . . . . . . . . . . . . . . . 7 4.2. Prioritization of the issues . . . . . . . . . . . . . . . 7 4.3. Reviewing changes . . . . . . . . . . . . . . . . . . . . 8 5. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 8 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 7. Security Considerations . . . . . . . . . . . . . . . . . . . 8 8. Normative References . . . . . . . . . . . . . . . . . . . . . 8 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 9 Intellectual Property and Copyright Statements . . . . . . . . . . 10 Krishnan, et al. Expires November 26, 2008 [Page 2] Internet-Draft IETF Review Guidelines May 2008 1. Introduction An Internet Draft's life cycle begins when the author(s) submit the document as an individual submission, and usually ends when the draft is published as an RFC, or is abandoned and eventually expires. During its lifetime, a draft is likely to receive review comments from a number people. In our experience, inefficient handling of review comments is one big source of delay and frustration in the IETF standards process. The goal of this document is to help document authors in dealing with reviews in effective manner. 2. Sources of reviews An internet draft gets reviewed at different stages of its lifecycle by different sets of people. When the draft is in the initial stages it gets reviewed mostly by the members of the working group that the draft targets. But as the draft progresses, it gets reviewed by a much more varied set of people with differing fields of expertise. These reviews are performed with a very different point of view than the working group reviews and are likely to bring out cross-area issues. The different sources from which the author can receive reviews are o Working Group participants: o Other IETF participants -- especially during IETF last call o Review from the area specific directorates. Some directorates try to review all documents -- these include Gen-ART (General Area Review Team) assisting the General Area Director, and SecDir (Security Directorate) assisting the Security Area Directors. Other directorates are more specialized: for example, MIB Doctors review only MIB documents. o Reviews from IESG members: after IETF last call, o Reviews from the IAB 3. Guidelines for authors 3.1. Who responds to reviews There are several persons who can respond to a received review. If the document is an individual document, it is customary for the author to respond to the review. If the document has several authors, they can co-ordinate among themselves when they respond. Otherwise the reviewer may receive conflicting responses from different authors. If the document is a working group document, Krishnan, et al. Expires November 26, 2008 [Page 3] Internet-Draft IETF Review Guidelines May 2008 ideally the working group chairs or the document shepherd should coordinate things, since the author and reviewer cannot unilaterally decide to make substantial changes. 3.2. Before reading the review Receiving a long review, or a critical review for your favorite document can be disheartening. Before reading the review -- or at the very least, before starting to reply -- it's good to remind yourself about a crucial difference between Internet Draft reviews, and say, reviews about books or movies. A good movie review attempts to give a balanced view: if the reviewer liked the movie, the review will say good things about it. However, a typical review of an Internet Draft will comment only the parts that the reviewer did not like and wants to see changed. The reviewer may well think that your work is excellent quality, and a valuable contribution to Internet engineering -- but it is not common to say this in a review. Thus, when you start reading a review, imagine that the review started with a positive comment -- after all, the reviewer considered this important enough to review it! Hint for reviewers: it helps if you start your review with a positive overall opinion. Although it may not seem so, there's a big difference between saying "The following things have to be fixed:" and "This is a well-written and valuable document; however, I'd like to propose some minor improvements for the following things:" 3.3. Responding to reviews Reviewers spend a non-trivial amount of time in performing a review, and reviews also play an important part in how IETF determines consensus. Thus, the single most important rule for handling reviews is the following: every review merits a response. Even if the review said "everything looks ok", saying "thank you for reviewing this" is part of common courtesy. If the reviewer raised issues, a response is deserved, irrespective whether the authors agree with the reviewer or not. 3.3.1. Timeframe for response The authors are expected to respond to the reviews within a reasonable amount of time. While there is no hard and fast rules concerning this period, two or three business days might be a reasonable amount of time to take for a response. It is possible Krishnan, et al. Expires November 26, 2008 [Page 4] Internet-Draft IETF Review Guidelines May 2008 that the authors may be unavailable to respond to a review within this timeline since they may be sick/on vacation/busy with dayjob etc. In this case, the document shepherd (if any) can respond to the review and follow up with the authors at a later time. 3.3.2. Contents of a response 3.3.2.1. The initial response The initial response that the author sends might be as simple as a single mail that acknowledges the receipt of the review and the intent of the author to look into the contents of the review. It would be nice for the author to come up with a timeline on when he/ she would be able to respond to the comments/issues contained in the review. 3.3.2.2. The substantial response In the substantial response the author tries to address the concerns raised in the review. There are several ways to do this. o Accept the concern as valid and propose a solution o Accept the concern as valid and propose a timeline for a solution o Accept the concern as valid and solicit text from the reviewer for a solution o Ask for more clarifying information to understand the concern o Dismiss the concern as invalid (out of scope, already covered in the document, misunderstanding etc.) o Any combination of the above The substantial response needs to be sent at least to the reviewer, the relevant working groups and the mailing list of the reviewing body (if any). It can also be sent to the other recipients of the original review. 3.3.2.3. Common issues with the substantial response This section tries to document some common mistakes that authors make while responding to a review. o If the reviewer has not understood some part of the draft, a very common response is to explain the topic in email. However, often that explanation should really be in the draft itself, not just in emails to the authors, so that future readers will also understand the text. o Soliciting text from the reviewer should be used sparingly. It can be helpful, if the authors, even after asking for more information, do not understand the reviewer's concern. However, Krishnan, et al. Expires November 26, 2008 [Page 5] Internet-Draft IETF Review Guidelines May 2008 if the author understands the concern, it is primarily his/her job to propose text. o Simply accepting the concern as valid is a response that is often seen, but is really good only when the solution is totally obvious (fixing a typo, etc.). For complex topics, saying "I'll change this in the next draft version" is not very good, since it makes life difficult for the reviewer to verify the fix. 3.4. Keeping track of the issues When the author makes an attempt in good faith to resolve all the issues raised by the reviewer, it is possible that some of the issues are left unresolved in the revised version of the draft. In order to prevent this from happening, it is useful to keep a systematic record of the issues and the associated resolutions. One common way of doing this is buy using an issue tracker. The IETF tools team provides an issue tracker on request to any of the working groups and the chairs can add authorized users for the tracker. When the issues are raised by the reviewer, the author can open the issue on the issue tracker and close it when it is resolved. The author can also add the suggested resolution text into the tracker. This way both the author and the reviewer can keep tabs on the change process without missing out any issues. For a document in Working Group Last Call or IETF wide Last Call it is considered good practice for the working group chair or the document shepherd to post a summary of all the comments received during the last call, and current proposal for handling them. Even if the working group chair or shepherd does not do this, it's highly recommended that the author does this.This summary could include links to mailing list archives, or if an issue tracker is used, issue numbers or links to "tickets". 3.5. New version of the draft One the author feels that all the open issues have been addressed she can submit a new revision of the draft that incorporates the changes. It is considered common courtesy to inform the reviewers that a new version of the draft is available. It is desirable for the author to summarize the list of changes in the new version of the draft either in the mail informing the reviewers or in a "Changes since version X" section of the draft. This makes it easier for the reviewers to go through just the changes instead of rereading the complete draft. Alternately or additionally, the rfcdiff tool can be used to identify the changes made in the latest version of the draft and the output of this tool can be provided to the reviewer. After the author and reviewer agree that the issues have been fixed, Krishnan, et al. Expires November 26, 2008 [Page 6] Internet-Draft IETF Review Guidelines May 2008 for working group documents, substantial issues need to be confirmed on the mailing list. Once the document has entered IESG processing, new versions should not be posted unless the document shepherd or AD explicitly says so. 3.6. Acknowledging the reviewer If the reviewer has contributed substantially to the improvement of the draft it is considered polite to add the reviewer's name to the Acknowledgements section of the draft, if there is one. There is no hard and fast rule about when to acknowledge a reviewer and hence the author should use her own judgement to decide whether or not to do this. 4. Guidelines for reviewers 4.1. Level of review The level of review performed on the draft varies greatly based on the source of the review and on the stage of the document lifecycle. In the earlier stages of the lifecycle, the document is usually reviewed by the members of the relevant working group(s) only. Let's call these reviews "early reviews". At this point the comments on the draft are deeply technical and mainly remove obvious errors from the draft. The reviewer and the author are usually from similar backgrounds and are able to understand the underlying technologies and jargon in a consistent fashion. The reviews that occur after the draft has been progressed from the working group have a very different perspective on the contents of the draft. They explore very different aspects of the draft such as security aspects, operational aspects, readability etc. that are not usually on the top of the list of priorities of the authors and the early reviewers. They also explore how the mechanisms proposed in the document fit into the Internet architecture as a whole and what detrimental effects they will have, if any. Let's call these "late reviews". The person performing the review should consider this when the review is being performed and concentrate her efforts on the right parts of the document. 4.2. Prioritization of the issues A review results in a list of issues or requested clarifications. The reviewer can choose to list them in the order they appear in the document. Alternately, and preferably, the reviewer can use another classification scheme where the issues are grouped together by their severity. This makes it easier for the eventual recipients of the review to attach the appropriate importance for resolving these Krishnan, et al. Expires November 26, 2008 [Page 7] Internet-Draft IETF Review Guidelines May 2008 issues. If the review is sent on behalf of a reviewing body this also helps the beneficiary of the review in taking a position on the document. e.g. The reviewer can classify the issues into the following categories o Major o Minor o Editorial o Nits 4.3. Reviewing changes Once the authors submit a new revision of the draft, the reviewer can look over the new revision to see if the agreed changes have been made to the draft. If the reviewer finds out that some changes have not been made, he/she can alert the authors of this fact. There are tools available that can make the reviewer's life easier in this regard. The rfcdiff tool can be used to identify the changes made in the latest version of the draft. The reviewer can just look over these changes instead of rereading the entire draft. The reviewers can also keep track of issues using the issue tracker used by the authors (if one was used). If the reviewer is satisfied with the changes, he/she can send out a mail to the original recipients of the review mentioning this. If not, a new generation of the review cycle is started and the steps described earlier are redone. 5. Acknowledgements The authors would like to thank Bernard Aboba and Thomas Narten for their contributions to this document. 6. IANA Considerations This document does not require any action from the IANA. 7. Security Considerations This document does not create any new security issues. 8. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. Krishnan, et al. Expires November 26, 2008 [Page 8] Internet-Draft IETF Review Guidelines May 2008 Authors' Addresses Suresh Krishnan Ericsson 8400 Decarie Blvd. Town of Mount Royal, QC Canada Phone: +1 514 345 7900 x42871 Email: suresh.krishnan@ericsson.com Pasi Eronen Nokia Research Center P.O. Box 407 FI-00045 Nokia Group Finland Email: pasi.eronen@nokia.com Eric Gray Ericsson 900 Chelmsford Street Lowell, MA USA Email: eric.gray@ericsson.com Krishnan, et al. Expires November 26, 2008 [Page 9] Internet-Draft IETF Review Guidelines May 2008 Full Copyright Statement Copyright (C) The IETF Trust (2008). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Krishnan, et al. Expires November 26, 2008 [Page 10]