Network Working Group John Loughney INTERNET-DRAFT Nokia Category: Informational Bernard Aboba Microsoft 23 June 2003 A Comprehensive Approach to Quality (COACH) This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC 2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. Copyright Notice Copyright (C) The Internet Society (2003). All Rights Reserved. Abstract This document describes an approach to improving the quality of IETF documents which is based on quality plans developed by Working Groups, and approved by the Area Director. This document describes the components of a quality plan, as well as a list of materials which need to be compiled in order to assist Working Groups in developing their plans. 1. Introduction This document describes an approach to improving the quality of IETF documents that centers around the development and execution of Working Group quality plans. Since in the IETF Working Groups are responsible for completing work, they are also accountable for the quality of that work. If the quality of IETF documents is to improve, the Working Loughney, et al. Informational [Page 1] INTERNET-DRAFT COACH 23 June 2003 Groups must plan for, and carry out, that quality improvement. This document describes how Working Group quality plans can be used to assist in that effort - and what materials the IETF needs to put together in order to assist Working Groups in developing their plans. The Working Group quality plan is a public document no more than 5 pages long that is developed in concert with the Working Group charter and is made available on the IETF web site. The quality plan lays out the challenges that the Working Group faces, and describes how it plans to overcome those challenges in order to deliver documents of high quality. The plan includes pointers to processes, tools, and templates that will assist in the effort, as well as the plan for assessing the quality of work at several checkpoints. Working Group quality plans are made publicly available so as to encourage sharing of best practices. Since each Working Group is different, there is no "one size fits all" quality plan, although it is expected that over time, as Working Groups share their plans with each other, that best practices will be more widely adopted. Typically, the quality plan will be revised when the Working Group Charter or schedule changes significantly. For example, if the Working Group has fallen badly behind in its schedule, then a revised Challenge Assessment may be required in order to determine where the Working Group resources have fallen short of those required to complete the work. 2. The quality plan The Working Group quality plan includes (but is not limited to) the following: [a] Working Group Charter. The Working Group Charter is a summary of what the Working Group plans to do, who will do it, and by when. It includes a description of the documents that the Working Group plans to produce, the individuals who have signed up to do the work, and the schedule for when the work will be completed. Since the Charter is a summary of the goals and objectives, as well as a description of the team, it is the foundation for the Working Group quality plan. [b] Challenge Assessment. This part of the quality plan describes the challenges that the Working Group faces in accomplishing the work outlined in the charter. For example, this part of the plan describes the technical issues that need to be solved in order to complete the work, the estimated size of the documents that are to be produced, the documents on which the work depends, and the areas of knowledge, skill or experience in which the working group is Loughney, et al. Informational [Page 2] INTERNET-DRAFT COACH 23 June 2003 weak. In general, it is expected that the Challenge Assessment will incorporate feedback from the Area Director and the IETF community, so as to ensure that the Working Group has a realistic view of the difficulty of the task. [c] The Tools Plan. This part of the plan describes the tools that the Working Group intends to use in order to manage the work. This includes the Working Group mailing list and web site; issue tracking and reporting tools; document revision control systems; and document production and build environments. [d] The Review plan. This part of the quality plan describes the checkpoints at which the work will be reviewed, the review process to be used, and how the reviewers will be recruited. The elements of the Working Group quality plan are discussed in more detail in the sections that follow. 2.1. Challenge Assessment The Working Group Charter is the statement of what the Working Group is attempting to achieve, as well as the resources at its disposal. The Charter is essentially a promise to deliver (the goals) and a down payment on that promise (the resources). The question is whether the promise is realistic or not, and whether the down payment is sufficient collateral against default. The purpose of the Challenge Assessment is to provide a realistic assessment of what is going to be necessary to deliver on the promises that the Working Group has made - with high quality. There are many reasons why Working Groups deliver poor quality documents. However, in most cases where poor quality documents are produced, it is unlikely that the plan was to deliver poor quality documents all along. Rather, it is more likely that the level of effort, skill and experience required to produce high quality documents was beyond the capability of the Working Group participants, and that poor quality was the result, not of bad intentions, but of insufficient resources. At the start of a project, it is easy to underestimate what will be required to finish the job. This seems particularly true in the IETF, where industries go from birth to maturity, and where individuals change jobs, change their marital status, buy or build a house, go through a mid-life crisis (or two), have children, or go trekking in Nepal, all in the time it takes to bring an initial Internet-Draft to the point where it is approved for publication as an RFC. Loughney, et al. Informational [Page 3] INTERNET-DRAFT COACH 23 June 2003 Often the group that is least able to estimate the resources required to finish the proposed work are the core Working Group participants - if only because a certain degree of naive optimism is required to start the process in the first place. The purpose of the Challenge Assessment is not to destroy that naive optimism, but rather to augment it with an external assessment of whether the resources identified in the Charter fall short of those required to complete the work successfully. An important element of the Challenge Assessment is a review of the extent to which the charter goes beyond the current state of the art. For example, are the security problems likely to be encountered ones which can be solved by the application of existing security technologies? Or are there unique aspects of the security problem for which existing techniques are inapplicable? Does the Charter require advances in research as well as engineering? To a large degree, the Challenge Assessment focuses on risks outside the core area of expertise of the Working Group. In general, IETF Working Groups tend to do a reasonable job within their area, but weaknesses, where they arise, are often the result of missing expertise in other areas. For example, a Working Group in the Security Area will tend handle security issues well, but may stumble over transport issues; an Internet Area Working Group will tend to handle things correctly at the IP layer, but may mishandle security; a Routing Area Working Group will tend to properly handle routing, but may have difficulty with Operations and Management. The goal of the Challenge Assessment is to garner feedback from the IETF community as a whole, in order to anticipate challenges that will need to be overcome in order to complete the proposed work successfully. Ideally, the Challenge Assessment will answer the question: "what does the Working Group need to do in order to have a high propability of completing the proposed work on schedule and with high quality?" As a result of the Challenge Assessment, the Working Group charter and quality plan may be modified so as to include additional resources in certain areas. For example, if security issues are anticipated, then security reviews may be added to the quality plan, and a security advisor may be named as part of the Working Group Charter. If production of large and complex documents is anticipated, then the Tools plan will include use of sophisticated issue tracking and document revision control systems, and the Working Group charter will include additional editors with experience producing documents of that complexity. Loughney, et al. Informational [Page 4] INTERNET-DRAFT COACH 23 June 2003 In addition to assessing the challenges that need to be overcome, the Challenge Assessment will also identify key assumptions that if wrong, would require the quality plan to be revised. For example, a key assumption might be that transmission layer security (IPsec, TLS) will be sufficient to address the security issues. If that assumption were to be invalidated, then the significantly greater security resources would be required - and the plan will need to be revised to take this into account. 2.2. Tools Plan The purpose of the Tools Plan is to identify the tools with which the work will be carried out. This includes: [a] The technology and host for the Working Group mailing list. While every Working Group has a mailing list, mailing list maintenance has become increasingly complex as issues of spam control and subscription spoofing have arisen. As a result, these and other issues need to be taken into account in selecting how the mailing list is run. [b] The Working Group web site. Many Working Groups now use their own web sites in order to track issues and provide access to work in progress in formats not available on the IETF archive, such as XML or HTML. In some cases, these web sites become integral parts of the document production environment. The quality plan should include a section on the plan for the web site. Ideally, the web site will be hosted in a location that would not be disrupted by a change in a participant's home address (home DSL or Cable links have proven to be particularly vulnerable in this regard). [c] The document production environment. Over the years, editors often develop their own document production environments. These can vary from the quick and dirty, suitable for the production of small documents, to comprehensive environments that handle the building of complex documents in multiple formats (e.g. text, HTML, XML, etc.) as well as generation of difference files and propagation of documents to the Working Group web site. Since each document is different, there is no single right answer to the problem of document production - but it does seem likely that Working Groups can benefit by sharing sample build environments and templates and documenting best practices. [d] The revision control system. Not every Working Group will need to maintain a revision control system, but where this is useful it is probably a good idea to describe how revision control will be Loughney, et al. Informational [Page 5] INTERNET-DRAFT COACH 23 June 2003 handled. [e] The issue tracking and reporting system. The use of issue tracking tools has had a substantial impact on Working Group productivity and document quality in a number of Working Groups. By focusing effort on the identification and resolution of issues, tracking tools encourage discussions that lead to document improvements. In addition, tracking tools make it possible to avoid rehashing old arguments and enable the quantitative measurement of progress towards a goal. Given the increasing popularity of document tracking tools, Working Groups are now experimenting with a number of tools, and as part of the quality plan, it is helpful for the chosen tools to be described along with the reasoning behind the selection and pointers to relevant templates. As Issue Tracking tools gain in popularity, use of reporting tools is also likely to become more common. Reporting tools are essential for quality improvement, since they supply the metrics by which progress can be gauged. Use of quality metrics is not yet widespread with the IETF, but where they are used, they should be discussed within the quality plan. 2.3. Review Plan The Review plan includes (but is not limited to) a description of the plans for the following: [a] Challenge Assessment review. This is plan for the challenge assessment review, which is based on the Working Group Charter. Ideally this plan should include reviewers from outside the area of the Working Group. [b] Work item review. In some cases, the Working Group may wish to review documents before they become work items in order to identify issues up front. If this is done, then the process by which these reviews are conducted should be described. [c] Midterm assessment. This is a review conducted when the document is mid-way through its life. [d] Late review. This is a review conducted late in the process, within a few months of the start of Working Group last call. As part of the review plan, the sources of reviewers should be identified. This could be members of the relevant directorates, noted experts, members of the IAB or IESG, volunteers in the process described in [SIRS], etc. Loughney, et al. Informational [Page 6] INTERNET-DRAFT COACH 23 June 2003 3. Post Mortems In order to increase the rate of quality improvement, it is important for Working Groups to assess their quality plans after the fact. Prior to dissolution, Working Groups are encouraged to produce a post mortem no longer than three pages including the following sections: [a] Quality assessment. This is an evaluation of the quality of working group output, written by the Area Director. [b] Challenge Assessment. This covers the unforseen challenges that the Working Group encountered. [c] Tools assessment. This is an evaluation of the tools that were used and how well they worked. [d] Reviews assessment. This is an assessment of the review process. [e] Recommendations. This section includes recommendations to future working groups looking to avoid similar problems. Post mortems will be made publicly available along with the working group quality plans. 4. Required materials In order to assist Working Groups in putting together their quality plans, the following materials would be helpful: [a] An archive of sample document build environments and templates. [b] A website covering issue tracking and reporting tools. [c] A document on use of tracking tools for document management, including metrics and reporting. [d] A document on the review process, including Challenge Assessment. [e] A website including sample quality plans and post-mortems. 5. Normative References [SIRS] Carpenter, B. and D. Crocker, "Careful Additional Review of Documents (CARD) by Senior IETF Reviewers (SIRS)", draft-carpenter-solution-sirs-01.txt, Internet draft (work in progress), June 2003. Loughney, et al. Informational [Page 7] INTERNET-DRAFT COACH 23 June 2003 Acknowledgments The authors would like to acknowledge Margaret Wasserman and the members of the IETF Problem Working Group for discussions relating to document quality. Authors' Addresses John Loughney Nokia It204merenkatu 11-13 00180 Helsinki Finland Email: john.loughney@nokia.com Bernard Aboba Microsoft Corporation One Microsoft Way Redmond, WA 98052 EMail: bernarda@microsoft.com Intellectual Property Statement The IETF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on the IETF's procedures with respect to rights in standards-track and standards- related documentation can be found in BCP-11. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementors or users of this specification can be obtained from the IETF Secretariat. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this standard. Please address the information to the IETF Executive Director. Loughney, et al. Informational [Page 8] INTERNET-DRAFT COACH 23 June 2003 Full Copyright Statement Copyright (C) The Internet Society (2003). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Expiration Date This memo is filed as , and expires December 22, 2003. Loughney, et al. Informational [Page 9]