Problem Statement E. Davies (ed.) Internet-Draft Nortel Networks Expires: November 8, 2003 May 10, 2003 IETF Problem Statement draft-ietf-problem-issue-statement-01.txt Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http:// www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on November 8, 2003. Copyright Notice Copyright (C) The Internet Society (2003). All Rights Reserved. Abstract This memo summarizes perceived problems in the structure, function and processes of the Internet Engineering Task Force (IETF). We are attempting to identify these problems, so that they can be addressed and corrected by the IETF community. The problems have been digested and categorized from an extensive discussion which took place on the 'problem-statement' mailing list from November 2002 to May 2003. The problem list has been further analyzed to try and determine the root causes, that are at the heart of the perceived problems: The result will be used to guide the next stage of the process in the Problem Statement working group which is to determine the structures and processes that will carry out the corrections. Davies (ed.) Expires November 8, 2003 [Page 1] Internet-Draft IETF Problem Statement May 2003 Table of Contents 1. Introduction: Issues/Problems in the IETF process . . . . . 3 1.1 Consequences of Past Growth . . . . . . . . . . . . . . . . 4 1.2 The Aim is Improvement, not Finger-pointing . . . . . . . . 4 1.3 Perception and not Consensus . . . . . . . . . . . . . . . . 4 1.4 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 5 1.5 Major Changes between Versions 00 and 01 . . . . . . . . . . 5 2. Root Cause Problems . . . . . . . . . . . . . . . . . . . . 7 2.1 Participants in the IETF do not Share a Common Understanding of its Mission . . . . . . . . . . . . . . . . 7 2.2 The IETF does not Consistently use Effective Engineering Practices . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 The IETF has Difficulty Handling Large and/or Complex Problems . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 The IETF's Workload Exceeds the Number of Fully Engaged Participants . . . . . . . . . . . . . . . . . . . . . . . . 10 2.5 The IETF Management Structure is not Matched to the Current Size and Complexity of the IETF . . . . . . . . . . 10 2.5.1 Span of Authority . . . . . . . . . . . . . . . . . . . . . 11 2.5.2 Procedural Blockages . . . . . . . . . . . . . . . . . . . . 11 2.5.3 Consequences of Low Throughput in IESG . . . . . . . . . . . 11 2.5.4 Lack of Formal Recognition outside IESG and IAB . . . . . . 12 2.5.5 Concentration of Influence in Too Few Hands . . . . . . . . 12 2.6 Working Group Practices can make Issue Closure Difficult . . 12 2.7 IETF Participants and Leaders are Inadequately Prepared for their Roles . . . . . . . . . . . . . . . . . . . . . . 13 3. Security Considerations . . . . . . . . . . . . . . . . . . 14 4. IANA Considerations . . . . . . . . . . . . . . . . . . . . 15 Normative References . . . . . . . . . . . . . . . . . . . . 16 Illustrative References . . . . . . . . . . . . . . . . . . 17 Author's Address . . . . . . . . . . . . . . . . . . . . . . 17 Intellectual Property and Copyright Statements . . . . . . . 18 Davies (ed.) Expires November 8, 2003 [Page 2] Internet-Draft IETF Problem Statement May 2003 1. Introduction: Issues/Problems in the IETF process Discussions over the last few months have revealed a significant number of perceived problems with the way the Internet Engineering Taskforce (IETF) operates. In advance of trying to change the IETF procedures and rules to deal with these problems, the IETF should have a clear, agreed-upon description of what problems we are trying to solve. The Problem Statement working group was chartered to create this document, which contains the description of the problems, and to use this analysis to suggest processes to address the identified problems. Taken in isolation, this document may appear to be exceedingly negative. Clearly the IETF needs to refresh its management structure and processes to address today's challenges, but it should not be forgotten that the IETF has produced a large body of high quality work which has lead to an extremely successful, all-pervasive network infrastructure. Against this background, we should see the current document as a necessary piece of self-criticism leading to renewal and continued success. The discussion of the positive aspects has been deliberately confined to the IETF Problem Resolution Processes document [4] which considers the core values that the IETF needs to maintain whilst correcting the problems that participants perceive as affecting the IETF at present. The raw material for this document was derived by summarizing the extensive discussions which took place initially on the 'wgchairs' mailing list and subsequently on the 'problem-statement' mailing list from November 2002 through to May 2003, incorporating additional input from relevant drafts published during this period (see [1], [2] and [3]) and the minutes of recent plenary discussions. This produced a, still extensive, list of perceived problems which were classified into a number of related groups using a classification suggested by the processes which go on in the IETF. This document has digested these perceived problems into a small set of root cause issues, and a short list of subsidiary issues which appear to be the most pressing items engendered by the root cause. This list is set out in Section 2. Section 1.1 gives a short explanation of the thinking that has taken place in coming to the current view of the root causes. The original summary of perceived problems has been posted to the Problem Statement Working Group mailing list so that it can be referred to in future. Note that it remains classified according the Davies (ed.) Expires November 8, 2003 [Page 3] Internet-Draft IETF Problem Statement May 2003 original scheme so that the raw data is available if alternative root cause analysis is needed. 1.1 Consequences of Past Growth As the problems of the IETF were examined, it became clear that they are neither new nor are they symptoms of a problem which is novel in the science of organizations. The IETF started off as a small, focused organization with a clearly defined mission and participants who had been working in this area for a significant period of time. Over the period 1989-1999 the IETF grew by a factor of ten or more in terms of number of participants, and in terms of work in progress. The effects of this growth have been compounded by the extension of the scope of the IETF which makes the work much more varied. Many of the problems and symptoms appear to be fundamentally caused by the organization failing to fully adapt its management structure and processes to its new larger size, and failing to clearly define its future mission once the initial mission had been completed or outgrown. These failures are just those that afflict many small organizations trying to make the transition from a small organization which can be run informally and where essentially all participants fully share the aims, values and motivations of the leadership, to a medium sized organization where there are too many participants for informal leadership and later arrivals either do not fully understand or have a different perception of the ethos of the organization. Some IETF participants have been aware of these issues for a long time. Extant evidence dating back to at least 1992 drew similar conclusions. 1.2 The Aim is Improvement, not Finger-pointing Many of the problems identified in this memo have been remarkably persistent over a 15-year period, surviving a number of changes in personnel. We see them as structural problems, not personnel problems. No blame for any of the perceived problems should be directed to any individual, and the sole aim of the review process is to identify how the IETF can improve itself so that it knows what it is about and becomes fit for that purpose in the shortest possible timeframe. 1.3 Perception and not Consensus The working group participants would like emphasize that both the long list of problems and the root cause issues that we have derived Davies (ed.) Expires November 8, 2003 [Page 4] Internet-Draft IETF Problem Statement May 2003 from them are problems that have been *perceived* to exist by a significant constituency, either on the mailing list and/or in private discussions. We also note that many of these problems appear to be of long standing, as a very similar list has survived from the discussions that took place prior to the changes documented in the Kobe agreement from 1992. We, in line with many contributors to the mailing list, do not believe that the process of problem resolution will be helped by continued rework of the root issues in what would probably be a vain attempt to achieve any sort of consensus. Instead, the general tenor and scope of the problems identified will provide a guide in setting up the processes needed to resolve the problems and provide input to the resolvers. 1.4 Acknowledgements Apart from the contributions of all those who provided input on the problem statement mailing list, the final reduction of the problems was especially assisted by the following people: Avri Doria (WG co-chair) Dave Crocker Jeanette Hoffmann Marc Blanchet Margaret Wasserman Melinda Shore (WG co-chair) Rob Austein . Spencer Dawkins Special thanks are due to Margaret Wasserman for extensive reviewing and contributions to the wording of Section 2. The early part of the reduction of the problem statement mailing list input was done by Harald Alvestrand and the latter part by Elwyn Davies and the team acknowledged above. There were a very large number of extensive and thoughtful contributions (some making several points). The thread was started by a call for volunteers in helping draft a problem statement, but quickly turned into a discussion of what the problems were. 1.5 Major Changes between Versions 00 and 01 o Section 1.3 added to clarify intentions of document. o Section 2.3 added to document additional root cause involving IETF's difficulties with large and/or complex problems. o Section 2.5 rewritten in line with mailing list comments. Davies (ed.) Expires November 8, 2003 [Page 5] Internet-Draft IETF Problem Statement May 2003 o Definition of SDO corrected - "Defining" rather than "Determining". o Appendix A removed and posted to mailing list to ensure survival, pending possible conversion into a separate Informational RFC. o Version 00 was perceived as excessively negative in tone. This was particularly true of the section headings in Section 2 which were resolutely absolutist and gave hostages to fortune in the form of neat sound bites readily accessible to detractors looking for ready ammunition. Consequently, the language in these headings has been modified. Additionally, some words have been added to the introduction noting the past successes of the IETF, pointing to the analysis of core values in the companion 'processes' draft [4] and positioning this document as a piece of vital self-criticism presaging effective renewal and rebirth. Davies (ed.) Expires November 8, 2003 [Page 6] Internet-Draft IETF Problem Statement May 2003 2. Root Cause Problems This section forms the heart of this analysis, and lists the issues which we believe lie at the core of the problems. Apart from the first issue which is fundamental, the problems are not necessarily in priority order, but they will be seen to be interlinked in various ways. 2.1 Participants in the IETF do not Share a Common Understanding of its Mission The IETF lacks a clearly defined and commonly understood Mission: As a result the goals and priorities for the IETF as a whole and any Working Groups (WGs) that are chartered are also unclear. The lack of a common mission has many consequences, of which the principal ones appear to be: o The IETF is unsure what it is trying to achieve and hence cannot know what its optimum internal organization should be to achieve its aims. o The IETF cannot determine what its 'scope' should be, and hence cannot decide whether a piece of proposed work is either in-scope or out-of-scope. o The IETF is unsure who its customers are, and consequently has managed to drive certain classes of customer, who would otherwise provide important input to the process, more or less completely out of the process. o Working Groups can potentially be hijacked by sectional interests to the detriment of the IETF's reputation. o The misty vision has restricted the associated architectural view to an outline top level view. It would be desirable to have roadmaps and architectural views for portions of work which extend beyond a single working group: It may also be the case that it is no longer possible to fit the whole Internet within a single architecture o The lack of precision regarding our goals leads to WG charters and requirements that are poorly thought out and/or not aligned with the overall architecture. The IETF creates standards and is therefore necessarily a Standards Development Organization (SDO) but many participants would like to differentiate the IETF and its way of working from the 'conventional' Davies (ed.) Expires November 8, 2003 [Page 7] Internet-Draft IETF Problem Statement May 2003 SDOs which emphasize corporate involvement and mandated delegates. Externally, the IETF is often placed in the same bracket as these conventional SDOs, especially by detractors, because the differentiation in the IETF's mission and processes and the rationale for those differences are not clear. This can lead to the IETF being shown in a poor light or communications between SDOs not being effective. 2.2 The IETF does not Consistently use Effective Engineering Practices For an organization with 'engineering' in its title and participants who are likely to trot out the statement "Trust me, I'm an engineer!" when confronted with the need to find a solution to a particularly knotty problem, the IETF has, at least in some cases, extremely ineffective Engineering Practices. The frequent inability of the IETF to deliver specifications within the timeframe that the markets need and the excessive perfectionism that is exhibited in some cases could both be improved if appropriate Engineering Practices were in use. Engineering requires appropriate trade-offs: Engineering success needs refinement only to the point of 'fitness for purpose' which should help to balance the tension between time to market and perfectionism. The use of appropriate Engineering Practices should, for example, prevent specifications being recycled in pursuit of perfection when they are already adequate improvements on the status quo. Some of the key areas where the IETF's practices appear to need tightening up include: o Lack of explicit quality auditing throughout the standards development process. o Lack of written guidelines or templates for the content of documents (as opposed to the overall layout) and matching lists of review criteria. o Poorly defined success criteria for WGs and individual documents. o Lack of criteria for determining when a piece of work is overrunning and/or is unlikely to be concluded successfully either at all or within an acceptable time frame: Lack of process for either extending the time frame, adjusting the scope or terminating the work item or the whole Working Group. o Tools to support the engineering process are minimal. Davies (ed.) Expires November 8, 2003 [Page 8] Internet-Draft IETF Problem Statement May 2003 In addition, IETF processes, and Working Group processes in particular, suffer because commonly accepted Project Management techniques are not regularly applied to the progress of work in the organization. o Project entry, goal setting and tracking processes are all either missing or implemented less effectively than the norm for commercial organizations in related activities. o Charters regularly fail to set sufficiently granular milestones at which progress of WGs, individuals and documents can be evaluated. o Better estimation procedures need to be used to determine what the expected delivery timescale for Working Group outputs should be. These estimates must be compared with the acceptable market and customer time frames for the work to be ready, and the scope of the work adjusted if timely delivery appears impossible. This process must be repeated from time to time during the project. One problem which the IETF does not appear to suffer from is excessive bureaucracy, especially written bureaucracy. It is important that any changes introduced do not significantly increase the bureaucratic load. Finally, even where the IETF does have Engineering Practices defined, there are frequently cases where they are ignored or distorted. 2.3 The IETF has Difficulty Handling Large and/or Complex Problems The IETF has historically been most successful when dealing with tightly focused problems that have few interactions with other parts of the total problem solution. IETF standardization procedures are optimized for tightly constrained working groups and are generally less effective if 'engineering in the large' is needed to reach a satisfactory solution. Engineering in the large can encompass many aspects of system design including: Architecture Frameworks Security Internationalization The IETF has historically standardized protocol components rather than complete systems but the current greater emphasis on work in the Applications area tends to require more engineering in the large. Part of the cause of this difficulty may be that the formal reporting Davies (ed.) Expires November 8, 2003 [Page 9] Internet-Draft IETF Problem Statement May 2003 structure of the IETF emphasises communication between the IESG through the ADs and the WGs and does not place much reliance on inter-WG communications: o The IETF is not consistently effective at resolving issues that cross WG or area boundaries. o The IETF does not posess effective mechanisms for inter-WG cooperation or communication. o The IETF does not have an effective means for defining architectures and frameworks that will shape the work of multiple WGs. 2.4 The IETF's Workload Exceeds the Number of Fully Engaged Participants There are a number of respects in which IETF participants and contributors appear to have become less fully engaged with the IETF processes than previously, for example: o Although there may be large attendances at many WG meetings, in many cases 5% or less of the participants have read the drafts which are under discussion or have a bearing on the decisions to be made. o Commitments to write, edit or review a document are not carried out in a timely fashion. o Little or no response is seen when a request for 'last-call' review is issued either at WG or IETF level. Whilst this might be put down to contributors having less time available in their work schedule during the downturn of the last two years, this is not the whole story because there were signs of this malaise back at the height of the boom in 2000. This problem exacerbates the problems which the IETF has had with timely delivery and may weaken the authority of IETF specifications if decisions are seen to be taken by badly informed participants and without widespread review. 2.5 The IETF Management Structure is not Matched to the Current Size and Complexity of the IETF The management and technical review processes currently in place were adequate for the older, smaller organization, but are apparently not scalable to the current size of the organization. One set of changes Davies (ed.) Expires November 8, 2003 [Page 10] Internet-Draft IETF Problem Statement May 2003 has already been made with the Internet Engineering Steering Group (IESG) taking over some of the functions of the Internet Architecture Board (IAB) as a result of the Kobe agreement in 1992, but that is now a long time ago, and the organization has undergone considerable further growth as well as extending the scope of its activities as the Internet has become more complex. 2.5.1 Span of Authority Overt authority in the IETF is concentrated in the small number of people sitting on the IESG at that time. Existing IETF processes work to funnel tasks on to this small number of people (primarily the Area Directors (ADs) in the IESG). This concentration slows up the process and puts a very large load of responsibility on to the shoulders of these people who are required to act as the senior management for Working Group (WG) chairs as well as acting as quality backstops for the large number of documents issued by the IETF. The situation has not been helped by the widening of the scope of IETF which has resulted in somewhat more WGs and a need for a very broad spectrum of knowledge within the set of ADs. 2.5.2 Procedural Blockages The current procedural rules combined with the management and quality roles of the ADs can lead to situations where WGs or document authors perceive that one or two ADs are deliberately blocking the progress of a WG document without good reason or public justification. Appeal processes in these circumstances are limited and the only sanction that could be applied to the relevant ADs is recall, which has almost always been perceived to be out of scale with the apparent offence and hence almost never invoked. This perception of invulnerability has led to a view that the IESG in general and the ADs in particular are insufficiently accountable for their actions to their WGs and the IETF at large, although the recent introduction of the Internet Draft Tracker tool makes it easier to determine if and how a document has become blocked, and hence to take appropriate steps to release it. 2.5.3 Consequences of Low Throughput in IESG If documents are inappropriately (or even accidentally) delayed or blocked as a result of IESG (in)action, this can cause much frustration inside the organization, a perception of disunity seen from outside the organization and delay of standards possibly to the point where they are too late to match market requirements: Work which has been properly authorized as being within scope of the IETF and properly quality checked during development should almost never come up against such a blockage. Davies (ed.) Expires November 8, 2003 [Page 11] Internet-Draft IETF Problem Statement May 2003 IESG delays must not be used to disguise or excuse slow work in WGs and ongoing cooperation between the ADs, WG chairs and WG members is essential to improving throughput. 2.5.4 Lack of Formal Recognition outside IESG and IAB The small number of formally recognized 'preferred' positions within the IETF, also limits the (intangible) rewards for participants. This can lead to useful and effective participants leaving because they cannot obtain any recognition (the only currency the IETF has to pay participants), which they use to fuel their own enthusiasm and help justify their continued attendance at IETF meetings to cost constrained employers. 2.5.5 Concentration of Influence in Too Few Hands Until the last couple of years, successive IETF Nominating Committees have chosen to give heavy weighting to continuity of IESG and IAB membership. Thus, the IETF appeared to have created a 'ruling class' system which tended to re-select the same leaders from a limited pool of people who had proved competent and committed in the past. Members of this 'ruling class' tend to talk more freely to each other and former members of the 'ruling class' - this may be because the 'ruling class' has also come to share a cultural outlook which matches the dominant ethos of the IETF. Newcomers to the organization and others outside the 'ruling class' are reluctant to challenge the apparent authority of the extended 'ruling class' during debates and consequently influence remains concentrated in a relatively small group of people. This reluctance may also be exacerbated if participants come from a different cultural background than the dominant one. 2.6 Working Group Practices can make Issue Closure Difficult The IETF appears to be poor at making timely and reasonable decisions that can be guaranteed to be adhered to during the remainder of a process or until shown to be incorrect. Participants are frequently allowed to re-open previously closed issues just to replay parts of the previous discussion without introducing new material. This may be either because the decision has not been clearly documented or it may be a maneuver to try to get a decision changed because the participant did not concur with the consensus originally. In either case revisiting decisions stops the process moving forward, and in the worst cases can completely derail a working group. On the other hand, the decision making process must allow discussions to be re-opened if significant new information Davies (ed.) Expires November 8, 2003 [Page 12] Internet-Draft IETF Problem Statement May 2003 comes to light or additional experience is gained which appears to justify alternative conclusions for a closed issue. 2.7 IETF Participants and Leaders are Inadequately Prepared for their Roles Participants and leaders at all levels in the IETF need to be taught the principles of the organization (Mission and Architecture(s)) and trained in carrying out the processes which they have to use in developing specifications, etc. The IETF currently has voluntary and inconsistent processes for educating its participants which may be why significant numbers of participants seem to fail to conform to the proper principles when working in the IETF context. The people in authority have generally been steeped in the principles of the IETF (as they see them) and first-time non-compliance by newer participants is sometimes treated as an opportunity for abuse rather than by recognition of a training failure. Lack of training compounded with concentration of influence in the 'ruling class' can lead to newcomers being ignored during discussions, consequently being ineffective either in their own eyes or their employers and so leaving the IETF. Davies (ed.) Expires November 8, 2003 [Page 13] Internet-Draft IETF Problem Statement May 2003 3. Security Considerations This document does not of itself have security implications, but it may have identified problems which raise security considerations for other work. Any such implications should be considered in the companion document which will be produced setting out how the IETF should set about solving the problems identified. Davies (ed.) Expires November 8, 2003 [Page 14] Internet-Draft IETF Problem Statement May 2003 4. IANA Considerations There are no IANA considerations defined in this document. Davies (ed.) Expires November 8, 2003 [Page 15] Internet-Draft IETF Problem Statement May 2003 Normative References [1] Huston, G. and M. Rose, "A Proposal to Improve IETF Productivity", draft-huston-ietf-pact-00 (work in progress), October 2002. [2] Blanchet, M., "Suggestions to Streamline the IETF Process", draft-blanchet-evolutionizeietf-suggestions-00 (work in progress), November 2002. [3] Hardie, T., "Working Groups and their Stuckees", draft-hardie-wg-stuckees-00 (work in progress), February 2003. Davies (ed.) Expires November 8, 2003 [Page 16] Internet-Draft IETF Problem Statement May 2003 Illustrative References [4] Wasserman, M., "IETF Problem Resolution Processes", draft-ietf-problem-process-00 (work in progress), May 2003. Author's Address Elwyn B. Davies Nortel Networks Harlow Laboratories London Road Harlow, Essex CM17 9NA UK Phone: +44 1279 405 498 EMail: elwynd@nortelnetworks.com Davies (ed.) Expires November 8, 2003 [Page 17] Internet-Draft IETF Problem Statement May 2003 Intellectual Property Statement The IETF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on the IETF's procedures with respect to rights in standards-track and standards-related documentation can be found in BCP-11. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementors or users of this specification can be obtained from the IETF Secretariat. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this standard. Please address the information to the IETF Executive Director. Full Copyright Statement Copyright (C) The Internet Society (2003). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assignees. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION Davies (ed.) Expires November 8, 2003 [Page 18] Internet-Draft IETF Problem Statement May 2003 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgement Funding for the RFC Editor function is currently provided by the Internet Society. Davies (ed.) Expires November 8, 2003 [Page 19]