Network Working Group K. Whistler Internet-Draft R. McGowan Expires: December 20, 2001 Unicode June 21, 2001 Unicode Transformation Format Seventeen draft-whistler-utf17-00.txt Status of this Memo This document is an Internet-Draft and is in full conformance with all provisions of Section 10 of RFC2026. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on December 20, 2001. Copyright Notice Copyright (C) The Internet Society (2001). All Rights Reserved. Abstract This memo describes a new transformation format that is much easier to read. UTF-17 converts each Unicode code point to a sequence of 1 synchronizing byte followed by 7 further bytes, for a total of 8 bytes per character. Whistler & McGowan Expires December 20, 2001 [Page 1] Internet-Draft UTF-17 June 2001 1. The Problem To date no one has proposed a suitable UTF that would work well with 64-bit architectures, but which would also be compatible with 8-bit string API's and streaming protocols. This oversight has the potential for significantly impeding the migration of UNICODE [1] software to the new generation of 64-bit machines. What the IT industry needs is something comparable to UTF-8 [2], but which could also function directly as a wide character as well in 64- bit contexts. Whistler & McGowan Expires December 20, 2001 [Page 2] Internet-Draft UTF-17 June 2001 2. The Solution To address this problem, this memo describes a new transformation format, UTF-17. UTF-17 converts each Unicode code point to a sequence of 1 synchronizing byte followed by 7 further bytes, for a total of 8 bytes per character. Each code point in the range 0..10FFFF is treated as a 21-bit integer, and the 21 bits are distributed according to the following formula: x xxxx xxxx xxxx xxxx xxxx ==> 00111000 00110xxx 00110xxx 00110xxx 00110xxx 00110xxx 00110xxx 00110xxx In UTF-17, for example, the Han character sequence ('17'), would be converted to: <38 30 31 31 31 36 30 31 38 30 31 30 37 30 30 33> Because all UTF-17 bytes are in the range 0x30..0x38, this UTF-17 byte sequence would also be visible displayed in ASCII (or Latin-1) as: "8011160180107003". Whistler & McGowan Expires December 20, 2001 [Page 3] Internet-Draft UTF-17 June 2001 3. Special Treatment of NULL for C Compatibility One special exception is provided. U+0000 is transformed into UTF-17 with 00000xxx as the pattern for the 8th byte, rather than 00110xxx. Thus U+0000 has the unique representation: <38 30 30 30 30 30 30 00> (or "8000000'\0'") This is for C compatibility, so that any null-terminated Unicode string will also be null-terminated in UTF-17. Whistler & McGowan Expires December 20, 2001 [Page 4] Internet-Draft UTF-17 June 2001 4. Benefits of UTF-17 UTF-17 is self-synchronizing, since it has a unique lead byte for each byte sequence, and all trail bytes start with a separate bit sequence. UTF-17 is fixed-width, and so could be implemented as a wchar_t processing code on new 64-bit systems (the wave of the future). UTF-17 is highly patterned, and would be easy to auto-identify for any charset converter. UTF-17 is easy to calculate, even for the hex-impaired, as only 8 bit combinations need be remembered, and they correspond directly to the second digits of the ASCII 0..7 as hex codes. UTF-17 will interoperate easily with UTF-64. UTF-17 has the same binary ordering as the Unicode characters themselves, as well as the same binary ordering as for UTF-8, UTF-32, and UTF-64. This makes possible multi-tiered applications, where server, app server, and client may not all share the same UTF, but require that binary-sorted lists be in the same order, to facilitate searching in those lists. UTF-17 is compatible with ASCII, as long as you avoid digits in your ASCII text. Since all UTF-17 bytes display as digits, it is programmer friendly and much easier to read than any other UTF. All UTF-17 values will display visibly and correctly in any debugger, and the programmer need only recall that "80111601" means U+5341, for instance, to get back to the original Unicode character. UTF-17 code units are a strict subset of the POSIX portable character set, and as such should work on all platforms. Furthermore, since they are also a subset of ASCII alphanumerics, they should work seamlessly with most Internet protocols. For example, UTF-17 could be used immediately to solve the IDN problem -- at least for domain names no longer than 8 characters in length. Whistler & McGowan Expires December 20, 2001 [Page 5] Internet-Draft UTF-17 June 2001 5. Potential Drawbacks of UTF-17 It is true that UTF-17 takes up twice the space of UTF-32, but with 64-bit machines and the continuing rapid progress in the lowering of cost/megabyte of storage, this should not really be a barrier to the rapid acceptance of UTF-17. The name "UTF-17" might mislead people into thinking it deals with 17-bit code units, which would clearly be absurd for modern computer architecture. This drawback could conceivably be addressed by changing the name to UTF-1+7 instead, which would mnemonically indicate the correct number of bits (1+7=8) for a code unit, the correct number of bytes (1+7=8) for a character representation, and the pattern (1+7) of lead and trail bytes used in the transformation. Whistler & McGowan Expires December 20, 2001 [Page 6] Internet-Draft UTF-17 June 2001 References [1] Unicode, Inc., "The Unicode Standard Version 3.0", January 2000. [2] Yergeau, F., "UTF-8, a transformation format of ISO 10646", January 1998. Authors' Addresses Kenneth Whistler Unicode, Inc. P.O. Box 391476 Mountain View, CA 94039-1476 US Phone: +1 650 693 3921 URI: http://www.unicode.org/ Rick McGowan Unicode, Inc. P.O. Box 391476 Mountain View, CA 94039-1476 US Phone: +1 650 693 3921 URI: http://www.unicode.org/ Whistler & McGowan Expires December 20, 2001 [Page 7] Internet-Draft UTF-17 June 2001 Full Copyright Statement Copyright (C) The Internet Society (2001). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the Internet Society or other Internet organizations, except as needed for the purpose of developing Internet standards in which case the procedures for copyrights defined in the Internet Standards process must be followed, or as required to translate it into languages other than English. The limited permissions granted above are perpetual and will not be revoked by the Internet Society or its successors or assigns. This document and the information contained herein is provided on an "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Acknowledgement Funding for the RFC Editor function is currently provided by the Internet Society. Whistler & McGowan Expires December 20, 2001 [Page 8]