Network Working Group K. Whistler
Internet-Draft R. McGowan
Expires: December 20, 2001 Unicode
June 21, 2001
Unicode Transformation Format Seventeen
draft-whistler-utf17-00.txt
Status of this Memo
This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.
This Internet-Draft will expire on December 20, 2001.
Copyright Notice
Copyright (C) The Internet Society (2001). All Rights Reserved.
Abstract
This memo describes a new transformation format that is much easier
to read. UTF-17 converts each Unicode code point to a sequence of 1
synchronizing byte followed by 7 further bytes, for a total of 8
bytes per character.
Whistler & McGowan Expires December 20, 2001 [Page 1]
Internet-Draft UTF-17 June 2001
1. The Problem
To date no one has proposed a suitable UTF that would work well with
64-bit architectures, but which would also be compatible with 8-bit
string API's and streaming protocols.
This oversight has the potential for significantly impeding the
migration of UNICODE [1] software to the new generation of 64-bit
machines.
What the IT industry needs is something comparable to UTF-8 [2], but
which could also function directly as a wide character as well in 64-
bit contexts.
Whistler & McGowan Expires December 20, 2001 [Page 2]
Internet-Draft UTF-17 June 2001
2. The Solution
To address this problem, this memo describes a new transformation
format, UTF-17.
UTF-17 converts each Unicode code point to a sequence of 1
synchronizing byte followed by 7 further bytes, for a total of 8
bytes per character. Each code point in the range 0..10FFFF is
treated as a 21-bit integer, and the 21 bits are distributed
according to the following formula:
x xxxx xxxx xxxx xxxx xxxx
==>
00111000 00110xxx 00110xxx 00110xxx 00110xxx 00110xxx 00110xxx
00110xxx
In UTF-17, for example, the Han character sequence <U+5341, U+4E03>
('17'), would be converted to:
<38 30 31 31 31 36 30 31 38 30 31 30 37 30 30 33>
Because all UTF-17 bytes are in the range 0x30..0x38, this UTF-17
byte sequence would also be visible displayed in ASCII (or Latin-1)
as: "8011160180107003".
Whistler & McGowan Expires December 20, 2001 [Page 3]
Internet-Draft UTF-17 June 2001
3. Special Treatment of NULL for C Compatibility
One special exception is provided. U+0000 is transformed into UTF-17
with 00000xxx as the pattern for the 8th byte, rather than 00110xxx.
Thus U+0000 has the unique representation:
<38 30 30 30 30 30 30 00> (or "8000000'\0'")
This is for C compatibility, so that any null-terminated Unicode
string will also be null-terminated in UTF-17.
Whistler & McGowan Expires December 20, 2001 [Page 4]
Internet-Draft UTF-17 June 2001
4. Benefits of UTF-17
UTF-17 is self-synchronizing, since it has a unique lead byte for
each byte sequence, and all trail bytes start with a separate bit
sequence.
UTF-17 is fixed-width, and so could be implemented as a wchar_t
processing code on new 64-bit systems (the wave of the future).
UTF-17 is highly patterned, and would be easy to auto-identify for
any charset converter.
UTF-17 is easy to calculate, even for the hex-impaired, as only 8 bit
combinations need be remembered, and they correspond directly to the
second digits of the ASCII 0..7 as hex codes.
UTF-17 will interoperate easily with UTF-64.
UTF-17 has the same binary ordering as the Unicode characters
themselves, as well as the same binary ordering as for UTF-8, UTF-32,
and UTF-64. This makes possible multi-tiered applications, where
server, app server, and client may not all share the same UTF, but
require that binary-sorted lists be in the same order, to facilitate
searching in those lists.
UTF-17 is compatible with ASCII, as long as you avoid digits in your
ASCII text.
Since all UTF-17 bytes display as digits, it is programmer friendly
and much easier to read than any other UTF.
All UTF-17 values will display visibly and correctly in any debugger,
and the programmer need only recall that "80111601" means U+5341, for
instance, to get back to the original Unicode character.
UTF-17 code units are a strict subset of the POSIX portable character
set, and as such should work on all platforms. Furthermore, since
they are also a subset of ASCII alphanumerics, they should work
seamlessly with most Internet protocols. For example, UTF-17 could
be used immediately to solve the IDN problem -- at least for domain
names no longer than 8 characters in length.
Whistler & McGowan Expires December 20, 2001 [Page 5]
Internet-Draft UTF-17 June 2001
5. Potential Drawbacks of UTF-17
It is true that UTF-17 takes up twice the space of UTF-32, but with
64-bit machines and the continuing rapid progress in the lowering of
cost/megabyte of storage, this should not really be a barrier to the
rapid acceptance of UTF-17.
The name "UTF-17" might mislead people into thinking it deals with
17-bit code units, which would clearly be absurd for modern computer
architecture. This drawback could conceivably be addressed by
changing the name to UTF-1+7 instead, which would mnemonically
indicate the correct number of bits (1+7=8) for a code unit, the
correct number of bytes (1+7=8) for a character representation, and
the pattern (1+7) of lead and trail bytes used in the transformation.
Whistler & McGowan Expires December 20, 2001 [Page 6]
Internet-Draft UTF-17 June 2001
References
[1] Unicode, Inc., "The Unicode Standard Version 3.0", January 2000.
[2] Yergeau, F., "UTF-8, a transformation format of ISO 10646",
January 1998.
Authors' Addresses
Kenneth Whistler
Unicode, Inc.
P.O. Box 391476
Mountain View, CA 94039-1476
US
Phone: +1 650 693 3921
URI: http://www.unicode.org/
Rick McGowan
Unicode, Inc.
P.O. Box 391476
Mountain View, CA 94039-1476
US
Phone: +1 650 693 3921
URI: http://www.unicode.org/
Whistler & McGowan Expires December 20, 2001 [Page 7]
Internet-Draft UTF-17 June 2001
Full Copyright Statement
Copyright (C) The Internet Society (2001). All Rights Reserved.
This document and translations of it may be copied and furnished to
others, and derivative works that comment on or otherwise explain it
or assist in its implementation may be prepared, copied, published
and distributed, in whole or in part, without restriction of any
kind, provided that the above copyright notice and this paragraph are
included on all such copies and derivative works. However, this
document itself may not be modified in any way, such as by removing
the copyright notice or references to the Internet Society or other
Internet organizations, except as needed for the purpose of
developing Internet standards in which case the procedures for
copyrights defined in the Internet Standards process must be
followed, or as required to translate it into languages other than
English.
The limited permissions granted above are perpetual and will not be
revoked by the Internet Society or its successors or assigns.
This document and the information contained herein is provided on an
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Acknowledgement
Funding for the RFC Editor function is currently provided by the
Internet Society.
Whistler & McGowan Expires December 20, 2001 [Page 8]