Label Generation Rules (lager) K. Davies
Internet-Draft ICANN
Intended status: Standards Track A. Freytag
Expires: June 11, 2016 ASMUS Inc.
December 9, 2015
Representing Label Generation Rulesets using XML
draft-ietf-lager-specification-05
Abstract
This document describes a method of representing rules for validating
identifier labels and alternate representations of those labels using
Extensible Markup Language (XML). These policies, known as "Label
Generation Rulesets" (LGRs), are used for the implementation of
Internationalized Domain Names (IDNs), for example. The rulesets are
used to implement and share that aspect of policy defining which
labels and specific Unicode code points are permitted for
registrations, which alternative code points are considered variants,
and what actions may be performed on labels containing those
variants.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on June 11, 2016.
Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
Davies & Freytag Expires June 11, 2016 [Page 1]
Internet-Draft Label Generation Rulesets in XML December 2015
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4
2. Design Goals . . . . . . . . . . . . . . . . . . . . . . . . 5
3. LGR Format . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.1. Namespace . . . . . . . . . . . . . . . . . . . . . . . . 6
3.2. Basic Structure . . . . . . . . . . . . . . . . . . . . . 6
3.3. Metadata . . . . . . . . . . . . . . . . . . . . . . . . 7
3.3.1. The version Element . . . . . . . . . . . . . . . . . 7
3.3.2. The date Element . . . . . . . . . . . . . . . . . . 8
3.3.3. The language Element . . . . . . . . . . . . . . . . 8
3.3.4. The scope Element . . . . . . . . . . . . . . . . . . 9
3.3.5. The description Element . . . . . . . . . . . . . . . 9
3.3.6. The validity-start and validity-end Elements . . . . 9
3.3.7. The unicode-version Element . . . . . . . . . . . . . 10
3.3.8. The references Element . . . . . . . . . . . . . . . 10
4. Code Points and Variants . . . . . . . . . . . . . . . . . . 11
4.1. Sequences . . . . . . . . . . . . . . . . . . . . . . . . 12
4.2. Conditional Contexts . . . . . . . . . . . . . . . . . . 13
4.3. Variants . . . . . . . . . . . . . . . . . . . . . . . . 14
4.3.1. Basic Variants . . . . . . . . . . . . . . . . . . . 14
4.3.2. The type attribute . . . . . . . . . . . . . . . . . 15
4.3.3. Null Variants . . . . . . . . . . . . . . . . . . . . 16
4.3.4. Variants with Reflexive Mapping . . . . . . . . . . . 17
4.3.5. Conditional Variants . . . . . . . . . . . . . . . . 18
4.4. Annotations . . . . . . . . . . . . . . . . . . . . . . . 20
4.4.1. The ref Attribute . . . . . . . . . . . . . . . . . . 20
4.4.2. The comment Attribute . . . . . . . . . . . . . . . . 21
4.5. Code Point Tagging . . . . . . . . . . . . . . . . . . . 21
5. Whole Label and Context Evaluation . . . . . . . . . . . . . 21
5.1. Basic Concepts . . . . . . . . . . . . . . . . . . . . . 21
5.2. Character Classes . . . . . . . . . . . . . . . . . . . . 22
5.2.1. Declaring and Invoking Named Classes . . . . . . . . 23
5.2.2. Tag-based Classes . . . . . . . . . . . . . . . . . . 23
5.2.3. Unicode Property-based Classes . . . . . . . . . . . 24
5.2.4. Explicitly Declared Classes . . . . . . . . . . . . . 25
5.2.5. Combined Classes . . . . . . . . . . . . . . . . . . 25
5.3. Whole Label and Context Rules . . . . . . . . . . . . . . 27
5.3.1. The rule Element . . . . . . . . . . . . . . . . . . 27
5.3.2. The Match Operators . . . . . . . . . . . . . . . . . 28
5.3.3. The count Attribute . . . . . . . . . . . . . . . . . 29
Davies & Freytag Expires June 11, 2016 [Page 2]
Internet-Draft Label Generation Rulesets in XML December 2015
5.3.4. The name and by-ref Attributes . . . . . . . . . . . 30
5.3.5. The choice Element . . . . . . . . . . . . . . . . . 31
5.3.6. Literal Code Point Sequences . . . . . . . . . . . . 31
5.3.7. The any Element . . . . . . . . . . . . . . . . . . . 32
5.3.8. The start and end Elements . . . . . . . . . . . . . 32
5.3.9. Example rule from IDNA2008 . . . . . . . . . . . . . 33
5.4. Parameterized Context or When Rules . . . . . . . . . . . 33
5.4.1. The anchor Element . . . . . . . . . . . . . . . . . 34
5.4.2. The look-behind and look-ahead Elements . . . . . . . 34
5.4.3. Omitting the anchor Element . . . . . . . . . . . . . 35
6. The action Element . . . . . . . . . . . . . . . . . . . . . 36
6.1. The match and not-match Attributes . . . . . . . . . . . 37
6.2. Actions with Variant Type Triggers . . . . . . . . . . . 37
6.2.1. The all-, any- and only-variants Attributes . . . . . 37
6.2.2. Example from RFC 3743 Tables . . . . . . . . . . . . 39
6.3. Recommended Disposition Values . . . . . . . . . . . . . 40
6.4. Precedence . . . . . . . . . . . . . . . . . . . . . . . 41
6.5. Implied Actions . . . . . . . . . . . . . . . . . . . . . 41
6.6. Default Actions . . . . . . . . . . . . . . . . . . . . . 42
7. Processing a Label against an LGR . . . . . . . . . . . . . . 42
7.1. Determining Eligibility for a Label . . . . . . . . . . . 42
7.2. Determining Variants for a Label . . . . . . . . . . . . 43
7.3. Determining a Disposition for a Label or Variant Label . 44
7.4. Duplicate Variant Labels . . . . . . . . . . . . . . . . 45
7.5. Checking Labels for Collision . . . . . . . . . . . . . . 45
8. Conversion to and from Other Formats . . . . . . . . . . . . 46
9. Media Type . . . . . . . . . . . . . . . . . . . . . . . . . 46
10. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 46
10.1. Media Type Registration . . . . . . . . . . . . . . . . 46
10.2. URN Registration . . . . . . . . . . . . . . . . . . . . 47
10.3. Disposition Registry . . . . . . . . . . . . . . . . . . 48
11. Security Considerations . . . . . . . . . . . . . . . . . . . 48
12. References . . . . . . . . . . . . . . . . . . . . . . . . . 49
12.1. Normative References . . . . . . . . . . . . . . . . . . 49
12.2. Informative References . . . . . . . . . . . . . . . . . 49
Appendix A. Example Tables . . . . . . . . . . . . . . . . . . . 50
Appendix B. How to Translate RFC 3743 based Tables into the XML
Format . . . . . . . . . . . . . . . . . . . . . . . 54
Appendix C. Indic Syllable Structure Example . . . . . . . . . . 58
Appendix D. RelaxNG Compact Schema . . . . . . . . . . . . . . . 61
Appendix E. Acknowledgements . . . . . . . . . . . . . . . . . . 61
Appendix F. Editorial Notes . . . . . . . . . . . . . . . . . . 61
F.1. Known Issues and Future Work . . . . . . . . . . . . . . 61
F.2. Change History . . . . . . . . . . . . . . . . . . . . . 61
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 64
Davies & Freytag Expires June 11, 2016 [Page 3]
Internet-Draft Label Generation Rulesets in XML December 2015
1. Introduction
This memo describes a method of using Extensible Markup Language
(XML) to describe the algorithm used to determine whether a given
identifier label is permitted, and under which conditions, based on
the code points it contains and their context. These algorithms are
comprised of a list of permissible code points, variant code point
mappings, and a set of rules acting on them. These algorithms form
part of an administrator's policies, and can be referred to as Label
Generation Rulesets (LGRs). In deploying internationalized domain
names (IDNs), they have also been known as IDN tables or variant
tables.
There are other kinds of policies relating to labels which are not
normally covered by Label Generation Rulesets and are therefore not
representable by the XML format described here. These include, but
are not limited to policies around trademarks, or prohibition of
fraudulent or objectionable words.
Administrators of the zones for top-level domain registries have
historically published their LGRs using ASCII text or HTML. The
formatting of these documents has been loosely based on the format
used for the Language Variant Table described in [RFC3743].
[RFC4290] also provides a "model table format" that describes a
similar set of functionality. Common to these formats is that the
algorithms used to evaluate the data therein are implicit or
specified elsewhere.
Through the first decade of IDN deployment, experience has shown that
LGRs derived from these formats are difficult to consistently
implement and compare due to their differing formats. A universal
format, such as one using a structured XML format, will assist by
improving machine-readability, consistency, reusability and
maintainability of LGRs.
When used to represent simple list of permitted code points, the
format is quite straightforward. At the cost of some complexity in
the resulting file, it also allows for an implementation of more
sophisticated handling of conditional variants that reflects the
known requirements of current zone administrator policies.
Another feature of this format is that it allows many of the
algorithms to be made explicit and machine implementable. A
remaining small set of implicit algorithms is described in this
document to allow commonality in implementation.
While the predominant usage of this specification is to represent IDN
label policy, the format is not limited to IDN usage and may also be
Davies & Freytag Expires June 11, 2016 [Page 4]
Internet-Draft Label Generation Rulesets in XML December 2015
used for describing ASCII domain name label rulesets, or other types
of identifier labels beyond those used for domain names.
2. Design Goals
The following goals informed the design of this format:
o The format needs to be implementable in a reasonably
straightforward manner in software.
o The format should be able to be automatically checked for
formatting errors, so that common mistakes can be caught.
o An LGR needs to be able to express the set of valid code points
that are allowed for registration under a specific administrator's
policies.
o Provide the ability to express computed alternatives to a given
identifier based on mapping relationships between code points,
whether one-to-one or many-to-many. These computed alternatives
are commonly known as "variants".
o Variant code points should be able to be tagged with specific
dispositions or categories that can be used to support registry
policy (such as whether to allocate the computed variant, or to
merely block it from usage or registration).
o Variants and code points must be able to be stipulated based on
contextual information. For example, specific variants may only
be applicable when they follow another specific code point, or
when the code point is displayed in a specific presentation form.
o The data contained within an LGR must be able to be interpreted
unambiguously, so that independent implementations that utilize
the contents will arrive at the same results.
o To the largest extent possible, policy rules should be able to be
specified in the XML format without relying hidden, or built-in
algorithms in implementations.
o LGRs should be suitable for comparison and re-use, such that one
could easily compare the contents of two or more to see the
differences, to merge them, and so on.
o As many existing IDN tables as practicable should be able to be
migrated to the LGR format with all applicable interpretation
logic retained.
Davies & Freytag Expires June 11, 2016 [Page 5]
Internet-Draft Label Generation Rulesets in XML December 2015
These requirements are partly derived from reviewing the existing
corpus of published IDN tables, plus the requirements of ICANN's work
to implement an LGR for the DNS Root Zone [LGR-PROCEDURE]. In
particular, Section B of that document identifies five specific
requirements for an LGR methodology.
The syntax and rules in [RFC5892] and [RFC3743] were also reviewed.
It is explicitly not the goal of this format to stipulate what code
points should be listed in an LGR by a zone administrator. Which
registration policies are used for a particular zone is outside the
scope of this memo.
3. LGR Format
An LGR is expressed as a well-formed XML Document [XML].
3.1. Namespace
The XML Namespace URI is "urn:ietf:params:xml:ns:lgr-1.0". [Note:
the examples and schemas for any non-final versions of this
specification use a namespace that is not guaranteed. Early
implementors should consider the need to revise the namespace in
subsequent revisions.]
See Section 10.2 for more information.
3.2. Basic Structure
The basic XML framework of the document is as follows:
...
The "lgr" element contains up to three sub-elements. First is an
optional "meta" element that contains all meta-data associated with
the LGR, such as its authorship, what it is used for, implementation
notes and references. This is followed by a "data" element that
contains the substantive code point data. Finally, an optional
"rules" element contains information on contextual and whole-label
evaluation rules, if any, along with any specific "action" elements
providing for the disposition of labels and computed variant labels.
Davies & Freytag Expires June 11, 2016 [Page 6]
Internet-Draft Label Generation Rulesets in XML December 2015
...
...
...
A document MUST contain exactly one "lgr" element. Each "lgr"
element MUST contain exactly one "data" element, optionally preceded
by one "meta" element and optionally followed by one "rules" element.
In the following descriptions, required, non-repeating elements or
attributes are generally not called out explicitly, in contrast to
optional ones or those that may be repeated. For attributes that
take lists as values the elements are space-delimited.
3.3. Metadata
The optional "meta" element is used to express meta-data associated
within the LGR. It can be used to identify the author or relevant
contact person, explain the intended usage of the LGR, and provide
implementation notes as well as references. With the exception of
"unicode-version" element, the data contained within is not required
by software consuming the LGR in order to calculate valid labels, or
to calculate variants. The "unicode-version" element MUST be used by
a consumer of the table to identify that it has the correct Unicode
property data to perform operations on the table.
3.3.1. The version Element
The "version" element is optional. It is used to uniquely identify
each version of the LGR. No specific format is required, but it is
RECOMMENDED that it be the decimal representation of a single
positive integer, which is incremented with each revision of the
file.
An example of a typical first edition of a document:
1
The "version" element may have an optional "comment" attribute.
Davies & Freytag Expires June 11, 2016 [Page 7]
Internet-Draft Label Generation Rulesets in XML December 2015
1
3.3.2. The date Element
The optional "date" element is used to identify the date the LGR was
posted. The contents of this element MUST be a valid ISO 8601 "full-
date" string as described in [RFC3339].
Example of a date:
2009-11-01
3.3.3. The language Element
The optional "language" element signals that the LGR is associated
with a specific language or script. The value of the "language"
element MUST be a valid language tag as described in [RFC5646]. The
tag may refer to a script plus undefined language if the LGR is not
referring to a specific language.
Example of an English language LGR:
en
If the LGR applies to a specific script, rather than a language, the
"und" language tag should be used followed by the relevant [RFC5646]
script subtag. For example, for a Cyrillic script LGR:
und-Cyrl
If the LGR covers a specific set of multiple languages or scripts,
the "language" element can be repeated. However, for cases of a
script-specific LGR exhibiting insignificant admixture of code points
from other scripts, it is RECOMMENDED to use a single "language"
element identifying the predominant script. In the exceptional case
of a multi-script LGR where no script is predominant, use Zyyy
(Common):
und-Zyyy
Note that that for the particular case of Japanese, a script tag
"Jpan" exists that matches the mixture of scripts used in writing
that language. The preferred "language" element would be:
und-Jpan
Davies & Freytag Expires June 11, 2016 [Page 8]
Internet-Draft Label Generation Rulesets in XML December 2015
3.3.4. The scope Element
This optional element refers to a scope, such as a domain, to which
this policy is applied. The "type" attribute specifies the type of
scope being defined. A type of "domain" means that the scope is a
domain that represents the apex of the DNS zone to which the LGR is
applied. For that type, the content of the "scope" element MUST be a
valid absolute domain name, without a trailing dot. However, the DNS
root zone is represented as ".".
example.com
There may be multiple "scope" tags used, for example to reflect a
list of domains to which the LGR is applied. Types of scope other
than "domain" are application-defined. Such application-defined type
values must be in a namespace "apptype" previously defined in the
document. An explanation of the application-defined type in the
"description" element is RECOMMENDED.
3.3.5. The description Element
The "description" element is an optional free-form element that
contains any additional relevant description that is useful for the
user in its interpretation. Typically, this field contains
authorship information, as well as additional context on how the LGR
was formulated and how it applies, such as citations and references
that apply to the LGR as a whole.
This field should not be relied upon for providing instructions on
how to parse or utilize the data contained elsewhere in the
specification. Authors of tables should expect that software
applications that parse and use LGRs will not use the description
field to condition the application of the LGR's data and rules.
The element has an optional "type" attribute, which refers to the
internet media type of the enclosed data. Typical types would be
"text/plain" or "text/html". The attribute SHOULD be a valid MIME
type. If supplied, it will be assumed that the contents are of that
media type. If the description lacks a type field, it will be
assumed to be plain text ("text/plain").
3.3.6. The validity-start and validity-end Elements
The "validity-start" and "validity-end" elements are optional
elements that describe the time period from which the contents of the
LGR become valid (i.e. are used in registry policy), and the contents
of the LGR cease to be used.
Davies & Freytag Expires June 11, 2016 [Page 9]
Internet-Draft Label Generation Rulesets in XML December 2015
The dates MUST confirm to the "full-date" format described in section
5.6 of [RFC3339].
2014-03-12
3.3.7. The unicode-version Element
Whenever an LGR depends on character properties from a given version
of the Unicode standard, the version number used in creating the LGR
MUST be listed in the form x.y.z, where x, y, and z are positive,
decimal integers (see [Unicode-Versions]). If any software
processing the table does not have access to character property data
of the requisite version, it MUST NOT perform any operations relating
to whole-label evaluation relying on Unicode properties
(Section 5.2.3).
The value of a given Unicode property in [UAX42] may change between
versions, unless such change has been explicitly disallowed in
[Unicode-Stability]. It is RECOMMENDED to only reference properties
defined as stable or immutable. As an alternative to referencing the
property, the information can be presented explicitly in the LGR.
6.2.0
It is not necessary to include a "unicode-version" element for LGRs
that do not make use of Unicode properties, however, it is
RECOMMENDED.
3.3.8. The references Element
A Label Generation Ruleset may define a list of references which are
used to associate various individual elements in the LGR to one or
more normative references. A common use for references is to
annotate that code points belong to an externally defined collection
or standard, or to give normative references for rules.
References are specified in an optional "references" element contains
any number of "reference" elements, each with a unique "id"
attribute. It is RECOMMENDED that the "id" attribute be a zero-based
integer, however, in addition to digits 0-9, it MAY contain uppercase
letters A-Z, as well as period, hyphen, colon or underscore. The
value of each "reference" element SHOULD be the citation of a
standard, dictionary or other specification in any suitable format.
In addition to an "id" attribute, a "reference" element may have a
"comment" attribute for an optional free-form annotation.
Davies & Freytag Expires June 11, 2016 [Page 10]
Internet-Draft Label Generation Rulesets in XML December 2015
The Unicode Standard, Version 7.0
Big-5: Computer Chinese Glyph and Character
Code Mapping Table, Technical Report C-26, 1984
ISO/IEC
10646:2012 3rd edition
...
...
...
A reference is associated with an element by using its id as part of
an optional "ref" attribute (see Section 4.4.1). The "ref" attribute
may be used with many kinds of elements in the "data" or "rules"
sections of the LGR, most notably those defining code points,
variants and rules. However, a "ref" attribute may not occur on
certain kinds of elements, including references to nameed character
classes or rules. See description of these elements below.
4. Code Points and Variants
The bulk of a label generation ruleset is a description of which set
of code points are eligible for a given label. For rulesets that
perform operations that result in potential variants, the code point-
level relationships between variants need to also be described.
The code point data is collected within the "data" element. Within
this element, a series of "char" and "range" elements describe
eligible code points, or ranges of code points, respectively.
Discrete permissible code points or code point sequences are declared
with a "char" element, e.g.
Ranges of permissible code points may be stipulated with a "range"
element, e.g.
The range is inclusive of the first and last code points. All
attributes defined for a "range" element act as if applied to each
code point within. A "range" element has no child elements.
Davies & Freytag Expires June 11, 2016 [Page 11]
Internet-Draft Label Generation Rulesets in XML December 2015
It is always possible to substitute a list of individually specified
code points for a range element. The reverse is not necessarily the
case. Whenever such a substitution is possible, it makes no
difference in processing the data. Tools reading or writing the LGR
format are free to aggregate sequences of consecutive code points of
the same properties into range elements.
Code points must be expressed in uppercase, hexadecimal, and zero
padded to a minimum of 4 digits. In other words, they are
represented according to the standard Unicode convention but without
the prefix "U+". The rationale for not allowing other encoding
formats, including native Unicode encoding in XML, is explored in
[UAX42]. The XML conventions used in this format, including the
element and attribute names, mirror this document where practical and
reasonable to do so. It is RECOMMENDED to list all "char" elements
in ascending order of the "cp" attribute.
All "char" elements in the data section MUST have distinct "cp"
attributes. The "range" elements MUST NOT specify code point ranges
that overlap either another range or any single code point "char"
elements.
4.1. Sequences
A sequence of two or more code points may be specified in an LGR, for
example, when defining the source for n:m variant mappings. Another
use of sequences would be in cases when the exact sequence of code
points is required to occur in order for the constituent elements to
be eligible, such as when a specific code point is only eligible when
preceded or followed by another code point. The following would
define the eligibility of the MIDDLE DOT (U+00B7) only when both
preceded and followed by the LATIN SMALL LETTER L (U+006C):
All sequences defined this way must be distinct, but sub-sequences
may be defined. Thus, the sequence defined here may coexist with
single code point definitions such as:
As an alternative to using sequences to define a required context, a
"char" or "range" element may specify conditional context using an
optional "when" attribute as described below in Section 4.2. The
latter method is more flexible in that such conditional context is
not limited to specific code point in addition to allowing both
prohibited as well as required context to be specified.
Davies & Freytag Expires June 11, 2016 [Page 12]
Internet-Draft Label Generation Rulesets in XML December 2015
As described below, the "char" element, whether or not it is used for
a single code point, or for a sequence, may have optional child
elements defining variants. Both the "char" and "range" elements can
take a number of optional attributes for conditional inclusion,
commenting, cross referencing and character tagging, as described
below.
4.2. Conditional Contexts
A conditional context is specified by a rule that must be satisfied
(or alternatively, must not be satisfied) for a code point in a given
label, often at a particular location in a label.
To specify a conditional context a "when" or "not-when" attribute may
be used. The value of each "when" or "not-when" attributes is a
whole label or parameterized context rule as described below in
Section 5.3. The content condition is met when the rule specified in
the "when" attribute is matched. Alternatively, a "not-when"
attribute may be used for a rule that must not be matched. It is an
error to reference a rule that is not actually defined in the "rules"
element.
A parameterized context rule (see Section 5.4) defines the context
immediately surrounding a given code point; unlike a sequence, the
context is not limited to a specific fixed code point, but for
example may be a code point that is a member of a certain subset or
has a certain Unicode property.
Given a suitable definition of a parameterized context rule named
"follows-virama" this example specifies that a ZERO-WIDTH JOINER
(U+200D) is restricted to immediately follow any of several code
points classified as virama:
For a complete example, see Appendix A.
In contrast, a whole label rule (see Section 5.3) specifies a
condition to be met by the entire label, for example that it must
contain at least one code point from a given script anywhere in the
label. In the following example no digit from either range must
occur in a label that mixes digits from both ranges:
Davies & Freytag Expires June 11, 2016 [Page 13]
Internet-Draft Label Generation Rulesets in XML December 2015
(See Section 5.3.9 for an example of the "mixed-digits" rule.)
The "when" or "not-when" attributes are mutually exclusive. They may
be applied to both "char" and "range" elements in the "data" element,
as well as to "var" elements (see Section 4.3.5).
If a contextual condition is not satisfied for any code point in a
label, the label is invalid, see Section 6.5. For variants, the
conditional context restricts the definition of the variant to the
case where the condition is met. Outside the specified context, a
variant is not defined .
4.3. Variants
Most LGRs typically only determine simple code point eligibility, and
for them, the elements described so far would be the only ones
required for their "data" section. Others additionally specify a
mapping of code points to other code points, known as "variants".
What constitutes a variant code point is a matter of policy, and
varies for each implementation. The following examples are intended
to demonstrate the syntax; they are not necessarily typical.
4.3.1. Basic Variants
Variant code points are specified using one of more "var" elements as
children of a "char" element. The target mapping is specified using
the "cp" attribute. Other, optional attributes for the "var" element
are described below.
For example, to map LATIN SMALL LETTER V (U+0076) as a variant of
LATIN SMALL LETTER U (U+0075):
A sequence of multiple code points can be specified as a variant of a
single code point. For example, the sequence of LATIN SMALL LETTER O
(U+006F) then LATIN SMALL LETTER E (U+0065) might hypothetically be
specified as a variant for an LATIN SMALL LETTER O WITH DIAERESIS
(U+00F6) as follows:
Davies & Freytag Expires June 11, 2016 [Page 14]
Internet-Draft Label Generation Rulesets in XML December 2015
The source and target of a variant mapping may both be sequences, but
not ranges.
If the source of one mapping is a prefix sequence of the source for
another, both variant mappings will be considered at the same
location in the input label when generating permuted variant labels.
If poorly designed, an LGR containing such an instance of a prefix
relation could generate multiple instances of the same variant label
for the same original label, but with potentially different
dispositions. Any duplicate variant labels encountered MUST be
treated as an error (see Section 7.4).
The "var" element specifies variant mappings in only one direction,
even though the variant relation is usually considered symmetric,
that is, if A is a variant of B then B should also be a variant of A.
The format requires that the inverse of the variant be given
explicitly to fully specify symmetric variant relations in the LGR.
This has the beneficial side effect of making the symmetry explicit:
Variant relations are normally not only symmetric, but also
transitive. If A is a variant of B and B is a variant of C, then A
is also a variant of C. As with symmetry, these transitive relations
are spelled out explicitly in the LGR.
All variant mappings are unique. For a given "char" element all
"var" elements MUST have a unique combination of "cp", "when" and
"not-when" attributes. It is RECOMMENDED to list the "var" elements
in ascending order of their target code point sequence. (For "when"
and "not-when" attributes, see Section 4.3.5).
4.3.2. The type attribute
Variants may be tagged with an optional "type" attribute. The value
of the "type" attribute may be any non-empty value not starting with
an underscore and not containing spaces. This value is used to
resolve the disposition of any variant labels created using a given
variant. (See Section 6.2.)
By default, the values of the "type" attribute directly describe the
target policy status (disposition) for a variant label that was
Davies & Freytag Expires June 11, 2016 [Page 15]
Internet-Draft Label Generation Rulesets in XML December 2015
generated using a particular variant, with any variant label being
assigned a disposition corresponding to the most restrictive variant
type. Several conventional disposition values are predefined below
in Section 6. Whenever these values can represent the desired
policy, they SHOULD be used.
By default, if a variant label contains any instance of one of the
variants of type "blocked" the label would be blocked, but if it
contained only instances of variants to be allocated it could be
allocated. See the discussion about implied actions in Section 6.6.
The XML format for the LGR makes the relation between the values of
the "type" attribute on variants and the resulting disposition of
variant labels fully explicit. See the discussion in Section 6.2.
Making this relation explicit allows a generalization of the "type"
attribute from directly reflecting dispositions to a more
differentiated intermediate value that used in the resolution of
label disposition. Instead of the default action of applying the
most restrictive disposition to the entire label, such a generalized
resolution can be used to achieve additional goals, such as limiting
the set of allocated variant labels, or to implement other policies
found in existing LGRs (see for example Appendix B).
Because variant mappings MUST be unique, it is not possible to define
the same variant for the same "char" element with different type
attributes (see however Section 4.3.5).
4.3.3. Null Variants
A null variant is a variant string that maps to no code point. This
is used when a particular code point sequence is considered
discretionary in the context of a whole label. To specify a null
variant, use an empty cp attribute. For example, to mark a string
with a ZERO WIDTH NON-JOINER (U+200C) to the same string without the
ZERO WIDTH NON-JOINER:
Davies & Freytag Expires June 11, 2016 [Page 16]
Internet-Draft Label Generation Rulesets in XML December 2015
This is useful in expressing the intent that some code points in a
label are to be mapped away when generating a canonical variant of
the label. However, in tables that are designed to have symmetric
variant mappings, this could lead to combinatorial explosion, if not
handled carefully.
The symmetric form of a null variant is expressed as follows:
A "char" element with an empty "cp" attribute MUST specify at least
one variant mapping. It is strongly RECOMMENDED to use a type of
"invalid" or equivalent when defining variant mappings from null
sequences, so that variant mapping from null sequences are removed in
variant label generation (see Section 4.3.2).
4.3.4. Variants with Reflexive Mapping
At first sight there seems to be no call for adding variant mappings
for which source and target code points are the same, that is for
which the mapping is reflexive, or, in other words, an identity
mapping. Yet such reflexive mappings occur frequently in LGRs that
follow [RFC3743].
Adding a "var" element allows both a type and a reference id to be
specified for it. While the reference id is not used in processing,
the type of the variant can be used to trigger actions. In permuting
the label to generate all possible variants, the type associated with
a reflexive variant mapping is applied to any of the permuted labels
containing the original code point.
In the following example, the code point U+3473 exists both as a
variant of U+3447 and as a variant of itself (reflexive mapping).
Assuming an original label of "U+3473 U+3447", the permuted variant
"U+3473 U+3473" would consist of the reflexive variant of U+3473
followed by a variant of U+3447. Accordingly, the types for both of
the variant mappings used to generate that particular permutation
would have the value "preferred" given the following definitions of
variant mappings:
Davies & Freytag Expires June 11, 2016 [Page 17]
Internet-Draft Label Generation Rulesets in XML December 2015
Having established the variant types in this way, a set of actions
could be defined that return a disposition of "allocatable" or
"activated" for a label consisting exclusively of variants with type
"preferred" for example. (For details on how to define actions based
on variant types see Section 6.2.1.)
In general, using reflexive variant mappings in this manner makes it
possible to calculate disposition values using a uniform approach for
all labels, whether they consist of mapped variant code points,
original code points, or a mixture of both. In particular, the
dispositions for two otherwise identical labels may differ based on
which variant mappings were executed in order to generate each of
them. (For details on how to generate variants and evaluate
dispositions, see Section 7.)
Another useful convention that uses reflexive variants is described
below in Section 6.2.1.
4.3.5. Conditional Variants
Fundamentally, variants are mappings between two sequences of code
points. However, in some instances for a variant relationship to
exist, some context external to the code point sequence must also be
considered. For example, a positional context may determine whether
two code point sequences are variants of each other.
An example of that are Arabic code points which can have different
forms based on position, with some code points sharing forms, thus
making them variants in the positions corresponding to those forms.
Such positional context cannot be solely derived from the code point
by itself, as the code point would be the same for the various forms.
As described in Section 4.2 a "when" or "not-when" attribute may be
given for any "var" element to specify required or prohibited
contextual conditions under which the variant defined.
Assuming the "rules" element contains suitably defined rules for
"arabic-isolated" and "arabic-final", the following example shows how
to mark ARABIC LETTER ALEF WITH WAVY HAMZA BELOW (U+0673) as a
Davies & Freytag Expires June 11, 2016 [Page 18]
Internet-Draft Label Generation Rulesets in XML December 2015
variant of ARABIC LETTER ALEF WITH HAMZA BELOW (U+0625), but only
when it appears in its isolated or final forms:
While only a single "when" or "not-when" attribute can be applied to
any "var" element, multiple "var" elements using the same mapping,
but different "when" or "not-when" attributes may be specified. The
combination of mapping and conditional context defines a unique
variant.
Care must be taken to ensure that for each variant label at most one
of the contextual conditions is met for variants with the same
mapping; otherwise duplicate variant labels would be created for the
same input label. Any such duplicate variant labels MUST be treated
as an error, see Section 7.4.
Two contexts may be complementary, as in the following example, which
shows ARABIC LETTER TEH MARBUTA (U+0629) as a variant of ARABIC
LETTER ALEF MAKSURA (U+0649), but with two different types.
The intent is that in final position a label that uses U+0629 instead
of U+0647 should be considered essentially the same label and
therefore allocatable to the same entity, while the same substitution
in non-final context leads to labels that are different, but
considered confusable so that either one, but not both should be
delegatable.
For symmetry, the reverse mappings must exist, and must agree in
their "when" or "not-when" attributes. However, symmetry does not
apply to the other attributes. For example, these are potential
reverse mappings for the above:
Here, both variants have the same "type" attribute. While it is
tempting to recognize that in this instance the "when" and "not-when"
Davies & Freytag Expires June 11, 2016 [Page 19]
Internet-Draft Label Generation Rulesets in XML December 2015
attributes are complementary and therefore between them cover every
single possible context, it is STRONGLY RECOMMENDED to use the format
shown in the example that makes the symmetry easily verifiable by
parsers and tools. (The same applies to entries created for
transitivity.)
Arabic is an example of a script for which such conditional variants
have been implemented based on the joining contexts for Arabic code
points. The mechanism defined here supports other forms of
conditional variants that may required by other scripts.
4.4. Annotations
Two attributes, the "ref" and "comment" attributes, can be used to
annotate individual elements in the LGR. They are ignored in
machine-processing or the LGR. The "ref" attribute is intended for
formal annotations and the "comment" attribute for free form
annotations. The latter can be applied more widely.
4.4.1. The ref Attribute
Reference information may optionally be specified by a "ref"
attribute, consisting of a space delimited sequence of reference
identifiers (see Section 3.3.8).
This facility is typically used to give source information for code
points or variant relations. This information is ignored when
machine-processing an LGR. If applied to a range the "ref" attribute
applies to every code point in the range. All reference identifiers
MUST be from the set declared in the "references" element (see
Section 3.3.8). It is an error to repeat a reference identifier in
the same "ref" attribute. It is RECOMMENDED that identifiers be
listed in ascending order.
In addition to "char", "range" and "var" elements in the data
section, a "ref" attribute may be present for a number of elements
types contained in the "rules" element as described below: actions,
literals ("char" inside a rule), as well as for definitions of rules
and classes, but not for references to named character classes or
rules using the "by-ref" attribute defined below. (The use of the
"by-ref" and "ref" attributes is mutually exclusive.) None of the
elements in the metadata take a "ref" attribute; to provide
additional information use the "description" element instead.
Davies & Freytag Expires June 11, 2016 [Page 20]
Internet-Draft Label Generation Rulesets in XML December 2015
4.4.2. The comment Attribute
Any "char", "range" or "variant" element in the data section may
contain an optional "comment" attribute. The contents of a "comment"
attribute are free-form plain text. Comments are ignored in machine
processing of the table. Comment attributes may also be placed on
all elements in the "rules" section of the document, such as actions
and match operators, such as literals ("char"), as well as
definitions of classes and rules, but not on child elements of the
"class" element. Finally, in the metadata, only the "version" and
"reference" elements may have "comment" attributes (to match the
syntax in [RFC3743]).
4.5. Code Point Tagging
Typically, LGRs are used to explicitly designate allowable code
points, where any label that contains a code point not explicitly
listed in the LGR is considered an ineligible label according to the
ruleset.
For more complex registry rules, there may be a need to discern one
or more subsets of code points. This can be accomplished by applying
an optional "tag" attribute to "char" or "range" elements that are
child elements of the "data" element. By collecting code points that
share the same tag value, character classes may be defined (see
Section 5.2.2) which can then be used in whole label evaluation rules
(see Section 5.3.2).
Each "tag" attribute may contain multiple values separated by white
space. A tag value is an identifier, which may also include certain
punctuation marks, such as colon. Formally, it MUST correspond to
the XML 1.0 Nmtoken (Name token) production. It is an error to
duplicate a value within the same "tag" attribute. A "tag" attribute
for a "range" element applies to all code points in the range.
Because code point sequences are not proper members of a set of code
points, a "tag" attribute MUST NOT be present in a "char" element
defining a code point sequence.
5. Whole Label and Context Evaluation
5.1. Basic Concepts
The code points in a label sometimes need to satisfy context-based
rules, for example for the label to be considered valid, or to
satisfy the context for a variant mapping (see the description of the
"when" attribute in Section 5.4).
Davies & Freytag Expires June 11, 2016 [Page 21]
Internet-Draft Label Generation Rulesets in XML December 2015
A Whole Label Evaluation rule (WLE) is applied to the whole label.
It is used to validate both original labels and variant labels
computed from them using a permutation over all applicable variant
mappings. A conditional context rules is a specialized form of WLE
specific to the context around a single code point or code point
sequence. For example, if a rule is referenced in the "when"
attribute of a variant mapping it is used to describe the conditional
context under which the particular variant mapping is defined to
exist.
Each rule is defined in a "rule" element. A rule may contain the
following as child elements:
o literal code points or code point sequences
o character classes, which define sets of code points to be used for
context comparisons
o context operators, which define when character classes and
literals may appear
o nested rules, whether defined in place or invoked by reference
Collectively, these are called match operators and are listed in
Section 5.3.2.
5.2. Character Classes
Character classes are sets of characters that often share a
particular property. While they function like sets in every way,
even supporting the usual set operators, they are called character
classes here in a nod to the use of that term in regular expression
syntax. (This also avoids confusion with the term "character set" in
the sense of character encoding.)
Character classes (or sets) can be specified in several ways:
o by defining the set via matching a tag in the code point data.
All characters with the same "tag" attribute are part of the same
class;
o by referencing one of the Unicode character properties defined in
the Unicode Character Database [UAX42];
o by explicitly listing all the code points in the class; or
o by defining the class as a set combination of any number of other
classes.
Davies & Freytag Expires June 11, 2016 [Page 22]
Internet-Draft Label Generation Rulesets in XML December 2015
5.2.1. Declaring and Invoking Named Classes
A character class has an optional "name" attribute, consisting of a
single, identifier not containing spaces. All names for classes must
be unique. If the "name" attribute is omitted, the class is
anonymous and exists only inside the rule or combined class where it
is defined. A named character class is defined independently and can
be referenced by name from within any rules or as part of other
character class definitions.
...
An empty "class" element with a "by-ref" attribute is a reference to
an existing named class. The "by-ref" attribute cannot be used in
the same "class" element with any of these attributes: "name", "from-
tag", "property" or "ref". The "name" attribute MUST be present, if
and only if the class is a direct child element of the "rules"
element. It is an error to reference a named class for which the
definition has not been seen.
5.2.2. Tag-based Classes
The "char" or "range" elements that are child elements of the "data"
element may contain a "tag" attribute that consists of one or more
space separated tag values, for example:
This defines two tags for use with code point U+0061, the tag
"letter" and the tag "lower". Use
to define two named character classes, "letter" and "lower",
containing all code points with the respective tags, the first with
0061 and 4E00 as elements and the latter with 0061, but not 4E00 as
an element. The "name" attribute may be omitted for an anonymous in-
place definition of a nested, tag-based class.
Davies & Freytag Expires June 11, 2016 [Page 23]
Internet-Draft Label Generation Rulesets in XML December 2015
Tag values are typically identifiers, with the addition of a few
punctuation symbols, such as colon. Formally they MUST correspond to
the XML 1.0 Nmtoken (Name token) production. While a "tag" attribute
may contain a list of tag values, the "from-tag" attribute always
contains a single tag value.
If the document contains no "char" or "range" elements with a
corresponding tag, the character class represents the empty set.
This is valid, to allow a common "rules" element to be shared across
files. However, it is RECOMMENDED that implementations allow for a
warning to ensure that referring to an undefined tag in this way is
intentional.
5.2.3. Unicode Property-based Classes
A class is defined in terms of Unicode properties by giving the
Unicode property alias and the property value or property value
alias, separated by a colon.
The example above selects all code points for which the Unicode
canonical combining class (ccc) value is 9. This value of the ccc is
assigned to all code points that encode viramas. The string "ccc" is
the short-alias for the canonical combining class, as defined in the
Unicode Character Database [UAX42].
Unicode properties may, in principle, change between versions of the
Unicode Standard. However, the values assigned for a given version
are fixed. If Unicode Properties are used, a Unicode version MUST be
declared in the "unicode-version" element in the header. (Note: some
Unicode properties are by definition stable across versions and do
not change once assigned (see [Unicode-Stability].)
It is RECOMMENDED that all implementations processing LGR files
provide support for the following minimal set of Unicode properties:
o General Category (gc)
o Script (sc)
o Canonical Combining Class (ccc)
o Bidi Class (bc)
o Arabic Joining Type (jt)
o Indic Syllabic Category (InSC)
Davies & Freytag Expires June 11, 2016 [Page 24]
Internet-Draft Label Generation Rulesets in XML December 2015
o Deprecated (Dep)
The short name for each property is given in parentheses.
If a program that is using an LGR to determine the validity of a
label encounters a property that it does not support, it MUST abort
with an error.
5.2.4. Explicitly Declared Classes
A class of code points may also be declared by listing the code
points that are a member of the class. This is useful when tagging
cannot be used because code points are not listed individually as
part of the eligible set of code points for the given LGR, for
example because they only occur in code point sequences.
To define a class in terms of an explicit list of code points use a
space separated list of hexadecimal code point values:
0061 0062 0063 0064
This defines a class named "abcd" containing the code points for
characters "a", "b", "c" and "d". The ordering of the code points is
not material, but it is RECOMMENDED to list them in ascending order.
In a class definition, ranges of code points are represented by a
hexadecimal start and end value separated by a hyphen. The following
declaration is equivalent to the preceding:
0061-0064
Range and code point declarations can be freely intermixed:
0061 0062-0063 0064
The contents of a class differ from a repertoire in that the latter
may contain sequences as elements, while the former may not.
Instead, they closely resemble character classes as found in regular
expressions.
5.2.5. Combined Classes
Classes may be combined using operators for set complement, union,
intersection, difference and symmetric difference (exclusive-or).
Because classes fundamentally function like sets, the union of
several character classes is itself a class, for example.
Davies & Freytag Expires June 11, 2016 [Page 25]
Internet-Draft Label Generation Rulesets in XML December 2015
+-------------------+----------------------------------------------+
| Logical Operation | Example |
+-------------------+----------------------------------------------+
| Complement | |
+-------------------+----------------------------------------------+
| Union | |
| | |
| | |
| | |
| | |
+-------------------+----------------------------------------------+
| Intersection | |
| | |
| | |
| | |
+-------------------+----------------------------------------------+
| Difference | |
| | |
| | |
| | |
+-------------------+----------------------------------------------+
| Symmetric | |
| Difference | |
| | |
| | |
+-------------------+----------------------------------------------+
Set Operators
The elements from this table may be arbitrarily nested inside each
other, subject to the following restriction: a "complement" element
MUST contain precisely one "class" or one of the operator elements,
while an "intersection", "symmetric-difference" or "difference"
element MUST contain precisely two, and a "union" element MUST
contain two or more of these elements.
An anonymous combined class can be defined directly inside a rule or
of the match operator elements that allow child elements (see
Section 5.3.2) by using the set combination as the outer element.
Davies & Freytag Expires June 11, 2016 [Page 26]
Internet-Draft Label Generation Rulesets in XML December 2015
The example shows the definition of an anonymous combined class that
represents the union of classes "xxx" and "yyy". There is no need to
wrap this union inside another "class" element, and, in fact, set
combination elements MUST NOT be nested inside a "class" element.
Lastly, to create a named combined class that can be referenced in
other classes or in rules as , add a "name"
attribute to the set combination element, for example and place it at the top level immediately below the
"rules" element (see Section 5.2.1).
. . .
Because (as for ordinary sets) a combination of classes is itself a
class, no matter by what combinations of set operators a combined
class is created, a reference to it always uses the "class" element
as described in Section 5.2.1. That is, a named class is always
referenced via an empty "class" element using the "by-ref" attribute
containing the name of the class to be referenced.
5.3. Whole Label and Context Rules
Each rule is comprised of a series of matching operators that must be
satisfied in order to determine whether a label meets a given
condition. Rules may reference other rules or character classes
defined elsewhere in the table.
5.3.1. The rule Element
A matching rule is defined by a "rule" element, the child elements of
which are one of the match operators from Section 5.3.2. In
evaluating a rule, each child element is matched in order. Rule
elements may be nested.
Rules may optionally be named using a "name" attribute containing a
single identifier string with no spaces. A named rule may be
incorporated into another rule by reference. If the "name" attribute
is omitted, the rule is anonymous and may not be incorporated by
reference into another rule or referenced by an action or "when"
attribute.
Davies & Freytag Expires June 11, 2016 [Page 27]
Internet-Draft Label Generation Rulesets in XML December 2015
A simple rule to match a label where all characters are members of
the class "preferred":
Rules are paired with explicit and implied actions, triggering these
actions when a rule matches a label. For example, a simple explicit
action for the rule shown above would be:
This has the effect of setting the policy disposition for a label
made up entirely of "preferred" code points to "allocatable".
Explicit actions are further discussed in Section 6 and the use of
rules in conditional contexts for implied actions is discussed in
Section 4.3.5 and Section 6.5.
5.3.2. The Match Operators
The child elements of a rule are a series of match operators, which
are listed here by type and name and with a basic example or two.
Davies & Freytag Expires June 11, 2016 [Page 28]
Internet-Draft Label Generation Rulesets in XML December 2015
+------------+-------------+------------------------------------+
| Type | Operator | Examples |
+------------+-------------+------------------------------------+
| logical | any | |
| +-------------+------------------------------------+
| | choice | |
| | | |
| | | |
| | | |
+--------------------------+------------------------------------+
| positional | start | |
| +-------------+------------------------------------+
| | end | |
+--------------------------+------------------------------------+
| literal | char | |
+--------------------------+------------------------------------+
| set | class | |
| | | 0061 0064-0065 |
+--------------------------+------------------------------------+
| group | rule | |
| | | |
+--------------------------+------------------------------------+
| contextual | anchor | |
| +-------------+------------------------------------+
| | look-ahead | |
| +-------------+------------------------------------+
| | look-behind | |
+--------------------------+------------------------------------+
Match Operators
Any element defining an anonymous class can be used as a match
operator, including any of the set combination operators (see
Section 5.2.5) as well as references to named classes.
All match operators shown as empty elements in the Examples column of
the table above do not support child elements of their own; otherwise
match operators may be nested. In particular, anonymous "rule"
elements can be used for grouping.
5.3.3. The count Attribute
The optional "count" attribute specifies the minimally required or
maximal permitted number of times a match operator is used to match
input. If the "count" attribute is
n the match operator matches the input exactly n times, where n is
1 or greater.
Davies & Freytag Expires June 11, 2016 [Page 29]
Internet-Draft Label Generation Rulesets in XML December 2015
n+ the match operator matches the input at least n times, where n
is 0 or greater.
n:m the match operator matches the input at least n times where n is
0 or greater, but matches the input up to m times in total,
where m > n. If m = n and n > 0, the match operator matches the
input exactly n times.
If there is no "count" attribute, the match operator matches the
input exactly once.
In matching, greedy evaluation is used in the sense defined for
regular expressions: beyond the required number or times, the input
is matched as many times as possible, but not so often as to prevent
a match of the remainder of the rule.
The optional "count" attribute MUST NOT be applied to any element
that contains a "name" attribute, but MAY be applied to operators
such as "class" that declare anonymous classes (including combined
classes) or invoke any predefined classes by reference. The count
attribute MUST not be applied to any "class" element, or element
defining a combined class, when it is nested inside a combined class.
The count attribute MUST NOT be applied to match operators of type
"start", "end", "anchor", "look-ahead" or "look-behind" or to any
operators, such as "rule" or "choice" that contain a nested instance
of them. This limitation applies recursively, and irrespective of
whether a rule element containing these nested instances is declared
in place or used by reference.
However, the "count" attribute MAY be applied to any other instances
of either an anonymous "rule" element or of a "choice" element,
including those instances nested inside other match operators. It
MAY also be applied to the elements "any" and "char", when used as
match operators.
5.3.4. The name and by-ref Attributes
Like classes (see Section 5.2.1), rules declared as immediate child
elements of the "rules" element MUST be named using a unique "name"
attribute, and all other instances MUST NOT be named. Anonymous
rules and classes or reference to named rules and classes can be
nested inside other match operators by reference.
To reference a named rule or class inside a rule or match operator
use a "rule" or "class" element with an optional "by-ref" attribute
containing the name of the referenced element. It is an error to
reference a rule or class for which the complete definition has not
Davies & Freytag Expires June 11, 2016 [Page 30]
Internet-Draft Label Generation Rulesets in XML December 2015
been seen. In other words, it is explicitly not possible to define
recursive rules or class definitions. The "by-ref" attribute cannot
appear in the same element as the "name" attribute, or in an element
that has any child elements.
Here's an example of a rule requiring that all labels be letters
(optionally followed by combining marks) and possibly digits. The
example shows rules and classes referenced by name.
5.3.5. The choice Element
The "choice" element is used to represent a list of two or more
alternatives:
Each child element of a "choice" represents one alternative. The
first matching alternative determines the match for the "choice"
element. To express a choice where an alternative itself consists of
a sequence of elements, the sequence must be wrapped in an anonymous
rule.
5.3.6. Literal Code Point Sequences
A literal code point sequence matches a single code point or a
sequence. It is defined by a "char" element, with the code point or
sequence to be matched given by the "cp" attribute. When used as a
literal, a "char" element may contain a "count" in addition to the
"cp" attribute and optional "comment" or "ref" attributes. No other
attributes or child elements are permitted.
Davies & Freytag Expires June 11, 2016 [Page 31]
Internet-Draft Label Generation Rulesets in XML December 2015
5.3.7. The any Element
The "any" element matches any single code point. It may have a
"count" attribute. For an example see Section 5.3.9
Unlike a literal, the "any" element" may not have a "ref" attribute.
5.3.8. The start and end Elements
To match the beginning or end of a label, use the "start" or "end"
element. An empty label would match this rule:
Conceptually, Whole Label Evaluation Rules evaluate the label as a
whole, but in practice, many rules do not actually need to be
specified to match the entire label. For example, to express a
requirement of not starting a label with a digit, a rule needs to
describe only the initial part of a label.
This example uses the previously defined rules, together with start
and end tag, to define a rule that requires that an entire label is
well-formed. For this example that means, that it must start with a
letter and contains no leading digits or combining marks, nor
combining marks placed on digits.
Each "start" or "end" element occurs at most once in a rule, except
if nested inside a "choice" element in such a way that in matching
each alternative at most one occurrence of each is encountered.
Otherwise, the result is an error; as is any case where a "start" or
"end" element is not encountered as first or last element to be
matched, respectively, in matching a rule. Start and end elements do
not have a "count" or any other attribute. It is an error for any
match operator enclosing a nested "start" or "end" element to have a
"count" attribute.
Davies & Freytag Expires June 11, 2016 [Page 32]
Internet-Draft Label Generation Rulesets in XML December 2015
5.3.9. Example rule from IDNA2008
This is an example of the whole label evaluation rule from [RFC5892]
forbidding the mixture of the Arabic-Indic and extended Arabic-Indic
digits in the same label. The example also demonstrates several
instances of the use of anonymous rules for grouping.
The effect of this example is that a label containing a code point
from either of the two digit ranges is invalid for any label matching
the "mixed-digits" rule, that is, anytime a code point from the other
range is also present. Note that this is not the same as
invalidating the definition of the "range" elements.
5.4. Parameterized Context or When Rules
A special type of rule provides a context for evaluating the validity
of a code point or variant mapping. This rule is invoked by the
"when" or "not-when" attributes described in Section 4.2. For "char"
and "range" elements, an action implied by a context rule always has
a disposition of "invalid" whenever the rule given by the "when"
attribute is not matched (see Section 6.5). Conversely, a "not-when"
attribute results in a disposition of "invalid" whenever the rule is
matched.
Davies & Freytag Expires June 11, 2016 [Page 33]
Internet-Draft Label Generation Rulesets in XML December 2015
5.4.1. The anchor Element
Such parameterized context or "When Rules" may contain a special
place holder represented by an "anchor" element. As each When Rule
is evaluated, the "anchor" element is replaced by a literal
corresponding to the "cp" attribute of the element containing the
"when" (or "not-when") attribute. The match to the "anchor" element
must be at the same position in the label as the code point or
variant mapping triggering the When Rule.
For example, the Greek lower numeral sign is invalid if not
immediately preceding a character in the Greek script. This is most
naturally addressed with a When Rule using look-ahead:
...
In evaluating this rule, the "anchor" element is treated as if it was
replaced by a literal
but only the instance of U+0375 at the given position is evaluated.
If a label had two instances of U+0375 with the first one matching
the rule and the second not, then evaluating the When Rule MUST
succeed for the first and fail for the second instance.
Unlike other rules, When Rules containing an "anchor" element MUST
only be invoked via the "when" or "not-when" attributes on code
points or variants; otherwise their "anchor" elements cannot be
evaluated. However, it is possible to invoke rules not containing an
"anchor" element from a "when" or "not-when" attribute. (See
Section 5.4.3)
5.4.2. The look-behind and look-ahead Elements
Context rules use the "look-behind" and "look-ahead" elements to
define context before and after the code point sequence matched by
the "anchor" element. If the "anchor" element is omitted, neither
the "look-behind" nor the "look-ahead" element may be present.
Davies & Freytag Expires June 11, 2016 [Page 34]
Internet-Draft Label Generation Rulesets in XML December 2015
Here is an example of a rule that defines an "initial" context for an
Arabic code point:
A "when rule" contains any combination of "look-behind", "anchor" and
"look-ahead" elements in that order. Each of these elements occurs
at most once, except if nested inside a "choice" element in such a
way that in matching each alternative at most one occurrence of each
is encountered. Otherwise, the result is undefined. None of these
elements takes a "count" attribute, nor does any enclosing match
operator. Otherwise, the result is undefined. If a context rule
contains a "look-ahead" or "look-behind" element, it MUST contain an
"anchor" element. If, because of a choice element, a required anchor
is not actually encountered, the results are undefined.
5.4.3. Omitting the anchor Element
If the "anchor" element is omitted, the evaluation of the context
rule is not tied to the position of the code point or sequence
associated with the "when" attribute.
According to [RFC5892] Katakana middle dot is invalid in any label
not containing at least one Japanese character anywhere in the label.
Davies & Freytag Expires June 11, 2016 [Page 35]
Internet-Draft Label Generation Rulesets in XML December 2015
Because this requirement is independent of the position of the middle
dot, the rule does not require an "anchor" element.
The Katakana middle dot is used only with Han, Katakana or Hiragana.
The corresponding When Rule requires that at least one code point in
the label is in one of these scripts, but the position of that code
point is independent of the location of the middle dot and no anchor
therefore required. (Note that the Katakana middle dot itself is of
script Common, that is, "sc:Zyyy").
6. The action Element
The purpose of a rule is to trigger a specific action. Often, the
action simply results in blocking or invalidating a label that does
not match a rule. An example of an action invalidating a label
because it does not match a rule named "leading-letter" is as
follows:
If an action is to be triggered on matching a rule, a "match"
attribute is used instead. Actions are evaluated in the order that
they appear in the XML file. Once an action is triggered by a label,
the disposition defined in the "disp" attribute is assigned to the
label and no other actions are evaluated for that label.
The goal of the Label Generation Rules is to identify all labels and
variant labels and to assign them disposition values. These
dispositions are then fed into a further process that ultimately
implements all aspects of policy. To allow this specification to be
used with the widest range of policies, the permissible values for
the "disp" attribute are neither defined nor restricted.
Nevertheless a set of commonly used disposition values is
RECOMMENDED. (See Section 6.3)
Davies & Freytag Expires June 11, 2016 [Page 36]
Internet-Draft Label Generation Rulesets in XML December 2015
6.1. The match and not-match Attributes
A "match" or "not-match" attribute specify a rule that must be
matched or not matched as a condition for triggering an action. Only
a single rule may be named as the value of a "match" or "not-match"
attribute. Because rules may be composed of other rules, this
restriction to a single attribute value does not impose any
limitation on the contexts that can trigger an action.
An action may contain a "match" or a "not-match" attribute, but not
both. An action without any attributes is triggered by all labels
unconditionally. For a very simple LGR, the following action would
allocate all labels that match the repertoire:
Since rules are evaluated for all labels, whether they are the
original label or computed by permuting the defined and valid variant
mappings for the label's code points, actions based on matching or
not matching a rule may be triggered for both original and variant
labels, but they the rules are not affected by the disposition
attributes of the variant mappings. To trigger any actions base on
these dispositions requires the use additional optional attributes
for actions described next.
6.2. Actions with Variant Type Triggers
6.2.1. The all-, any- and only-variants Attributes
An action may contain one of the optional attributes "any-variant",
"all-variants", or "only-variants" defining triggers based on variant
types. The permitted value for these attributes consists of one or
more variant type values, separated by spaces. When a variant label
is generated, these variant type values are compared to the set of
type values on the variant mappings used to generate the particular
variant label (see Section 7).
Any single match may trigger an action that contains an "any-variant"
attribute, while for an "all-variants" or "only-variants" attribute,
the variant type for all variant code points must match one or
several of the type values specified in the attribute to trigger the
action. There is no requirement that the entire list of variant type
values be matched, as long as all variant code points match at least
one of the values.
An "only-variants" attribute will trigger the action only if all code
points of the variant label have variant mappings from the original
Davies & Freytag Expires June 11, 2016 [Page 37]
Internet-Draft Label Generation Rulesets in XML December 2015
code points. In other words, the label contains no original code
points other than those with a reflexive mapping (see Section 4.3.4).
. . .
In the example above, the label "xx" would have variant labels "xx",
"xy", "yx" and "yy". The first action would result in blocking any
variant label containing "y", because the variant mapping from "x" to
"y" is of type "blocked", triggering the "any-variant" condition.
Because in this example "x" has a reflexive variant mapping to itself
of type "allocatable" the original label "xx" has a reflexive variant
"xx" that would trigger the "only_variants" condition on the second
action.
A label "yy" would have the variants "xy", "yx" and "xx". Because
the variant mapping from "y" to "x" is of type "allocatable" and a
mapping from "y" to "y" is not defined, the labels "xy" and "yx"
trigger the "any-variant" condition on the third label. The variant
"xx", being generated using the mapping from "y" to "x" of type
"allocatable", would trigger the "only-variants" condition on the
section action. As there is no reflexive variant "yy", the original
label "yy" cannot trigger any variant type triggers. However, it
could still trigger an action defined as matching or not matching a
rule.
In each action, one variant type trigger may be present by itself or
in conjunction with an attribute matching or not-matching a rule. If
variant triggers and rule-matching triggers are used together, the
label MUST "match" or respectively "not-match" the specified rule,
AND satisfy the conditions on the variant type values given by the
"any-variant", "all-variants", or "only-variants" attribute.
A useful convention combines the "any-variant" trigger with reflexive
variant mappings (Section 4.3.4). This convention is used, for
example, when multiple LGRs are defined within the same registry and
for overlapping repertoire. In some cases, the delegation of a label
from one LGR must prohibit the delegation of another label in some
other LGR. This can be done using a variant of type "blocked" as in
Davies & Freytag Expires June 11, 2016 [Page 38]
Internet-Draft Label Generation Rulesets in XML December 2015
this example from an Armenian LGR, where the Armenian, Latin and
Cyrillic letters all look identical:
The issue is that the target code points for these two variants are
both outside the Armenian repertoire. By using a reflexive variant
with the following convention:
...
and associating this with an action of the form:
it is possible to list the symmetric and transitive variant mappings
in the LGR even where they involve out-of-repertoire code points. By
associating the action shown with the special type for these
reflexive mappings any original labels containing one or more of the
out-of-repertoire code points are filtered out -- just as if these
code points had not been listed in the LGR in the first place.
Nevertheless, they do participate in the permutation of variant
labels for n-repertoire labels (Armenian in the example), and these
permuted variants can be used to detect collisions with out-of-
repertoire labels (see Section 7).
6.2.2. Example from RFC 3743 Tables
This section gives an example of using variant type triggers,
combined with variants with reflexive mappings (Section 4.3.4) to
achieve LGRs that implement tables like those defined according to
[RFC3743] where the goal is to allow as variants only labels that
consist entirely of simplified or traditional variants, in addition
to the original label.
Assuming an LGR where all variants have been given suitable "type"
attributes of "blocked", "simplified", "traditional", or "both",
similar to the ones discussed in Appendix B. Given such an LGR, the
Davies & Freytag Expires June 11, 2016 [Page 39]
Internet-Draft Label Generation Rulesets in XML December 2015
following example actions evaluate the disposition for the variant
label:
The first action matches any variant label for which at least one of
the code point variants is of type "blocked". The second matches any
variant label for which all of the code point variants are of type
"simplified" or "both", in other words an all-simplified label. The
third matches any label for which all variants are of type
"traditional" or "both", that is all traditional. These two actions
are not triggered by any variant labels containing some original code
points, unless each of those code points has a variant defined with a
reflexive mapping (Section 4.3.4).
The final two actions rely on the fact that actions are evaluated in
sequence, and that the first action triggered also defines the final
disposition for a variant label (see Section 6.4). They further rely
on the assumption that the only variants with type "both" are also
reflexive variants.
Given these assumptions, any remaining simplified or traditional
variants must then be part of a mixed label, and so are blocked; all
labels surviving to the last action are original code points only
(that is the original label). The example assumes that an original
label may be a mixed label; if that is not the case, the disposition
for the last action would be set to "blocked".
There are exceptions where the assumption on reflexive mappings made
above does not hold, so this basic scheme needs some refinements to
cover all cases. For a more complete example, see Appendix B.
6.3. Recommended Disposition Values
The precise nature of the policy action taken in response to a
disposition and the name of the corresponding "disp" attributes are
only partially defined here. It is strongly RECOMMENDED to use the
following dispositions only with their conventional sense.
invalid The resulting string is not a valid label. This disposition
may be assigned implicitly, see Section 6.5. No variant labels
should be generated from a variant mapping with this type.
Davies & Freytag Expires June 11, 2016 [Page 40]
Internet-Draft Label Generation Rulesets in XML December 2015
blocked The resulting string is a valid label, but should be blocked
from registration. This would typically apply for a derived
variant that has is undesirable as having no practical use or
being confusingly similar to some other label.
allocatable The resulting string should be reserved for use by the
same operator of the origin string, but not automatically
allocated for use.
activated The resulting string should be activated for use. (This
is the typical default action if no dispositions are defined and
is known as a "preferred" variant in [RFC3743])
6.4. Precedence
Actions are applied in the order of their appearance in the file.
This defines their relative precedence. The first action triggered
by a label defines the disposition for that label. To define a
specific order of precedence, list the actions in the desired order.
The conventional order of precedence for the actions defined in
Section 6.3 is "invalid", "blocked", "allocatable", then "activated".
This default precedence is used for the default actions defined in
Section 6.6.
6.5. Implied Actions
The context rules on code points ("not-when" or "when" rules) carry
an implied action with a disposition of "invalid" (not eligible).
These rules are evaluated at the time the code points for a label or
its variant labels are checked for validity (see Section 7). In
other words, they are evaluated before any of the whole-label
evaluation rules and with higher precedence. The context rules for
variant mappings are evaluated when variants are generated and/or
when variant tables are made symmetric and transitive. They have an
implied action with a disposition of "invalid" which means a putative
variant mapping does not exist whenever the given context matches a
"not-when" rule or fails to match a "when" rule specified for that
mapping. The result of that disposition is that the variant mapping
is ignored in generating variant labels and the value is therefore
not accessible to trigger any explicit actions.
Note that such non-existing variant mapping is different from a
blocked variant, which is a variant code point mapping that exists
but results in a label that may not be allocated.
Davies & Freytag Expires June 11, 2016 [Page 41]
Internet-Draft Label Generation Rulesets in XML December 2015
6.6. Default Actions
As described in Section 6 any variant mapping may be given a "type"
attribute. An action containing an "any-variant" or "all-variants"
attribute relates these type values to a resulting disposition for
the entire variant label.
If no actions are defined for the standard disposition values of
"invalid", "blocked", "allocatable" and "activated", then the
following default actions exist that are shown below in their default
order of precedence (see Section 6.4). This default order for
evaluating dispositions applies only to labels that triggered no
explicitly defined actions and which are therefore handled by default
actions. Default actions have a lower order of precedence than
explicit actions (see Section 7.3).
The default actions for variant labels are defined as follows:
A final default action sets the disposition to "allocatable" for any
label matching the repertoire for which no other action has been
triggered (catch-all).
7. Processing a Label against an LGR
7.1. Determining Eligibility for a Label
In order to test a specific label for membership in the LGR, a
consumer of the LGR must iterate through each code point within a
given label, and test that each code point is a member of the LGR.
If any code point is not a member of the LGR, the label shall be
deemed as invalid.
An individual code point is deemed a member of the LGR when it is
listed using a "char" element, or is part of a range defined with a
"range" element, and all necessary condition in any "when" or "not-
when" attributes are correctly satisfied.
Alternatively, a code point is also deemed a member of the LGR when
it forms part of a sequence that corresponds to a sequence listed
using a "char" element for which the "cp" attribute defines a
Davies & Freytag Expires June 11, 2016 [Page 42]
Internet-Draft Label Generation Rulesets in XML December 2015
sequence, and all necessary condition in any "when" or "not-when"
attributes are correctly satisfied.
A label must also not trigger any action that results in a
disposition of "invalid", otherwise it is deemed not eligible. (This
step may need to be deferred, until variant code point dispositions
have been determined).
For LGRs that contain reflexive variant mappings (defined in
Section 4.3.4), the final evaluation of eligibility for the label
must be deferred until variants are generated. In essence, LGRs that
use this feature treat the original label as the (identity) variant
of itself. For such LGRs, the ordinary iteration over code points
would generally only exclude a subset of invalid labels, but it could
be used effectively as a pre-screening.
To check the validity of a label with reflexive mappings, it is not
necessary to generate all variant labels. Only a single variant
needs to be created, where for each code point, any reflexive
variants are applied, and the label disposition is evaluated. A
disposition of "invalid" results in the label being not eligible.
(In the exceptional case where context rules are present on reflexive
mappings, multiple reflexive variants may be defined, but for each
original label, at most one of these can be valid at each code
position. However, see Section 7.4).
7.2. Determining Variants for a Label
For a given eligible label, the set of variant labels is deemed to
consist of each possible permutation of original code points and
substituted code points or sequences defined in "var" elements,
whereby all "when" and "not-when" attributes are correctly satisfied
for each "char" or "var" element in the given permutation and all
applicable whole label evaluation rules are satisfied as follows:
1. Create each possible permutation of a label, by substituting each
code point or code point sequence in turn by any defined variant
mapping (including any reflexive mappings)
2. Apply variant mappings with "when" or "not-when" attributes only
if the conditions are satisfied; otherwise they are not defined
3. Record each of the "type" values on the variant mappings used in
creating a given variant label in a disposition set; for any
unmapped code point record the "type" value of any reflexive
variant (see Section 4.3.4)
4. Determine the disposition for each variant label per Section 7.3
Davies & Freytag Expires June 11, 2016 [Page 43]
Internet-Draft Label Generation Rulesets in XML December 2015
5. If the disposition is "invalid", remove the label from the set
6. If final evaluation of the disposition for the unpermuted label
per Section 7.3 results in a disposition of "invalid", remove all
associated variant labels from the set.
The number of potential permutations can be very large. In practice,
implementations would use suitable optimizations to avoid having to
actually create all permutations.
7.3. Determining a Disposition for a Label or Variant Label
For a given label (variant or original), its disposition is
determined by evaluating in order of their appearance all actions for
which the label or variant label satisfies the conditions.
1. For any label, the disposition is given by the value of the
"disp" attribute for the first action triggered by the label. An
action is triggered, if all of the following are true:
* the label matches the whole label evaluation rule given in the
"match" attribute for that action;
* the label does not match the whole label evaluation rule given
in the "not-match" attribute for that action;
* any of the recorded variant types for a variant label match
the types given in the "any-variant" attribute for that
action;
* all of the recorded variant types for a variant label match
the types given in the "all-variants" or "only-variants"
attribute given for that action;
* in case of an "only-variants" attribute, the label contains
only code points that are the target of applied variant
mappings;
or
* the action does not contain any "match", "not-match", "any-
variant", "all-variants", or "only-variants" attributes:
catch-all.
2. For any remaining variant label, assign the variant label the
disposition using the default actions defined in Section 6.6.
For this step, variant types outside the predefined recommended
set (see Section 6.3) are ignored.
Davies & Freytag Expires June 11, 2016 [Page 44]
Internet-Draft Label Generation Rulesets in XML December 2015
3. For any remaining label, set the disposition to "allocatable".
7.4. Duplicate Variant Labels
For a poorly designed LGR, it is possible to generate duplicate
variant labels from the same input label, but with different, and
potentially conflicting dispositions. Implementations MUST treat any
duplicate variant labels encountered as an error, irrespective of
their dispositions.
This situation can arise in two ways. One is described in
Section 4.3.5 and involves defining the same variant mapping with two
context rules that are formally distinct, but nevertheless overlap so
that they are not mutually exclusive for the same label.
The other case involves variants defined for sequences, where one
sequence is a prefix of another (see Section 4.3.1). The following
shows such an example resulting in conflicting reflexive variants:
A label "ab" would generate the variant labels "{a}{b}" and "{ab}"
where the curly braces show the sequence boundaries as they were
applied during variant mapping. The result is a duplicate variant
label "ab", one based on a variant of type "allocatable" plus an
original code point "b" that has no variant, and another one based on
a single variant of type "blocked", thus creating two variant labels
with conflicting dispositions.
In the general case it is difficult to impossible to prove by
mechanical inspection of the LGR that duplicate variant labels will
never occur, so implementations have to be prepared to detect this
error during variant label generation. The condition is easily
avoided by careful design of context rules and special attention to
the relation among code point sequences with variants.
7.5. Checking Labels for Collision
The obvious method for checking collision between labels is to
generate the fully permuted set of variants for one of them and see
whether it contains the other label as a member. As discussed above,
this can be prohibitive, and is not necessary.
Davies & Freytag Expires June 11, 2016 [Page 45]
Internet-Draft Label Generation Rulesets in XML December 2015
Because of symmetry and transitivity, all variant mappings form
disjoint sets in which each of them is a variant of all other
members. As a consequence, the set of variant labels likewise forms
disjoint subsets, based on which set of mappings was used.
Instead of generating all permutations, that is, use each variant
mapping in each set, it is sufficient to substitute an "index"
mapping, in effect identifying the set of variant code points. Such
an index mapping could be, for example, the variant mapping for which
the target code point (or sequence) comes first n some sorting order.
To check collision then means generating a single variant label from
the original, by substituting the "index" value as the target for
mapping from any code point. This results in an "index label". Two
labels collide whenever the index labels for them are the same.
8. Conversion to and from Other Formats
Both [RFC3743] and [RFC4290] provide different grammars for IDN
tables. These formats are unable to fully cater for the increased
requirements of contemporary IDN variant policies.
This specification is a superset of functionality provided by these
IDN table formats, thus any table expressed in those formats can be
expressed in this format. Automated conversion can be conducted
between tables conformant with the grammar specified in each
document.
For notes on how to translate an RFC 3743-style table, see
Appendix B.
9. Media Type
Transmission of a well-formed LGR in accordance with this
specification SHOULD be transmitted with a media type of
"application/lgr+xml". This media type will signal to an LGR-aware
client that the content is designed to be interpreted as an LGR.
10. IANA Considerations
This document requests the following actions from IANA:
10.1. Media Type Registration
The media type "application/lgr+xml" should be registered to denote
transmission of label generation rulesets that are compliant with
this specification, in accordance with [RFC6838].
Davies & Freytag Expires June 11, 2016 [Page 46]
Internet-Draft Label Generation Rulesets in XML December 2015
Type name: application
Subtype name: lgr+xml
Required parameters: None.
Optional parameters: charset (as for application/xml per [RFC7303])
Security considerations: As for application/xml per [RFC7303]
Interoperability considerations: As for application/xml per [RFC7303]
Published specification: This document.
Applications which use this media type: Software using label
generation rulesets, including registry applications and client
validators.
Additional information: None.
Magic number: None.
File extension: .lgr or .xml
Macintosh file-type code: None.
Object Identifiers: None.
Intended Usage: Common
Personal and email address for further information: See the Authors
of this document.
Change Controller: World Wide Web Consortium (W3C)
10.2. URN Registration
This specification uses a URN to describe the XML namespace, in
accordance with [RFC3688].
URI: urn:ietf:params:xml:ns:lgr-1.0
Registrant Contact: See the Authors of this document.
XML: None.
Davies & Freytag Expires June 11, 2016 [Page 47]
Internet-Draft Label Generation Rulesets in XML December 2015
10.3. Disposition Registry
This document establishes a vocabulary of "Label Generation Ruleset
Dispositions" which should be reflected as a new IANA registry. This
registry should be divided into two sub-registries:
o Standard Dispositions - This registry shall list dispositions that
have been defined in Standards Track documents. The initial set
of registrations shall be the four dispositions in this document
described in Section 6.3.
o Private Dispositions - This registry shall list dispositions that
have been registered on a first-come first-served basis by third
parties with the IANA. Such dispositions must take the form
"entity:disposition" where the entity is a prefix that uniquely
identifies the private user of the namespace. For example,
"acme:reserved" could be a private extension used by the
organisation ACME to denote a disposition relating to reserved
labels. These extensions are not intended to be interoperable,
but registration is designed to minimize potential conflicts. It
is strongly recommended any new dispositions that require
interoperability and have applicability beyond a single
organization be defined as Standard Dispositions.
All private dispositions MUST be registered using the prefix-colon
notation to distinguish them from standard dispositions.
The IANA registry should provide data on the name of the disposition,
the intended purposes, and the registrant or defining specification
for the disposition.
11. Security Considerations
If a system that is querying an identifier list (such as a domain
zone) that uses the rules in this memo, and those rules are not
implemented correctly, and that system is relying on the rules being
applied, the system might fail if the rules are not applied in a
predictable fashion. This could cause security problems for the
querying system.
A naive implementation attempting to generate all variant labels for
a given label could lead to the possibility of exhausting the
resources on the machine running the LGR processor, potentially
causing a DoS on the server. For many operations, brute force
generation can be avoided by optimization, and if needed, the number
of permuted labels can be estimated more cheaply ahead of time.
Davies & Freytag Expires June 11, 2016 [Page 48]
Internet-Draft Label Generation Rulesets in XML December 2015
The implementation of Whole Label Evaluation rules, using certain
backtracking algorithms, can take exponential time for pathological
rules or labels and exhaust stack resources. This can be mitigated
by proper implementation and enforcing the restrictions on
permissible label length.
12. References
12.1. Normative References
[RFC3339] Klyne, G. and C. Newman, "Date and Time on the Internet:
Timestamps", RFC 3339, DOI 10.17487/RFC3339, July 2002,
.
[RFC3688] Mealling, M., "The IETF XML Registry", BCP 81, RFC 3688,
DOI 10.17487/RFC3688, January 2004,
.
[RFC5646] Phillips, A., Ed. and M. Davis, Ed., "Tags for Identifying
Languages", BCP 47, RFC 5646, DOI 10.17487/RFC5646,
September 2009, .
[RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type
Specifications and Registration Procedures", BCP 13,
RFC 6838, DOI 10.17487/RFC6838, January 2013,
.
[RFC7303] Thompson, H. and C. Lilley, "XML Media Types", RFC 7303,
DOI 10.17487/RFC7303, July 2014,
.
[UAX42] Unicode Consortium, "Unicode Character Database in XML".
[Unicode-Stability]
Unicode Consortium, "Unicode Encoding Stability Policy,
Property Value Stability".
[Unicode-Versions]
Unicode Consortium, "Unicode Version Numbering".
[XML] World Wide Web Consortium, "Extensible Markup Language
(XML) 1.0".
12.2. Informative References
[ASIA-TABLE]
DotAsia Organisation, ".ASIA ZH IDN Language Table".
Davies & Freytag Expires June 11, 2016 [Page 49]
Internet-Draft Label Generation Rulesets in XML December 2015
[LGR-PROCEDURE]
Internet Corporation for Assigned Names and Numbers,
"Procedure to Develop and Maintain the Label Generation
Rules for the Root Zone in Respect of IDNA Labels".
[RFC3743] Konishi, K., Huang, K., Qian, H., and Y. Ko, "Joint
Engineering Team (JET) Guidelines for Internationalized
Domain Names (IDN) Registration and Administration for
Chinese, Japanese, and Korean", RFC 3743,
DOI 10.17487/RFC3743, April 2004,
.
[RFC4290] Klensin, J., "Suggested Practices for Registration of
Internationalized Domain Names (IDN)", RFC 4290,
DOI 10.17487/RFC4290, December 2005,
.
[RFC5564] El-Sherbiny, A., Farah, M., Oueichek, I., and A. Al-Zoman,
"Linguistic Guidelines for the Use of the Arabic Language
in Internet Domains", RFC 5564, DOI 10.17487/RFC5564,
February 2010, .
[RFC5892] Faltstrom, P., Ed., "The Unicode Code Points and
Internationalized Domain Names for Applications (IDNA)",
RFC 5892, DOI 10.17487/RFC5892, August 2010,
.
[TDIL-HINDI]
Technology Development for Indian Languages (TDIL)
Programme, "Devanagari Script Behaviour for Hindi".
[WLE-RULES]
Internet Corporation for Assigned Names and Numbers, "WLE
Rules".
Appendix A. Example Tables
The following presents a minimal LGR table defining the lower case
LDH (letter-digit-hyphen) repertoire and containing no rules or
metadata elements. Many simple LGR tables will look quite similar,
except that they would contain some metadata.
Davies & Freytag Expires June 11, 2016 [Page 50]
Internet-Draft Label Generation Rulesets in XML December 2015
The following sample LGR shows a more complete collection of the
elements and attributes defined in this specification in a somewhat
typical context.
1
2010-01-01
sv
example.com
2010-01-01
2013-12-31
Swedish
examples institute.
]]>
6.3.0
The
Unicode Standard 6.2
RFC 5892
Big-5: Computer Chinese Glyph
and Character Code Mapping Table, Technical Report
C-26, 1984
Davies & Freytag Expires June 11, 2016 [Page 51]
Internet-Draft Label Generation Rulesets in XML December 2015
Davies & Freytag Expires June 11, 2016 [Page 52]
Internet-Draft Label Generation Rulesets in XML December 2015
0061-007A
0061 0065 0069 006F 0075
Davies & Freytag Expires June 11, 2016 [Page 53]
Internet-Draft Label Generation Rulesets in XML December 2015
Appendix B. How to Translate RFC 3743 based Tables into the XML Format
As a background, the [RFC3743] rules work as follows:
1. The Original (requested) label is checked to make sure that all
the code points are a subset of the repertoire.
2. If it passes the check, the Original label is allocatable.
3. Generate the all-simplified and all-traditional variant labels
(union of all the labels generated using all the simplified
variants of the code points) for allocation.
To illustrate by example, here is one of the more complicated set of
variants:
U+4E7E
U+4E81
U+5E72
U+5E79
U+69A6
U+6F27
The following shows the relevant section of the Chinese language
table published by the .ASIA registry [ASIA-TABLE]. Its entries
read:
;;;
These are the lines corresponding to the set of variants listed above
U+4E7E;U+4E7E,U+5E72;U+4E7E;U+4E81,U+5E72,U+6F27,U+5E79,U+69A6
U+4E81;U+5E72;U+4E7E;U+5E72,U+6F27,U+5E79,U+69A6
U+5E72;U+5E72;U+5E72,U+4E7E,U+5E79;U+4E7E,U+4E81,U+69A6,U+6F27
U+5E79;U+5E72;U+5E79;U+69A6,U+4E7E,U+4E81,U+6F27
U+69A6;U+5E72;U+69A6;U+5E79,U+4E7E,U+4E81,U+6F27
U+6F27;U+4E7E;U+6F27;U+4E81,U+5E72,U+5E79,U+69A6
The corresponding data section XML format would look like this:
Davies & Freytag Expires June 11, 2016 [Page 54]
Internet-Draft Label Generation Rulesets in XML December 2015
Here the simplified variants have been given a type of "simp", the
traditional variants one of "trad" and all other ones are given
"blocked".
Davies & Freytag Expires June 11, 2016 [Page 55]
Internet-Draft Label Generation Rulesets in XML December 2015
Because some variant mappings show in more than one column, while the
XML format allows only a single type value, they have been given the
type of "both".
Note that some variant mappings map to themselves (identity), that is
the mapping is reflexive (see Section 4.3.4). In creating the
permutation of all variant labels, these mappings have no effect,
other than adding a value to the variant type list for the variant
label containing them.
In the example so far, all of the entries with type="both" are also
mappings where source and target are identical. That is, they are
reflexive mappings as defined in Section 4.3.4.
Given a label "U+4E7E U+4E81", the following labels would be ruled
allocatable under [RFC3743] based on how that standard is commonly
implemented in domain registries:
Original label: U+4E7E U+4E81
Simplified label 1: U+4E7E U+5E72
Simplified label 2: U+5E72 U+5E72
Traditional label: U+4E7E U+4E7E
However, if allocatable labels were generated simply by a straight
permutation of all variants with type other than type="blocked" and
without regard to the simplified and traditional variants, we would
end up with an extra allocatable label of "U+5E72 U+4E7E". This
label is comprised of a both Simplified Chinese character and a
Traditional Chinese code point and therefore shouldn't be
allocatable.
To more fully resolve the dispositions requires several actions to be
defined as described in Section 6.2.2 which will override the default
actions from Section 6.6. After blocking all labels that contain a
variant with type "blocked", these actions will set to allocatable
labels based on the following variant types: "simp", "trad" and
"both". Note that these variant types do not directly relate to
dispositions for the variant label, but that the actions will resolve
them to the standard dispositions on labels, to with "blocked" and
"allocatable".
To resolve label dispositions requires five actions to be defined (in
the rules section of this document) these actions apply in order and
the first one triggered, defines the disposition for the label. The
actions are:
1. block all variant labels containing at least one blocked variant.
Davies & Freytag Expires June 11, 2016 [Page 56]
Internet-Draft Label Generation Rulesets in XML December 2015
2. allocate all labels that consist entirely of variants that are
"simp" or "both"
3. also allocate all labels that are entirely "trad" or "both"
4. block all surviving labels containing any one of the dispositions
"simp" or "trad" or "both" because they are now known to be part
of an undesirable mixed simplified/traditional label
5. allocate any remaining label; the original label would be such a
label.
The rules declarations would be represented as:
Up to now, variants with type "both" have occurred only associated
with reflexive variant mappings. The "action" elements defined above
rely on the assumption that this is always the case. However,
consider the following set of variants:
U+62E0;U+636E;U+636E;U+64DA
U+636E;U+636E;U+64DA;U+62E0
U+64DA;U+636E;U+64DA;U+62E0
The corresponding XML would be:
Davies & Freytag Expires June 11, 2016 [Page 57]
Internet-Draft Label Generation Rulesets in XML December 2015
To make such variant sets work requires a way to selectively trigger
an action based on whether a variant type is associated with an
identity or reflexive mapping, or is associated with an ordinary
variant mapping. This can be done by adding a prefix "r-" to the
"type" attribute on reflexive variant mappings. For example the
"trad" for code point U+64DA in the preceding figure would become
"r-trad".
With the dispositions prepared in this way, only a slight
modification to the actions is needed to yield the correct set of
allocatable labels:
The first three actions get triggered by the same labels as before.
The fourth action blocks any label that combines an original code
point with any mix of ordinary variant mappings; however no labels
that are a combination of only original code points (code points
having either no variant mappings or a reflexive mapping) would be
affected. These are the original labels and they are allocated in
the last action.
Using this scheme of assigning types to ordinary and reflexive
variants, all RFC 3743-style tables can be converted to XML. By
defining a set of actions as outlined above, the LGR will yield the
correct set of allocatable variants: all variants consisting
completely of variant code points preferred for simplified or
traditional, respectively, will be allocated, as will be the original
label. All other variant labels will be blocked.
Appendix C. Indic Syllable Structure Example
In LGRs for Indic scripts it may be desirable to restrict valid
labels to sequences of valid Indic syllables, or aksharas. This
appendix gives a sample set of rules designed to enforce this
restriction.
An example of a BNF from for an akshara which has been published in
"Devanagari Script Behavior for Hindi" [TDIL-HINDI]. The rules for
other languages and scripts used in India are expected to be
generally similar.
For Hindi, the BNF has the form:
Davies & Freytag Expires June 11, 2016 [Page 58]
Internet-Draft Label Generation Rulesets in XML December 2015
V[m]|{C[N]H}C[N](H|[v][m])
Where:
V (upper case) is any independent vowel
m is any vowel modifier (Devanagari Anusvara, Visarga, and
Candrabindu)
C is any consonant (with inherent vowel)
N is Nukta
H is a Halant (or Virama)
v (lower case) is any dependent vowel sign (matra)
{} encloses items which may be repeated one or more times
[ ] encloses items which may or may not be present
| separates items, out of which only one can be present
By using the Unicode property "InSC" or "Indic_Syllabic_Category"
which corresponds rather directly to the classification of characters
in the BNF above, we can translate the BNF into a set of WLE rules
matching the definition of an akshara.
Davies & Freytag Expires June 11, 2016 [Page 59]
Internet-Draft Label Generation Rulesets in XML December 2015
With the rules and classes as defined above, the final action assigns
a disposition of "invalid" to all labels that are not composed of a
sequence of well-formed aksharas, optionally interspersed with other
characters, perhaps digits, for example.
The relevant Unicode property could be replicated by tagging
repertoire values directly in the LGR which would remove the
dependency on any specific version of the Unicode Standard.
Generally, dependent vowels may only follow consonant expressions,
however, for some scripts, like Bengali, the Unicode standard
supports sequences of dependent vowels or their application on
independent vowels. This makes the definition of akshara less
restrictive.
It is possible to reduce the complexity of these rules by defining
alternate rules which simply define the permissible pair-wise context
Davies & Freytag Expires June 11, 2016 [Page 60]
Internet-Draft Label Generation Rulesets in XML December 2015
of adjacent code points by character class--such as the rule that a
Halant can only follow a (nuktated) consonant. (See the example in
[WLE-RULES]).
Appendix D. RelaxNG Compact Schema
%schema%
Appendix E. Acknowledgements
This format builds upon the work on documenting IDN tables by many
different registry operators. Notably, a comprehensive language
table for Chinese, Japanese and Korean was developed by the "Joint
Engineering Team" [RFC3743] that is the basis of many registry
policies; and a set of guidelines for Arabic script registrations
[RFC5564] was published by the Arabic-language community.
Contributions that have shaped this document have been provided by
Francisco Arias, Mark Davis, Martin Duerst, Paul Hoffman, Alexander
Mayrhofer, Nicholas Ostler, Thomas Roessler, Audric Schiltknecht,
Steve Sheng, Michel Suignard, Andrew Sullivan, Wil Tan and John
Yunker.
Appendix F. Editorial Notes
This appendix to be removed prior to final publication.
F.1. Known Issues and Future Work
These are being tracked at http://trac.tools.ietf.org/wg/lager/trac/
report/1
F.2. Change History
draft-davies-idntables-00 Initial draft.
draft-davies-idntables-01 Add an XML Namespace, and fix other XML
nits. Add support for sequences of code points. Improve on
consistently using Unicode nomenclature.
draft-davies-idntables-02 Add support for validity periods.
draft-davies-idntables-03 Incorporate requirements from the Label
Generation Ruleset Procedure for the DNS Root Zone. These
requirements include a detailed grammar for specifying whole-
label variants, and the ability to explicitly declare of the
actions associated with a specific variant. The document also
consistently applies the term "Label Generation Ruleset", rather
Davies & Freytag Expires June 11, 2016 [Page 61]
Internet-Draft Label Generation Rulesets in XML December 2015
than "IDN table", to reflect the policy term now being used to
describe these.
draft-davies-idntables-04 Support reference information per
[RFC3743]. Update description in response to feedback. Extend
the context rules to "char" elements and allow for inverse
matching ("not-when"). Extend the description of label
processing and implied actions, and allow for actions that
reference disposition attributes on any or all variant mappings
used in the generation of a variant label.
draft-davies-idntables-05 Change the name of the "disposition"
attribute to "disp". Add comment attribute on version and
reference elements. Allow empty "cp" attributes in char
elements to support expressing symmetric mapping of null
variants. Describe use of variants that map identically.
Clarify how actions are triggered, in particular based on
variant dispositions, as well as description of default actions.
Revise description of processing a label and its variants. Move
example table at the head of appendices. Add "only-variants"
attribute. Change "name" attribute to "by-ref" attribute for
referencing named classes and rules. Change "not" to
"complement". Remove "match" attribute on rules as redundant if
"start" and "end" are supported. Rename "match" element to
"anchor" as better fitting its function and removing confusion
with both the "match" attribute on actions as well as the
generic term Match Operator. Augmented the examples relevant to
[RFC3743].
draft-davies-idntables-06 Extend the discussion of reflexive
variants and their use; includes update of the appendix on
converting tables in the style of [RFC3743]. Improve
description of tagging and clarify that it doesn't apply to
sequences. Specify that root zone uses ".". Add an appendix
with an Indic Syllable Structure example. Extend count
attribute to allow maximal counts.
draft-davies-idntables-07 Change "byref" to "by-ref". Add list of
recommended properties. Change "location" to "positional" for
collective name of start/end match operators. Use from-tag
instead of by-ref for tag-based classes. Made optional or
mutually exclusive nature of some attributes more explicit.
Allowing "comment" attributes on all child elements of "rules"
except "char" and "range" elements used as child elements of
"class". Recast the design goals and requirements at the start
of the document. Reword aspects of the document to make it
clear the format's application is not limited only to domain
names.
Davies & Freytag Expires June 11, 2016 [Page 62]
Internet-Draft Label Generation Rulesets in XML December 2015
draft-davies-idntables-08 Change "domain" to scope with
type="domain". Reword in several places for clarity. Flesh out
note on security. Change "disp" to "type" for variants, to mark
that these attributes do not necessarily correspond one-to-one
to variant label dispositions. Add example of variant type
triggers. Remove "long form" of class definition.
draft-davies-idntables-09 Grammatical updates, clarity improvements.
Altered some DNS-specific terminology.
draft-davies-idntables-10 Added convention for out-of-repertoire
variants, additional examples of when rules in the context of
symmetry, isolated minor copy editing. Use a URN as the XML
namespace (provisional). Specify a media type for the file.
draft-ietf-lager-specification-00 Update to reflect adoption as a
work item by the IETF LAGER working group.
draft-ietf-lager-specification-01 Update to reflect decisions in
first interim meeting of IETF LAGER working group. Correcting a
number of typos, added section on contextual conditions,
clarified language on how actions are triggered, and changed
"block", "allocate" and "activate" to "blocked", "allocatable".
and "activated". Other minor changes.
draft-ietf-lager-specification-02 Minor changes.
draft-ietf-lager-specification-03 Update to fix a typo in the
schema, and clarify the use of reflexive variants in checking
label validity. Added security consideration for naive
implementations of permuted labels and WLE rules. Added
discussion of error conditions under which duplicate variant
labels might be created. Other minor changes.
draft-ietf-lager-specification-04 Updated XML namespace in the
RelaxNG schema.
draft-ietf-lager-specification-05 Add IANA Considerations for media
type registration, URN registration and instantiating a
dispositions registry. Split references into normative and
informative. Describe a tighter restriction on permissible
values for "ref" attributes. Clarify when "count" attributes
are permitted. Typos fixed. Checked the schema against the
specification and made corrections as well as replaced the
"text" datatype with less permissive types for most attributes.
Davies & Freytag Expires June 11, 2016 [Page 63]
Internet-Draft Label Generation Rulesets in XML December 2015
Authors' Addresses
Kim Davies
Internet Corporation for Assigned Names and Numbers
12025 Waterfront Drive
Los Angeles, CA 90094
US
Phone: +1 310 301 5800
Email: kim.davies@icann.org
URI: http://www.icann.org/
Asmus Freytag
ASMUS Inc.
Email: asmus@unicode.org
Davies & Freytag Expires June 11, 2016 [Page 64]