CBOR C. Bormann
Internet-Draft Universität Bremen TZI
Intended status: Experimental 17 August 2023
Expires: 18 February 2024
Common CBOR Deterministic Encoding and Application Profiles
draft-bormann-cbor-dcbor-02
Abstract
CBOR (STD 94, RFC 8949) defines "Deterministically Encoded CBOR" in
its Section 4.2, providing some flexibility for application specific
decisions. To facilitate Deterministic Encoding to be offered as a
selectable feature of generic encoders, the present document
discusses a Common CBOR Deterministic Encoding Profile that can be
shared by a large set of applications with potentially diverging
detailed requirements. The concept of Application Profiles is
layered on top of the Common CBOR Deterministic Encoding Profile and
can address those more application specific requirements. The
document defines the application profile "Gordian dCBOR" as an
example of an application profile built on the Common CBOR
Deterministic Encoding Profile.
About This Document
This note is to be removed before publishing as an RFC.
Status information for this document may be found at
https://datatracker.ietf.org/doc/draft-bormann-cbor-dcbor/.
Discussion of this document takes place on the Concise Binary Object
Representation Maintenance and Extensions (CBOR) Working Group
mailing list (mailto:cbor@ietf.org), which is archived at
https://mailarchive.ietf.org/arch/browse/cbor/. Subscribe at
https://www.ietf.org/mailman/listinfo/cbor/.
Source for this draft and an issue tracker can be found at
https://github.com/cabo/det.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Bormann Expires 18 February 2024 [Page 1]
Internet-Draft CBOR Profiles August 2023
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 18 February 2024.
Copyright Notice
Copyright (c) 2023 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Conventions and Definitions . . . . . . . . . . . . . . . 3
2. Common CBOR Deterministic Encoding Profile . . . . . . . . . 3
3. Application Profiles . . . . . . . . . . . . . . . . . . . . 5
3.1. Gordian dCBOR . . . . . . . . . . . . . . . . . . . . . . 6
3.1.1. Removing Simple Values . . . . . . . . . . . . . . . 6
3.1.2. Removing Integer Values . . . . . . . . . . . . . . . 6
3.1.3. Numeric Reduction of Floating-Point Values . . . . . 7
3.2. Extensibility . . . . . . . . . . . . . . . . . . . . . . 8
4. CDDL support . . . . . . . . . . . . . . . . . . . . . . . . 8
5. Implementation Status . . . . . . . . . . . . . . . . . . . . 9
5.1. Gordian dCBOR Application Profile . . . . . . . . . . . . 9
5.1.1. TypeScript . . . . . . . . . . . . . . . . . . . . . 9
5.1.2. Swift . . . . . . . . . . . . . . . . . . . . . . . . 10
5.1.3. Rust . . . . . . . . . . . . . . . . . . . . . . . . 10
5.1.4. Ruby . . . . . . . . . . . . . . . . . . . . . . . . 10
6. Security Considerations . . . . . . . . . . . . . . . . . . . 11
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 11
8. References . . . . . . . . . . . . . . . . . . . . . . . . . 11
8.1. Normative References . . . . . . . . . . . . . . . . . . 11
Bormann Expires 18 February 2024 [Page 2]
Internet-Draft CBOR Profiles August 2023
8.2. Informative References . . . . . . . . . . . . . . . . . 12
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 13
Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 14
1. Introduction
CBOR (STD 94, RFC 8949) defines "Deterministically Encoded CBOR" in
its Section 4.2, providing some flexibility for application specific
decisions. To facilitate Deterministic Encoding to be offered as a
selectable feature of generic encoders, the present document
discusses a Common CBOR Deterministic Encoding Profile that can be
shared by a large set of applications with potentially diverging
detailed requirements. The concept of Application Profiles is
layered on top of the Common CBOR Deterministic Encoding Profile and
can address those more application specific requirements. The
document defines the application profile "Gordian dCBOR" as an
example of an application profile built on the Common CBOR
Deterministic Encoding Profile.
1.1. Conventions and Definitions
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
2. Common CBOR Deterministic Encoding Profile
Section 4.2.1 of [STD94] defines _Core Deterministic Encoding
Requirements_ for CBOR. It mandates to keep with what are only
recommendations for Preferred Encoding for regular CBOR encoders. It
adds mandates for definite-length encoding and for a map ordering
based on lexicographic ordering of the (deterministically) encoded
map keys.
Note that this specific set of requirements is elective — in
principle, other variants of deterministic encoding can be defined
(and have been, now being phased out slowly, as detailed in
Section 4.2.3 of [STD94]), or (as many applications of CBOR do today)
deterministic encoding is not used at all.
The core requirements are designed, however, to provide well-
understood and easy to implement rules while maximizing coverage,
i.e., the subset of CBOR data items that are fully specified by these
rules, and also placing minimal burden on implementations.
Bormann Expires 18 February 2024 [Page 3]
Internet-Draft CBOR Profiles August 2023
Section 4.2.2 of [STD94] picks up on the interaction of extensibility
(CBOR tags) and deterministic encoding. CBOR itself uses some tags
to increase the range of its basic generic data types, e.g., tag 2/3
extend the range of basic major types 0/1 in a seamless way.
Section 4.2.2 of [STD94] recommends handling this transition the same
way as with the transition between different integer representation
lengths in the basic generic data model, i.e., by mandating the
Preferred Encoding (Section 3.4.3 of [STD94]).
1. The Common CBOR Deterministic Encoding Profile turns this
recommendation into a mandate: Integers that can be represented
by basic major type 0 and 1 are encoded using the deterministic
encoding defined for them, and integers outside this range are
encoded using the preferred serialization (Section 3.4.3 of
[STD94]) of tag 2 and 3 (i.e., no leading zeros).
Most tags capture more specific application semantics and therefore
may be harder to define a deterministic encoding for. While the
deterministic encoding of their tag internals is often covered by the
_Core Deterministic Encoding Requirements_, the mapping of diverging
platform application data types on the tag contents may be hard to do
in a deterministic way; see Section 3.2 of [I-D.bormann-cbor-det] for
more explanation as well as examples. As the Common CBOR
Deterministic Encoding Profile would continually need to address
additional issues raised by the registration of new tags, this
specification RECOMMENDS that new tag registrations address
deterministic encoding in the context of this Profile.
A particularly difficult field to obtain deterministic encoding for
is floating point numbers, partially because they themselves are
often obtained from processes that are not entirely deterministic
between platforms. See Section 3.2.2 of [I-D.bormann-cbor-det] for
more details. Section 4.2.2 of [STD94] presents a number of choices,
which need to be made to obtain a Common CBOR Deterministic Encoding
Profile. Specifically (in the order of the bullet list at the end of
Section 4.2.2 of [STD94]):
2. Besides the mandated use of preferred encoding, there is no
further specific action for the two different zero values, e.g.,
an encoder that is asked by an application to represent a
negative floating point zero will generate 0xf98000.
3. There is no attempt to mix integers and floating point numbers,
i.e., all floating point values are encoded as the preferred
floating-point representation that accurately represents the
value, independent of whether the floating point value is,
mathematically, an integral value (choice 2 of the second
bullet).
Bormann Expires 18 February 2024 [Page 4]
Internet-Draft CBOR Profiles August 2023
4. There is no special handling of NaN values, except that the
preferred encoding rules also apply to NaNs with payloads, using
the canonical encoding of NaNs as defined in [IEEE754].
Typically, most applications that employ NaNs in their storage
and communication interfaces will only use the NaN with payload
0, which encodes as 0xf97e00.
5. There is no special handling of subnormal values.
6. The Common CBOR Deterministic Encoding Profile does not presume
equivalence of floating point values with other representation
(e.g., tag 4/5) with basic floating point values.
The main intent here is to preserve the basic generic data model, so
application profiles can make their own decisions. For an example of
that, see Section 3.1.3.
While [I-D.rundgren-deterministic-cbor] is a relatively terse
document that is not always easy to interpret, to this author its
intent appears to be aligned with that of the Common CBOR
Deterministic Encoding Profile defined here.
3. Application Profiles
The dCBOR Application Profile specifies the use of Deterministic
Encoding as defined in Section 4.2 of [STD94] (see also
[I-D.bormann-cbor-det] for more information) together with some
application-level rules. As an example, the rules for Gordian dCBOR
[I-D.mcnally-deterministic-cbor] are specified in this section.
The application-level rules specified by an Application Profile are
based on the same Common CBOR Deterministic Encoding Profile; they do
not "fork" CBOR.
An Application Profile implementation produces well-formed,
deterministically encoded CBOR according to [STD94], and existing
generic CBOR decoders will therefore be able to decode it, including
those that check for Deterministic Encoding. Similarly, generic CBOR
encoders will be able to produce valid CBOR that can be processed by
Application Profile implementations, if handed Application Profile
conforming data model level information from an application.
Please note that the separation between standard CBOR processing and
the processing required by the Application Profile is a conceptual
one: Both Application Profile processing and standard CBOR processing
can be combined into a special dCBOR/CBOR encoder/decoder.
Bormann Expires 18 February 2024 [Page 5]
Internet-Draft CBOR Profiles August 2023
An Application Profile is intended to be used in conjunction with an
application, which typically will use a subset of the CBOR generic
data model, which in turn influences which subset of the application
profile is used. As a result, an Application Profile places no
direct requirement on what subset of CBOR is implemented. For
instance, while the dCBOR application profile defines rules for the
processing of floating point values, there is no requirement that
dCBOR implementations support floating point numbers (or any other
kind of number, such as arbitrary precision integers or 64-bit
negative integers) when they are used with applications that do not
use them.
3.1. Gordian dCBOR
Gordian dCBOR [I-D.mcnally-deterministic-cbor] provides an
application profile that requires encoders to produce valid CBOR in
deterministic encoding as defined in the Common CBOR Deterministic
Encoding Profile. Gordian dCBOR also requires dCBOR decoders to
reject CBOR data items that were not deterministically encoded.
Beyond the Common CBOR Deterministic Encoding Profile, dCBOR imposes
certain limitations on the CBOR basic generic data model. Some items
that can be represented in the CBOR basic generic data model are
entirely outlawed by this application profile. Other items are
represented by what are considered equivalent data items by the dCBOR
equivalence model, so a recipient application might receive data that
may not be the same data in the CBOR equivalence model as the ones
the generating application produced.
These restrictions mainly are about numeric values, which are
therefore the subject of the main subsection of this section.
3.1.1. Removing Simple Values
Only the three simple values false (0xf4), true (0xf5), and null
(0xf6) are allowed at the application level; the remaining 253 values
must be rejected.
3.1.2. Removing Integer Values
Only the integer values in range [-2^63, 2^64-1] can be expressed in
dCBOR ("basic dCBOR integers"). Note that the range is asymmetric,
with only 2^63 negative values, but 2^64 unsigned (non-negative)
values, creating an (approximately) 64.6 bit integer.
Bormann Expires 18 February 2024 [Page 6]
Internet-Draft CBOR Profiles August 2023
This maps to a choice between a platform 64-bit two's complement
signed integer (often called int64) and a 64-bit unsigned integer
(uint64). (Specific applications will, of course, further restrict
valid ranges of integers, based on their position and semantics in
the CBOR data item.)
3.1.3. Numeric Reduction of Floating-Point Values
dCBOR implementations that do support floating point numbers MUST
perform the following two reductions of numeric values when
constructing CBOR data items:
1. When representing integral floating point values (floating point
values with a zero fractional part), check whether the
mathematically identical value can be represented as a dCBOR
integer value, i.e., is in the range [-2^63, 2^64-1] given above.
If that is the case, convert the integral floating point to that
mathematically identical integer value before encoding it.
(Deterministic Encoding will then ensure the shortest length
encoding is used.) This means that if a floating point value has
a non-zero fractional part, or an exponent that takes it out of
the given range of basic dCBOR integers, the original floating
point value is used for encoding. (Specifically, conversion to a
bignum is never considered.)
This also means that the three representations of a zero number
in CBOR (0, 0.0, -0.0 in diagnostic notation) are all reduced to
the basic integer 0 (with preferred encoding 0x00).
Note that this reduction can turn valid maps into invalid ones,
as it can create duplicate keys, e.g., for:
{
10: "integer ten",
10.0: "floating ten"
}
This means that, at the application level, the application MUST
prevent the creation of maps that would turn invalid in dCBOR
processing.
2. In addition, before encoding, represent all NaN values by using
the quiet NaN value having the half-width CBOR representation
0xf97e00.
Bormann Expires 18 February 2024 [Page 7]
Internet-Draft CBOR Profiles August 2023
dCBOR-based applications MUST accept these "reduced" numbers in place
of the original value, e.g., a dCBOR-based application that expects a
floating point value needs to accept a basic dCBOR integer in its
place (and, if needed, convert it to a floating point value for its
own further processing).
dCBOR-based applications MUST NOT accept numbers that have not been
reduced as specified in this section, except maybe by making the
unreduced numbers available for their diagnostic value when there has
been an explicit request to do so. This is similar to a checking
flag mentioned in Section 5.1 (API Considerations) of
[I-D.bormann-cbor-det] being set by default.
3.2. Extensibility
[I-D.mcnally-deterministic-cbor] does not discuss extensibility. A
meaningful way to handle extensibility in this application profile
would be to lift value range restrictions, keeping the profile-
specific equivalence rules show here intact and possibly adding
equivalences as needed for newly allowed values. This requires
further discussion.
4. CDDL support
[RFC8610] defines control operators to indicate that the contents of
a byte string carries a CBOR-encoded data item (.cbor) or a sequence
of CBOR-encoded data items (.cborseq).
CDDL specifications may want to specify that the data items should be
encoded in Common CBOR Deterministic Encoding, or with the dCBOR
application profile applied as well. This specification adds four
CDDL control operators that can be used for this.
The control operators .cde and .cdeseq are exactly like .cbor and
.cborseq except that they also require the encoded data item(s) to be
in Common CBOR Deterministic Encoding.
The control operators .dcbor and .dcborseq are exactly like .cde and
.cdeseq except that they also require the encoded data item(s) to
conform to the dCBOR application profile.
For example, the normative comment in Section 3 of
[I-D.draft-mcnally-envelope-03]:
leaf = #6.24(bytes) ; MUST be dCBOR
...can now be formalized as:
Bormann Expires 18 February 2024 [Page 8]
Internet-Draft CBOR Profiles August 2023
leaf = #6.24(bytes .dcbor any)
More importantly, if the encoded data item also needs to have a
specific structure, this can be expressed by the right hand side
(instead of using the most general CDDL type any here).
(Note that the ...seq control operators do not enable specifying
different deterministic encoding requirements for the elements of the
sequence. If a use case for such a feature becomes known, it could
be added.)
5. Implementation Status
This section is to be removed before publishing as an RFC.
(Boilerplate as per Section 2.1 of [RFC7942]:)
This section records the status of known implementations of the
protocol defined by this specification at the time of posting of this
Internet-Draft, and is based on a proposal described in [RFC7942].
The description of implementations in this section is intended to
assist the IETF in its decision processes in progressing drafts to
RFCs. Please note that the listing of any individual implementation
here does not imply endorsement by the IETF. Furthermore, no effort
has been spent to verify the information presented here that was
supplied by IETF contributors. This is not intended as, and must not
be construed to be, a catalog of available implementations or their
features. Readers are advised to note that other implementations may
exist.
According to [RFC7942], "this will allow reviewers and working groups
to assign due consideration to documents that have the benefit of
running code, which may serve as evidence of valuable experimentation
and feedback that have made the implemented protocols more mature.
It is up to the individual working groups to use this information as
they see fit".
5.1. Gordian dCBOR Application Profile
5.1.1. TypeScript
* Implementation Location: [bc-dcbor-ts]
* Primary Maintainer:
* Languages: TypeScript (transpiles to JavaScript)
* Coverage:
Bormann Expires 18 February 2024 [Page 9]
Internet-Draft CBOR Profiles August 2023
* Testing:
* Licensing:
5.1.2. Swift
* Implementation Location: [BCSwiftDCBOR]
* Primary Maintainer:
* Languages: Swift
* Coverage:
* Testing:
* Licensing: BSD-2-Clause-Patent
5.1.3. Rust
* Implementation Location: [bc-dcbor-rust]
* Primary Maintainer:
* Languages: Rust
* Coverage:
* Testing:
* Licensing: Custom
5.1.4. Ruby
* Implementation Location: [cbor-dcbor]
* Primary Maintainer: Carsten Bormann
* Languages: Ruby
* Coverage: Complete specification; complemented by CBOR encoder/
decoder and command line interface from [cbor-diag] and
deterministic encoding from [cbor-deterministic]. Checking of
dCBOR exclusions not yet implemented.
* Testing: Also available at https://cbor.me (https://cbor.me)
* Licensing: Apache-2.0
Bormann Expires 18 February 2024 [Page 10]
Internet-Draft CBOR Profiles August 2023
6. Security Considerations
TODO Security
7. IANA Considerations
// RFC Editor: please replace RFCXXXX with the RFC number of this RFC
// and remove this note.
This document requests IANA to register the contents of Table 1 into
the registry "CDDL Control Operators" of [IANA.cddl]:
+===========+===========+
| Name | Reference |
+===========+===========+
| .cde | [RFCXXXX] |
+-----------+-----------+
| .cdeseq | [RFCXXXX] |
+-----------+-----------+
| .dcbor | [RFCXXXX] |
+-----------+-----------+
| .dcborseq | [RFCXXXX] |
+-----------+-----------+
Table 1: New control
operators to be
registered
8. References
8.1. Normative References
[IANA.cddl]
IANA, "Concise Data Definition Language (CDDL)",
<https://www.iana.org/assignments/cddl>.
[IEEE754] IEEE, "IEEE Standard for Floating-Point Arithmetic", IEEE
Std 754-2019, DOI 10.1109/IEEESTD.2019.8766229,
<https://ieeexplore.ieee.org/document/8766229>.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/rfc/rfc2119>.
Bormann Expires 18 February 2024 [Page 11]
Internet-Draft CBOR Profiles August 2023
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.
[RFC8610] Birkholz, H., Vigano, C., and C. Bormann, "Concise Data
Definition Language (CDDL): A Notational Convention to
Express Concise Binary Object Representation (CBOR) and
JSON Data Structures", RFC 8610, DOI 10.17487/RFC8610,
June 2019, <https://www.rfc-editor.org/rfc/rfc8610>.
[STD94] Bormann, C. and P. Hoffman, "Concise Binary Object
Representation (CBOR)", STD 94, RFC 8949,
DOI 10.17487/RFC8949, December 2020,
<https://www.rfc-editor.org/rfc/rfc8949>.
8.2. Informative References
[bc-dcbor-rust]
"Blockchain Commons Deterministic CBOR ("dCBOR") for
Rust", n.d.,
<https://github.com/BlockchainCommons/bc-dcbor-rust>.
[bc-dcbor-ts]
"Blockchain Commons Deterministic CBOR ("dCBOR") for
TypeScript", n.d.,
<https://github.com/BlockchainCommons/bc-dcbor-ts>.
[BCSwiftDCBOR]
"Blockchain Commons Deterministic CBOR ("dCBOR") for
Swift", n.d.,
<https://github.com/BlockchainCommons/BCSwiftDCBOR>.
[cbor-dcbor]
Bormann, C., "PoC of the McNally/Allen "dCBOR"
application-level CBOR representation rules", n.d.,
<https://github.com/cabo/cbor-dcbor>.
[cbor-deterministic]
Bormann, C., "cbor-deterministic gem", n.d.,
<https://github.com/cabo/cbor-deterministic>.
[cbor-diag]
Bormann, C., "CBOR diagnostic utilities", n.d.,
<https://github.com/cabo/cbor-diag>.
Bormann Expires 18 February 2024 [Page 12]
Internet-Draft CBOR Profiles August 2023
[I-D.bormann-cbor-det]
Bormann, C., "CBOR: On Deterministic Encoding", Work in
Progress, Internet-Draft, draft-bormann-cbor-det-01, 9
August 2023, <https://datatracker.ietf.org/doc/html/draft-
bormann-cbor-det-01>.
[I-D.draft-mcnally-envelope-03]
McNally, W. and C. Allen, "The Gordian Envelope Structured
Data Format", Work in Progress, Internet-Draft, draft-
mcnally-envelope-03, 17 August 2023,
<https://datatracker.ietf.org/doc/html/draft-mcnally-
envelope-03>.
[I-D.mcnally-deterministic-cbor]
McNally, W. and C. Allen, "Gordian dCBOR: A Deterministic
CBOR Application Profile", Work in Progress, Internet-
Draft, draft-mcnally-deterministic-cbor-05, 8 August 2023,
<https://datatracker.ietf.org/doc/html/draft-mcnally-
deterministic-cbor-05>.
[I-D.rundgren-deterministic-cbor]
Rundgren, A., "Deterministically Encoded CBOR (D-CBOR)",
Work in Progress, Internet-Draft, draft-rundgren-
deterministic-cbor-17, 16 August 2023,
<https://datatracker.ietf.org/doc/html/draft-rundgren-
deterministic-cbor-17>.
[RFC7942] Sheffer, Y. and A. Farrel, "Improving Awareness of Running
Code: The Implementation Status Section", BCP 205,
RFC 7942, DOI 10.17487/RFC7942, July 2016,
<https://www.rfc-editor.org/rfc/rfc7942>.
Acknowledgments
Section 3.1 of this document is based on the work of Wolf McNally and
Christopher Allen as documented in [I-D.mcnally-deterministic-cbor]
and discussed in 2023 in the CBOR working group.
Contributors
Wolf McNally
Blockchain Commons
Email: wolf@wolfmcnally.com
Christopher Allen
Blockchain Commons
Email: christophera@lifewithalacrity.com
Bormann Expires 18 February 2024 [Page 13]
Internet-Draft CBOR Profiles August 2023
Anders Rundgren
Independent
Montpellier
France
Email: anders.rundgren.net@gmail.com
URI: https://www.linkedin.com/in/andersrundgren/
Author's Address
Carsten Bormann
Universität Bremen TZI
Postfach 330440
D-28359 Bremen
Germany
Phone: +49-421-218-63921
Email: cabo@tzi.org
Bormann Expires 18 February 2024 [Page 14]