DataCanon Core

Version 1 Draft

Introduction

DataCanon (DC) is a data model for structured scientific data with a canonical identity encoding.

DataCanon defines a set of data types and their equality relations, together with a unique canonical byte encoding for every instance.

The type and encoding definitions provide a common, representation-independent basis for communicating, identifying, comparing, and preserving structured scientific data across software implementations and persistence mechanisms.

The identity of a DataCanon Instance is determined solely by its type and its constituent information. It does not depend on any particular programming language, textual notation, parser, renderer, or persistence format.

Textual syntaxes and persistence formats may represent DataCanon Instances, but those representations are outside the core DataCanon specification.

DataCanon is intended for scientific software whose data must remain stable, portable, reproducible, and meaningful over years or decades of development.

Goals

DataCanon is designed to provide

• rigorously defined data types,
• deterministic data identity,
• implementation-independent comparison,
• reliable interchange between software systems,
• deterministic fingerprinting,
• long-term persistence.

The goals support scientific software whose data must remain robust over the lifetime of a project rather than the lifetime of a particular programming language, textual syntax, or persistence format.

Non-Goals

DataCanon intentionally does not provide

• a general-purpose programming language,
• a required textual syntax,
• symbolic algebra,
• general computation,
• variables,
• user-defined functions,
• expression evaluation,
• units,
• dimensional analysis.

DataCanon represents structured data.

It does not manipulate that data or define mathematical operations between instances except where such structure is explicitly part of a DataCanon Type.

DataCanon canonicalizes identity, not mathematics. Mathematical relationships between distinct DataCanon Instances are outside the DataCanon specification.

The non-goals apply to the DataCanon data model and canonical identity. Optional textual representations and preprocessing facilities may provide conveniences such as substitutions and expression evaluation without extending the DataCanon Type system or altering DCI identity.

Terminology

The following terms are used throughout this specification.

DataCanon Type (DCT)

A DataCanon Type (DCT) is a data type defined by the DataCanon specification.

Each DCT defines

• the information constituting its instances,
• the invariants those instances MUST satisfy, and
• equality between its instances.

A DCT may specify information directly and may contain constituent instances of other DCTs. Direct information has a domain prescribed by the containing DCT; a constituent DCI retains its own DataCanon type and identity.

DataCanon Instance (DCI)

A DataCanon Instance (DCI) is an instance of exactly one DCT.

A DCI is a DataCanon value. This specification makes no separate distinction between a DataCanon instance and a DataCanon value.

DCI equality is defined by the corresponding DCT.

DCI identity is independent of any implementation-language object, textual representation, persistence format, or other realization of the DCI.

Canonical Encoding

The canonical encoding of a DCI is the unique byte sequence assigned to that DCI by this specification.

Let E(x) denote the canonical encoding of DCI x.

For any DCIs x and y,

x = y  if and only if  E(x) = E(y)

where equality on the left is DCI equality as defined by the corresponding DCTs and equality on the right is byte-for-byte equality.

The canonical encoding does not define DCI equality. Equality is defined by the DCTs themselves. The canonical encoding provides a unique, portable encoding of that identity.

The canonical encoding is fully decodable and may be stored or transmitted directly. No additional representation is required to reconstruct the encoded DCI. For DCIs referencing external resources, canonical decoding reconstructs the DCI without requiring access to those resources. Retrieval and verification of external contents are separate operations.

Representation

A representation of a DCI is any form from which that DCI can be recovered without loss of DataCanon identity.

Representations may include textual syntaxes, implementation-language objects, or persistence formats.

No such representation is privileged by the core DataCanon specification.

Conventional Types

A conventional type is a type used directly for data and computation in a programming language. Conventional types may be built into the language, provided by libraries, or defined by users. An implementation may supply additional conventional types where suitable ones are missing.

An implementation may define conversions between DCIs and instances of conventional types. Each conventional type retains its own construction, equality, and operation rules, which need not coincide with the rules of an associated DCT. A conversion relationship does not establish type equivalence, a one-to-one correspondence, or a lossless round trip.

The term describes the practical types used through an implementation’s conversion interface; it does not introduce another DCT or a closed type system. A programming-language object implementing a DCI and an instance of a conventional type used in conversion can be distinct objects of different types within the same language.

Conventional-type operations do not change DCI equality or canonical encoding. Conversion to a conventional object may discard distinctions needed to recover the original DCI. Such an object is not necessarily a representation of that original DCI under the definition above.

The optional DataCanon Conventional Types Version 1 companion defines construction, arithmetic, comparison, collection behavior, and conversion requirements for implementations claiming support for those conventional types. Core conformance does not require support for that companion.

Design Principles

Representation Independence

DataCanon identity is independent of the representation used to express, communicate, or persist a DCI.

Different representations of the same DCI may differ arbitrarily in syntax, layout, storage organization, or implementation details.

Different representations recover the same DataCanon data when they recover equal DCIs.

Implementation Independence

The DataCanon specification does not prescribe how a software implementation represents DCIs internally.

Different implementations may use different classes, structures, memory layouts, or other implementation-specific mechanisms.

Implementations representing equal DCIs MUST nevertheless agree on DCI equality and canonical encoding.

Deterministic Identity

Every valid DCI has exactly one canonical encoding.

The canonical encoding depends only on the DCI and its DCT.

It does not depend on the implementation that produced it or on any other representation of the DCI.

Structural Identity

Unless explicitly stated otherwise by a DCT, DCI identity is determined by the structure and constituents defined by that DCT rather than by mathematical, application-specific, or implementation-specific equivalence.

Constituent Structure

A DCT specifies fixed-domain information directly. A field is a constituent DCI when its DataCanon type or recursive DataCanon structure participates in the containing value’s identity. For example, Fraction uses direct mathematical integer fields, while Complex uses Numerical DCIs whose individual DCTs matter.

Encoding rules reuse shared payload definitions for direct information and complete canonical encodings for constituent DCIs.

Minimality

DataCanon defines only what is required to establish portable structured data and its identity.

Facilities for textual input, textual rendering, preprocessing, application semantics, computation, and specialized persistence are defined separately when needed.

DataCanon Types

DataCanon defines the types described in this section.

Each DataCanon Type (DCT) defines its instances, the invariants those instances must satisfy, and equality between its instances.

Instances of different DCTs are never equal.

DataCanon Types are classified according to the organization of their information and constituent DCIs. Classification does not alter identity or encoding rules.

Primitive Types

A primitive DCT specifies a basic value directly, without constituent DCIs. Its information may have multiple prescribed fields.

Version 1 defines the following primitive DCTs:

• Null
• Boolean
• Integer
• Float
• String
• Bytes
• Fraction
• Constant

Null

The Null DCT contains exactly one DCI.

All Null DCIs are therefore equal.

Boolean

The Boolean DCT contains exactly two DCIs, representing the Boolean values false and true.

Two Boolean DCIs are equal if and only if they represent the same Boolean value.

Integer

The Integer DCT represents the mathematical integers (\mathbb{Z}).

An Integer DCI consists of exactly one mathematical integer. Integer has unlimited precision.

Two Integer DCIs are equal if and only if they represent the same mathematical integer.

Float

The Float DCT represents finite IEEE-754 binary64 values.

A Float DCI consists of exactly one finite IEEE-754 binary64 value, including its sign.

Two Float DCIs are equal if and only if their IEEE-754 binary64 bit representations are identical.

Positive and negative zero are therefore distinct Float DCIs.

NaN and positive and negative infinity are not Float DCIs in Version 1.

Floating-point values are inherently approximate. Applications requiring an exact rational representation should use a suitable exact DCT.

String

The String DCT represents finite sequences of Unicode scalar values.

A String DCI consists of exactly one such sequence.

Two String DCIs are equal if and only if they contain identical sequences of Unicode scalar values.

DataCanon performs no Unicode normalization. Strings that are visually identical but consist of different sequences of Unicode scalar values are therefore distinct String DCIs.

Bytes

The Bytes DCT represents finite sequences of bytes. A Bytes DCI consists of exactly one such sequence, which MAY be empty.

Two Bytes DCIs are equal if and only if their complete byte sequences are identical, including length and byte order.

Bytes are opaque. DataCanon does not assign a character encoding, file format, filename, media type, or application-specific interpretation to their contents. Applications may store such metadata alongside a Bytes DCI, for example in a Record. That metadata participates in the containing DCI’s identity according to its DCT, but is not a constituent of the Bytes DCI.

Bytes is distinct from String, Array, and ExternalFile. Its byte sequence is contained directly in the DCI, rather than referenced by a path and digest.

Fraction

The Fraction DCT represents a structured ratio of two mathematical integers.

A Fraction DCI consists of

• a mathematical integer numerator, and
• a positive mathematical integer denominator.

Two Fraction DCIs are equal if and only if their corresponding constituents are equal.

Fraction identity is structural rather than mathematical. Distinct numerator and denominator pairs remain distinct even when they represent the same mathematical rational number.

DataCanon does not reduce, normalize, or otherwise mathematically simplify Fractions.

Constant

The Constant DCT contains exactly two DCIs in Version 1, named pi and e. The name identifies the exact mathematical constant: pi is the ratio of a circle’s circumference to its diameter in Euclidean geometry; e is Euler’s number, the base of the natural logarithm. No decimal approximation is stored.

Two Constant DCIs are equal if and only if their names agree. Constant is a primitive DCT: its prescribed name is direct information, not a String DCI. A Constant is never equal to a Float containing an approximation of its value. No other names, aliases, user-defined constants, or physical constants are included in Version 1.

Numerical Instances

A Numerical DCI is an Integer, Float, Fraction, Constant, Product, Power, Quotient, Array, Sum, or OrderedProduct DCI. Sum and OrderedProduct operands MUST themselves be Numerical DCIs. Boolean and Complex are not members of this category. A Boolean Array is still an Array DCI; category membership does not promise that a conventional arithmetic implementation supports it.

A Scalar Numerical DCI is one of:

• an Integer, Float, Fraction, or Constant DCI;
• a Product, Power, or Quotient DCI, whose invariants require scalar constituents;
• a Sum or OrderedProduct DCI whose every operand is a Scalar Numerical DCI.

An empty Sum or OrderedProduct satisfies the scalar classification. An Array, including a rank-zero Array, does not. Classification depends on constituent structure, without evaluating values or inferring ranks or shapes of results. For example, a Sum containing an Array through a nested OrderedProduct is not scalar, regardless of the Array’s shape or the factors’ mathematical values.

The categories specify permitted constituent DCIs; neither is an additional DCT. Scalar structural validity does not certify that an interpretation is mathematically defined or real. Types listed here may be defined later.

Atomic Types

An atomic DCT has a fixed set of named fields and at least one constituent DCI. Fields may contain directly specified information or constituent DCIs, as prescribed by the DCT.

Version 1 defines the following atomic DCTs:

• Power
• Quotient
• Complex

Power

The Power DCT represents a base with a structured positive rational exponent. A Power DCI consists of

• a base that is a Scalar Numerical DCI,
• a positive mathematical integer numerator, and
• a positive mathematical integer denominator.

The exponent may be interpreted as the numerator divided by the denominator. Both exponent constituents MUST be positive and participate independently in identity. They MUST NOT be reduced or otherwise normalized.

Two Power DCIs are equal if and only if all corresponding constituents are equal. Power is distinct from its base and from other numerical DCTs, including when its exponent is one. Nested Powers MUST NOT be collapsed.

The base MUST be a Scalar Numerical DCI. Power does not evaluate the base, select a mathematical branch, or certify that a real value exists. Its structural invariants impose no additional sign restriction on the base.

Quotient

The Quotient DCT represents a structured numerator and denominator, each a Scalar Numerical DCI. The constituents may belong to different numerical DCTs.

Two Quotient DCIs are equal if and only if corresponding numerators and corresponding denominators are equal. Constituent roles participate in identity. Quotient is distinct from Fraction, even when both constituents are Integers.

DataCanon MUST NOT evaluate, cancel, normalize, or rationalize Quotient constituents. This draft imposes numerical constituent-type restrictions but no nonzero-denominator invariant. Structural validity does not certify that the mathematical division is defined. Any stronger denominator restriction must be specified explicitly as a DCT invariant.

Complex

The Complex DCT represents a structured complex number consisting of a real component and an imaginary component.

A Complex DCI consists of

• a real component, and
• an imaginary component.

Each component MUST be a Scalar Numerical DCI.

Two Complex DCIs are equal if and only if their corresponding components are equal.

Because instances of different DCTs are never equal, the DCT of each component participates in Complex identity.

DataCanon does not convert Complex components between numerical DCTs or otherwise mathematically simplify them.

Composite Types

A composite DCT has variable-size constituent structure.

The organization and permitted contents of that structure are defined by the DCT.

Version 1 defines the following composite DCTs:

• List
• Tuple
• Set
• Multiset
• Sum
• Product
• OrderedProduct
• Mapping
• UnorderedMapping
• Record
• Array
• ExternalFile
• ExternalArray

List

The List DCT represents a finite ordered sequence of DCIs.

A List DCI consists of zero or more DCIs in a defined order.

The constituent DCIs may belong to any DCT and need not belong to the same DCT.

Two List DCIs are equal if and only if

• they contain the same number of items, and
• corresponding items are equal.

Thus, item order participates in List identity.

Lists may recursively contain DCIs of any DCT, including other composite DCTs.

Tuple

The Tuple DCT represents a finite ordered sequence of DCIs.

A Tuple DCI consists of zero or more DCIs in a defined order.

The constituent DCIs may belong to any DCT and need not belong to the same DCT.

Two Tuple DCIs are equal if and only if

• they contain the same number of items, and
• corresponding items are equal.

Thus, item order participates in Tuple identity.

List and Tuple are distinct DCTs. Their instances are therefore never equal, even when they contain equal constituent DCIs in the same order.

Tuples may recursively contain DCIs of any DCT, including other composite DCTs.

Set

The Set DCT represents a finite unordered collection of distinct DCIs.

A Set DCI consists of zero or more DCIs.

The constituent DCIs may belong to any DCT and need not belong to the same DCT.

No two members of a Set DCI may be equal.

Two Set DCIs are equal if and only if they contain the same members, independent of member order.

Thus, member order does not participate in Set identity.

Sets may recursively contain DCIs of any DCT, including other composite DCTs.

Multiset

The Multiset DCT represents a finite unordered collection of DCIs in which members may occur more than once.

A Multiset DCI consists of zero or more member occurrences. Each occurrence is a DCI of any DCT; members need not belong to the same DCT.

Two Multiset DCIs are equal if and only if, for every DCI, they contain the same number of occurrences equal to that DCI under DataCanon equality.

Thus, member order does not participate in Multiset identity, while multiplicity does. An empty Multiset contains no member occurrences.

Multiset and Set are distinct DCTs. Their instances are never equal, even when a Multiset contains no repeated members.

Multisets may recursively contain DCIs of any DCT, including other Multisets. Nesting participates in identity; nested Multisets are not flattened.

Sum and Product

Sum and Product are finite unordered collections of operand occurrences. Multiplicity and nesting participate in identity. Two instances of the same DCT are equal if and only if their operands match under DCI equality with the same multiplicities, independent of operand order.

Every Sum operand MUST be a Numerical DCI. Sum therefore accepts scalar expressions, Arrays, and OrderedProducts, including recursively nested Sums. Every Product operand MUST be a Scalar Numerical DCI. Product remains an unordered scalar structure and MUST NOT contain an Array through any path.

No operands are evaluated, flattened, distributed, deduplicated, or simplified. Array ranks and shape compatibility are not Sum invariants. Scalar terms may be stored alongside Arrays; Core assigns no implicit broadcasting or identity matrix convention to their mathematical interpretation.

OrderedProduct

The OrderedProduct DCT is a finite ordered sequence of zero or more Numerical DCIs. Scalars, Arrays, Sums, and nested OrderedProducts are permitted. Operands are complete DCIs, not native information fields.

Two OrderedProduct DCIs are equal if and only if they have the same number of operands and corresponding operands are equal. Order, multiplicity, nesting, and each operand’s DCT participate in identity. OrderedProduct is distinct from Product and List, including for empty or scalar-only instances.

Core MUST NOT multiply factors, commute scalars past Arrays, flatten nested products, remove identities, or check mathematical shape compatibility. No rank restriction is imposed on Array operands. A rank-three Array is valid here even when a conventional binding cannot evaluate its ordered product.

Mapping

The Mapping DCT represents a finite ordered collection of key-value entries.

A Mapping DCI consists of zero or more entries in a defined order.

Each entry consists of

• a key DCI, and
• a value DCI.

Keys and values may belong to any DCT.

No two entries in a Mapping DCI may have equal keys.

Two Mapping DCIs are equal if and only if

• they contain the same number of entries,
• corresponding keys are equal, and
• corresponding values are equal.

Thus, entry order participates in Mapping identity.

UnorderedMapping

The UnorderedMapping DCT represents a finite unordered collection of key-value entries.

An UnorderedMapping DCI consists of zero or more entries.

Each entry consists of

• a key DCI, and
• a value DCI.

Keys and values may belong to any DCT.

No two entries in an UnorderedMapping DCI may have equal keys.

Two UnorderedMapping DCIs are equal if and only if they contain the same key-value entries, independent of entry order.

Thus, entry order does not participate in UnorderedMapping identity.

Mapping and UnorderedMapping are distinct DCTs.

Record

The Record DCT represents a named structured value consisting of an ordered sequence of named fields.

A Record DCI consists of

• a name that is a finite sequence of Unicode scalar values, and
• zero or more fields in a defined order.

Each field consists of

• a field name that is a finite sequence of Unicode scalar values, and
• a value DCI.

Field values may belong to any DCT.

Field names within a Record MUST be unique.

Two Record DCIs are equal if and only if

• their Record names are equal,
• they contain the same number of fields,
• corresponding field names are equal, and
• corresponding field values are equal.

Thus, both the Record name and field order participate in Record identity.

Record is distinct from Mapping. A Record represents a named structure with a defined sequence of named fields, whereas a Mapping represents an ordered collection of associations between arbitrary DCI keys and values.

The Record name is semantic data. DataCanon does not assign application-specific meaning to that name and does not interpret it as the name of another DCT.

A Record name may correspond to a type, class, or other application-defined concept in an implementation. Such correspondences are defined by the implementation or application rather than by the Record DCT.

Array

The Array DCT represents dense, regular multidimensional machine data.

An Array DCI consists of

• an Array type,
• a finite multidimensional shape, and
• one element for every position defined by that shape.

The shape is a finite ordered sequence of nonnegative mathematical integers.

The number of entries in the shape is the rank of the Array, and the product of the entries is the total number of elements.

An empty shape denotes a rank-zero Array containing one element.

The Array type determines the domain and equality relation of its elements.

Two Array DCIs are equal if and only if

• they have the same Array type,
• they have the same shape, and
• corresponding elements are equal according to the equality rule of that Array type.

Thus, Array type and shape participate in Array identity independently of the values of the elements.

Arrays are distinct from Lists. A List recursively contains DCIs. The elements of an Array are instead values of its Array type and are not themselves DCIs.

DataCanon does not define a universal type system or coercion policy for Array construction. In particular, DataCanon does not prescribe

• which additional native Array types an implementation may support,
• how mixed input values are promoted or coerced, or
• how an implementation represents unsupported or implementation-specific Array types.

The Array construction choices are delegated to the implementation.

Applications requiring DataCanon-defined type identity for every individual element should use nested Lists rather than Arrays.

Portable Array Types

DataCanon Version 1 defines the following portable Array types:

• bool
• int8
• int16
• int32
• int64
• uint8
• uint16
• uint32
• uint64
• float16
• float32
• float64
• complex64
• complex128

The portable Array type names denote DataCanon Array types rather than types belonging to any particular implementation.

The bool Array type contains exactly two values, corresponding to false and true. Two bool elements are equal if and only if they represent the same Boolean value.

The signed and unsigned integer Array types represent mathematical integers within the ranges determined by their respective fixed widths. Two integer Array elements of the same Array type are equal if and only if they represent the same mathematical integer.

The floating-point Array types represent finite IEEE-754 binary16, binary32, and binary64 values, respectively.

Two floating-point Array elements of the same Array type are equal if and only if their IEEE-754 bit representations are identical.

Positive and negative zero are therefore distinct floating-point Array elements.

NaN and positive and negative infinity are not elements of the portable floating-point Array types in Version 1.

The complex64 Array type consists of ordered pairs of float32 elements representing the real and imaginary components.

The complex128 Array type consists of ordered pairs of float64 elements representing the real and imaginary components.

Two complex Array elements of the same Array type are equal if and only if their corresponding real and imaginary components are equal.

DataCanon Version 1 defines Array DCIs only for the portable Array types listed above. Implementations MAY support additional native Array types, but values of those types are not Version 1 Array DCIs unless their types and canonical encodings are defined by an applicable extension to this specification.

ExternalFile

The ExternalFile DCT represents an externally stored finite sequence of bytes whose relative path and content digest are part of its identity.

An ExternalFile DCI consists of

• a DataCanon relative path, and
• a SHA-256 content digest.

The path identifies an external resource within a namespace determined by the representation or application resolving the DCI.

The content digest is the SHA-256 digest of the exact byte sequence contained in the external file.

The path MUST be relative.

Two ExternalFile DCIs are equal if and only if

• their paths are equal, and
• their content digests are equal.

Thus, both the relative path and content digest participate in ExternalFile identity.

The contents of an ExternalFile are opaque to DataCanon. DataCanon assigns no meaning to the byte sequence and does not prescribe how an application interprets it.

The content digest binds the ExternalFile DCI to a particular byte sequence. A file found at the specified path whose SHA-256 digest differs from the content digest is not the external file identified by that DCI.

The existence or availability of the external file does not alter the identity of the ExternalFile DCI. A missing or inaccessible external file prevents the external content from being retrieved or verified but does not change the DCI.

ExternalArray

The ExternalArray DCT represents dense, regular multidimensional machine data stored externally using the DataCanon Array byte representation.

An ExternalArray DCI consists of

• a DataCanon relative path,
• a portable Array type,
• a finite multidimensional shape, and
• a SHA-256 content digest.

The path identifies an external resource within a namespace determined by the representation or application resolving the DCI.

The path MUST be relative.

The Array type and shape satisfy the same requirements as the corresponding constituents of an Array DCI.

The logical external element byte stream MUST be the DataCanon-defined byte representation of the Array elements for the specified Array type and shape.

The content digest is the SHA-256 digest of that logical byte stream, before storage compression. Persistence may store the stream directly or in a losslessly compressed representation. Compression does not add a DCI constituent; the complete relative path, including any suffix, still participates in identity.

Two ExternalArray DCIs are equal if and only if

• their paths are equal,
• their Array types are equal,
• their shapes are equal, and
• their content digests are equal.

Thus, relative path, Array type, shape, and content digest all participate in ExternalArray identity.

ExternalArray and Array are distinct DCTs. An ExternalArray DCI and an Array DCI are therefore never equal, even when the external byte sequence represents the same numerical data as the Array DCI.

The content digest binds the ExternalArray DCI to a particular logical element byte stream. After any persistence-defined decompression, a stream whose SHA-256 digest differs from the content digest does not satisfy that DCI.

The existence or availability of the external data does not alter the identity of the ExternalArray DCI. Missing or inaccessible external data prevents the Array contents from being retrieved or verified but does not change the DCI.

Relative Paths

External DCTs identify external resources using a DataCanon relative path.

A DataCanon relative path is a finite nonempty ordered sequence of path components.

Each path component is a nonempty finite sequence of Unicode scalar values.

The components "." and ".." are not permitted.

The characters / and \ are not permitted within a path component.

Two DataCanon relative paths are equal if and only if they contain the same number of components and corresponding components are equal according to String equality.

A DataCanon relative path is independent of host operating-system path syntax.

The namespace used to resolve a DataCanon relative path is not part of DCI identity. The mechanism by which a representation or implementation resolves a DataCanon relative path to an external resource is outside the core DataCanon specification.

Canonical Encoding

Every DataCanon Instance has exactly one canonical encoding as a finite sequence of bytes.

The canonical encoding provides a representation-independent encoding of DCI identity.

Let E(x) denote the canonical encoding of DCI x. For any DCIs x and y,

x = y  if and only if  E(x) = E(y)

where equality on the left is DCI equality and equality on the right is byte-for-byte equality.

The canonical encoding MUST be

• deterministic — a DCI always produces the same byte sequence;
• injective — unequal DCIs produce different byte sequences;
• implementation independent — the encoding does not depend on implementation-language representations;
• unambiguous — the encoded DCT and all information constituting the DCI can be recovered uniquely from the byte sequence;
• recursive — constituent DCIs are encoded according to the same canonical encoding rules;
• stable — the encoding of a Version 1 DCI does not change between conforming Version 1 implementations.

The canonical encoding is not intended as a human-readable representation, general-purpose interchange format, or optimized persistence format.

Its purpose is to provide a minimal, stable, and unique byte representation of DCI identity.

Version Context

The canonical encoding contains no version identifier or framing header. The applicable DataCanon specification version is determined by the context in which the encoding is used. Persistence and interchange formats MAY provide version information independently of the canonical encoding.

Future specifications MUST NOT silently reinterpret Version 1 identifiers or payload encodings when those bytes are interpreted as Version 1 canonical encodings.

Unsigned Integer Encoding

Canonical encoding uses a variable-length representation of nonnegative mathematical integers for lengths, counts, ranks, dimensions, and other nonnegative integer metadata explicitly designated by this specification.

A nonnegative integer is encoded using unsigned LEB128.

The binary representation of the integer is divided into groups of seven bits, beginning with the least-significant bits.

Each group is encoded as one byte. The low seven bits contain the group value. The high bit is the continuation bit.

The continuation bit is

• 1 when another group follows, and
• 0 for the final byte.

The encoding MUST use the minimum possible number of bytes. Redundant high-order zero groups are not permitted.

Thus, every nonnegative mathematical integer has exactly one canonical unsigned integer encoding.

Examples include

There is no upper bound on the represented integer.

Signed Integer Encoding

Mathematical integers are first mapped bijectively to nonnegative mathematical integers using zigzag encoding.

For mathematical integer (n), define

$$ Z(n) = \begin{cases} 2n, & n \ge 0,\ -2n - 1, & n < 0. \end{cases} $$

Thus,

The canonical signed integer encoding of (n) is the canonical unsigned integer encoding of (Z(n)).

This representation has no implementation-defined integer width and therefore supports the unlimited-precision Integer DCT.

Unicode String Information Encoding

Let S(t) encode a finite sequence of Unicode scalar values t as its UTF-8 byte count using U, followed by the exact UTF-8 bytes:

S(t) = U(number_of_UTF8_bytes) || UTF8(t)

This is the payload encoding used by String, without the String DCT identifier. It is also used for Record names, field names, and relative-path components. All UTF-8 validity rules apply; no Unicode normalization is performed.

Encoding Structure

The canonical encoding of every DCI consists of

1. a one-byte DCT identifier, followed by
2. the canonical payload defined for that DCT.

Thus,

E(x) = DCT(x) || payload(x)

where || denotes byte concatenation.

The DCT identifier determines how the following payload is interpreted.

Direct information is encoded using the payload rules prescribed by its containing DCT. No DCT identifier is included unless explicitly specified.

Every constituent that is itself a DCI is encoded using its complete canonical encoding, including its own DCT identifier.

Consequently, nested DCIs are self-describing with respect to their DCTs.

Lengths and counts required to delimit variable-size payloads are encoded using the canonical unsigned integer encoding defined above.

DCT Identifiers

DataCanon Version 1 assigns the following one-byte identifiers:

The DCT identifiers are part of the DataCanon Version 1 specification and MUST NOT be reassigned within Version 1.

Identifier 06 is reserved and MUST NOT be used for a Version 1 DCI. A canonical decoder MUST reject that identifier.

The numerical ordering of DCT identifiers has no semantic meaning.

Primitive Types

Null

The canonical encoding of a Null DCI consists solely of the Null DCT identifier.

00

Because Null has exactly one DCI, no payload is required.

Boolean

The canonical encoding of a Boolean DCI consists of

1. the Boolean DCT identifier, followed by
2. one byte identifying the Boolean value.

The Boolean payload is

00    false
01    true

Thus,

01 00    false
01 01    true

are the complete canonical encodings.

No other Boolean payload byte is valid.

Integer

The canonical encoding of an Integer DCI consists of

1. the Integer DCT identifier, followed by
2. the canonical signed integer encoding of its mathematical integer value.

Thus,

02 || I(n)

is the canonical encoding of Integer DCI (n), where I is the signed integer encoding defined above.

There is no implementation-defined integer width.

Float

The canonical encoding of a Float DCI consists of

1. the Float DCT identifier, followed by
2. the eight bytes of its IEEE-754 binary64 representation in big-endian byte order.

Thus every Float DCI occupies exactly nine bytes.

The payload preserves the exact IEEE-754 representation, including the sign of zero.

Because NaN and positive and negative infinity are not Float DCIs in Version 1, their IEEE-754 representations are not valid Float payloads.

String

The canonical encoding of a String DCI consists of

1. the String DCT identifier,
2. the number of bytes in the UTF-8 encoding of the String, encoded using the canonical unsigned integer encoding, and
3. the UTF-8 bytes of the String.

Thus,

04 || U(N) || UTF8

where N is the number of bytes in UTF8 and U is the canonical unsigned integer encoding.

Because a String DCI is a sequence of Unicode scalar values, its UTF-8 representation is uniquely determined.

DataCanon performs no Unicode normalization before encoding.

Bytes

The canonical encoding of a Bytes DCI consists of

1. the Bytes DCT identifier,
2. the number of bytes, encoded using the canonical unsigned integer encoding, and
3. the exact byte sequence.

16 || U(N) || BYTES

No text encoding or transformation is applied. Every byte value from 0 through 255 is permitted. An empty Bytes DCI is encoded as 16 00.

Constant

A Constant DCI is encoded as its DCT identifier 17 followed by exactly one constant identifier byte: 00 for pi, 01 for e. No length, String DCT identifier, or name text is included. All other constant identifier bytes are invalid and MUST be rejected. A missing identifier byte is invalid.

pi: 17 00
e:  17 01

Fraction

A Fraction DCI is encoded as

05 || I(numerator) || I(denominator)

I is the canonical signed integer encoding defined above. Both fields are mathematical integers; the denominator MUST be positive. Neither field has an Integer DCT identifier. No reduction is performed.

Atomic Types

Power

A Power DCI is encoded as

14 || E(base) || I(numerator) || I(denominator)

The base is a complete Scalar Numerical DCI. The positive mathematical integer exponent fields use the signed integer encoding without Integer DCT identifiers. Both are always encoded, including denominator one, and are not reduced or otherwise normalized.

Quotient

A Quotient is encoded as its identifier followed by the complete encodings of its Scalar Numerical numerator and denominator in that order:

15 || E(numerator) || E(denominator)

No constituent is omitted or mathematically transformed.

Complex

The canonical encoding of a Complex DCI consists of

1. the Complex DCT identifier,
2. the canonical encoding of its real component, and
3. the canonical encoding of its imaginary component.

Thus,

07 || E(real) || E(imaginary)

Because each component is encoded as a complete DCI, its DCT participates directly in the canonical encoding.

For example, a Complex whose real component is an Integer necessarily has a different canonical encoding from one whose otherwise numerically equal real component is a Float.

Composite Types

List

The canonical encoding of a List DCI consists of

1. the List DCT identifier,
2. the number of items encoded using the canonical unsigned integer encoding, and
3. the canonical encoding of each item in semantic order.

Thus, for a List containing (N) items,

08 || U(N) || E(item_0) || E(item_1) || ... || E(item_(N-1))

An empty List therefore consists of

08 00

Because each item is self-describing and the number of items is encoded explicitly, no additional separators or item lengths are required.

Tuple

The canonical encoding of a Tuple DCI consists of

1. the Tuple DCT identifier,
2. the number of items encoded using the canonical unsigned integer encoding, and
3. the canonical encoding of each item in semantic order.

Thus, for a Tuple containing (N) items,

09 || U(N) || E(item_0) || E(item_1) || ... || E(item_(N-1))

An empty Tuple therefore consists of

09 00

List and Tuple use the same payload structure but have different DCT identifiers and therefore cannot have identical canonical encodings.

Set

The canonical encoding of a Set DCI consists of

1. the Set DCT identifier,
2. the number of members encoded using the canonical unsigned integer encoding, and
3. the canonical encodings of its members in lexicographic byte order.

To encode a Set, the canonical encoding of each member is first computed independently.

The member encoding byte sequences are then sorted lexicographically, comparing each byte as an unsigned integer from 0 through 255.

For a Set containing (N) members whose sorted canonical encodings are (e_0, e_1, \ldots, e_{N-1}),

0a || U(N) || e_0 || e_1 || ... || e_(N-1)

Because Set members are distinct DCIs and canonical encoding is injective, no two member encodings may be identical.

An empty Set therefore consists of

0a 00

The byte ordering used during Set encoding has no semantic meaning and does not introduce order into the Set DCI.

Multiset

The canonical encoding of a Multiset DCI consists of

1. the Multiset DCT identifier,
2. the total number of member occurrences encoded using the canonical unsigned integer encoding, and
3. the complete canonical encoding of every member occurrence, sorted in nondecreasing lexicographic byte order.

Each byte is compared as an unsigned integer from 0 through 255. Duplicate member encodings MUST be retained, once for each occurrence. The count is the number of occurrences, not the number of distinct members.

For a Multiset containing (N) occurrences with sorted canonical encodings (e_0, e_1, \ldots, e_{N-1}),

11 || U(N) || e_0 || e_1 || ... || e_(N-1)

An empty Multiset consists of

11 00

The sorting rule provides a unique encoding without introducing semantic member order. Nested members use their complete canonical encodings and are not flattened.

OrderedProduct

The encoding is the identifier 18, a canonical unsigned operand count, and complete operand encodings in their stored order:

18 || U(N) || E(factor_0) || ... || E(factor_(N-1))

No sorting occurs. Every operand MUST be a Numerical DCI. Duplicate factors and nested OrderedProducts are preserved. The empty instance is 18 00.

Sum and Product

A Sum or Product is encoded as its DCT identifier, the total number of operand occurrences using canonical unsigned integer encoding, and every complete operand encoding in nondecreasing lexicographic byte order:

12 || U(N) || e_0 || e_1 || ... || e_(N-1)    Sum
13 || U(N) || e_0 || e_1 || ... || e_(N-1)    Product

Each byte is compared as an unsigned integer. Duplicate operand encodings MUST be retained. Sorting does not assign semantic order or flatten nested nodes.

Empty Sum and Product encodings are 12 00 and 13 00, respectively.

Mapping

The canonical encoding of a Mapping DCI consists of

1. the Mapping DCT identifier,
2. the number of entries encoded using the canonical unsigned integer encoding, and
3. the canonical encodings of each entry in semantic entry order.

Each entry is encoded as

E(key) || E(value)

Thus, for a Mapping containing (N) entries,

0b
|| U(N)
|| E(key_0) || E(value_0)
|| E(key_1) || E(value_1)
|| ...
|| E(key_(N-1)) || E(value_(N-1))

where the entries appear in the order belonging to the Mapping DCI.

An empty Mapping therefore consists of

0b 00

No separate entry delimiter or entry-length encoding is required because each key and value is itself an unambiguously decodable DCI.

UnorderedMapping

The canonical encoding of an UnorderedMapping DCI consists of

1. the UnorderedMapping DCT identifier,
2. the number of entries encoded using the canonical unsigned integer encoding, and
3. the canonical encodings of its entries in canonical key order.

To encode an UnorderedMapping, the canonical encoding of each key is first computed independently.

Entries are then sorted lexicographically by their canonical key encodings, comparing each byte as an unsigned integer from 0 through 255.

Each entry is encoded as

E(key) || E(value)

Thus, for an UnorderedMapping containing (N) entries whose canonically ordered keys are (k_0, k_1, \ldots, k_{N-1}),

0c
|| U(N)
|| E(k_0) || E(value_0)
|| E(k_1) || E(value_1)
|| ...
|| E(k_(N-1)) || E(value_(N-1))

Because keys within an UnorderedMapping are distinct DCIs and canonical encoding is injective, no two key encodings may be identical. The canonical key ordering therefore cannot contain ties.

An empty UnorderedMapping therefore consists of

0c 00

The byte ordering used during encoding has no semantic meaning and does not introduce entry order into the UnorderedMapping DCI.

Record

A Record DCI with N fields is encoded as

0d
|| S(record_name)
|| U(N)
|| S(field_name_0) || E(field_value_0)
|| S(field_name_1) || E(field_value_1)
|| ...
|| S(field_name_(N-1)) || E(field_value_(N-1))

S is the Unicode string information encoding. Record and field names have no String DCT identifiers. Field values are complete DCIs. Fields retain their semantic order; names MUST be unique according to Unicode scalar sequence equality. A Record with no fields is 0d || S(record_name) || 00.

Array

The canonical encoding of an Array DCI consists of

1. the Array DCT identifier,
2. a one-byte Array type identifier,
3. the Array rank encoded using the canonical unsigned integer encoding,
4. each dimension of the Array shape encoded using the canonical unsigned integer encoding, and
5. the canonical element byte stream.

Thus, for an Array of rank (R) with shape

$$ (d_0,d_1,\ldots,d_{R-1}), $$

the canonical encoding is

0e
|| ARRAY_TYPE
|| U(R)
|| U(d_0) || U(d_1) || ... || U(d_(R-1))
|| ELEMENT_BYTES

The number of Array elements is determined by the shape and is therefore not encoded separately.

For a rank-zero Array, the rank is zero and no dimensions follow. A rank-zero Array contains exactly one element.

Array Type Identifiers

DataCanon Version 1 assigns the following one-byte Array type identifiers:

The Array type identifiers are part of the DataCanon Version 1 specification and MUST NOT be reassigned within Version 1.

The numerical ordering of Array type identifiers has no semantic meaning.

Element Ordering

Array elements are encoded in C-contiguous logical order.

For an Array with shape

$$ (d_0,d_1,\ldots,d_{R-1}), $$

the rightmost index varies fastest.

The canonical element ordering depends only on the logical Array shape and element values. It does not depend on the memory layout, strides, contiguity, or storage representation used by an implementation.

Boolean Elements

Each bool element is encoded as exactly one byte:

00    false
01    true

No other byte value is valid.

Integer Elements

Fixed-width integer elements are encoded using exactly their specified number of bits.

Signed integer types use two’s-complement representation.

Unsigned integer types use ordinary unsigned binary representation.

Types wider than one byte are encoded in big-endian byte order.

Thus, each element occupies

Floating-Point Elements

Floating-point elements are encoded using their exact IEEE-754 representations.

float16, float32, and float64 use binary16, binary32, and binary64, respectively.

Types wider than one byte are encoded in big-endian byte order.

The exact bit representation is preserved, including the sign of zero.

NaN and positive and negative infinity are not valid elements of portable floating-point Arrays in Version 1.

Complex Elements

A complex64 element consists of

1. its real component encoded as a float32 element, followed by
2. its imaginary component encoded as a float32 element.

A complex128 element consists of

1. its real component encoded as a float64 element, followed by
2. its imaginary component encoded as a float64 element.

Thus, a complex64 element occupies exactly 8 bytes and a complex128 element occupies exactly 16 bytes.

Canonical Element Byte Stream

The canonical element byte stream is the concatenation of the canonical element encodings in C-contiguous logical order.

No separators, element counts, or element lengths occur within the byte stream because the Array type and shape determine both the size of each element and the total number of elements.

An Array containing zero elements therefore has an empty canonical element byte stream.

External Types

Relative Path Encoding

A relative path with N components is encoded as

U(N) || S(component_0) || S(component_1) || ... || S(component_(N-1))

Each component is Unicode string information, without a String DCT identifier. The path MUST be nonempty and each component MUST satisfy all relative-path invariants. No path separator is encoded.

SHA-256 Digest Encoding

A SHA-256 digest is encoded as exactly 32 bytes.

The bytes appear in the order defined by SHA-256.

No length field is encoded because the digest length is fixed.

The canonical encoding contains the digest bytes themselves, not a hexadecimal, Base64, or other textual representation.

ExternalFile

The canonical encoding of an ExternalFile DCI consists of

1. the ExternalFile DCT identifier,
2. the canonical encoding of its relative path, and
3. the 32-byte SHA-256 digest of the external file contents.

Thus,

0f
|| PATH
|| SHA256

where PATH is the relative-path encoding defined above and SHA256 is the digest of the exact external file byte sequence.

No file contents are included in the canonical encoding of the ExternalFile DCI.

The content digest participates directly in ExternalFile identity. Different external byte sequences are expected to have different SHA-256 digests under the cryptographic collision-resistance assumption; this is not a mathematical guarantee.

The path also participates directly in ExternalFile identity. Two ExternalFile DCIs with identical external contents but different relative paths are distinct.

ExternalArray

The canonical encoding of an ExternalArray DCI consists of

1. the ExternalArray DCT identifier,
2. the canonical encoding of its relative path,
3. the one-byte portable Array type identifier,
4. the Array rank encoded using the canonical unsigned integer encoding,
5. each dimension of the Array shape encoded using the canonical unsigned integer encoding, and
6. the 32-byte SHA-256 digest of the external Array element byte stream.

Thus, for an ExternalArray of rank (R) with shape

$$ (d_0,d_1,\ldots,d_{R-1}), $$

the canonical encoding is

10
|| PATH
|| ARRAY_TYPE
|| U(R)
|| U(d_0) || U(d_1) || ... || U(d_(R-1))
|| SHA256

where

• PATH is the canonical relative-path encoding,
• ARRAY_TYPE is the one-byte portable Array type identifier,
• U is the canonical unsigned integer encoding, and
• SHA256 is the digest of the canonical Array element byte stream.

The logical external payload is exactly the canonical Array element byte stream defined for the specified Array type and shape. That stream contains no DCT identifier, Array type identifier, rank, shape, digest, header, or other framing metadata. A storage compression envelope is outside the logical stream.

The Array type and shape determine the exact logical byte length, after any decompression. SHA-256 is computed over that logical stream. Persistence defines uncompressed .bin, gzip .bin.gz, and HDF5 storage; the canonical DCI encoding itself is unchanged.

Two ExternalArray DCIs are equal if and only if their

• relative paths,
• Array types,
• shapes, and
• content digests

are equal.

ExternalArray and Array are distinct DCTs and therefore never have equal instances, even when an ExternalArray refers to the same numerical element values represented by an Array DCI.

The path and digest both participate in ExternalArray identity.

A resource found at the specified path whose byte sequence has a different SHA-256 digest does not satisfy the ExternalArray DCI.

A missing or inaccessible external resource does not change the identity of the ExternalArray DCI; it prevents the external contents from being retrieved or verified.

Canonical Encoding Validity

A byte sequence is a valid canonical encoding of a DCI if and only if it satisfies all encoding rules defined by this specification and encodes a valid instance of the identified DCT.

A decoder of canonical DataCanon encoding MUST reject any byte sequence that violates these requirements.

Complete Consumption

When a byte sequence is presented as the canonical encoding of exactly one DCI, decoding MUST consume the entire byte sequence.

Bytes remaining after a complete DCI has been decoded make the input invalid as the canonical encoding of exactly one DCI.

Thus,

E(x) || additional_bytes

is not the canonical encoding of x.

DCT Identifiers

The first byte of every DCI encoding MUST be a DCT identifier defined by the applicable version of the DataCanon specification.

An unassigned DCT identifier is invalid.

The payload following the identifier MUST satisfy the encoding rules of that DCT.

Canonical Unsigned Integer Encoding

Every unsigned LEB128 value MUST use the minimum number of bytes required to represent its value.

An encoding containing a redundant high-order zero group is invalid.

A continuation bit indicating another byte MUST be followed by another byte.

An unterminated unsigned integer encoding is invalid.

Canonical Signed Integer Encoding

A signed Integer payload MUST consist of the canonical unsigned integer encoding of a value produced by the zigzag mapping.

Because the zigzag mapping is bijective over the mathematical integers, every canonically encoded nonnegative zigzag value determines exactly one mathematical integer.

Floating-Point Values

A Float payload MUST contain exactly eight bytes encoding a finite IEEE-754 binary64 value.

A payload representing NaN or positive or negative infinity is invalid in Version 1.

The same requirement applies at the corresponding width to portable floating-point Array elements.

Strings

A String payload MUST contain

1. a canonical unsigned integer specifying the number of UTF-8 bytes, followed by
2. exactly that number of bytes.

Those bytes MUST form a valid UTF-8 encoding of a sequence of Unicode scalar values.

Overlong UTF-8 encodings, surrogate code points, malformed UTF-8 sequences, and other byte sequences not representing Unicode scalar values are invalid.

No Unicode normalization is performed.

Bytes

A Bytes payload MUST contain a canonical unsigned integer byte count followed by exactly that number of bytes. Every byte value is valid; UTF-8 validation does not apply. Truncated payloads and nonminimal length encodings are invalid.

Fraction

A Fraction payload MUST contain two minimally encoded signed integer fields. The denominator MUST be positive. Missing fields, nonminimal encodings, and nonpositive denominators are invalid.

Atomic Types

The payload of an atomic DCI MUST contain exactly the number and DCTs of constituents required by its DCT.

Each constituent MUST itself have a valid canonical encoding and MUST satisfy all invariants required by the containing DCT.

For example, Power exponent fields MUST be positive mathematical integers. Direct fields use their prescribed payload encodings, not nested DCI encodings.

Ordered Composite Types

The encoded item, entry, or field count MUST equal the number of constituents that follow.

Each constituent DCI MUST have a valid canonical encoding.

For Mapping and Record, all uniqueness requirements defined by the corresponding DCT MUST also be satisfied.

Compositional numerical types

Every Sum and Product operand MUST have a valid complete canonical encoding. Sum operands MUST be Numerical DCIs; Product operands MUST be Scalar Numerical DCIs, including recursively scalar Sum and OrderedProduct operands. Operand encodings MUST be in nondecreasing lexicographic byte order; equal adjacent encodings are permitted and MUST retain their multiplicities.

An OrderedProduct contains its declared number of Numerical DCIs in order, without sorting or deduplication.

A Power MUST contain a Scalar Numerical base DCI followed by two minimally encoded positive signed integer fields. A Quotient MUST contain two Scalar Numerical DCIs in numerator/denominator order.

Decoding MUST preserve constituent types, exponent structure, multiplicity, and nesting without evaluation or mathematical normalization.

Set

A Set payload MUST contain exactly the encoded number of members.

Each member MUST be a valid canonically encoded DCI.

Member encodings MUST occur in strictly increasing lexicographic byte order.

Because equal DCIs have identical canonical encodings, strict ordering also ensures that no duplicate Set members occur.

A Set whose members are not in canonical byte order is not canonically encoded.

Multiset

A Multiset payload MUST contain exactly the encoded number of member occurrences. Each occurrence MUST be a valid canonically encoded DCI.

Member encodings MUST occur in nondecreasing lexicographic byte order. Adjacent equal encodings are valid and represent repeated members. Decreasing member encodings are invalid.

A decoder MUST preserve every occurrence and MUST NOT deduplicate or flatten members.

UnorderedMapping

An UnorderedMapping payload MUST contain exactly the encoded number of entries.

Each key and value MUST be a valid canonically encoded DCI.

Entries MUST occur in strictly increasing lexicographic order of their complete canonical key encodings.

No two entries may have equal keys.

An UnorderedMapping whose entries are not in canonical key order is not canonically encoded.

Record

A Record payload MUST contain exactly the encoded number of fields.

The Record name and every field name MUST use valid Unicode string information encoding without a DCT identifier.

Field names MUST satisfy the uniqueness requirement of the Record DCT.

Field values MUST be valid canonically encoded DCIs.

Array

An Array payload MUST use a portable Array type identifier defined by this specification.

The encoded rank MUST equal the number of encoded dimensions.

Each dimension MUST be a canonically encoded nonnegative integer.

The number of elements is determined by the product of the dimensions, with an empty shape defining a rank-zero Array containing one element.

The canonical element byte stream MUST contain exactly

number_of_elements * bytes_per_element

bytes for the specified Array type.

Every element encoding MUST satisfy the requirements of that Array type.

In particular, floating-point and complex elements MUST NOT contain NaN or positive or negative infinity in Version 1.

The Array decoder consumes exactly the number of element bytes determined by its type and shape.

Relative Paths

A canonically encoded DataCanon relative path MUST contain at least one component.

The encoded component count MUST equal the number of components that follow.

Each component MUST be valid Unicode string information satisfying the DataCanon relative path requirements.

In particular,

• a component MUST NOT be empty,
• a component MUST NOT equal "." or "..", and
• a component MUST NOT contain / or \.

ExternalFile

An ExternalFile payload MUST contain

1. a valid DataCanon relative path, followed by
2. exactly 32 bytes representing a SHA-256 content digest.

The external file itself is not required to be available in order for the ExternalFile DCI or its canonical encoding to be valid.

Availability and verification of the referenced external resource are separate operations.

ExternalArray

An ExternalArray payload MUST contain

1. a valid DataCanon relative path,
2. a portable Array type identifier,
3. a valid rank and shape, and
4. exactly 32 bytes representing the SHA-256 digest of the external canonical Array element byte stream.

The expected external byte-stream length is determined by the Array type and shape.

The external resource itself is not required to be available in order for the ExternalArray DCI or its canonical encoding to be valid.

Availability, byte-length verification, and content-digest verification of the referenced resource are separate operations.

Decoding

Canonical decoding is the inverse of canonical encoding.

For every DCI x,

decode(E(x)) = x

A conforming canonical decoder MUST reject any byte sequence that is not the canonical encoding of exactly one valid DCI.

Canonical decoding performs no mathematical simplification, type coercion, Unicode normalization, or other transformation of the represented DCI.

Fingerprinting

Every DataCanon Instance has a deterministic fingerprint.

The fingerprint of a DCI is the SHA-256 digest of its canonical encoding.

For DCI x with canonical encoding E(x),

$$ F(x) = \operatorname{SHA256}(E(x)). $$

A fingerprint is a sequence of exactly 32 bytes. Its conventional textual representation is the corresponding sequence of 64 lowercase hexadecimal characters. The textual representation is not itself the fingerprint.

Because equal DCIs have identical canonical encodings, equal DCIs necessarily have identical fingerprints.

Fingerprint equality does not define DCI equality. DCI equality is defined by the corresponding DCT.

The canonical encoding, rather than the fingerprint, is the complete representation of DCI identity.

Cryptographic Digests and Identity

DataCanon uses SHA-256 in two distinct roles:

• to identify the contents of external resources, and
• to compute fingerprints of DataCanon Instances.

SHA-256 is not mathematically injective. Distinct byte sequences may therefore have identical SHA-256 digests.

The equality of DataCanon Instances is defined exactly by their DCTs and constituents. For external DCTs, the content digest is a constituent of the DCI; the external byte sequence itself is not.

Consequently, the identification of external contents by their digests, and the use of fingerprints as practical identifiers, rely on the cryptographic collision resistance of SHA-256.

The cryptographic assumptions do not alter the definitions of DCI equality or canonical encoding.

Appendix A: Canonical Encoding Test Vectors

This appendix specifies fixed Version 1 canonical byte encodings for selected DCIs. Hexadecimal byte pairs are separated by spaces for readability; the spaces are not part of the encoded byte sequences. The test vectors provide independent conformance checks and do not change the encoding rules above.

The following additional vectors are displayed as concatenated fields to make their structure explicit.

ExternalFile with path components ( "x.bin" ) and a 32-byte all-zero digest:

0f
01
05 78 2e 62 69 6e
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

ExternalArray with path components ( "x.bin" ), Array type float64, shape (2,), and a 32-byte all-zero digest:

10
01
05 78 2e 62 69 6e
0b
01 02
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The all-zero digests in the last two vectors are illustrative digest constituents. The test vectors specify DCI metadata encoding and do not assert the existence of external resources with those digests.