YAML Interchange

DataCanon YAML Interchange — Version 1 Draft

Status: Draft for independent implementation review
Depends on: DataCanon Core Version 1; DataCanon Shorthand Version 1
Scope: Interchange between YAML documents and DataCanon Instances (DCIs)
Out of scope: Canonical binary encoding rules, HDF5 persistence, Shorthand preprocessing, and application-specific YAML tags

Introduction

DataCanon YAML Interchange permits ordinary YAML values to represent DataCanon Instances (DCIs), while preserving DataCanon-specific structures through an embedded Shorthand convention called the $ airlock.

The interchange is not a new DataCanon Type system and does not define DCI identity. All DCI validity and equality requirements come from DataCanon Core Version 1. Embedded values use strict DataCanon Shorthand Version 1, without preprocessing.

A conforming writer MUST produce YAML from which a conforming reader reconstructs an equal DCI. YAML documents need not be byte-for-byte identical across implementations. Indentation, quoting, flow versus block style, and other identity-preserving YAML presentation choices are not standardized.

A conforming reader is required to support the portable YAML subset specified here; it is not required to accept every construct permitted by the broader YAML language.

Conformance and terminology

The terms MUST, MUST NOT, SHOULD, and MAY indicate normative requirements and recommendations.

Let Y(x) denote any conforming YAML rendering of DCI x, and let P_Y denote YAML interchange parsing. The primary conformance requirement is:

P_Y(Y(x)) = x

where equality is DataCanon equality. A conforming implementation MUST preserve, among other things:

• distinctions between DCTs, such as Integer and Float, or List and Array;
• unreduced Fraction and unevaluated numerical structure;
• the sign of floating-point zero;
• order in ordered collections;
• member and key identity in unordered collections; and
• the paths, types, shapes, and digests of external DCIs.

The interchange reader MUST NOT implicitly perform Shorthand substitutions, compact List expansion, or eval(...) processing on $ content.

Portable YAML data model

YAML-native values

The following YAML-native values are mapped to DCIs:

YAML sequences MUST be interpreted as Lists, never inferred to be Arrays, regardless of their dimensions or numerical contents.

YAML mappings MUST preserve the order in which their entries are presented when converted to Mapping DCIs. YAML mapping keys and values are converted recursively using the same rules as other YAML values, subject to the portable-subset limitations below.

The interchange defines no native YAML representation for Bytes, Fraction, Sum, Product, Power, Quotient, Complex, Tuple, Set, Multiset, UnorderedMapping, Record, Array, ExternalFile, or ExternalArray. Such DCIs are represented using the $ airlock.

YAML scalar resolution

For the portable subset, implementations MUST agree on whether a scalar is Null, Boolean, Integer, Float, or String before applying the $ convention. The Python reference implementation uses PyYAML’s SafeLoader scalar resolution and SafeDumper emission behavior.

In particular, the following scalar forms have the indicated interpretations:

The reference loader also recognizes some additional YAML scalar spellings, including yes, no, on, and off as Booleans. Such spellings are not recommended for portable hand-authored documents. A conforming writer SHOULD use the unambiguous forms in the table and MUST quote strings that would otherwise be resolved as a different YAML scalar type.

The exact PyYAML reference version and the complete boundary of portable input scalar spellings remain Version 1 finalization items. Independent implementations SHOULD use the reference behavior and the conformance vectors rather than silently applying a different YAML schema. In particular, YAML 1.1-style and YAML 1.2-style implicit resolution are not always interchangeable.

Any resolved numeric value MUST satisfy the invariants of its corresponding DataCanon DCT. In particular, nonfinite floating-point values are invalid DataCanon Floats.

YAML features outside the portable subset

The portable subset does not standardize arbitrary YAML tags, application-defined object construction, timestamps, binary tags, merge keys, recursive aliases, or duplicate YAML mapping keys. Implementations MAY reject such input. An implementation’s acceptance of these constructs does not expand the required portable subset.

The Python reference implementation may accept additional YAML forms through yaml.safe_load; this does not make those forms normative DataCanon interchange syntax. Implementations SHOULD reject or clearly document any input whose conversion would silently lose DCI-relevant information.

The $ airlock

Purpose

The embedded DataCanon marker is $. After optional leading whitespace, the marker MUST be followed by at least one whitespace character before the strict Shorthand value. Whitespace is recognized as Unicode whitespace, consistently with removal of leading whitespace. A decoded YAML String beginning with this marker and boundary represents one embedded strict DataCanon Shorthand DCI, unless escaped as defined below. A marker at the end of the String is also reserved, but is invalid as embedded DC because the required DCI is missing.

The marker performs neither substitution nor evaluation. $100, ${HOME}, $HOME, $1/2, and $$ 1/2 are ordinary Strings because the initial $ has no whitespace boundary. A dollar sign elsewhere in a String has no special meaning. Only $ is recognized as the embedded DataCanon marker.

Examples:

fraction: $ 2/4
product: $ 1/2 * sqrt(3)
array: $ array([1.0, 2.0])

The resulting values are respectively a Fraction DCI, a Product DCI, and an Array DCI. They are not YAML Strings in the resulting DataCanon Mapping.

Quoting alone does not escape the marker. The prefix is recognized in the decoded YAML string value, not by searching raw YAML source characters. Consequently, YAML quoting and block-scalar formatting do not alter the airlock semantics.

Embedded Shorthand parsing

For an unescaped airlock String:

1. Remove leading whitespace before $.
2. Remove the $ prefix, whose whitespace boundary MUST have been recognized.
3. Remove leading whitespace from the remainder.
4. Parse the remainder as one complete strict Shorthand DCI.

The embedded text MUST satisfy the strict Shorthand specification, including complete consumption. An invalid embedded DCI makes the interchange input invalid.

For example, the following YAML scalar can be emitted as a literal block:

|-
  $
  {
    "a": 1,
    "b": 2
  }

Its decoded YAML String begins with $ and contains a Shorthand Mapping. The |- marker is YAML presentation syntax, not part of the embedded Shorthand.

Escaping literal Strings

The prefix $ is reserved at the whitespace boundary defined above, or at the end of the String, including after leading whitespace. To preserve an ordinary DataCanon String that would otherwise be interpreted as embedded Shorthand, a writer MUST add one backslash immediately before the prefix, after any existing backslashes.

More precisely, for a decoded YAML String, divide its beginning into:

• zero or more leading whitespace characters;
• zero or more consecutive backslashes; and
• the remaining text.

If the remaining text begins with $ followed by whitespace, or consists solely of $:

• zero preceding backslashes means the remainder after $ is embedded Shorthand;
• one or more preceding backslashes means an ordinary String: remove exactly one of those backslashes and preserve all other characters.

If that boundary condition is not satisfied, the String is unchanged, including all preceding backslashes. In particular, \$100 and \${HOME} are preserved literally.

Thus, at the level of decoded YAML String values (not YAML source quoting):

The writer MUST apply this escaping convention to String DCIs in all positions, including mapping keys. YAML itself may additionally quote or escape backslashes; those YAML-level escapes are decoded before applying the DataCanon rule.

Airlock rendering

To represent a DCI through the airlock, a writer MUST:

1. Render the DCI as valid strict Shorthand that preserves its identity.
2. Prefix the resulting text with $ and whitespace separating the prefix from the Shorthand value.
3. Emit the result as a YAML String scalar.

A writer MAY use inline YAML scalar presentation or YAML literal-block presentation. A writer SHOULD use a block scalar when the embedded Shorthand contains line breaks.

The Python reference implementation uses $ for single-line Shorthand and $\n for multiline Shorthand. Its embedded Shorthand uses explicit standard layout. Callers may select alignment and same-line delimiter padding without changing DCI identity. Other conforming renderers MAY choose any identity-preserving Shorthand layout.

The reference writer emits nonempty native YAML Lists and Mappings in block style and does not wrap single-line scalars to a fixed width. Empty collections retain [] and {}. When alignment is requested, rectangular groups of nonempty, flat Tuple rows within a native YAML List share column widths; ragged rows and mixed List/Tuple rows retain ordinary rendering. Native YAML Lists remain native Lists. Same-line padding applies inside embedded Shorthand, including keys, and does not control YAML indentation. YAML syntax and scalar-resolution rules may still require quoting. Layout, alignment, padding, and quoting are presentation choices, not identity constituents.

An airlock String MUST NOT be interpreted using optional Shorthand preprocessing.

DataCanon-to-YAML rendering

Native primitive DCIs

Null, Boolean, Integer, Float, and String DCIs SHOULD be rendered using their corresponding native YAML scalar forms, subject to identity preservation.

A writer MUST distinguish native Integer and Float values, including integral-valued Floats such as 1.0. It MUST preserve floating-point signed zero. It MUST quote or otherwise mark Strings so that YAML scalar resolution does not change their DCT.

A String DCI additionally undergoes the $ prefix-escaping rule above.

List DCIs

A List DCI MUST be rendered as a YAML sequence whose members are recursively rendered in their semantic order. An empty List is represented as an empty YAML sequence.

A Tuple DCI MUST NOT be rendered as an ordinary YAML sequence, because that would change its DCT. It uses the airlock instead.

Mapping DCIs

A Mapping DCI MAY be rendered as a native YAML mapping only if all of its keys and values can be recursively converted and the converted keys can be represented without loss or collision.

For compatibility with the Python reference implementation, the native-mapping decision is made by constructing the recursively converted key/value pairs, then checking that every converted key is hashable in the reference mapping model and that no two converted keys compare equal as mapping keys.

If any converted key is unhashable, or any two keys collide under the reference mapping’s equality rules, the writer MUST render the entire Mapping DCI through the $ airlock instead of silently dropping or merging entries.

In particular, the following distinct DataCanon keys collide as Python dictionary keys:

• Boolean true and Integer 1;
• Integer 1 and Float 1.0;
• Float 0.0 and Float -0.0.

Such a Mapping MUST use the airlock, even though it is a perfectly valid DataCanon Mapping.

A writer MUST preserve the semantic entry order of an ordered Mapping, whether using native YAML or the airlock. An UnorderedMapping DCI always uses the airlock; it MUST NOT silently become an ordered Mapping DCI.

Bytes DCIs

Bytes DCIs MUST use the $ airlock, for example:

payload: $ bytes("00ff80")

The complete byte sequence is embedded in the document. A native YAML String is not inferred to be Bytes. YAML binary tags remain outside the portable subset; the explicit Shorthand construct provides the portable representation.

Other DCTs

Every DCI not explicitly given a faithful native YAML representation above MUST be rendered through the $ airlock. This includes Bytes, Fraction, Sum, Product, Power, Quotient, Complex, Tuple, Set, Multiset, UnorderedMapping, Record, Array, ExternalFile, and ExternalArray.

The airlock preserves the DCI, not necessarily any referenced external payload. An ExternalFile or ExternalArray airlock value contains its Shorthand path and digest metadata; resource retrieval and verification are governed separately by the DataCanon persistence specification.

YAML-to-DataCanon parsing

A conforming interchange reader MUST:

1. Parse the YAML document into the portable YAML value model, preserving scalar types and mapping order.
2. Convert native Null, Boolean, Integer, and Float values to their corresponding DCIs.
3. Process each YAML String using the $ airlock and escaping rules.
4. Convert YAML sequences recursively to List DCIs.
5. Convert YAML mappings recursively to ordered Mapping DCIs.
6. Validate the resulting DCIs under DataCanon Core Version 1.

The reader MUST NOT infer Arrays from nested YAML sequences. It MUST NOT simplify embedded numerical structures. It MUST NOT apply the conversion rules for conventional programming-language types to coerce distinct DCTs into one another.

A reader MAY report a malformed document as a YAML parsing error, an invalid embedded Shorthand error, or a DataCanon invariant error. Exact error-message wording is not normative.

Identity and interoperability

The YAML interchange is not a canonical textual encoding. Different YAML byte sequences can represent equal DCIs. Conforming implementations MUST agree on the resulting DCI for documents within the portable subset.

In particular, a conforming implementation MUST preserve the DCI across a YAML write–read round trip, including any $ embedded values:

P_Y(Y(x)) = x

No requirement is imposed on matching YAML whitespace, key quoting, block versus flow layout, or exact embedded Shorthand formatting.

A YAML document does not contain canonical DataCanon bytes merely by virtue of representing a DCI. Fingerprints are always computed from the DCI’s canonical binary encoding, not from YAML source text.

Security and resource handling

YAML readers SHOULD use safe data-only parsing and MUST NOT execute arbitrary application objects or code merely because a YAML tag or scalar requests it.

Embedded $ content MUST be processed by strict Shorthand parsing; it MUST NOT invoke the Shorthand preprocessor or arbitrary evaluation.

Implementations MAY enforce resource limits on YAML document size, nesting depth, aliases, and embedded Shorthand size. Such limits are implementation policy and do not change DCI equality.

Reading a YAML document containing ExternalFile or ExternalArray DCIs MUST NOT require the referenced resources to exist or be loaded. Resource resolution, integrity checking, and copying are separate operations.

Conformance examples

The following examples illustrate the portable interchange semantics. The exact YAML presentation produced by a writer may differ.

Native values and nested Lists

name: calculation
energy: -12.5
positions:
  - [0.0, 0.0, 0.0]
  - [0.5, 0.5, 0.5]

The result is a Mapping DCI containing String, Float, and nested List DCIs. The positions are not an Array DCI.

Embedded mathematical structures

fraction: $ 2/4
product: $ 1/2 * sqrt(3)
complex: $ complex(1/2, 1/2 * sqrt(3))

The result contains Fraction, Product, and Complex DCIs, respectively. The Fraction remains unreduced.

Literal reserved prefix

message: "\\$ 2/4"

After YAML string decoding and DataCanon prefix unescaping, message is a String DCI whose value is $ 2/4, not a Fraction DCI.

Mapping-key collision

A Mapping DCI containing both Boolean true and Integer 1 as keys cannot be safely represented as a native YAML mapping under the reference key semantics. A writer may emit the entire Mapping as a block-scalar airlock value:

|-
  $
  {
    true: "boolean",
    1: "integer"
  }

The reader recovers the two distinct keys and their values.

UnorderedMapping

value: $ umapping("a": 1, "b": 2)

The value is an UnorderedMapping DCI, not a native YAML Mapping DCI.

Multiset

A Multiset uses the $ airlock rather than a native YAML sequence:

samples: $ multiset(1, 1, 2)

The value contains three member occurrences, including two equal Integer members. A writer and reader MUST preserve multiplicity and nested structure. Reordering members does not change Multiset identity.

An ordinary YAML sequence such as [1, 1, 2] remains a List DCI, not a Multiset. Multisets may also be represented through the airlock in mapping-key positions, subject to the same native-key collision rules as other DCIs.

External resource

payload: $ external_file("inputs/config.txt", "0000000000000000000000000000000000000000000000000000000000000000")

The value is an ExternalFile DCI with a relative path and 32-byte digest. No file is accessed while parsing this value.

Invalid and nonportable cases

The following MUST be rejected as valid portable interchange, or are explicitly outside its guaranteed portable subset:

• A $ scalar containing malformed or incomplete strict Shorthand: invalid.
• An embedded DCI violating a core invariant, such as a zero Fraction denominator: invalid.
• A native YAML floating-point value that is NaN or infinity: invalid as a DataCanon Float.
• A native YAML value that cannot be mapped to a DataCanon DCT under the portable subset: outside the portable subset.
• Arbitrary executable YAML tags or application-object constructors: outside the portable subset and MUST NOT be executed by a conforming safe reader.
• Duplicate YAML mapping keys or input relying on merge-key resolution: outside the portable subset; behavior is not guaranteed by this draft.

Version 1 finalization items

The following are intentionally left for a final compatibility pass rather than expanded into a separate YAML standard:

1. Reference YAML scalar resolver: Pin the PyYAML reference version and establish portable scalar-resolution vectors, particularly for legacy Boolean spellings, exponent notation, and timestamp-like Strings.
2. Mapping-key collisions: Verify cross-language behavior against a compact set of distinct DCI keys that collide in Python’s conventional dictionary model.
3. YAML parser subset: Decide whether duplicate YAML mapping keys should be explicitly rejected by all conforming readers rather than left outside the portable subset.
4. Additional interchange vectors: Test YAML produced by Python against an independent implementation, checking resulting DCI canonical bytes rather than YAML text.

The finalization items do not alter the DataCanon Core DCTs or canonical binary encoding.