Python API

A DataCanon Type (DCT) defines its instances and their identity. The Python reference implementation provides construction of DataCanon Instances (DCIs), conversion, Shorthand, canonical bytes, fingerprints, persistence, and additional conventional numerical and collection types.

A conventional type is a type used directly for data and computation in a programming language, including built-in, library-provided, and user-defined types. Conventional list, Fraction, and Power objects are distinct from DCIs represented by ListDC, FractionDC, and PowerDC objects. DCIs and conventional objects are both Python objects; conversion connects types with different contracts.

For a step-by-step introduction, use Conventional Types.

Public API

Classes implementing DCTs, such as RecordDC and BytesDC, are available from data_canon.instance. Field is a helper class for Record fields. Conversion and Shorthand parsing can construct DCIs without requiring direct use of those classes.

Shared queries, transformations, and output

The functions below are available from data_canon. No function invokes argparse or calls the dc executable. Applications can select, transform, and display a DCI using the same operations as the command-line tool.

Numerical transformations preserve Mapping keys and external references, without retrieving payloads. Set collisions fail rather than dropping members; Multiset occurrences are retained. Rounding retains numerical expression structure; call to_float() first to approximate complete expressions. Scalar Sum and Product evaluation follows canonical operand order; scalar OrderedProduct evaluation retains factor order. Array-valued Sums and OrderedProducts retain their structure while scalar subexpressions are converted. The input DCI and its cached fingerprint remain unchanged. Evaluation failures, nonfinite results, and Array overflow are reported as exceptions.

from data_canon import (
    parse_shorthand, get_values, index_value, to_float, round_floats,
    render_output,
)
from data_canon.instance import StringDC

source = parse_shorthand('{"coordinates": [(Pi, 1/2), (E, 0)]}')
selected = index_value(get_values(source, StringDC("coordinates")), 0)
result = round_floats(to_float(selected), 3)
assert render_output(result, padding=0) == "(3.142, 0.5)"

Conversion between conventional objects and DCIs

as_dc_instance(obj, *, registry=None)

Construct a DCI from an instance of a supported conventional type. Passing an existing DCI returns that instance unchanged. An optional ConversionRegistry supplies application-owned conventional-type rules, applied recursively.

from fractions import Fraction

from data_canon import as_dc_instance

instance = as_dc_instance({
    ("Eg", 0): 2.37,
    "position": (Fraction(1, 2), Fraction(1, 4), 0.199),
    "label": "O",
    "payload": b"\x00\xff",
})

Floats and portable floating-point Array elements must be finite. Conventional bytes is supported directly; convert mutable byte buffers to bytes explicitly. Construction of a DCI cannot recover information already lost when the source conventional object was constructed or manipulated.

dc_instance_to_python(node, source=None, *, registry=None)

Construct an instance of the corresponding conventional type from a DCI, recursively converting constituent DCIs. The DCI is already a Python object; the function name identifies the implementation language, not a change from one language to another.

from data_canon import dc_instance_to_python

converted = dc_instance_to_python(instance)
assert type(converted) is dict
assert type(converted["position"]) is tuple
assert isinstance(converted["position"][0], Fraction)

The conversion directions are not necessarily symmetric. A conventional Fraction reduces ratios; Sum, Product, and Power apply their own simplifications. A generated named tuple is accepted as a conventional Tuple input by as_dc_instance(), so it does not reconstruct the original Record name and fields through that function.

Conventional dictionaries and sets apply their own hashability and equality rules. A hashable List DCI key becomes an unhashable conventional list, which can make conversion to a dict fail. Distinct DCI keys or Set members can also be merged if their converted conventional objects compare equal.

Without a matching custom Record handler, names and fields must satisfy namedtuple restrictions for conventional Record conversion to succeed. Array conversion returns the stored read-only NumPy array; use .copy() when an independently mutable array is wanted. Do not circumvent the stored array’s read-only protection, because mutation would invalidate the DCI’s identity and any cached fingerprint or hash.

External resources are materialized during this conversion. Their bytes are retrieved and verified using retained source context or the supplied source. ExternalFile conversion can additionally use the implementation’s filename-based content loaders. Keep the external DCI for metadata-only work.

Conversion to conventional objects and construction of a new DCI do not promise a lossless round trip for every DCI. The DCI model retains its own information and equality rules independently of the conventional result types.

NumPy scalar conversion

as_dc_instance() supports NumPy Boolean, signed/unsigned integer, portable floating-point and complex, Unicode, and byte scalars. The conversions preserve their values using Boolean, Integer, Float, Complex, String, and Bytes DCIs. Every supported NumPy integer width converts exactly to the unlimited-precision Integer DCT. Finite binary16/32 values widen exactly to binary64, preserving signed zero; complex64/128 become Complex DCIs with Float components.

Scalar width does not become identity metadata. A zero-dimensional ndarray is still an Array DCI and retains its Array type and empty shape.

Nonfinite numerical scalars are rejected. Wider floating-point/complex formats, temporal scalars (datetime64, timedelta64), and structured or opaque void scalars require an explicit conversion or registry handler. Portable dtype aliases follow their corresponding dtype rules. Custom exact-type handlers take precedence over the built-in NumPy conversions.

See NumPy scalar values.

ConversionRegistry()

Create an application-owned registry. The conversion functions accept it through registry=registry; omission retains the existing default rules.

Both methods receive a handler callable as converter(value, convert). The second argument recursively converts using the same registry. A reverse handler’s recursive converter also retains the outer source context. Conventional-to-DCI handlers must return an InstanceDC; reverse handlers receive the original RecordDC and are responsible for validating their schema.

Rules for exact conventional types take precedence over built-in conversion. Rules do not apply to subclasses implicitly. Existing DCIs always pass through as_dc_instance() unchanged, and registering a DCI implementation class is rejected. Record matching uses the complete name, without import or class lookup. Duplicate registrations, invalid types or names, and noncallable handlers raise errors. Handler exceptions propagate without fallback.

Registries do not alter global behavior, DCT definitions, canonical encoding, or parsing and persistence. No handler is serialized into the DCI. An unregistered Record retains the generated-namedtuple fallback, including its name restrictions. Writers and codecs do not accept a registry: convert to a DCI explicitly before writing and convert decoded DCIs afterward.

The Custom Conversion Rules guide defines both directions of a date convention and demonstrates nested conversion, schema validation, and canonical and file round trips.

Shorthand parsing

parse_shorthand(text, *, preprocess=False, substitutions=None)

Parse exactly one complete Shorthand expression into a DCI:

from data_canon import parse_shorthand

phase = parse_shorthand("complex(1/2, 1/2 * sqrt(3))")

The parser accepts explicit constructors and their shorter forms in any mixture:

assert parse_shorthand("fraction(1, 2)") == parse_shorthand("1/2")
assert parse_shorthand("list(1, 2)") == parse_shorthand("[1, 2]")
assert parse_shorthand("tuple(1)") == parse_shorthand("(1)")
assert parse_shorthand("set()") == as_dc_instance(set())
assert parse_shorthand('mapping("x": 1)') == parse_shorthand('{"x": 1}')

Collection constructor arguments are their members or entries. For example, list([1, 2]) is a List containing one List, not a conversion of that inner List. set() is the empty Set; {} is the empty Mapping. The old emptyset spelling is not accepted.

Strict parsing rejects trailing input and invalid DCT information. Numerical parentheses are not general grouping: (1 + 2) is a singleton Tuple containing a Sum. Use constructors when nesting needs to be explicit.

Preprocessing can be requested directly:

prepared_instance = parse_shorthand("<1 2 3>", preprocess=True)

The dc CLI preprocesses Shorthand by default and offers --strict; that frontend policy does not change this function’s strict default.

Supplying substitutions requires preprocess=True. For a clear two-stage workflow, call preprocess_shorthand() separately as shown below.

Shorthand rendering

render_shorthand(node, *, layout="standard", custom_layout=None, align=False, indent=2, padding=1, longhand=False)

Render a DCI as identity-preserving Shorthand. A conventional input object is first converted with as_dc_instance(), and the resulting DCI is rendered:

from data_canon import render_shorthand

text = render_shorthand(phase)
assert parse_shorthand(text) == phase

Shorthand has no canonical text layout. Rendering must preserve DCI equality; fingerprints use canonical bytes rather than this text.

Longhand

longhand is a Boolean, defaulting to False. With True, all structured DCIs use explicit constructors. Null, Boolean, Integer, Float, and String retain their literal spellings. Fraction uses fraction(...), and Bytes retains bytes(...). Array elements retain their specialized numerical syntax. The boundary is syntactic, not the primitive/atomic/composite classification. For example, sum(1, 1.0, fraction(1, 2)) is longhand; literals need no additional wrappers. Version 1 defines neither integer(...), float(...), nor string(...) wrappers, nor a separate fully wrapped rendering mode. The same parser accepts both outputs.

value = parse_shorthand("1/2 * sqrt(3)")
assert render_shorthand(value, layout="standard") == "1/2 * sqrt( 3 )"
assert render_shorthand(value, layout="standard", longhand=True) == (
    "product( fraction(1, 2), power( 3, 1, 2 ) )"
)

Default rendering uses constructors to retain empty or singleton Sums and Products, nesting, and unambiguous Power bases. Quotient retains quotient(...). No separate notation or power_style option is provided.

Independent layout, alignment, indentation, and padding

layout controls line breaking and delimiter placement:

For example, a semi-compact Mapping renders as:

{ "a": 1,
  "b": 2
}

Scalar-only collections and empty collections normally stay inline. Lists, Tuples, Sets, and Multisets normally become multiline when they directly contain a List or Tuple, or an item already spans multiple lines. An inline subtree suppresses those line breaks. Nonempty Mappings are multiline unless their effective layout is inline. Multiline Mapping values start below their keys, so indentation does not depend on key length. Multiline keys remain valid Shorthand and use the same recursive rendering rules.

align="left" or align="right" pads cells in rectangular List/Tuple rows and supported Array tables. align=False disables column padding. Alignment can be combined with every layout and with longhand. Ragged rows receive no column padding. Alignment does not introduce a line-width wrapping threshold.

indent controls extra indentation when structural nesting places content on new lines, defaulting to 2. For example, it controls content after a standalone opener and a multiline value below its Mapping key. It does not add spacing after an opener that shares its content line. Compact continuation items align beneath the first item, whose column is determined by the opening token and padding. Consequently, changing indentation does not move compact Mapping keys; it moves nested blocks below them.

padding controls gaps beside same-line collection and constructor delimiters, defaulting to 1. For example, a flat List is [ 1, 2 ]; padding=0 gives [1, 2]. It also supplies the gap after a shared multiline opener and before a shared multiline closer. A standalone delimiter receives no padding. Continuation items begin at relative column len(opening) + padding.

Both options accept nonnegative integers, including zero, and reject Boolean, negative, and noninteger arguments. Empty collections retain [], (), set(), and similar spellings. Primitive literals and their fixed constructor spellings, item separators, and specialized Array element brackets retain their existing forms. align independently controls table-cell padding.

packed = parse_shorthand('{1, 2, [0, 1]}')
print(render_shorthand(packed, layout="compact", indent=0, padding=0))

Output:

{1,
 2,
 [0, 1]}

rows = parse_shorthand('{"rows": [(1, 20), (30, 4)]}')
text = render_shorthand(
    rows,
    custom_layout={"list": "compact", "tuple": "compact", "mapping": "semi-compact"},
    align="left",
    indent=2,
)
print(text)
assert parse_shorthand(text) == rows

Output:

{ "rows":
    [ ( 1 , 20 ),
      ( 30, 4  ) ]
}

layout is a single global layout name, defaulting to "standard". custom_layout is an optional conventional mapping from lowercase type names to layout names. Each override applies only to the named DCT; all unspecified types use the global layout. The arguments do not change one another.

Accepted override names are list, tuple, mapping, unordered_mapping, record, set, multiset, sum, product, ordered_product, power, quotient, complex, array, external_file, and external_array. Sets, Multisets, Records, and other constructor forms follow the same four layout contracts. Mapping overrides do not implicitly affect UnorderedMappings or Records. Ordinary scalar literals remain unchanged. The three multiline layouts do not force a flat sequence to break; inline suppresses structural line breaks.

text = render_shorthand(
    rows,
    layout="compact",
    custom_layout={"mapping": "semi-compact", "set": "standard"},
    align="right",
    indent=4,
)
assert parse_shorthand(text) == rows

Constructor continuation arguments align beneath the first argument, at the opening token’s length plus padding. Indentation does not change that column. Unknown layouts or type names are rejected. layout does not accept a mapping. align defaults to False; only False, "left", and "right" are accepted. Settings are passed per call.

Force one line globally or for a subtree

layout="inline" renders the selected value and every descendant on one line. An inline parent overrides child layout choices inside that subtree. Alignment is disabled there, and indentation has no effect; padding still applies. String line breaks remain escaped within their literals. No input is evaluated or mathematically simplified.

text = render_shorthand(rows, layout="inline", padding=0)
print(text)
assert parse_shorthand(text) == rows

Output:

{"rows": [(1, 20), (30, 4)]}

Use custom_layout={"list": "inline"} to keep Lists on one line while other values follow their selected layouts. A Mapping or Array inside such a List is also inline, even if its own type override requests standard. Siblings outside that List retain their selected layouts. A type override is consulted at each node unless an enclosing inline value has already forced the subtree.

Array element brackets retain their specialized numerical syntax; Array elements are not List DCIs. The Array constructor follows its selected layout, and structural Array nesting uses the requested indentation. Scalars retain their existing spellings.

Rectangular rows and line breaking

Outside an inline subtree, Lists, Tuples, Sets, and Multisets become multiline when they directly contain a List or Tuple, including an empty one, or when an item already renders across multiple lines. Scalar and inline numerical items do not independently force breaks. Flat sequences have no automatic line-width threshold.

Alignment recognizes rectangular List or Tuple rows inside any of those four collection types and preserves each row’s DCT. Short Sums, Products, and Powers can occupy a single cell. Ragged rows receive no table padding. The rules also apply to longhand constructors. Column padding may change across a table when one cell grows; use align=False when minimizing such whitespace changes matters for version-control diffs.

Numerical conventional types

from data_canon import Sum, Product, OrderedProduct, Power, Quotient, ComplexNumber

The numerical conventional types accept int, finite float, fractions.Fraction, and instances of the supported numerical conventional types. Boolean is not a numerical operand. Their construction, equality, and arithmetic rules are distinct from the associated DCT rules.

Sum(*items) and Product(*items)

Construct variadic sums and products:

a = Power(2, 1, 2)
b = Power(3, 1, 2)
assert Sum(b, a, 3*b) == Sum(4*b, a)
assert Product(a, a) == Product(Power(2, 1))

For scalar-only operands, conventional Sum and Product flatten instances of their own type, combine matching terms or factors, and preserve first-occurrence presentation order. Equality compares exact normalized values across supported numerical types and ignores operand order. Conventional float contributions remain separate from exact rational quantities; matching float contributions can combine with one another. Scalar multiplication distributes over conventional scalar Sum objects. Array-valued Sums retain structure until explicit evaluation.

.items is the immutable tuple of simplified operands. Conventional Sum and Product objects are frozen and unhashable. Sum and Product DCIs are hashable and retain their prescribed DCI structure independently of conventional simplification.

Power(base, numerator, denominator=1)

Construct a symbolic positive rational power:

root = Power(3, 1, 2)
cube_root = Power(5, 1, 3)
assert Power(5, 2, 4) == Power(5, 1, 2)

The exponent fields must be positive Python int values; conventional Power reduces their ratio. A Power DCI instead retains its direct integer exponent fields exactly. Conventional Power extracts an exact root of a rational base when the reduced exponent’s denominator permits it, retaining the remaining integer exponent. For example, Power(4, 1, 2) stores base=2, numerator=1, denominator=1, and Power(16, 3, 2) stores base=4, numerator=3, denominator=1. Power(4, 8, 1) retains all three fields; construction does not expand it to Power(65536, 1, 1). Without an exact denominator root, the base and reduced exponent remain unchanged.

Float bases follow the same construction rule, provided the extracted root is exactly representable as a float. Even roots of known negative scalar bases are rejected, and odd roots use the real branch. Nested scalar-base powers combine exponents with the real branch accounted for.

Equality and explicit numerical conversion still recognize the value of a retained integer power:

integer_power = Power(4, 8)
assert (integer_power.base, integer_power.numerator, integer_power.denominator) == (4, 8, 1)
assert integer_power == 65536
assert int(integer_power) == 65536

Surrounding Sum, Product, and Quotient arithmetic may evaluate retained powers when simplifying its operands. DCI conversion stores the conventional object’s current base and exponent fields; it does not use the cached comparison form.

Quotient(numerator, denominator)

Represent a structured ratio, including a Sum or Product denominator:

ratio = Quotient(1, Sum(a, b))
assert 1 / Sum(a, b) == ratio

The conventional constructor reduces rational coefficients, cancels common factors, and rejects denominators recognized as zero. It retains the Quotient type and does not rationalize a Sum denominator during construction. Exact cross multiplication can recognize a rationalized equivalent during comparison. The Quotient DCT’s structural validity rules are separately defined by Core.

Numerical equality

Comparison uses exact rational-expression normalization, with cached comparison forms, rather than comparing float() results. Float leaves contribute their actual binary rational values: Quotient(1, 2) == 0.5, but Quotient(1, 10) != 0.1. Float and rational storage remain separate during construction.

The supported normalization is not complete mathematical equality for arbitrary expressions. Fractional powers of nonrational bases remain symbolic; integer factorization is bounded to trial divisors through 1000 and residual perfect powers through degree 64. Unresolved factors are preserved exactly, so some mathematical equivalences may not be recognized. See Conventional Types for examples and comparison limits.

Explicit float() and int() conversion

assert float(Sum(1, 1.0)) == 2.0
approximation = float(ratio)
assert int(Quotient(-7, 3)) == -2

Conversions evaluate without modifying the object. int() truncates toward zero. Exact integer and Fraction intermediates remain exact where possible; fractional powers follow math.pow. Domain errors, division by zero, and overflow can raise Python exceptions. Numerical evaluation operates on the conventional objects, not on DCIs.

ComplexNumber(real, imaginary)

Construct a conventional ComplexNumber object with separately typed numerical components. Equality compares numerical component values, including against supported real scalars and built-in complex values. Conventional ComplexNumber is unhashable. Built-in complex stores floating-point components:

z = ComplexNumber(Fraction(1, 2), Product(Fraction(1, 2), Power(3, 1, 2)))
assert as_dc_instance(z) == phase
approximate_complex = complex(z)

The conventional ComplexNumber type supports arithmetic and conjugation. complex(z) requests floating-point complex components.

OrderedProduct and Array-valued Sum

OrderedProduct(*factors) stores real conventional scalar values, NumPy arrays, and other array-expression objects in order. Sum(*terms) also accepts Arrays and OrderedProducts. A scalar-only Sum retains its existing normalization rules. Scalar Product, Power, Quotient, and ComplexNumber components reject Array-valued structures.

• expression.evaluate() evaluates Array structure into an object-dtype NumPy Array, retaining exact conventional scalar entries.
• np.asarray(expression, dtype=float) or dtype=complex explicitly requests machine values. Without a dtype, exact object entries remain.
• Array-valued Sum requires matching shapes at any rank. OrderedProduct requires rank-two Arrays and compatible multiplication dimensions. Scalar factors scale the result; no broadcasting, implicit identity insertion, or reshaping occurs.
• Construction does not infer shapes or multiply Arrays. Conversion from a valid DCI can still fail if a conventional scalar subexpression is invalid, as with a Quotient having a recognized zero denominator.
• Conventional whole-Array equality returns a Boolean using exact scalar comparisons. Array-valued expressions cannot be converted to scalar float/int.

is_numerical_dci(node) and is_scalar_numerical_dci(node) are available from data_canon.instance. Both classify structure without mathematical evaluation. OrderedProduct DCIs use OrderedProductDC(items_tuple); their immutable operands are exposed through .items. Sum DCIs remain unordered and preserve multiplicity.

Bytes and collections

Conventional bytes

payload = as_dc_instance(b"\x00\xff\x80")
assert payload == parse_shorthand('bytes("00ff80")')

A Bytes DCI has byte-for-byte identity and no interpretation metadata. To store metadata alongside a payload, use a Record:

from data_canon.instance import BytesDC, Field, RecordDC, StringDC

blob = RecordDC("Payload", (
    Field("media_type", StringDC("application/octet-stream")),
    Field("filename", StringDC("sample.bin")),
    Field("data", BytesDC(b"\x00\xff\x80")),
))

Record and field names are direct Unicode information. Field values remain DCIs whose DCTs participate in identity.

Multiset(iterable)

from data_canon import Multiset

members = Multiset(["O", "Sr", "O"])
members.add("O")
assert as_dc_instance(members) == parse_shorthand('multiset("Sr", "O", "O", "O")')

Multiset() is empty. Duplicates are retained, and order is insignificant for equality. Conventional Multiset uses its members’ equality rules and supports unhashable members when their equality yields a usable Boolean. It does not impose DCI equality on conventional members.

UnorderedDict

from data_canon import UnorderedDict

first = as_dc_instance(UnorderedDict({"a": 1, "b": 2}))
second = as_dc_instance(UnorderedDict({"b": 2, "a": 1}))
assert first == second

Conventional dict objects construct ordered Mapping DCIs. UnorderedDict retains Python dictionary-key restrictions and cannot recover keys already merged by conventional equality.

Canonical encoding and fingerprints

canonical_encode(node) and canonical_decode(data)

from data_canon import canonical_encode, canonical_decode

encoded = canonical_encode(instance)
assert type(encoded) is bytes
assert canonical_decode(encoded) == instance

Encoding includes every DCI’s one-byte DCT identifier and its prescribed payload. Variable-type constituents use complete DCI encodings. Fixed integer fields in Fraction and Power use signed integer payloads directly. Record names and path components use UTF-8 byte counts and bytes directly, without String identifiers.

Array elements use logical C order and portable element representations, with multibyte values in big-endian order. Canonical encoding is independent of Shorthand and NumPy memory layout.

canonical_decode() requires conventional bytes, consumes exactly one complete DCI, and rejects malformed or noncanonical input. External metadata decodes without resource retrieval.

fingerprint(node)

from data_canon import fingerprint

identifier = fingerprint(instance)
assert type(identifier) is str
assert len(identifier) == 64

The Python return value is lowercase hexadecimal SHA-256 of the canonical bytes. The abstract fingerprint is 32 digest bytes. Equal DCIs have equal fingerprints; a matching digest is not the definition of equality.

Python hashing of DCIs

All 24 DCT implementations support Python hash() and can be used as dictionary keys, including Arrays, Lists, Sets, and nested structures:

from data_canon.instance import FloatDC, IntegerDC, ListDC

key = ListDC((IntegerDC(1), FloatDC(-0.0)))
lookup = {key: "sample"}
assert lookup[ListDC((IntegerDC(1), FloatDC(-0.0)))] == "sample"

Hash computation is lazy. Constructing a DCI does not compute its hash or fingerprint. Null, Boolean, Integer, Float, String, Bytes, and Fraction DCIs use direct hashes; Float uses its binary64 bits. Structured DCIs use their cached fingerprint bytes, computing the fingerprint on the first hash request if it is not already available. Nonsemantic source context and payload caches do not participate in the hash.

Python hashes satisfy the equality requirement that equal DCIs have equal hashes. Hash values need not distinguish unequal DCIs and are not stable cross-process identifiers. Use fingerprint() for the portable identifier. Hashability of a DCI does not imply hashability of its converted conventional object: a List DCI and a conventional list remain different types.

Persistence

write(data, filename=None, format="shorthand")

from data_canon import read, write

written = write(instance, filename="output_data.dcb")
assert read(written) == instance

A recognized filename suffix overrides format selection. .hdf5 is also recognized. Without a suffix, the selected format’s suffix is appended. Without a filename, the fingerprint supplies the filename stem. The return value is a pathlib.Path.

The current writer verifies and persists referenced external resources. For .dcs and .dcb, their paths are relative to the output directory; HDF5 embeds them separately. This public interface has no metadata-only export flag. A write can fail when required resources are missing or conflicting.

read(filename)

Select the format by suffix and reconstruct a DCI. Reads preserve nonsemantic source context for later external-resource access. They do not materialize external payloads.

write_hdf5(node, filename, *, store_fingerprint=False) and read_hdf5(filename)

from data_canon import read_hdf5, write_hdf5

write_hdf5(instance, "example.h5", store_fingerprint=True)
assert read_hdf5("example.h5") == instance

Arrays use native HDF5 numerical datasets; Bytes uses a rank-one uint8 dataset. Fixed metadata fields reuse integer/string payload storage without DCI kind attributes. Genuine constituent DCIs retain their required kind attributes. HDF5 file bytes are not canonical DCI bytes.

store_fingerprint adds optional file-root metadata. It does not add a DCI constituent. Complete HDF5 writes verify external payloads before embedding them.

External-reference construction types

ExternalFile(path, _data=None)

Construct a conventional ExternalFile object from a relative path to an existing file, or from contained input bytes or text:

from data_canon import ExternalFile

resource = ExternalFile.from_bytes(b"input data", filename="input.bin")
resource_instance = as_dc_instance(resource)

from_bytes(data, filename=None, directory="external_dc") retains input bytes as a nonsemantic cache. from_string(text, filename=None, directory="external_dc") encodes UTF-8. If omitted, the filename is derived from the content digest. The resulting DCI contains a relative path and digest, not the payload itself.

ExternalArray(data, directory="external_dc", filename=None, compression=False)

import numpy as np
from data_canon import ExternalArray

external_array = as_dc_instance(ExternalArray(
    np.array([1.0, -0.0], dtype="float64"),
    filename="values.bin",
))

Conversion records portable type, shape, relative path, and SHA-256 of the canonical Array element stream. Cached data and source context do not affect identity. Metadata reads need no payload; conversion of the ExternalArray DCI to a conventional NumPy array materializes and verifies it.

Compress external Array storage

compression is a Boolean and defaults to False. Choose a filename stem; the constructor appends .bin or .bin.gz and stores the finalized filename:

forces = np.zeros((32, 32), dtype="float64")
compressed = ExternalArray(forces, filename="forces", compression=True)
assert compressed.filename == "forces.bin.gz"
compressed_instance = as_dc_instance(compressed)
assert compressed_instance.path.components[-1] == "forces.bin.gz"

A filename is a nonempty single component. Explicit suffixes must match the setting; suffixes other than .bin and .bin.gz are rejected. Use directory for path prefixes. Compression is encoded by the finalized path, not a new DCI constituent. The digest always covers the uncompressed canonical element bytes.

write(compressed_instance, filename="forces.dcs")
compressed_restored = read("forces.dcs")
assert compressed_restored == compressed_instance
assert np.array_equal(dc_instance_to_python(compressed_restored), forces)

Directory output writes external_dc/forces.bin.gz; retrieval transparently decompresses, verifies the digest, and validates shape, type, and elements. Malformed gzip and excess decompressed data are rejected. Existing conflicting resources are not overwritten. HDF5 output embeds logical bytes in a chunked gzip-compressed dataset (level 6), retaining the same DCI path and digest:

write(compressed_instance, filename="forces.h5")
assert read("forces.h5") == compressed_instance

Exporting that HDF5 DCI to a directory recreates the gzip file. Empty and scalar Arrays are supported. This option does not compress native Array DCIs or opaque ExternalFiles, and does not add .dcb.gz representation support. A .gz ExternalFile still means its exact opaque bytes. Raw canonical element payloads use .bin, not .dcb, because they are not complete DCI encodings. The runnable examples/external_array_compression.py demonstrates directory/HDF5/directory transfer with a zero-heavy force-constant Array.

Preprocessing

preprocess_shorthand()

The common call form is:

from data_canon import preprocess_shorthand

prepared = preprocess_shorthand("<1 2 3>")
assert parse_shorthand(prepared) == parse_shorthand("[1, 2, 3]")

Transformations run in order: compact List expansion, external and inline ordered textual substitution, then explicit scalar eval(...) in the body. Local definitions may use external names and earlier local definitions. Each local definition resolves its own explicit eval(...) before becoming available to later definitions. Self-references and forward references are rejected. Replacements are literal text; use constructors to preserve nested numerical expression boundaries. The result is text. Quoted Strings, including Bytes hexadecimal arguments, are protected.

See the Shorthand Preprocessing guide for separate introductions to each feature.

YAML interchange

parse_yaml(text) and render_yaml(node, *, align=False, padding=1)

from data_canon import parse_yaml, render_yaml

yaml_instance = parse_yaml("""
'$ ("Eg", 0)': 2.37
name: calculation
phase: $ complex(1/2, 1/2 * sqrt(3))
payload: $ bytes("00ff80")
""")
assert parse_yaml(render_yaml(yaml_instance)) == yaml_instance
assert dc_instance_to_python(yaml_instance)[("Eg", 0)] == 2.37

Native YAML sequences are Lists; mappings become ordered Mappings. The $ prefix is interpreted after YAML String decoding and invokes strict Shorthand. The $ marker requires a whitespace boundary; $100 and ${HOME} remain literal Strings. A lone $ is reserved but lacks the required embedded DCI. Quoting alone does not escape a marker; a backslash does (\$ 1/2). Literal reserved prefixes are escaped by the writer. Native key collisions or unhashable converted keys cause the entire Mapping to use the airlock.

Bytes, Multiset, and all structured numerical DCTs use embedded Shorthand. YAML does not implicitly preprocess embedded text or retrieve external resources. Its presentation is not canonical and is not fingerprinted directly.

YAML uses block-style Lists and Mappings, keeps single-line scalars unwrapped, and uses literal blocks for multiline Shorthand. Empty collections retain [] and {}. Scalar quoting is left to YAML’s syntax and scalar-resolution rules.

rows = parse_shorthand("[(1, 2), (30, 4)]")
yaml_text = render_yaml(rows, align="left", padding=0)
assert parse_yaml(yaml_text) == rows

align accepts False (default), "left", or "right". Lists of rectangular, nonempty, flat Tuple rows share column alignment; ragged or mixed rows retain ordinary formatting. Embedded Shorthand also receives the alignment setting. padding is a nonnegative integer, defaulting to 1, with the same same-line delimiter contract as render_shorthand(). Padding affects $ text rather than YAML indentation, and applies to embedded keys as well as values.

Specification and draft compatibility

The seven specifications define the language-independent contracts. The optional Conventional Types companion defines common conventional behavior; this API documents the Python binding. Core conformance does not require conventional arithmetic. The current fixed-field revision changes canonical bytes and fingerprints from earlier drafts. Regenerate affected old-draft .dcb files; automatic legacy-format detection is not provided.