DataCanon

• 1: Getting Started
• 2: Guides
• 2.1: Identity and Fingerprinting
• 2.2: Persistence and External Resources
• 2.3: Conventional Types
• 2.4: Shorthand Preprocessing
• 2.5: Custom Conversion Rules
• 2.6: Command-Line Interface
• 3: Python API
• 4: Publication Outline
• 5: Reference
• 6: Specifications
• 6.1: DataCanon Core
• 6.2: Shorthand
• 6.3: Shorthand Preprocessing
• 6.4: YAML Interchange
• 6.5: HDF5
• 6.6: Persistence
• 6.7: Conventional Types

1/2 * sqrt(3)
quotient(1, sqrt(3) + sqrt(2))
complex(1/2, 1/2 * sqrt(3))

1 - Getting Started

Scientific data moves between programs, readable input files, and storage formats. A different file layout need not mean different data; a changed Array type or coordinate structure can mean different data even when the numbers look the same.

DataCanon gives structured scientific data a precisely defined identity, independent of its programming language or storage representation. Each DataCanon Instance (DCI) has one unique canonical byte encoding. A fingerprint derived from those bytes lets you identify the data across representations.

This guide constructs a dataset, identifies it, stores and inspects it, then shows how to author scientific input directly.

Installation

Clone the DataCanon code repository, then run these commands from its root:

python3 -m venv DC
source DC/bin/activate
python -m pip install .

The Makefile also provides environment setup, editable installation, and tests.

Construct and identify a DCI

Start with a Python dictionary containing a Tuple key, a List of species, a coordinate Tuple, and a numerical Array:

import numpy as np

from data_canon import as_dc_instance, fingerprint

data = {
    ("Eg", 0): 2.37,
    "species": ["Sr", "Ti", "O"],
    "origin": (0.0, 0.0, 0.0),
    "lattice": np.eye(3, dtype="float64"),
}
instance = as_dc_instance(data)
identifier = fingerprint(instance)
print(identifier)

Output:

eb8f306c801ca4452e283f052b51a6834a846399cc3aac43720fb7cf48faeca4

The whole dataset is one DCI: a Mapping containing other DCIs. Mapping keys can be structured values, including the Tuple ("Eg", 0).

Python’s dict, tuple, and NumPy arrays are conventional types used by the application. Conversion constructs a DCI whose information and equality rules are defined by DataCanon, independently of those Python types.

The fingerprint is SHA-256 of the DCI’s canonical bytes, written as 64 lowercase hexadecimal characters. You can record it alongside a calculation to identify its input data. It is derived from the data’s defined identity, not from the memory address of a Python object or the layout of a text file.

Type participates in identity. Replacing the origin Tuple with a List changes the DCI even though its coordinates have the same numerical values:

changed_data = dict(data)
changed_data["origin"] = [0.0, 0.0, 0.0]
changed_instance = as_dc_instance(changed_data)

assert changed_instance != instance
assert fingerprint(changed_instance) != identifier

Array type and shape also participate in identity. Equal DCIs have identical canonical bytes and fingerprints; DCI equality itself is defined by the data model, not by comparing fingerprints.

Write a DCI to disk as binary data

Use .dcb to store the canonical bytes directly:

from data_canon import read, write

write(instance, filename="output_data.dcb")
restored = read("output_data.dcb")

assert restored == instance
assert fingerprint(restored) == identifier

The file is binary, not text. Reading reconstructs the same DCI, including its types, Array shape, and element values. DataCanon decoding reconstructs DCIs without invoking application-defined code.

Render a DCI as Shorthand

Shorthand is a human-readable representation of a DCI:

from data_canon import parse_shorthand, render_shorthand

text = render_shorthand(instance)
print(text)

Output:

{
  ( "Eg", 0 ): 2.37,
  "species": [ "Sr", "Ti", "O" ],
  "origin": ( 0.0, 0.0, 0.0 ),
  "lattice":
    array(
      [[1.0, 0.0, 0.0],
       [0.0, 1.0, 0.0],
       [0.0, 0.0, 1.0]]
    )
}

Parsing the text recovers the same identity:

parsed = parse_shorthand(text)
assert parsed == instance
assert fingerprint(parsed) == identifier

Text layout is presentation. Canonical bytes determine the fingerprint, so spacing does not become part of the data’s identity.

Inspect a DCI via the command line

Installing the package also installs dc. With the environment active, display the binary file as the Shorthand shown above:

dc output_data.dcb

Inspect particular values without writing a Python script:

dc output_data.dcb --get '("Eg", 0)'
dc output_data.dcb --get species --index 0

Output:

2.37
"Sr"

Print the same fingerprint calculated in Python:

dc output_data.dcb --fingerprint

The CLI guide covers searches, selection, and format transfers in more detail.

Preserve identity through YAML and HDF5

The same DCI can be represented in other formats without changing its identity. YAML uses native values where faithful and a $ marker for embedded strict Shorthand, including Tuple keys and Array values:

from data_canon import parse_yaml, render_yaml

yaml_text = render_yaml(instance)
print(yaml_text)

Output:

$ ( "Eg", 0 ): 2.37
species:
- Sr
- Ti
- O
origin: $ ( 0.0, 0.0, 0.0 )
lattice: |-
  $
  array(
    [[1.0, 0.0, 0.0],
     [0.0, 1.0, 0.0],
     [0.0, 0.0, 1.0]]
  )

from_yaml = parse_yaml(yaml_text)
assert from_yaml == instance
assert fingerprint(from_yaml) == identifier

For numerical storage, HDF5 represents Arrays as native numerical datasets:

write(instance, filename="output_data.h5")
from_hdf5 = read("output_data.h5")
assert from_hdf5 == instance
assert fingerprint(from_hdf5) == identifier

Use .dcs with the same write() and read() functions to store Shorthand text. The .dcb, Shorthand, YAML, and HDF5 representations recover equal DCIs and therefore the same canonical bytes and fingerprint. Hashing the YAML or HDF5 file bytes directly would instead identify that particular file representation.

Author scientific data directly

So far, we have constructed data in Python. DataCanon also lets us author structured scientific data directly in Shorthand:

scientific_input = parse_shorthand("""
{
  "position": (1/2, 1/4, 0),
  "phase": complex(1/2, 1/2 * sqrt(3))
}
""")

The position contains exact Fractions. The Product 1/2 * sqrt(3) exactly represents an irrational quantity, and the Complex retains the structure of its real and imaginary components.

DataCanon does not do math. Parsing constructs and preserves Fraction, Power, Product, and Complex DCIs. Applications can convert DCIs to instances of conventional types for computation; DC identity belongs to the stored data structure.

Make scientific input easier to maintain

Optional Shorthand preprocessing lets an input file use named parameters. An excerpt of a crystal-structure input illustrates the idea: a parameter controls several coordinates while exact reference positions remain visible.

source = """
u = 0.305 |
{
  "basis_atoms": {
    "O": [
      (u,        u,        0),
      (1/2 +  u, 1/2 + -u, 1/2)
    ]
  }
}
"""

authored = parse_shorthand(source, preprocess=True)
write(authored, filename="structure.dcb")

Preprocessing substitutes u before strict parsing. The resulting DCI stores the coordinate values and their numerical structure; the parameter definition is an input convenience. The same identity and persistence operations now apply to this authored data.

The CLI enables preprocessing for .dcs input files. The Shorthand Preprocessing guide develops substitutions, compact Lists, and explicit eval(...) separately, and includes a complete crystal-structure example.

Next steps

• Identity and Fingerprinting: identify calculation inputs and associate results with their inputs.
• Persistence and External Resources: store complete datasets and move verified external content between directories and HDF5.
• Conventional Types: convert DCIs to conventional objects and use numerical operations.
• Custom Conversion Rules: connect your own classes to Records without changing the DC type system.
• Command-Line Interface: inspect, search, select, and transfer data from the shell.
• Python API and Specifications: implementation details and the formal contracts.

2 - Guides

The guides develop individual features step by step:

• Identity and Fingerprinting: identify calculation inputs, distinguish data changes from presentation changes, and associate results with their inputs.
• Persistence and External Resources: write complete packages, retrieve verified payloads, and transfer compressed external Arrays between directories and HDF5.
• Conventional Types: distinguish conventional objects from DCIs, use numerical and collection operations, and understand conversion.
• Custom Conversion Rules: define application-owned Record conventions and recursively convert conventional objects, using dates as an example.
• Command-Line Interface: inspect files, transfer formats, search nested mappings, and compute fingerprints.
• Shorthand Preprocessing: compact Lists, substitutions, and scalar eval(...).

Start with Getting Started for binary persistence, a first conversion example, and the command-line interface. The Reference describes DCT structure and identity; conventional objects follow their own type-specific rules.

2.1 - Identity and Fingerprinting

A calculation’s inputs may appear in a readable text file, a binary archive, and several programming languages. Comparing the files tells you whether their bytes agree. DataCanon lets you ask a different question: do the representations describe the same structured data?

This guide identifies a calculation’s inputs, distinguishes changes in data from changes in presentation, and uses fingerprints to associate results with their inputs. Start with Getting Started for installation and the first persistence workflow.

Identify the inputs to a calculation

Consider an application that evaluates a response at several sampling points:

import numpy as np

from data_canon import as_dc_instance, canonical_encode, fingerprint

inputs = {
    "model": "response-v1",
    "temperature": 300.0,
    "sampling_points": np.array(
        [[0.0, 0.0, 0.0], [0.5, 0.0, 0.0]],
        dtype="float64",
    ),
    ("solver", "tolerance"): 1e-8,
}
instance = as_dc_instance(inputs)
input_id = fingerprint(instance)

The DCI identifies the whole input structure: its keys, values, types, Array shape, and Mapping entry order. The fingerprint is SHA-256 of the canonical encoding, returned as 64 lowercase hexadecimal characters.

Canonical encoding preserves all distinctions in DCI identity. Equal DCIs have identical canonical bytes; unequal DCIs have different canonical bytes. SHA-256 compresses those bytes to a fixed-size identifier. Fingerprint equality is a practical identification mechanism based on collision resistance, not the definition of DCI equality.

Change the presentation without changing the data

Render the input using different layouts, then parse both versions:

from data_canon import parse_shorthand, render_shorthand

readable = render_shorthand(instance)
single_line = render_shorthand(instance, layout="inline")

first = parse_shorthand(readable)
second = parse_shorthand(single_line)

assert first == second == instance
assert canonical_encode(first) == canonical_encode(second)
assert fingerprint(first) == fingerprint(second) == input_id

Whitespace and layout belong to the text representation. They do not change the DCI. The same principle applies when the DCI is stored in HDF5 rather than Shorthand: the physical file bytes can differ while the reconstructed data has the same identity.

Hash a file’s bytes when you need to identify that exact file. Fingerprint its DCI when you need to identify its DataCanon data across representations.

Recognize changes that matter

A change in type can reveal an unintended change in a calculation pipeline:

changed_inputs = dict(inputs)
changed_inputs["sampling_points"] = inputs["sampling_points"].astype("float32")
changed = as_dc_instance(changed_inputs)

assert changed != instance
assert fingerprint(changed) != input_id

All coordinates in this example are exactly representable in both Array types. The type change still changes the DCI. DataCanon retains a distinction that a comparison of numerical values alone could miss.

Other identity rules follow each DCT’s contract:

Choose the DCT that expresses the application’s intended distinctions. If association order should not matter, an UnorderedMapping expresses that choice explicitly. The Reference summarizes all DCT identity rules.

Choose what the identifier covers

An input identifier covers exactly the information placed in its DCI. Adding a run timestamp changes that DCI even when the calculation inputs stay the same. Keep operational metadata separate when it should not affect input identity:

run_record = {
    "input_fingerprint": input_id,
    "inputs": instance,
    "operator_note": "Convergence study, second run",
}
run_instance = as_dc_instance(run_record)

The input DCI remains available unchanged inside the run record. The run record has its own identity, which also covers the note.

Application conventions determine what is required to reproduce a calculation. Include a model version, solver settings, or other dependencies when they affect the result. DataCanon cannot infer missing information or decide that two different parameterizations describe the same physical system. An application can perform its own normalization before constructing a DCI when it wants such equivalences to share an identifier.

Records provide named structures for application conventions. See Custom Conversion Rules for mapping an application’s classes to explicit Record structures.

Associate results with their inputs

A fingerprint is useful as a lookup key for a result cache or provenance record:

results_by_input = {
    input_id: {"response": 12.5, "converged": True},
}
result = results_by_input[fingerprint(instance)]
assert result["converged"]

For a cache to be valid, its key must cover every dependency that can affect the cached result. If a code version or random seed matters, include it in the identified inputs or in a larger cache-key DCI.

Within one Python process, DCIs can also be dictionary keys directly:

point = as_dc_instance(np.array([0.5, 0.0, 0.0], dtype="float64"))
responses = {point: 12.5}
same_point = as_dc_instance(np.array([0.5, 0.0, 0.0], dtype="float64"))
assert responses[same_point] == 12.5

All DCIs are Python-hashable, including Array and List DCIs. Python’s hash() supports dictionary lookup; use fingerprint() for a portable identifier to store or communicate between processes and implementations. Preserve DCI immutability when using a DCI as a key.

External content has a separate integrity check

An ExternalFile or ExternalArray DCI includes a relative path and a declared content digest. Its fingerprint covers that metadata. Constructing the DCI or computing its fingerprint does not retrieve the external bytes.

Materialization verifies the retrieved content against the declared digest. Missing or corrupt content prevents materialization without changing the metadata DCI’s identity. The Persistence and External Resources guide develops that workflow, including complete packages and cross-format transfer.

For the formal encoding and fingerprint contract, see DataCanon Core.

2.2 - Persistence and External Resources

A scientific dataset often combines a small description with a large numerical Array or an existing file. DataCanon can store the data together or preserve an explicit reference to external content. The DCI retains its identity as a complete dataset moves between a directory and an HDF5 container.

This guide constructs a dataset with an external Array, writes a complete package, retrieves its numerical contents, and transfers the package. The Identity and Fingerprinting guide explains what its identifier covers.

Decide what belongs inside the DCI

An Array DCI contains its numerical elements. An ExternalArray DCI contains the information needed to identify and interpret an external element stream: a relative path, portable Array type, shape, and SHA-256 content digest.

Array and ExternalArray are distinct DCTs. Converting an Array into an external reference changes the DCI. Moving an existing ExternalArray’s payload between supported storage representations can preserve its identity.

Construct a dataset with an external Array

Force-constant matrices provide a useful example: a calculation may produce a large matrix with many zeros, accompanied by relatively small descriptive data. Use a small matrix here to keep the example easy to run:

import numpy as np

from data_canon import ExternalArray, as_dc_instance, fingerprint

forces = np.zeros((32, 32), dtype="float64")
forces[0, 0] = 2.5
forces[0, 1] = -0.0

dataset = as_dc_instance({
    "description": "Force-constant example",
    "force_constants": ExternalArray(
        forces,
        filename="forces",
        compression=True,
    ),
})
dataset_id = fingerprint(dataset)

The conventional ExternalArray object supplies the numerical data during conversion. The resulting ExternalArray DCI records the path ("external_dc", "forces.bin.gz"), type float64, shape (32, 32), and content digest. The implementation retains the supplied data as a nonsemantic cache so the writer can persist it.

The content digest covers the uncompressed canonical Array element bytes. Those bytes contain only elements: no type, shape, or DCI header. A .bin or .bin.gz payload is therefore different from a .dcb file, which contains one complete canonical DCI encoding.

Write a complete directory package

from pathlib import Path
from data_canon import read, write

source = Path("force_example") / "dataset.dcs"
write(dataset, filename=source)

The writer creates the Shorthand representation at force_example/dataset.dcs and the compressed payload at force_example/external_dc/forces.bin.gz. Together, the representation and all referenced payloads form a complete package.

Use dataset.dcb instead to store the DCI as canonical binary bytes. Both directory-backed formats resolve relative paths beneath the representation’s containing directory by default.

The current Python write() verifies and persists required resources. It does not expose a metadata-only export option. Missing resources or conflicting contents can cause a write to fail. An existing external file with conflicting contents is not silently overwritten.

Read metadata before retrieving content

restored = read(source)
assert restored == dataset
assert fingerprint(restored) == dataset_id

Reading reconstructs the DCI hierarchy without materializing external payloads. The dataset’s metadata can be read even if the payload is unavailable. The reader retains the representation’s source location as nonsemantic context for later retrieval.

Converting the restored dataset to conventional objects requests the external Array’s numerical contents:

from data_canon import dc_instance_to_python

conventional = dc_instance_to_python(restored)
loaded_forces = conventional["force_constants"]
assert np.array_equal(loaded_forces, forces)
assert np.signbit(loaded_forces[0, 1])

Materialization decompresses the payload, verifies its digest, and validates its type, shape, required byte length, and element validity. It returns a NumPy array; the original DCI remains an ExternalArray DCI.

A missing file, malformed compressed stream, incorrect digest, or invalid element stream causes retrieval to fail. The metadata DCI and its fingerprint remain unchanged.

Transfer the package to HDF5

container = Path("force_example") / "dataset.h5"
write(restored, filename=container)

embedded = read(container)
assert embedded == dataset
assert fingerprint(embedded) == dataset_id

The HDF5 writer retrieves and verifies the source payload before embedding it. The container holds the DCI hierarchy and an embedded payload collection, so the complete package can travel as one file. External references retain their paths and digests; they are not converted into Array DCIs.

Reading the HDF5 hierarchy still does not materialize its external Arrays. Subsequent conversion retrieves the embedded payload by its complete relative path and verifies it.

Export the package back to a directory:

destination = Path("force_export") / "dataset.dcb"
write(embedded, filename=destination)

assert (destination.parent / "external_dc" / "forces.bin.gz").exists()
assert read(destination) == dataset

The writer recreates the compressed external file using the same relative path. The Shorthand, HDF5, and binary representations recover equal DCIs and therefore share a fingerprint. Their physical file bytes need not agree.

Understand compression and identity

ExternalArray(..., compression=True) selects .bin.gz storage. With compression=False, the default, a filename stem receives .bin instead. Explicit .bin and .bin.gz suffixes must match the compression setting.

In a directory, .bin.gz stores a gzip stream. In HDF5, the same reference uses a gzip-compressed dataset containing the logical element bytes. The HDF5 library handles dataset decompression; the implementation does not interpret those returned bytes as another gzip file.

Changing forces.bin to forces.bin.gz changes the path and therefore changes DCI identity. Once the path is chosen, transferring that reference between directory and HDF5 storage preserves it. Compression does not change the content digest’s definition.

The option applies to ExternalArrays. It does not automatically compress every Array or ExternalFile, or introduce support for .dcb.gz representations.

Keep opaque files opaque

Use an ExternalFile when the payload is a file whose interpretation belongs to the application:

from data_canon import ExternalFile

notes = as_dc_instance(ExternalFile.from_bytes(
    b"Calibration instrument: A17\n",
    filename="calibration.txt",
))
write(notes, filename=Path("force_example") / "calibration.dcb")

An ExternalFile digest covers the file’s exact bytes. A .gz filename does not cause DataCanon to reinterpret that digest as a digest of decompressed content. The Python conversion interface can apply a filename-based content loader after verification; for example, .txt content becomes text. Such interpretation is separate from the external DCI’s identity.

Move resources without changing references

Move a directory package with its relative resource tree intact. Moving only dataset.dcs or dataset.dcb may leave a valid metadata representation whose payloads cannot be found in the new directory.

If a DCI was decoded without a source location, supply the namespace explicitly when converting it:

from data_canon import canonical_decode, canonical_encode

detached = canonical_decode(canonical_encode(dataset))
recovered = dc_instance_to_python(detached, source=source.parent)
assert np.array_equal(recovered["force_constants"], forces)

The source selects where to look. The DCI still determines which bytes are acceptable through its digest and Array metadata. Source context never changes equality or fingerprinting.

Within a complete package, references to the same path must agree on one shared payload. Different paths remain distinct locations, even when their payloads agree. Writers reject conflicting references rather than rename paths or select one payload silently.

For method signatures, see the Python API. The Persistence specification and HDF5 specification define the portable requirements for independent implementations.

2.3 - Conventional Types

DataCanon implementations connect DataCanon Instances (DCIs) with instances of conventional types in a programming language. Conventional types include built-in types, types provided by libraries, and user-defined types used directly for data and computation. Python examples include list, fractions.Fraction, and numpy.ndarray.

A DataCanon Type (DCT) defines information, invariants, and equality. A DCI is an instance of a DCT. The Python implementation represents DCIs using classes such as ListDC and FractionDC. A ListDC object and a conventional list object are both Python objects, but belong to different types with different rules. Conversion connects the types; it does not make their semantics identical.

The examples use Python. Implementations in Julia, Rust, C, or another language can define conversions involving conventional types from their own language and libraries. The DataCanon DCTs and DCI equality rules remain independent of those choices.

The optional Conventional Types specification defines common construction, arithmetic, comparison, and collection requirements for independent implementations. Support is declared per conventional type; Core conformance does not require conventional arithmetic. The guide introduces the Python binding, including its interfaces and conversion choices.

Distinguish DCTs, their implementations, and conventional types

The table identifies conversion relationships, not type equivalences or a universal one-to-one correspondence. For example, both built-in complex and ComplexNumber objects can be converted to Complex DCIs, but their component information differs. The Python API specifies the conversion directions and resulting types.

Exact named mathematical constants

The conventional Constant type preserves a named mathematical constant. Pi and E are instances representing pi and Euler’s number; neither stores a decimal approximation. Conventional arithmetic retains the constant:

import math
from data_canon import Constant, Pi, E, Sum, Product, as_dc_instance, render_shorthand

assert Pi == Constant("pi")
assert Sum(Pi, Pi) == Product(2, Pi)
assert Pi / Pi == 1
assert Pi != math.pi
assert float(E) == math.e

instance = as_dc_instance(2 * Pi)
assert render_shorthand(instance, padding=0) == "2 * Pi"

float(Pi) explicitly requests an approximation. The corresponding Constant DCI preserves its name, and its shorthand spelling is Pi (E for Euler’s number). Longhand uses constant("pi") or constant("e"). Physical constants and user-defined names are not included.

Begin with a List DCI and a conventional list

Construct a List DCI from a conventional list, then convert the DCI to a new list object:

from data_canon import as_dc_instance, dc_instance_to_python
from data_canon.instance import ListDC

source = [1, 2, 3]
instance = as_dc_instance(source)
converted = dc_instance_to_python(instance)

assert isinstance(instance, ListDC)
assert type(converted) is list
assert converted == source

Both instance and converted are Python objects. The function constructs an instance of the conventional list type from the List DCI represented by instance.

The difference affects equality:

assert [1] == [1.0]
assert as_dc_instance([1]) != as_dc_instance([1.0])

Conventional lists use the equality rules of their members. List DCIs compare the identities of their constituent DCIs, so Integer and Float constituents remain distinct. Conversion must be understood in terms of both types’ rules.

Use existing conventional types

Most scientific data already has useful conventional types in Python. Use built-in scalars and containers, Fraction for rational numbers, and NumPy arrays for machine data:

from fractions import Fraction
import numpy as np
from data_canon.instance import MappingDC

measurements = {
    ("Eg", 0): 2.37,
    "position": (Fraction(1, 2), Fraction(1, 4), 0.199),
    "lattice_vectors": np.eye(3, dtype="float64"),
}
measurement_instance = as_dc_instance(measurements)
recovered = dc_instance_to_python(measurement_instance)

assert isinstance(measurement_instance, MappingDC)
assert type(recovered) is dict
assert recovered[("Eg", 0)] == 2.37
assert type(recovered["position"]) is tuple
assert isinstance(recovered["position"][0], Fraction)
assert isinstance(recovered["lattice_vectors"], np.ndarray)

The Tuple key is a DCI within the Mapping DCI and a conventional tuple within the recovered dictionary. Array type and shape are preserved by the specified Array conversion. The returned NumPy array is read-only; .copy() creates an independently mutable array. Mutable buffers or unsupported numerical types may require explicit conversion before they can be represented by a DCI.

as_dc_instance() accepts an existing DCI too, returning that instance unchanged. The identity case does not eliminate the distinction between a DCI and a conventional object of another type.

NumPy scalar values

NumPy scalars convert by value to the corresponding scalar DCTs:

For example:

from data_canon.instance import FloatDC, IntegerDC

assert as_dc_instance(np.uint64(2**64 - 1)) == IntegerDC(2**64 - 1)
assert as_dc_instance(np.float32(0.5)) == FloatDC(0.5)

Binary16 and binary32 values widen exactly to binary64. Scalar conversions do not preserve the original NumPy width as identity metadata. For a numerical element whose type must be preserved, use a rank-zero Array:

from data_canon.instance import ArrayDC

scalar = as_dc_instance(np.float32(0.5))
typed_scalar = as_dc_instance(np.array(0.5, dtype="float32"))
assert isinstance(scalar, FloatDC)
assert isinstance(typed_scalar, ArrayDC)
assert typed_scalar.data.shape == ()
assert typed_scalar.data.dtype == np.dtype("float32")

NaN and infinities remain invalid numerical DCIs. Extended-precision scalar formats, such as a platform’s wider longdouble or clongdouble, require an explicit application conversion or custom registry rule. A platform alias of a supported portable dtype follows that portable dtype’s rule.

NumPy datetime64, timedelta64, and structured or opaque void scalars have no default conversion. Their units or field structure must be handled explicitly, for example by a registered conversion to a Record. A duration never implicitly becomes a unitless Integer. Exact-type registry rules take precedence over the default NumPy conversions and apply recursively in collections and Mapping keys.

Supply conventional types where suitable ones are missing

A language may lack a conventional type for a structure that DataCanon can represent. The Python implementation supplies Power, Sum, Product, Quotient, ComplexNumber, Multiset, and UnorderedDict for such cases. The additional classes are conventional types, just like types provided by other libraries; they are distinct from PowerDC, SumDC, and the other classes implementing DCTs.

DataCanon defines structured data and its identity; it does not define mathematical operations on that data. Working with structured numbers naturally creates a need to calculate with them. The additional numerical conventional types therefore support arithmetic under their own construction, equality, and simplification rules, similar in spirit to Python’s Fraction.

The arithmetic examples below construct conventional objects directly. No Shorthand parser, DCI construction, or persistence operation is needed to use Power, Sum, or the other numerical conventional types. Arithmetic operations on those objects do not evaluate or simplify stored DCIs.

Arithmetic with structured numbers

Retain a radical exactly

Construct a conventional Power object with a base and a positive rational exponent:

from data_canon import Power

sqrt_two = Power(2, 1, 2)
sqrt_three = Power(3, 1, 2)
cube_root_five = Power(5, 1, 3)

Power(3, 1, 2) represents three raised to the power one half. Its base remains a conventional int object here. Omitting the denominator gives an integer exponent: Power(3, 2) means three squared.

Conventional Power reduces the exponent fraction and extracts an exact root of the base when that makes the exponent an integer. Construction retains the Power type and the remaining integer exponent:

assert Power(5, 2, 4) == Power(5, 1, 2)
root_four = Power(4, 1, 2)
assert type(root_four) is Power
assert (root_four.base, root_four.numerator, root_four.denominator) == (2, 1, 1)
assert root_four == 2
assert Power(8, 1, 3) == 2

cubed = Power(16, 3, 2)
assert (cubed.base, cubed.numerator, cubed.denominator) == (4, 3, 1)

integer_power = Power(4, 8)
assert (integer_power.base, integer_power.numerator, integer_power.denominator) == (4, 8, 1)
assert integer_power == 65536
assert int(integer_power) == 65536

An integer exponent is preserved, rather than expanded merely to initialize the object. Equality and explicit int() or float() conversion can still compute its value. If the requested root is not exact, the constructor retains the base and reduced exponent: Power(8, 1, 2) keeps those fields.

Float bases follow the same rule, with an exactness check on the replacement float. For example, Power(4.0, 1, 2) becomes Power(2.0, 1, 1) and Power(16.0, 3, 2) becomes Power(4.0, 3, 1). Integer powers such as Power(0.1, 2) remain powers; construction introduces no floating-point rounding.

The behavior follows the model of Fraction: construction retains the requested conventional type, while equality can identify an equivalent scalar. A Power DCI has different rules and retains its original base and exponent constituents.

Calculate with exact geometric quantities

For an equilateral triangle of side length one, retain the exact height and area using conventional Fraction and Power objects:

from fractions import Fraction

height = Fraction(1, 2) * sqrt_three
area = height / 2

assert area == Fraction(1, 4) * sqrt_three
assert height + height == sqrt_three

The multiplication produces a conventional Product object. Division by a conventional int adjusts the rational factor; adding the two identical heights combines their factors. No decimal approximation of the square root is needed.

Add terms and combine matching contributions

Python operators construct conventional Sum and Product objects:

from data_canon import Sum

value = sqrt_two + sqrt_three + 3 * sqrt_three
assert value == Sum(sqrt_two, 4 * sqrt_three)
assert sqrt_three - sqrt_three == 0

Conventional Sum combines matching terms and rational constants, and flattens nested Sum objects. Extractable radical factors are simplified before terms are collected:

assert Sum(Power(12, 1, 2), sqrt_three).items == (3 * sqrt_three,)
assert Sum(1, Power(4, 1, 2)).items == (3,)

Explicit constructors are useful when assembling a collection of terms:

terms = [sqrt_two, sqrt_three, 3 * sqrt_three]
assembled = Sum(*terms)
assert assembled == value

The constructor retains a conventional Sum object even if simplification leaves one term. Arithmetic operators may return that remaining term directly:

assert Sum(sqrt_three, 3 * sqrt_three) == Sum(4 * sqrt_three)
assert sqrt_three + 3 * sqrt_three == 4 * sqrt_three

Sum(*items) and Product(*items) take separate operands, rather than one iterable argument. Conventional Sum and Product objects are immutable and unhashable, independently of the hashable Sum and Product DCIs.

Multiply factors and combine powers with a common base

Conventional Product combines rational factors and combines exponents for matching bases:

from data_canon import Product

assert Product(Fraction(1, 2), Fraction(2, 3)) == Product(Fraction(1, 3))
assert Power(5, 1, 2) * Power(5, 1, 3) == Power(5, 5, 6)
assert sqrt_two * sqrt_three == Product(sqrt_two, sqrt_three)

The last product can retain its two factors while comparing equal to the square root of six. Surrounding arithmetic can evaluate an integer power and combine the result with other terms or factors. Retaining the exponent in a standalone Power does not prevent that simplification:

assert sqrt_two * sqrt_three == Power(6, 1, 2)
assert sqrt_three * sqrt_three == 3
assert Product(3, Power(4, 1, 2)).items == (6,)

Multiplication distributes over conventional Sum objects:

assert 2 * Sum(sqrt_two, sqrt_three) == Sum(2 * sqrt_two, 2 * sqrt_three)

The simplification rules support exact structured calculations while keeping the scope of simplification limited and predictable.

Divide by a structured denominator

A reciprocal of a sum needs more than a conventional Fraction. Quotient retains a structured numerator and denominator:

from data_canon import Quotient

denominator = sqrt_three + sqrt_two
reciprocal = 1 / denominator
assert reciprocal == Quotient(1, Sum(sqrt_three, sqrt_two))

The denominator remains a conventional Sum; construction does not rationalize it. Equality can nevertheless recognize the rationalized equivalent by exact cross multiplication:

assert reciprocal == sqrt_three - sqrt_two
assert Quotient(4, 6) == Fraction(2, 3)
assert Quotient(2 * sqrt_three, 4 * sqrt_three) == Fraction(1, 2)

The Quotient constructor reduces rational coefficients and cancels common factors. It retains the Quotient type even when the result equals a scalar. Division rejects denominators recognized as zero.

Division by a conventional int or Fraction adjusts exact rational factors, as in the triangle example. Division by a conventional float follows floating-point arithmetic.

Keep floating-point contributions separate

Exact rational and floating-point contributions retain their separate roles:

mixed = Sum(Fraction(1, 4), 0.125)
assert mixed.items == (Fraction(1, 4), 0.125)

with_offset = Sum(sqrt_three, 0.25, 0.5)
assert with_offset == Sum(sqrt_three, 0.75)

Conventional float contributions can combine with one another using Python floating-point arithmetic, with its usual rounding. Float contributions remain separate from exact rational coefficients and are not converted to large rational numbers. The numerical conventional types accept finite float values and reject bool values as numerical operands.

Compare numerical values exactly

Conventional numerical equality uses exact normalization and cross multiplication. Equality can hold across Sum, Product, Power, Quotient, int, Fraction, and float; it is independent of presentation order and does not require matching conventional types.

Float comparison uses the exact binary value already stored by the float, following Python’s Fraction behavior:

assert Quotient(1, 2) == 0.5
assert Quotient(1, 10) != 0.1
assert Sum(2**53 + 1) != float(2**53 + 1)
assert mixed == Fraction(3, 8)

Keeping float and rational contributions separate during construction does not require them to compare unequal. No rounded floating-point evaluation is used as an equality test. Exact comparison forms are cached when needed without changing the object’s displayed operands or its conversion to a DCI.

Normalization handles rational arithmetic, distributive expressions, rational radicals, and quotient cross multiplication. It is not a complete procedure for recognizing every mathematical equivalence. Fractional powers with nonrational bases remain symbolic. Integer factorization uses trial divisors through 1000 and extracts residual perfect powers through degree 64; a large unresolved factor stays intact. Comparisons can therefore miss equivalences requiring further factorization or algebraic identities. No numerical tolerance or floating-point fallback is used to guess equality.

Choose when to request a machine number

Use float() at the point where a numerical routine or display needs a machine floating-point result:

import math

assert math.isclose(float(area), math.sqrt(3) / 4)
assert math.isclose(float(reciprocal), 1 / (math.sqrt(3) + math.sqrt(2)))

The conventional numerical object remains unchanged. Fractional powers use the real-valued domain and rounding behavior of math.pow during explicit evaluation.

int() also evaluates, then truncates toward zero, following Python conventions:

assert float(Sum(1, 1.0)) == 2.0
assert int(Quotient(-7, 3)) == -2

Explicit numerical conversions can raise errors for undefined operations or overflow. Construction rejects an even root of a known negative scalar; odd roots of negative scalars use the real branch. A symbolic expression with a more complicated base may require further evaluation to determine its domain.

Use structured complex components

Use conventional ComplexNumber when real or imaginary components should retain Fraction, Power, Sum, or other conventional numerical objects:

from data_canon import ComplexNumber

phase = ComplexNumber(Fraction(1, 2), Fraction(1, 2) * sqrt_three)
shifted = phase + 1
assert shifted.real == Fraction(3, 2)
assert shifted.imaginary == phase.imaginary

Component equality follows the same numerical rules. For example, ComplexNumber(Power(4, 1, 2), 0) compares equal to 2. Comparisons with built-in complex values compare the actual component values. Conventional ComplexNumber objects are unhashable.

Addition, subtraction, multiplication, division, negation, and conjugation are available. For example, multiplication by the imaginary unit rotates the components:

rotated = phase * ComplexNumber(0, 1)
assert rotated.real == -phase.imaginary
assert rotated.imaginary == phase.real
assert phase.conjugate() == ComplexNumber(phase.real, -phase.imaginary)

Request a built-in Python complex number explicitly when needed:

machine_phase = complex(phase)
assert math.isclose(machine_phase.real, 0.5)
assert math.isclose(machine_phase.imag, math.sqrt(3) / 2)

Component operations follow their underlying Python types. In particular, division of two conventional int values or float operations can introduce floating-point results; use Fraction or structured numerical operands when exact rational division is required.

Array expressions and ordered multiplication

The conventional Sum class supports scalar terms and Array expressions. The conventional OrderedProduct class preserves factor order, while Product remains a scalar type. Construction preserves the expression; .evaluate() performs conventional arithmetic. For example, combine the Pauli matrices:

import numpy as np
from data_canon import Sum, OrderedProduct, Power, as_dc_instance

sigma_x = np.array([[0, 1], [1, 0]])
sigma_y = np.array([[0, -1j], [1j, 0]])
operator = Sum(OrderedProduct(2, sigma_x), OrderedProduct(3, sigma_y))
np.testing.assert_array_equal(np.asarray(operator, dtype=complex),
                              2 * sigma_x + 3 * sigma_y)
instance = as_dc_instance(operator)

Factor order matters in conventional matrix multiplication as well as DCI identity. OrderedProduct(A, B) and OrderedProduct(B, A) can give different results. Scalar factors may occur anywhere; NumPy array factors currently must have rank two when an OrderedProduct is evaluated. Conventional Sum supports NumPy arrays of any rank with exactly matching shapes. Neither operation broadcasts or reshapes data. Nonzero scalar terms mixed into an Array Sum require an explicit identity Array.

Exact coefficients are retained in object-array entries:

exact = OrderedProduct(Power(2, 1, 2), np.eye(2, dtype=int)).evaluate()
assert exact.dtype == object
assert exact[0, 0] == Power(2, 1, 2)

from fractions import Fraction
exact_y = OrderedProduct(Fraction(1, 3), sigma_y).evaluate()
assert exact_y[1, 0].imaginary == Fraction(1, 3)

Use an explicit dtype when a machine approximation is wanted. An exact object Array cannot itself be encoded as a portable Array DCI; use a supported machine dtype or nested Lists to preserve exact per-element DCTs. Scalar-valued fields such as Power bases and Complex components do not accept Array-valued Sums.

Additional conventional collection types

Retain repeated collection members

A conventional Multiset preserves the number of occurrences without assigning semantic order. For example, it can describe a composition:

from data_canon import Multiset

composition = Multiset(["Ti", "O", "O"])
assert composition == Multiset(["O", "Ti", "O"])
assert composition != Multiset(["Ti", "O"])

composition.add("O")
assert composition.count("O") == 3

Conventional Multiset is mutable. It accepts an iterable and supports list-like operations. Its members need not be hashable:

samples = Multiset([[1, 2], [1, 2]])
assert samples == Multiset([[1, 2], [1, 2]])

Member comparison follows Python equality, so members must provide usable Boolean equality results. This differs from DCI equality; for example, Python Boolean and integer equality can identify values that DataCanon distinguishes.

Mark an unordered dictionary

Python dict already compares equal dictionaries independently of insertion order. Its insertion order is nevertheless retained when converting it to a DataCanon Mapping. Use UnorderedDict to request an UnorderedMapping instead:

from data_canon import UnorderedDict

settings = UnorderedDict({"temperature": 300, "pressure": 0})
settings["temperature"] = 350
assert settings["temperature"] == 350

Conventional UnorderedDict behaves as a Python dictionary, with the same key hashability and equality rules. Its additional role is to express the intended DataCanon collection type during conversion.

Construct DCIs from the additional conventional objects

The arithmetic and collection examples above created instances of conventional types. Use the conversion interface when their results need DataCanon identity:

from data_canon.instance import ProductDC, MultisetDC, UnorderedMappingDC

area_instance = as_dc_instance(area)
assert isinstance(area_instance, ProductDC)
assert dc_instance_to_python(area_instance) == area
assert isinstance(as_dc_instance(composition), MultisetDC)
assert isinstance(as_dc_instance(settings), UnorderedMappingDC)

A conventional Power object follows the arithmetic implementation’s rules. A Power DCI retains its base DCI and prescribed exponent fields. Likewise, conventional Sum objects simplify matching terms, while Sum DCIs retain the specified numerical structure and multiplicity.

The conventional objects can be placed in a dictionary with Tuple keys and converted together:

from data_canon import render_shorthand

results = {
    ("triangle", "height"): height,
    ("triangle", "area"): area,
    "phase": phase,
    "composition": composition,
    "settings": settings,
}
result_instance = as_dc_instance(results)
print(render_shorthand(result_instance))
assert render_shorthand(results) == render_shorthand(result_instance)

render_shorthand() accepts conventional objects by constructing a DCI through the same conversion interface before rendering. The DCI’s identity is governed by its DCT, rather than by the original object’s equality rules.

Preserve the distinction during conversion

Conventional types have their own construction, equality, and operation rules. Conversion can simplify structure, lose distinctions, or fail when a target type cannot hold the required information. A round trip through conventional objects is not guaranteed to preserve an arbitrary DCI’s identity.

For example, converting a Sum DCI invokes the conventional Sum constructor:

from data_canon import parse_shorthand, Sum

original = parse_shorthand("sum(1, 1)")
calculated_sum = dc_instance_to_python(original)
assert calculated_sum == Sum(2)
assert as_dc_instance(calculated_sum) != original

No original DCI is modified by the conversion. The newly constructed Sum DCI has the structure of the conventional result, rather than that of original. Python’s Fraction reduces ratios, and conventional Power reduces its rational exponent; their conversion rules can also change DCI structure.

Collection conversion has similar consequences:

first = {"temperature": 300, "pressure": 0}
second = {"pressure": 0, "temperature": 300}

assert first == second
assert as_dc_instance(first) != as_dc_instance(second)
assert as_dc_instance(UnorderedDict(first)) == as_dc_instance(UnorderedDict(second))

Python dictionary equality ignores insertion order, while Mapping DCI equality includes entry order. UnorderedDict requests an UnorderedMapping DCI instead. A conventional Multiset uses its members’ conventional equality rules, which need not coincide with DCI equality.

All DCIs in the reference implementation can be used as Python dictionary keys. Converting a DCI key to a conventional object can nevertheless produce an unhashable list or NumPy array. Distinct Boolean, Integer, or Float DCI keys can also collide under conventional dictionary-key equality. Conversion to a conventional dictionary is therefore not always a suitable way to retain a DataCanon Mapping. See the Python API for conversion details.

Store and recover the results

Writers accept conventional objects through as_dc_instance(). Readers reconstruct DCIs from the stored representation:

from data_canon import read, write

write(results, filename="exact_results.dcb")
restored = read("exact_results.dcb")
assert restored == result_instance

recovered_results = dc_instance_to_python(restored)
assert recovered_results[("triangle", "area")] == area

The storage round trip preserves DCI identity. The subsequent conversion from the Mapping DCI to a conventional dict is a separate operation, subject to the target types’ rules. Keep the DCI when its complete structural identity is needed; use conventional objects when their collection or arithmetic operations are wanted.

ExternalFile and ExternalArray are conventional types for constructing external references. Their reverse conversions are different operations: dc_instance_to_python() retrieves and verifies an ExternalFile payload, returning bytes or a content loader’s result, and materializes an ExternalArray as a NumPy array. The conversion does not return another ExternalFile or ExternalArray construction object. Metadata-only work can use the external DCIs without retrieving the payloads.

For constructor signatures, complete conversion tables, and resource handling, use the Python API. For numerical DCT structure, nesting, and identity, use the Reference and Core specification. For the language-independent conventional behavior and its required comparison cases, use the Conventional Types specification.

Application-defined Record conventions

An application can connect its conventional types to ordinary Records through an explicit ConversionRegistry. For example, a conventional date can become a Record holding year, month, and day fields, while another convention might also store calendar metadata. DC preserves that structure without choosing a calendar or defining date operations.

Rules apply recursively through supported collections and Record fields, using as_dc_instance(obj, registry=registry) and dc_instance_to_python(node, registry=registry). Reading a stored DCI does not execute those rules. The Custom Conversion Rules guide works through a complete date example and explains validation and defaults.

Compressed external Arrays

Use ExternalArray(array, filename="forces", compression=True) to select gzip storage for zero-heavy numerical data such as force constants. The conventional constructor finalizes the name as forces.bin.gz; compression defaults to off, which produces forces.bin. Explicit suffixes must agree with the setting.

Convert to a DCI before writing. Directory-backed packages contain the gzip file, while HDF5 packages apply dataset compression to the logical element bytes. The digest covers uncompressed canonical bytes in both representations. The DCI’s finalized path and identity are preserved through transfers. A name ending in .bin identifies a different DCI from the corresponding .bin.gz name. ExternalFiles retain opaque-byte semantics.

See the Python API for filename examples, validation, and compression behavior.

2.4 - Shorthand Preprocessing

Overview

Shorthand preprocessing provides three conveniences for entering scientific data by hand: compact List notation, named textual substitutions, and scalar mathematical evaluation with eval(...).

Preprocessing transforms text into strict Shorthand. Parsing that text then constructs a DataCanon Instance (DCI). This guide introduces each feature separately before showing how they work together.

Start with one compact List

Ordinary Shorthand Lists use brackets and commas:

[1, 2, 3]

Compact notation lets you enter the same List using angle brackets and spaces:

<1 2 3>

Use preprocess_shorthand() to expand it:

from data_canon import preprocess_shorthand

source = "<1 2 3>"
prepared = preprocess_shorthand(source)
print(prepared)

The resulting text is:

[1, 2, 3]

No DataCanon Instance has been constructed yet. prepared is a Python String. Parse that String to construct the List DCI:

from data_canon import parse_shorthand

instance = parse_shorthand(prepared)

Preprocessing and strict parsing are separate steps: preprocessing expands input notation; strict parsing interprets the resulting Shorthand and validates the data.

A compact group always produces a List, even when it contains only one item:

<6>  -> [6]

Empty compact groups are invalid. Use ordinary [] for an empty List.

Enter nested Lists

A comma can separate rows whose items are separated by spaces:

<1 2 3, 4 5 6>

expands to:

[
  [1, 2, 3],
  [4, 5, 6]
]

A semicolon can introduce another level:

<1 2, 3 4; 5 6, 7 8>

expands to:

[
  [
    [1, 2],
    [3, 4]
  ],
  [
    [5, 6],
    [7, 8]
  ]
]

At each level, the preprocessor looks for separators in this order:

1. Newline.
2. Semicolon.
3. Comma.
4. Horizontal whitespace: space, tab, or carriage return.

The first separator that splits the contents into multiple nonempty parts establishes the outer level. Each part is then processed in the same way. Newlines therefore introduce a level outside semicolons, commas, and spaces. Explicitly nested angle-bracket groups are also supported.

The layouts above illustrate the resulting structure; exact output spacing is not important to strict parsing. Avoid repeated or trailing separators while their treatment remains a Version 1 draft finalization item.

Existing structures remain intact

Separators inside quoted Strings or balanced (), [], and {} do not split items. For example:

<[1,2] 3>

becomes:

[[1,2], 3]

The inner [1,2] is one item. Likewise,

<"Sr Ti O" 3>

becomes:

["Sr Ti O", 3]

The spaces inside the String do not introduce List items.

Construct an Array when needed

Compact notation alone creates Lists. To enter an Array, place it inside an Array expression:

source = 'array(<1.0 0.0, 0.0 1.0>, dtype="float64")'
prepared = preprocess_shorthand(source)
instance = parse_shorthand(prepared)

The prepared text represents:

array([[1.0, 0.0], [0.0, 1.0]], dtype="float64")

The strict Array parser constructs the Array. Compact notation supplies its numerical body without itself performing Array inference.

Substitute one name

A substitution associates a name with replacement text. Define the name before |, then use it in the document body:

species="Sr" | {"species": species}

Preprocessing produces:

{"species": "Sr"}

In Python:

source = 'species="Sr" | {"species": species}'
prepared = preprocess_shorthand(source)
instance = parse_shorthand(prepared)

The name species is an input convenience. After substitution, the parser receives an ordinary Shorthand Mapping.

Define several names

Separate definitions with whitespace. Whitespace around = is permitted:

species = "Sr"
position = [0.5, 0.5, 0.5] |
{
  "species": species,
  "position": position
}

becomes:

{
  "species": "Sr",
  "position": [0.5, 0.5, 0.5]
}

Definitions can contain Strings, Tuples, Arrays, Mappings, or other text:

label = "oxygen"
site = (1/2, 1/2, 0)
lattice = array([[1.0, 0.0], [0.0, 1.0]], dtype="float64") |
{
  "label": label,
  "site": site,
  "lattice": lattice
}

Replacement text is inserted literally. It is not separately parsed as a DCI; only the final document must be valid strict Shorthand. Replacement text can therefore also be an expression fragment used later inside eval(...).

Name rules and protected Strings

A name begins with an ASCII letter or underscore, followed by ASCII letters, digits, or underscores. Names are case-sensitive and must be unique.

Substitution replaces complete identifier-like tokens. A definition for a does not replace part of alpha or a2, and nothing is substituted inside quoted Strings:

a=3 | {"value": a, "text": "a"}

becomes:

{"value": 3, "text": "a"}

Build on earlier definitions

Definitions are processed from left to right. Each definition may use externally supplied names and any earlier local definitions:

a        = 4.593
c_over_a = 0.644
c        = a * c_over_a |
[c]

This produces [4.593 * 0.644], which represents a List containing a Product. To compute the lattice parameter for a machine Array, request evaluation:

source = """a=4.593 c_over_a=0.644 c=eval(a * c_over_a) |
array(<a 0 0, 0 a 0, 0 0 c>)"""
instance = parse_shorthand(source, preprocess=True)

The third diagonal entry is 2.957892. An evaluated definition is resolved before later definitions use it, so d=eval(c * 2) would also work. Self-references and references to later definitions are rejected; arrange parameters before the definitions that depend on them.

Replacements remain literal text. For example, a=1 + 2 b=3 * a | b becomes 3 * 1 + 2, without implicit grouping. Use constructors when a nested numerical structure matters: a=sum(1, 2) b=product(3, a) | b preserves the Sum as one Product constituent. Ordinary Shorthand parentheses create Tuples, not groups. Inside eval(...), arithmetic parentheses provide ordinary grouping.

Readable input files with named parameters

Named parameters make an input file easier to read and revise: repeated values have one definition, and relationships remain visible in the data.

A crystal structure provides a useful example. Atomic positions are often specified as fractions of the repeating cell, with several positions sharing one displacement parameter. Define delta once and write each coordinate in terms of its reference position and displacement:

from data_canon import parse_shorthand, preprocess_shorthand, render_shorthand

source = """
delta = 0.199 |
{
  "basis_atoms": [
    (delta,         1/4 + delta,  1/8),
    (1/2 + -delta,  3/4 + -delta, 5/8),
    (-delta,        3/4 + delta,  3/8),
    (1/2 + delta,   1/4 + -delta, 7/8)
  ]
}
"""

prepared = preprocess_shorthand(source)
instance = parse_shorthand(prepared)
print(render_shorthand(instance, layout="standard", align="left"))

After substitution, the strict parser receives the following structure (whitespace may differ):

{
  "basis_atoms": [
    (0.199,         1/4 + 0.199,  1/8),
    (1/2 + -0.199,  3/4 + -0.199, 5/8),
    (-0.199,        3/4 + 0.199,  3/8),
    (1/2 + 0.199,   1/4 + -0.199, 7/8)
  ]
}

The List contains four Tuple DCIs. Coordinates such as 1/4 + 0.199 are Sum DCIs whose constituents retain the exact Fraction reference position and the Float displacement separately. The resulting DCI no longer contains the substitution name; the source text remains useful for editing the parameter.

The ASCII name delta is used because substitution names currently follow the ASCII identifier rules. With the positive value defined here, -delta becomes the negative Float literal -0.199 by textual substitution. This example needs neither Compact List notation nor eval(...).

Supply substitutions from Python

Instead of placing definitions in the input, supply a mapping of names to replacement Strings:

prepared = preprocess_shorthand(
    '{"species": species, "position": position}',
    substitutions={
        "species": '"Sr"',
        "position": "[0.5, 0.5, 0.5]",
    },
)
instance = parse_shorthand(prepared)

The result is the same Mapping as in the inline example. The values in substitutions must be text. The quotation marks in '"Sr"' are intentional: they make the replacement a Shorthand String literal.

External substitutions may also be used in inline definitions:

prepared = preprocess_shorthand(
    'site=(x, x, 0) | {"position": site}',
    substitutions={"x": "1/2"},
)

This produces:

{"position": (1/2, 1/2, 0)}

Each inline definition expands external names and earlier local names in one pass, then resolves its explicit eval(...) expressions. After definitions are resolved, external names are substituted once in the body, followed by local names once. The external mapping itself is not recursively expanded. An inline name must not conflict with an externally supplied name.

Array presets and linear combinations

Substitutions can name Arrays, making input files concise without changing DCI identity. For example, the Pauli matrices provide a small linear-operator example:

Sigma_x = array([[0, 1], [1, 0]])
Sigma_y = array([[0, 0-1j], [0+1j, 0]]) |
{
  "operator": 2 * Sigma_x + 3 * Sigma_y
}

After substitution, * constructs OrderedProduct because an Array is present; + constructs Sum. DC preserves those structures without multiplying or adding anything. Their conventional counterparts can evaluate the linear combination. The same mechanism supports 2 * I when I is defined as an identity Array.

Evaluate one scalar expression

Use eval(...) when you want to compute a numerical result before parsing:

prepared = preprocess_shorthand("eval(2 + 3)")
print(prepared)

Output:

5

Parsing prepared constructs Integer 5:

instance = parse_shorthand(prepared)

Evaluation is explicit. The preprocessor does not perform arithmetic on expressions outside eval(...).

Supported arithmetic

The evaluator supports real integer and floating-point literals, parentheses, unary + and -, binary +, -, *, /, and **, and real sqrt(...):

Operator precedence follows Python. Division / performs true division; exponentiation uses **, not ^.

Intermediate values use arbitrary-precision integers and binary64 floating-point arithmetic. If a final Float has an exactly integral value, it is emitted as an Integer. Otherwise, a finite floating-point result is emitted as a Float literal. No tolerance is used to decide whether a value is integral.

Evaluation versus stored structure

Strict Shorthand can store an irrational value as a Product containing a Fraction and a Power:

1/2 * sqrt(3)

That expression retains its Product and constituent structure when preprocessed. To compute a floating-point approximation instead, request evaluation:

eval(1/2 * sqrt(3))

This distinction lets you choose between preserving scientific structure and performing numerical arithmetic before constructing the DCI.

Strict structure and preprocessing arithmetic

Outside eval(...), 1 + 2 constructs a Sum DCI and 2 * 3 constructs a Product DCI. Inside eval(...), those operators perform scalar arithmetic before the result is parsed:

assert parse_shorthand("1 + 2") != parse_shorthand("3")
assert parse_shorthand(preprocess_shorthand("eval(1 + 2)")) == parse_shorthand("3")

The evaluator follows Python operator precedence. For example, eval(1/2**3) produces 0.125; strict Shorthand rejects 1/2**3 because a Fraction base requires an explicit power(...) constructor. They are different input grammars used at different stages.

Evaluation limits

The evaluator supports real scalar arithmetic only. It excludes unresolved names, arbitrary function calls, Arrays, indexing, attribute access, and general code execution.

Division by zero, negative real square roots, complex results, and nonfinite floating-point results are rejected. Implementations may limit expression length, nesting, and intermediate integer size. Cross-language numerical conformance details remain recorded in the Version 1 draft specification.

Combine the features

Once each feature is familiar, they can be used together. Consider entering a symmetric matrix with repeated numerical parameters:

a=1.1 b=0.2 |
<a 0 b, 0 a 0, b 0 eval(-a)>

First, compact List notation expands:

[
  [a, 0, b],
  [0, a, 0],
  [b, 0, eval(-a)]
]

Next, substitutions are inserted:

[
  [1.1, 0, 0.2],
  [0, 1.1, 0],
  [0.2, 0, eval(-1.1)]
]

Finally, eval(...) is evaluated:

[
  [1.1, 0, 0.2],
  [0, 1.1, 0],
  [0.2, 0, -1.1]
]

Strict parsing now constructs a List of Lists. To construct an Array, wrap the compact body in array(..., dtype="float64").

The order also matters when replacement text contains spaces. For example:

term=1/2 * sqrt(3) | <term 0>

Compact List expansion happens first:

[term, 0]

Substitution then produces:

[1/2 * sqrt(3), 0]

The spaces within the replacement Product do not split it into additional List items because compact List expansion has already finished.

Longhand and preprocessing

Longhand is the explicit-constructor subset of strict Shorthand. Preprocessing can insert constructors just as it can insert shorter forms:

prepared = preprocess_shorthand('values=list(1, 2) | tuple(values, fraction(1, 3))')
assert parse_shorthand(prepared) == parse_shorthand('([1, 2], 1/3)')

The longhand renderer flag has no effect on preprocessing. Compact List notation refers specifically to angle-bracket input groups; it is unrelated to the renderer’s layout="compact" delimiter policy.

Python parsing and CLI defaults

Call parse_shorthand() directly for strict Shorthand. When preprocessing is wanted, call preprocess_shorthand() first and parse its result.

Quoted Strings are protected during every stage. Text such as "<1 2>", "eval(2 + 3)", or "a | b" remains ordinary String content.

Preprocessing does not change DataCanon’s types or identity rules. YAML $ values and .dcs persistence files use strict Shorthand and do not implicitly perform preprocessing when read through the ordinary Python persistence API. The CLI is a human-input frontend and enables preprocessing by default:

echo '<1 2, 3 4>' | dc
dc rutile.dcs --dump rutile.dcb

Use --strict to require strict Shorthand input in the CLI. The option does not change the Python API defaults or enable preprocessing in key/count queries.

For the detailed rules, see the Shorthand Preprocessing specification.

2.5 - Custom Conversion Rules

DataCanon Records let applications describe structured values without adding new DataCanon Types. An application can define a Record convention and supply explicit rules connecting that Record to a conventional type.

A calendar date is a useful example. Python provides datetime.date, but DC need not prescribe a calendar, year range, or date arithmetic. The application can make those choices and store the necessary information in a Record.

This guide defines an illustrative "example.Date" convention. The name is application data, not a built-in DC type or a universally prescribed date format.

1. Choose the Record structure

The example stores three Integer fields in the order year, month, day:

record(
  "example.Date",
  "year": 2026,
  "month": 10,
  "day": 22
)

DC defines the Record’s structure and identity. The application defines how those fields describe a date and which dates are valid for its purposes. Other applications can choose different names, additional calendar metadata, or different representations without changing the DC type system.

2. Write the conventional-to-DCI rule

A handler receives the conventional object and a convert() function. Calling convert() applies the same registry recursively to a constituent object.

from datetime import date

from data_canon import ConversionRegistry, as_dc_instance, dc_instance_to_python
from data_canon.instance import Field, IntegerDC, RecordDC

DATE_RECORD = "example.Date"
DATE_FIELDS = ("year", "month", "day")


def date_to_record(value, convert):
    return RecordDC(DATE_RECORD, tuple(
        Field(name, convert(getattr(value, name)))
        for name in DATE_FIELDS
    ))

The handler constructs an ordinary Record DCI. Integer field values use the usual conversion rules. A conventional-to-DCI handler must return a DCI; returning a conventional dictionary or another unsupported object is an error.

3. Write and validate the reverse rule

A reverse handler receives the original Record DCI and a recursive convert() function. Validate the expected schema before constructing a conventional object:

def date_from_record(node, convert):
    if tuple(field.name for field in node.fields) != DATE_FIELDS:
        raise ValueError("example.Date requires year, month, day in that order.")
    if not all(isinstance(field.value, IntegerDC) for field in node.fields):
        raise TypeError("example.Date fields must be Integer DCIs.")
    return date(*(convert(field.value) for field in node.fields))

Python’s date constructor checks its calendar and supported date range. An invalid calendar date raises an error during this application conversion. The Record itself can still be a valid DCI: DC does not assign date semantics to its name or perform the application’s calendar validation.

The example deliberately requires exactly three fields in the stated order. An application’s own handler can validate additional calendar or provenance fields according to its documented convention.

4. Register both directions explicitly

registry = ConversionRegistry()
registry.register_type(date, date_to_record)
registry.register_record(DATE_RECORD, date_from_record)

original = date(2026, 10, 22)
instance = as_dc_instance(original, registry=registry)
assert isinstance(instance, RecordDC)
assert dc_instance_to_python(instance, registry=registry) == original

The registry belongs to the application. Registration does not change global conversion behavior or affect another registry.

Conventional input rules match the exact type. Registering date does not also register datetime.datetime, which carries additional time information. Register a separate rule for each supported conventional type. An explicit rule for an exact built-in type takes precedence over its usual conversion, but existing DCIs are always returned unchanged by as_dc_instance().

Reverse rules match the full Record name using exact String equality. There is no import, class lookup, or prefix-based dispatch derived from that name. Duplicate registrations are rejected so a rule cannot be silently replaced.

5. Convert a nested structure

Rules apply throughout supported collections, including Mapping keys and values:

source = {
    ("run", 7): {
        "recorded_on": date(2026, 10, 22),
        "samples": [date(2026, 10, 21), date(2026, 10, 22)],
    },
    date(2026, 10, 22): "completed",
}

instance = as_dc_instance(source, registry=registry)
restored = dc_instance_to_python(instance, registry=registry)
assert restored == source

A custom handler for a container-like conventional type should call its supplied convert() function for each constituent that needs conversion. Calling the public conversion function without registry=registry would instead use the default rules. Reverse convert() calls also retain the caller’s external resource source context.

Arrays keep their specialized machine elements. Custom conversion does not walk raw Array elements, Fraction integers, Record names, or other fixed metadata as though that information consisted of separate DCIs.

6. Store and recover the DCI

Conversion rules are separate from canonical encoding and persistence:

from data_canon import canonical_decode, canonical_encode

encoded = canonical_encode(instance)
decoded = canonical_decode(encoded)
assert decoded == instance
assert dc_instance_to_python(decoded, registry=registry) == source

Decoding recovers Record DCIs without executing application handlers. Another implementation can preserve, fingerprint, and inspect the same Records without knowing the application’s date convention.

The same sequence works with files:

from data_canon import read, write

write(instance, filename="dated_data.dcb")
loaded = read("dated_data.dcb")
assert loaded == instance
assert dc_instance_to_python(loaded, registry=registry) == source

Pass the registry to the conversion functions. read(), write(), renderers, and canonical codecs do not accept a registry; convert conventional objects to DCIs first and convert decoded DCIs afterward.

Convert your own class

The same mechanism works for an application-defined class. Consider a simple DataPoint whose coordinates are stored in underscore-prefixed attributes:

from fractions import Fraction

from data_canon import (
    ConversionRegistry, as_dc_instance, dc_instance_to_python,
    render_shorthand, read, write,
)
from data_canon.instance import Field, RecordDC


class DataPoint:
    def __init__(self, x, y, z):
        self._x = x
        self._y = y
        self._z = z

Choose an application Record name and stable field names. The Record need not expose the class’s attribute naming convention: x, y, and z describe the stored data, while _x, _y, and _z remain implementation details.

POINT_FIELDS = ("x", "y", "z")


def point_to_record(value, convert):
    return RecordDC("example.DataPoint", tuple(
        Field(name, convert(getattr(value, "_" + name)))
        for name in POINT_FIELDS
    ))


def point_from_record(node, convert):
    if tuple(field.name for field in node.fields) != POINT_FIELDS:
        raise ValueError("example.DataPoint requires x, y, z in that order.")
    return DataPoint(*(convert(field.value) for field in node.fields))


point_registry = ConversionRegistry()
point_registry.register_type(DataPoint, point_to_record)
point_registry.register_record("example.DataPoint", point_from_record)

The class imposes no coordinate-type restrictions, so the reverse handler only validates its field names and order. If your class requires particular coordinate types or other invariants, validate those requirements in the handler or its conventional constructor. Recursive convert() calls retain each coordinate’s usual conventional-to-DCI conversion.

Construct and inspect the DCI:

point = DataPoint(Fraction(1, 2), 0, 1.25)
point_instance = as_dc_instance(point, registry=point_registry)
print(render_shorthand(point_instance, padding=0))

Output:

record(
  "example.DataPoint",
  "x": 1/2,
  "y": 0,
  "z": 1.25
)

Write the Record DCI, read it back, and explicitly reconstruct a DataPoint:

write(point_instance, filename="point.dcb")
loaded_point = read("point.dcb")
assert isinstance(loaded_point, RecordDC)

restored_point = dc_instance_to_python(loaded_point, registry=point_registry)
assert type(restored_point) is DataPoint
assert (restored_point._x, restored_point._y, restored_point._z) == (
    Fraction(1, 2), 0, 1.25
)

Reading returns a Record DCI. The registered reverse handler calls your class’s constructor only during the subsequent conventional conversion. A Record name never causes DC to import or instantiate a class automatically.

The rules also apply inside collections:

cloud = {"points": [point, DataPoint(1, 2, 3)]}
cloud_instance = as_dc_instance(cloud, registry=point_registry)
restored_cloud = dc_instance_to_python(cloud_instance, registry=point_registry)
assert all(type(value) is DataPoint for value in restored_cloud["points"])
assert restored_cloud["points"][1]._z == 3

This example does not define DataPoint.__eq__, so the assertions inspect the restored class and attributes rather than comparing separately constructed objects using Python’s default identity-based equality. The runnable file examples/custom_datapoint.py performs the file round trip in a temporary directory.

Defaults, errors, and limits

With no registry, existing conversion rules are unchanged. An unsupported conventional object raises TypeError. An unregistered Record follows the existing generated-namedtuple conversion, which requires suitable Python class and field names. In particular, a qualified Record name such as "example.Date" needs an application handler for conventional conversion; it remains valid for DCI reading, writing, and identity operations without one.

Application handlers are ordinary Python functions, explicitly chosen by the caller. Handler exceptions propagate; the implementation does not silently fall back to another rule or repair the Record. A handler is responsible for its own schema, conversion policy, and any information it discards.

A custom convention does not make every conversion lossless. Conventional collections still apply their own equality and hashability rules, and a handler can choose a lossy conventional representation. The example date convention has a tested round trip because both directions agree on its schema.

The runnable date example is examples/custom_conversion.py; the user-defined class example is examples/custom_datapoint.py in the code repository. See the Python API for registry signatures.

2.6 - Command-Line Interface

The dc command loads a DataCanon file and displays, converts, searches, or selects items from its DCI. It works with Shorthand (.dcs), canonical binary bytes (.dcb), and DataCanon HDF5 files (.h5 or .hdf5), and YAML interchange files (.yml or .yaml).

Put an input filename first, or omit it to read Shorthand from standard input. With no action option, dc prints Shorthand:

dc structure.dcb

Installation and completion

Installing the Python package installs the dc command in the same environment. Activate that environment before using it. For a development installation from the code repository:

source DC/bin/activate
make install
dc --help

Enable shell completion in the active environment:

eval "$(register-python-argcomplete dc)"

The Makefile also provides make completion, which prints that command for you to execute. Add the registration to your shell initialization after environment activation if you want it applied automatically. Completion suggests command-line options, filenames, and choices such as rendering styles. It does not inspect file contents to suggest mapping keys. Option names appear when you type -; completion does not fill an empty argument position with every CLI option. After --get, further bare arguments are keys until another option begins.

Read Shorthand from standard input

Omit the filename to try a DCI directly at the command line:

echo 'power(8, 1, 3)' | dc

Output:

8**(1/3)

The command preserves the Power DCI and displays it in Shorthand. Constructor names are lowercase. All existing actions and rendering options also work with piped input:

echo 'power(8, 1, 3)' | dc --longhand
echo '{"outer": {"x": 3}}' | dc --get x
echo '[1, 2, 1]' | dc --count 1
echo 'power(8, 1, 3)' | dc --fingerprint
echo 'power(8, 1, 3)' | dc --dump root.dcb

Use - to select standard input explicitly, including with shell redirection:

dc - --layout standard --align left < structure.dcs

The CLI preprocesses Shorthand input by default, so compact Lists, substitutions, and explicit eval(...) expressions need no extra option:

echo '<1 2, 3 4>' | dc --dump matrix.dcb

The resulting DCI is a List of Lists. Standard input defaults to Shorthand; select YAML with --input-format yaml. Use a filename for canonical binary or HDF5 input. A displayed DCI can feed another dc command:

dc matrix.dcb | dc --longhand

Input may span multiple lines and must contain exactly one complete expression. The command reads through end-of-file; when typing directly in a terminal, finish with Ctrl-D on Linux or macOS. Empty input and malformed expressions are errors.

External-resource paths in standard input resolve from the current working directory. Piped Shorthand carries the DCI’s metadata; it does not carry the original file’s resource namespace or its external payloads.

Example input file

A crystal structure illustrates how named parameters keep a scientific input file readable. The example describes rutile, a form of titanium dioxide (TiO₂). Three lattice vectors define a repeating cell; atom positions are given as fractions of those vectors and grouped by chemical element.

Save the following input as rutile.dcs:

a        = 4.593
c_over_a = 0.644
c        = a * c_over_a
u        = 0.305 |
{
  "lattice_vectors": [
    (a, 0, 0),
    (0, a, 0),
    (0, 0, c)
  ],

  "basis_atoms": {
    "Ti": [
      (0,   0,   0),
      (1/2, 1/2, 1/2)
    ],

    "O": [
      (u,        u,        0),
      (1   + -u, 1   + -u, 0),
      (1/2 +  u, 1/2 + -u, 1/2),
      (1/2 + -u, 1/2 +  u, 1/2)
    ]
  }
}

The lattice_vectors and basis_atoms keys follow the Principia Materia (PM) conventions. Lattice vectors and atom positions are written explicitly as Lists of Tuples. The exact halves remain Fractions, and the parameter u makes the relationships between oxygen positions visible in the input.

Load the input file directly:

dc rutile.dcs

Definitions are resolved in source order, so c can use a and c_over_a. The definition c = a * c_over_a retains a Product in the third lattice vector. No evaluation is needed to load or store that structure. Use c = eval(a * c_over_a) if a computed numerical lattice parameter is wanted instead.

The CLI preprocesses .dcs and standard-input Shorthand by default. Use --strict to disable compact Lists, substitutions, and eval(...) and require strict Shorthand:

dc structure.dcs --strict
echo '<1 2, 3 4>' | dc --strict

The second command reports a syntax error. --strict controls input parsing; it does not change rendering or mapping-key interpretation. Canonical binary and HDF5 files use their existing readers in either mode. The Python parse_shorthand() function, read() function, and YAML airlocks retain their strict defaults. See Shorthand Preprocessing for the individual input features.

Dump to another format

Write the resolved DCI as canonical binary bytes:

dc rutile.dcs --dump rutile.dcb

The .dcb file is binary. It stores the DCI, including its numerical structure; substitution definitions and source formatting are not stored. Subsequent reads of this file do not need preprocessing.

The destination suffix selects the output format:

For example:

dc rutile.dcb --dump rutile.h5
dc rutile.h5 --dump restored.dcs
dc rutile.dcb --dump rutile.yaml

A successful dump prints nothing. The resulting representation preserves the DCI’s identity. For DCIs with external references, dumping produces a complete package: resources are verified and copied beside .dcs, .dcb, or YAML output, or embedded in HDF5. Missing or invalid required resources cause the dump to fail.

ExternalArray paths ending in .bin.gz select gzip storage for directory packages and compressed embedded datasets for HDF5. The digest covers the uncompressed canonical element bytes. Transfers retain the same path and digest; an HDF5-to-directory dump recreates the gzip file. Configure compression when constructing the conventional ExternalArray; no additional CLI flag is needed. ExternalFile payloads remain opaque even when their paths end in .gz.

Display YAML

Use --yaml to print the DCI as YAML instead of Shorthand:

dc structure.dcb --yaml
dc structure.dcb --get basis_atoms --yaml
echo '{("Eg", 0): 2.37, "position": (1/2, 0, 0)}' | dc --yaml

The existing YAML interchange rules apply: native YAML values are used where faithful, and $ introduces strict Shorthand for other DCTs, including Tuple keys and values. The $ marker must be followed by whitespace. $100 and ${HOME} remain literal Strings. The marker performs no substitution or evaluation; quoting alone does not escape it. A literal reserved prefix uses a backslash, such as \$ 1/2. The writer handles literal-prefix escaping.

YAML output uses block-style Lists and Mappings, with one List member per - entry; empty collections remain [] or {}. Single-line $ values are not wrapped to a fixed width. YAML quotes scalars only when needed by its syntax or scalar-resolution rules, and multiline Shorthand uses block scalars.

A .yml or .yaml dump target selects YAML without requiring --yaml:

dc structure.dcb --dump structure.yml
dc structure.dcb --yaml --dump structure

The second command creates structure.yaml. Combining --yaml with an explicit non-YAML suffix is an error. YAML export preserves the DCI and does not convert it to conventional Python objects. Terminal output needs only external metadata; a YAML file dump verifies and copies referenced payloads beside the output file.

--yaml works with display and --get, but not --count or --fingerprint. YAML output accepts --align left, --align right, --align none, and --padding N:

dc rutile.dcs --yaml --align left --padding 0
dc rutile.dcb --dump rutile.yaml --align left --padding 1

Alignment is disabled by default. Within a native YAML List, rectangular groups of nonempty, flat Tuple rows share column widths. Each List aligns independently. Ragged rows, mixed List/Tuple rows, or cells requiring multiline Shorthand fall back to normal rendering. Native YAML Lists remain Lists; alignment does not replace them with Tuple escapes. Embedded Shorthand uses the same alignment rules as render_shorthand().

Padding defaults to 1 and controls spaces beside same-line delimiters inside $ escapes, including Tuple keys and numerical constructors. --padding 0 produces $ (1, 2) instead of $ ( 1, 2 ). Padding does not change YAML indentation or the space separating $ from its value. Fraction literals and other fixed forms keep their usual spelling. --longhand, --layout, --custom-layout, and --indent remain Shorthand-only options and are rejected for YAML output.

Read or write YAML

A .yaml or .yml input suffix selects YAML automatically. The default output remains Shorthand, so conversion needs no output option:

dc structure.yaml
dc structure.yaml --dump structure.dcb
dc structure.yaml --yaml --align left

For standard input, choose YAML explicitly:

cat structure.yaml | dc --input-format yaml
cat structure.yaml | dc --input-format yaml --yaml --align left --padding 0

--input-format yaml and --input-format shorthand override text input format selection for stdin or a named file. Input and output choices are independent; --yaml always selects output, never input. The CLI does not guess format from text contents. Canonical binary and HDF5 input still require filenames and their existing readers.

YAML output uses block-style Lists and Mappings. Native multiline Strings use YAML literal block scalars where supported by PyYAML, with quoted fallback when necessary to preserve their contents. Multiline $ escapes likewise use literal block scalars when possible. The --fold-string N option enables width-controlled folded scalars for eligible long single-line Strings. YAML’s native folding and chomping rules apply; these differ from DataCanon Shorthand block-string rules. YAML $ escapes always use strict Shorthand; they never run preprocessing. --strict applies only to Shorthand input and is rejected with YAML input. External paths in a YAML file resolve relative to its containing directory; YAML received on stdin uses the current working directory. Metadata reads do not retrieve external payloads. Python YAML import remains parse_yaml(); this CLI support does not change generic Python read()/write() dispatch.

Choose the displayed notation and layout

Choose delimiter layout, column alignment, and indentation independently:

dc rutile.dcb --layout standard --align left
dc rutile.dcb --layout compact --align left --indent 4
dc rutile.dcb --longhand

The default is standard layout with shorter notation where appropriate. --layout standard puts multiline delimiters on separate lines; --layout semi-compact shares the opening line with the first item or entry; --layout compact also shares the closing line with the final content. --layout inline forces the selected value and its whole subtree onto one line.

dc rutile.dcb --layout inline --padding 0
dc rutile.dcb --custom-layout '{"list":"inline"}'

A List forced inline also renders nested Mappings, Arrays, and other values inline. Child layout overrides yield to that enclosing inline requirement; siblings outside the List retain their own layouts. Column alignment and structural indentation have no effect inside an inline subtree.

--align left, --align right, and --align none control column padding. --align without a value selects left alignment. Alignment is disabled unless requested. --indent N controls structural nesting on new lines, defaulting to 2. It does not add spaces after a same-line opener. Compact continuation items align beneath the first item; indentation moves nested blocks below keys rather than changing that alignment.

--padding N controls spaces beside same-line collection and constructor delimiters, defaulting to 1. It applies to flat values too: [ 1, 2 ] becomes [1, 2] with --padding 0. Standalone delimiters receive no padding. Both options accept nonnegative integers, including zero.

echo '{1, 2, [0, 1]}' | dc --layout compact --indent 0 --padding 0

Output:

{1,
 2,
 [0, 1]}

Use --custom-layout for per-type overrides:

dc rutile.dcb --layout standard --custom-layout '{"mapping":"semi-compact","list":"compact","tuple":"compact","set":"compact"}' --align left

The argument is a strict DataCanon Mapping or UnorderedMapping with String keys and String values. Quote the entire argument for the shell. Constructor form, such as mapping("set": "compact"), is also accepted. The CLI does not use JSON, Python evaluation, or preprocessing to read this configuration.

Every structured type accepts the same four layout contracts. For example, "set", "multiset", "unordered_mapping", "record", and "array" can be configured alongside Lists, Tuples, and Mappings. The Python API lists all accepted type names. An override affects only its named DCT; unspecified types use --layout (default standard). No separate per-type command-line flags are needed. Unknown names, invalid values, and duplicate keys are errors before input loading. Multiline Mapping values start below their keys unless an enclosing inline layout suppresses the line breaks.

--longhand selects explicit constructors for structured values. Null, Boolean, Integer, Float, and String remain literals: sum(1, 1.0, fraction(1, 2)) is longhand. Fraction uses fraction(...), Bytes keeps bytes(...), and Array elements retain their specialized syntax. Layout and notation are independent:

dc rutile.dcb --longhand --layout semi-compact --align left

Lists, Tuples, Sets, and Multisets use one row per item when they contain a List or Tuple, or an item that already spans multiple lines, unless forced inline. Coordinate Tuples remain inline, so the rutile lattice vectors and atom positions appear as separate rows. Alignment adds column padding for rectangular rows, including coordinates containing Sums or Products. For example:

echo '[(1, 0), (0, -1)]' | dc --layout semi-compact --align left

Output:

[ ( 1,  0 ),
  ( 0, -1 )
]

Using --layout compact also puts the final ] on the last row.

The same options apply when dumping Shorthand:

dc rutile.dcb --dump readable.dcs --layout standard --align left
dc rutile.dcb --dump constructors.dcs --longhand

Rendering options affect textual representation, not DCI identity. The options also apply to values printed by --get. They are rejected with --count, --fingerprint, and binary or HDF5 output. YAML accepts alignment and padding as described above; Shorthand layout options do not apply to YAML.

Render multiline and folded Strings

By default, standard Shorthand layout renders Strings containing LF characters as literal blocks when possible:

{
  "message":
    """
    First line
    Second line
    """
}

Literal blocks preserve the String’s actual line breaks and whitespace. Strings that cannot safely use block syntax retain their ordinary quoted representation.

For long Strings without semantic line breaks, use --fold-string N to request physical wrapping at a preferred width of N characters:

dc input_data.dcs --fold-string 80
dc input_data.dcs --dump readable.dcs --fold-string 80

The renderer uses folded blocks when it can do so without changing the String DCI:

{
  "description":
    """>
    This is a long description that can be wrapped across
    physical lines without introducing semantic newlines
    into the underlying String.
    """
}

The requested width is a preference, not a strict maximum. Long words and significant whitespace may require longer lines or fallback to quoted rendering.

Block Strings are emitted only under standard Shorthand layout. The semi-compact, compact, and inline policies retain quoted Strings with escapes. Enclosing inline layouts take precedence over child rendering preferences.

The same --fold-string option is available for YAML output:

dc input_data.dcs --yaml --fold-string 80
dc input_data.dcs --dump readable.yaml --fold-string 80

YAML uses its native literal (|) and folded (>) scalar syntax. PyYAML may choose quoted scalars when block formatting is unsuitable, particularly when significant trailing whitespace is present. Width-controlled folding is applied selectively to eligible Strings, not globally to all YAML output.

These options affect textual presentation only. Parsing the output recovers the same DCI and therefore the same canonical bytes and fingerprint.

Find a mapping key

Supply a bare name for a String key. For example, find the titanium positions in the nested basis_atoms Mapping:

dc rutile.dcb --get Ti

The search visits Mappings and UnorderedMappings throughout the DCI, so it finds "Ti" inside "basis_atoms". It also visits mappings nested in collections, Record field values, and mapping keys. It does not require a Mapping at the root.

One matching entry prints its value. If the same key occurs in multiple mappings, the command prints a List containing all matching values, in traversal order. Equal repeated values remain separate occurrences. It continues searching inside a matched value, so deeper matches are included too. Traversal follows the recovered constituent order; unordered collections do not promise an application-level ordering of results.

For example, save this strict Shorthand as repeated.dcs:

{"left": {"x": 1}, "right": {"x": 2}}

Then:

dc repeated.dcs --get x

prints a List containing 1 and 2. A single match whose value is itself a List is printed directly; multiple List-valued matches form an outer List.

Keys can belong to any DCT. For example, when a file contains a Tuple key:

dc results.dcb --get '(1/2, 0, 0)'

Valid Shorthand keys retain their DCTs. To select String keys whose text would otherwise parse as another DCT, include explicit Shorthand quotes:

dc results.dcb --get 1          # Integer key
dc results.dcb --get true       # Boolean key
dc results.dcb --get '"1"'      # String key
dc results.dcb --get '"true"'   # String key

Implicit String handling accepts bare identifiers, including Unicode letters and names such as basis_atoms or site_1. Names with spaces or punctuation need explicit Shorthand quotes, for example --get '"site label"'. Malformed structured keys such as power(1, remain errors; they do not become Strings.

Key comparison uses DCI equality. Integer 1, Float 1.0, and Boolean true remain distinct keys. Record names and field names are fixed metadata, so --get does not treat them as mapping keys. A missing key is an error.

Follow a chain of mapping keys

Supply several keys to --get to search for the first and then follow direct Mapping entries:

dc rutile.dcb --get basis_atoms O
dc results.dcb --get measurements '("Eg", 0)'

The first key is searched recursively using the existing single-key behavior. Every subsequent key is looked up directly in each matching Mapping or UnorderedMapping value. Later steps do not search recursively or descend through Lists. Each key uses the same strict Shorthand or bare-String-name rules. Negative integer keys work within the chain. For negative expressions that argument parsing could mistake for options, use constructor form, such as --get measurements 'fraction(-1, 2)'. Dots are not separators; a String key containing a dot needs Shorthand quotes, for example --get '"basis.atoms"'.

Branches missing a subsequent key, or whose intermediate value is not a Mapping, are skipped. If no branch completes the chain, the command reports an error. Results stay separate during traversal: one final match is printed directly, while multiple final matches become a List in traversal order. Equal repeated matches remain separate, and List-valued matches are not flattened.

For example, this input has two complete chains and one incomplete branch:

echo '[{"a": {"b": 1}}, {"a": {"b": 2}}, {"a": 3}]' | dc --get a b

Output:

[ 1, 2 ]

Put an input filename before --get; subsequent bare arguments are consumed as keys. For stdin, omit the filename or use -. Another option ends the key sequence:

dc rutile.dcb --get basis_atoms O --yaml --align

Pipelines remain useful, but a second dc --get searches recursively again. A multi-key --get follows direct entries after the first recursive search.

Select or slice a List, Tuple, or Array

--index SELECTOR [SELECTOR ...] applies integers or slices successively. Integer indices are zero-based and accept negative values from the end:

echo '[(1, 2), (3, 4)]' | dc --index 0
echo '[(1, 2), (3, 4)]' | dc --index=-1
echo '[(1, 2), (3, 4)]' | dc --index 1 0

Without --get, indexing applies to the root DCI. With --get, it applies to the final result after the entire key chain:

dc rutile.dcb --get basis_atoms O --index 2
dc rutile.dcb --get basis_atoms O --index=-1 --yaml

If there are multiple final matches, --index selects one member of the aggregate List; it does not index each match individually. A single List-valued match is indexed directly. Selection order is always --get, then --index, then rendering, independently of where the options occur on the command line.

For Lists and Tuples, integer indexing selects a contained DCI. For Arrays, integer indexing selects along the first dimension. If the result is an Array, it is returned as an Array DCI; if the result is an Array element, that value is converted to a DCI. Strings, Sets, Multisets, and Mappings are not indexed. An incompatible target or an out-of-range integer index reports an error; malformed selectors are command-line usage errors. Empty sequences have no valid indices.

Slicing operates along the first dimension and preserves the DCT: a List slice is a List, a Tuple slice is a Tuple, and an Array slice is an Array. Out-of-range bounds are clipped, and slices may produce an empty result. A zero step or malformed slice is a usage error before input is loaded. Subsequent selectors operate on the slice result itself, without distributing across its members.

Shorthand formatting and YAML output options apply to the selected DCI. --index cannot be combined with --dump, --count, or --fingerprint. Each index is applied to the preceding result: --index 1 0 selects item 1, then item 0 from that value. Every intermediate target must be a List or Tuple; otherwise the command fails without printing a partial result. Another option ends the index sequence. Put the filename before --index as well as --get. Use --index -1 -1 for several negative integer indices.

Slices use start:stop[:step], with Python’s exclusive stop bound, omitted bounds, negative bounds, and negative steps:

dc rutile.dcb --get basis_atoms O --index 0:2
echo '(0, 1, 2, 3, 4)' | dc --index ::2
echo '[0, 1, 2, 3, 4]' | dc --index ::-1

Integer indexing selects a contained DCI for Lists and Tuples. For Arrays, integer indexing selects along the first dimension. If the result is an Array, it is returned as an Array DCI; if the result is an Array element, that value is converted to a DCI. Slicing operates along the first dimension and preserves the DCT: a List slice is a List, a Tuple slice is a Tuple, and an Array slice is an Array. Out-of-range bounds are clipped, and slices may produce an empty result. A zero step or malformed slice is a usage error before input is loaded. Subsequent selectors operate on the slice result itself, without distributing across its members:

echo '[(0, 1), (2, 3), (4, 5)]' | dc --index 1: 0 1

That command selects the suffix List, its first Tuple, then the Tuple’s second item, producing 3.

A slice beginning with - can look like an option to argparse, so attach it with =. Repeat --index to continue the same selection chain:

echo '[0, 1, 2, 3, 4]' | dc --index=-2:
echo '[[0, 1, 2], [3, 4, 5]]' | dc --index 1 --index=-2: --index 0

Repeated --index options concatenate their selectors in order. Integers and slices are the entire selection syntax; there is no expression evaluation or filter language. Put the input filename before the selection arguments.

Approximate and round numerical values

Use --round N to round floating-point values to N decimal places, including floating-point and complex Array elements. Negative N rounds to powers of ten. Array type and shape are preserved. Rounding uses Python’s round rule on binary floating-point values, not exact decimal arithmetic.

dc results.dcb --get reciprocal_vectors --round 3 --align
dc results.dcb --round 3 --yaml

Use --to-float to collapse a complete scalar Fraction, Constant, Sum, Product, Power, or Quotient value to a Float. Integer and Boolean DCIs remain unchanged; Complex components are processed individually. Existing Arrays retain their types. Array-valued Sums and OrderedProducts retain their structure; scalar subexpressions inside them are approximated without evaluating Array arithmetic. Lists, Tuples, Records, and other collections are processed recursively. For example:

echo '2 * Pi + 1/2' | dc --to-float --round 3

produces:

6.783

Sum and Product operands are evaluated in canonical order so equal DCIs have equal approximations, independent of their stored operand order.

Selection (--get and --index) happens first, then conversion, then rounding, regardless of the order of flags. Mapping keys and external references are unchanged, and no external payload is loaded. A Set transformation that would produce duplicate members fails explicitly; Multisets retain occurrences.

Both options create a transformed DCI. They work with Shorthand, YAML, and --dump, including binary or HDF5 output. The original DCI is unchanged; a transformed DCI can have a different fingerprint. Neither option can be combined with --count or --fingerprint. Omit --to-float to round Float leaves while retaining the surrounding numerical structures and exact fields.

Count occurrences of a DCI

--count searches for a complete DCI using its type-defined equality:

dc rutile.dcb --count '1/2'

It prints an integer count. Zero matches is a successful search that prints 0.

The search includes the root and all constituent DCIs: collection members, mapping keys and values, Record field values, Complex components, Power bases, and Quotient constituents. Repeated occurrences count separately, including repeated members of Multisets, Sums, and Products.

Fixed information prescribed directly by a DCT is not a separate DCI. Searching for Integer 1 does not match a Fraction numerator or a Power exponent field. Likewise, Array elements, Record names, relative-path components, and byte payload contents are not searched as constituent DCIs.

Queries are never preprocessed. --count requires strict Shorthand; --get additionally accepts bare String names as described above. For a negative numeric query, use an equals sign so argument parsing keeps the value with its option:

dc results.dcb --count=-1

The same convention works for negative keys: --get=-1.

Print the fingerprint

dc rutile.dcb --fingerprint

The output is the 64-character lowercase hexadecimal SHA-256 fingerprint of the DCI’s canonical bytes. It is independent of Shorthand layout and HDF5 storage layout. It is not a hash of the input file’s representation bytes, except when that file is itself the raw canonical .dcb encoding.

Displaying, searching, and fingerprinting external DCIs use their metadata. Metadata operations do not retrieve external payloads. Dumping a complete package does require those resources.

Options and errors

Choose at most one action: --dump, --get, --count, or --fingerprint. --get accepts one or more keys. --index optionally selects from the root or the final --get result; it is not valid with the other actions. With none supplied, the action is to display Shorthand, or YAML with --yaml. Omitting the filename, or supplying -, reads text from standard input (Shorthand by default; --input-format yaml selects YAML). Help and version output do not read input:

dc --help
dc --version

Successful output goes to standard output and diagnostics go to standard error. Displayed Shorthand and YAML are suitable for shell redirection, but use --dump when creating a complete package with external resources.

A malformed Shorthand query, invalid input file, or failed resource verification produces a diagnostic and a nonzero exit status.

3 - Python API

A DataCanon Type (DCT) defines its instances and their identity. The Python reference implementation provides construction of DataCanon Instances (DCIs), conversion, Shorthand, canonical bytes, fingerprints, persistence, and additional conventional numerical and collection types.

A conventional type is a type used directly for data and computation in a programming language, including built-in, library-provided, and user-defined types. Conventional list, Fraction, and Power objects are distinct from DCIs represented by ListDC, FractionDC, and PowerDC objects. DCIs and conventional objects are both Python objects; conversion connects types with different contracts.

For a step-by-step introduction, use Conventional Types.

Public API

Classes implementing DCTs, such as RecordDC and BytesDC, are available from data_canon.instance. Field is a helper class for Record fields. Conversion and Shorthand parsing can construct DCIs without requiring direct use of those classes.

Shared queries, transformations, and output

The functions below are available from data_canon. No function invokes argparse or calls the dc executable. Applications can select, transform, and display a DCI using the same operations as the command-line tool.

Numerical transformations preserve Mapping keys and external references, without retrieving payloads. Set collisions fail rather than dropping members; Multiset occurrences are retained. Rounding retains numerical expression structure; call to_float() first to approximate complete expressions. Scalar Sum and Product evaluation follows canonical operand order; scalar OrderedProduct evaluation retains factor order. Array-valued Sums and OrderedProducts retain their structure while scalar subexpressions are converted. The input DCI and its cached fingerprint remain unchanged. Evaluation failures, nonfinite results, and Array overflow are reported as exceptions.

from data_canon import (
    parse_shorthand, get_values, index_value, to_float, round_floats,
    render_output,
)
from data_canon.instance import StringDC

source = parse_shorthand('{"coordinates": [(Pi, 1/2), (E, 0)]}')
selected = index_value(get_values(source, StringDC("coordinates")), 0)
result = round_floats(to_float(selected), 3)
assert render_output(result, padding=0) == "(3.142, 0.5)"

Conversion between conventional objects and DCIs

as_dc_instance(obj, *, registry=None)

Construct a DCI from an instance of a supported conventional type. Passing an existing DCI returns that instance unchanged. An optional ConversionRegistry supplies application-owned conventional-type rules, applied recursively.

from fractions import Fraction

from data_canon import as_dc_instance

instance = as_dc_instance({
    ("Eg", 0): 2.37,
    "position": (Fraction(1, 2), Fraction(1, 4), 0.199),
    "label": "O",
    "payload": b"\x00\xff",
})

Floats and portable floating-point Array elements must be finite. Conventional bytes is supported directly; convert mutable byte buffers to bytes explicitly. Construction of a DCI cannot recover information already lost when the source conventional object was constructed or manipulated.

dc_instance_to_python(node, source=None, *, registry=None)

Construct an instance of the corresponding conventional type from a DCI, recursively converting constituent DCIs. The DCI is already a Python object; the function name identifies the implementation language, not a change from one language to another.

from data_canon import dc_instance_to_python

converted = dc_instance_to_python(instance)
assert type(converted) is dict
assert type(converted["position"]) is tuple
assert isinstance(converted["position"][0], Fraction)

The conversion directions are not necessarily symmetric. A conventional Fraction reduces ratios; Sum, Product, and Power apply their own simplifications. A generated named tuple is accepted as a conventional Tuple input by as_dc_instance(), so it does not reconstruct the original Record name and fields through that function.

Conventional dictionaries and sets apply their own hashability and equality rules. A hashable List DCI key becomes an unhashable conventional list, which can make conversion to a dict fail. Distinct DCI keys or Set members can also be merged if their converted conventional objects compare equal.

Without a matching custom Record handler, names and fields must satisfy namedtuple restrictions for conventional Record conversion to succeed. Array conversion returns the stored read-only NumPy array; use .copy() when an independently mutable array is wanted. Do not circumvent the stored array’s read-only protection, because mutation would invalidate the DCI’s identity and any cached fingerprint or hash.

External resources are materialized during this conversion. Their bytes are retrieved and verified using retained source context or the supplied source. ExternalFile conversion can additionally use the implementation’s filename-based content loaders. Keep the external DCI for metadata-only work.

Conversion to conventional objects and construction of a new DCI do not promise a lossless round trip for every DCI. The DCI model retains its own information and equality rules independently of the conventional result types.

NumPy scalar conversion

as_dc_instance() supports NumPy Boolean, signed/unsigned integer, portable floating-point and complex, Unicode, and byte scalars. The conversions preserve their values using Boolean, Integer, Float, Complex, String, and Bytes DCIs. Every supported NumPy integer width converts exactly to the unlimited-precision Integer DCT. Finite binary16/32 values widen exactly to binary64, preserving signed zero; complex64/128 become Complex DCIs with Float components.

Scalar width does not become identity metadata. A zero-dimensional ndarray is still an Array DCI and retains its Array type and empty shape.

Nonfinite numerical scalars are rejected. Wider floating-point/complex formats, temporal scalars (datetime64, timedelta64), and structured or opaque void scalars require an explicit conversion or registry handler. Portable dtype aliases follow their corresponding dtype rules. Custom exact-type handlers take precedence over the built-in NumPy conversions.

See NumPy scalar values.

ConversionRegistry()

Create an application-owned registry. The conversion functions accept it through registry=registry; omission retains the existing default rules.

Both methods receive a handler callable as converter(value, convert). The second argument recursively converts using the same registry. A reverse handler’s recursive converter also retains the outer source context. Conventional-to-DCI handlers must return an InstanceDC; reverse handlers receive the original RecordDC and are responsible for validating their schema.

Rules for exact conventional types take precedence over built-in conversion. Rules do not apply to subclasses implicitly. Existing DCIs always pass through as_dc_instance() unchanged, and registering a DCI implementation class is rejected. Record matching uses the complete name, without import or class lookup. Duplicate registrations, invalid types or names, and noncallable handlers raise errors. Handler exceptions propagate without fallback.

Registries do not alter global behavior, DCT definitions, canonical encoding, or parsing and persistence. No handler is serialized into the DCI. An unregistered Record retains the generated-namedtuple fallback, including its name restrictions. Writers and codecs do not accept a registry: convert to a DCI explicitly before writing and convert decoded DCIs afterward.

The Custom Conversion Rules guide defines both directions of a date convention and demonstrates nested conversion, schema validation, and canonical and file round trips.

Shorthand parsing

parse_shorthand(text, *, preprocess=False, substitutions=None)

Parse exactly one complete Shorthand expression into a DCI:

from data_canon import parse_shorthand

phase = parse_shorthand("complex(1/2, 1/2 * sqrt(3))")

The parser accepts explicit constructors and their shorter forms in any mixture:

assert parse_shorthand("fraction(1, 2)") == parse_shorthand("1/2")
assert parse_shorthand("list(1, 2)") == parse_shorthand("[1, 2]")
assert parse_shorthand("tuple(1)") == parse_shorthand("(1)")
assert parse_shorthand("set()") == as_dc_instance(set())
assert parse_shorthand('mapping("x": 1)') == parse_shorthand('{"x": 1}')

Collection constructor arguments are their members or entries. For example, list([1, 2]) is a List containing one List, not a conversion of that inner List. set() is the empty Set; {} is the empty Mapping. The old emptyset spelling is not accepted.

Strict parsing rejects trailing input and invalid DCT information. Numerical parentheses are not general grouping: (1 + 2) is a singleton Tuple containing a Sum. Use constructors when nesting needs to be explicit.

Preprocessing can be requested directly:

prepared_instance = parse_shorthand("<1 2 3>", preprocess=True)

The dc CLI preprocesses Shorthand by default and offers --strict; that frontend policy does not change this function’s strict default.

Supplying substitutions requires preprocess=True. For a clear two-stage workflow, call preprocess_shorthand() separately as shown below.

Shorthand rendering

render_shorthand(node, *, layout="standard", custom_layout=None, align=False, indent=2, padding=1, longhand=False)

Render a DCI as identity-preserving Shorthand. A conventional input object is first converted with as_dc_instance(), and the resulting DCI is rendered:

from data_canon import render_shorthand

text = render_shorthand(phase)
assert parse_shorthand(text) == phase

Shorthand has no canonical text layout. Rendering must preserve DCI equality; fingerprints use canonical bytes rather than this text.

Longhand

longhand is a Boolean, defaulting to False. With True, all structured DCIs use explicit constructors. Null, Boolean, Integer, Float, and String retain their literal spellings. Fraction uses fraction(...), and Bytes retains bytes(...). Array elements retain their specialized numerical syntax. The boundary is syntactic, not the primitive/atomic/composite classification. For example, sum(1, 1.0, fraction(1, 2)) is longhand; literals need no additional wrappers. Version 1 defines neither integer(...), float(...), nor string(...) wrappers, nor a separate fully wrapped rendering mode. The same parser accepts both outputs.

value = parse_shorthand("1/2 * sqrt(3)")
assert render_shorthand(value, layout="standard") == "1/2 * sqrt( 3 )"
assert render_shorthand(value, layout="standard", longhand=True) == (
    "product( fraction(1, 2), power( 3, 1, 2 ) )"
)

Default rendering uses constructors to retain empty or singleton Sums and Products, nesting, and unambiguous Power bases. Quotient retains quotient(...). No separate notation or power_style option is provided.

Independent layout, alignment, indentation, and padding

layout controls line breaking and delimiter placement:

For example, a semi-compact Mapping renders as:

{ "a": 1,
  "b": 2
}

Scalar-only collections and empty collections normally stay inline. Lists, Tuples, Sets, and Multisets normally become multiline when they directly contain a List or Tuple, or an item already spans multiple lines. An inline subtree suppresses those line breaks. Nonempty Mappings are multiline unless their effective layout is inline. Multiline Mapping values start below their keys, so indentation does not depend on key length. Multiline keys remain valid Shorthand and use the same recursive rendering rules.

align="left" or align="right" pads cells in rectangular List/Tuple rows and supported Array tables. align=False disables column padding. Alignment can be combined with every layout and with longhand. Ragged rows receive no column padding. Alignment does not introduce a line-width wrapping threshold.

indent controls extra indentation when structural nesting places content on new lines, defaulting to 2. For example, it controls content after a standalone opener and a multiline value below its Mapping key. It does not add spacing after an opener that shares its content line. Compact continuation items align beneath the first item, whose column is determined by the opening token and padding. Consequently, changing indentation does not move compact Mapping keys; it moves nested blocks below them.

padding controls gaps beside same-line collection and constructor delimiters, defaulting to 1. For example, a flat List is [ 1, 2 ]; padding=0 gives [1, 2]. It also supplies the gap after a shared multiline opener and before a shared multiline closer. A standalone delimiter receives no padding. Continuation items begin at relative column len(opening) + padding.

Both options accept nonnegative integers, including zero, and reject Boolean, negative, and noninteger arguments. Empty collections retain [], (), set(), and similar spellings. Primitive literals and their fixed constructor spellings, item separators, and specialized Array element brackets retain their existing forms. align independently controls table-cell padding.

packed = parse_shorthand('{1, 2, [0, 1]}')
print(render_shorthand(packed, layout="compact", indent=0, padding=0))

Output:

{1,
 2,
 [0, 1]}

rows = parse_shorthand('{"rows": [(1, 20), (30, 4)]}')
text = render_shorthand(
    rows,
    custom_layout={"list": "compact", "tuple": "compact", "mapping": "semi-compact"},
    align="left",
    indent=2,
)
print(text)
assert parse_shorthand(text) == rows