Persistence

DataCanon Persistence — Version 1 Draft

Status: Draft for independent implementation review
Depends on: DataCanon Core Version 1; DataCanon Shorthand Version 1; DataCanon HDF5 Version 1
Related: DataCanon YAML Interchange Version 1; DataCanon Shorthand Preprocessing Version 1
Scope: Storage-format selection, reading and writing, external-resource namespaces, deferred retrieval, integrity verification, and identity-preserving transfer
Out of scope: DCT definitions, canonical byte encoding, Shorthand grammar, HDF5 object layout, YAML scalar semantics, and implementation-language APIs

Introduction

DataCanon Persistence defines how DataCanon Instances (DCIs) are read, written, and transferred among supported storage representations while preserving their identity. It also defines how the relative paths belonging to ExternalFile and ExternalArray DCIs are interpreted within different resource namespaces.

DataCanon Core Version 1 alone defines DCI equality, canonical encoding, and fingerprints. Persistence MUST NOT redefine or alter those properties.

A DCI may be represented in memory, as canonical binary bytes, as Shorthand text, or within HDF5. External resources may be stored as separate filesystem files or embedded in an HDF5 container. The source from which an implementation can retrieve such resources is not part of the DCI.

This specification concerns the behavior of conforming implementations, not their choice of internal classes, APIs, or filesystem utilities. The Python reference implementation provides operations commonly named read and write, but those names are not normative.

Conformance and terminology

The terms MUST, MUST NOT, SHOULD, and MAY indicate normative requirements and recommendations.

A representation is a stored form from which a DCI can be reconstructed. A resource namespace is the context in which an external DCI’s DataCanon relative path is resolved. A source is an implementation-level handle or description of such a namespace. A payload is the byte sequence referred to by an external DCI.

A metadata read reconstructs a DCI without necessarily retrieving external payloads. Materialization retrieves and validates a referenced payload. A complete package consists of a DCI representation together with all external payloads needed to satisfy its external references.

For every supported format, a conforming writer and reader MUST satisfy the identity-preserving round trip:

read(write(x)) = x

where equality is DataCanon equality. The equation assumes that the write succeeds and the stored representation is subsequently available and intact. It does not imply that a metadata read materializes external resources.

Distinct representations of equal DCIs need not have equal file bytes. In particular, Shorthand formatting and HDF5 physical layout are not canonical encodings.

Supported representations and filenames

Version 1 defines the following conventional file suffixes:

A .dcs file MUST contain exactly one valid strict Shorthand DCI expression, encoded as UTF-8 text. The file MAY contain permitted Shorthand whitespace. The file MUST NOT require substitutions, compact List expansion, or eval(...) to be interpreted.

A .dcb file MUST contain exactly one complete canonical Version 1 DCI encoding. It MUST NOT prepend a magic header, length prefix, or version marker to those bytes. Its bytes are precisely the canonical encoding defined by Core Version 1.

A .h5 file MUST follow the HDF5 Version 1 layout, including its required /root object and the rules for embedded external payloads.

The suffixes above are conventional format-selection signals, not semantic DCI constituents. An implementation MAY provide explicit format selection, and MAY recognize additional suffix aliases, but MUST interpret a selected format according to its governing specification.

YAML interchange is separately specified and is not a required format for the generic Version 1 persistence dispatch. An implementation MAY provide explicit YAML import and export operations. It MUST NOT silently treat arbitrary YAML as strict Shorthand or vice versa.

Reading a DCI

Identity-preserving metadata reads

A conforming read operation MUST reconstruct the DCI described by the selected representation and MUST validate its semantic constituents according to Core Version 1.

In particular, a reader MUST preserve:

• the DCT of every constituent, including distinctions between Integer, Float, Fraction, Sum, Product, Power, Quotient, List, Tuple, and Array;
• ordered collection and Record field order;
• unordered collection membership and key/value identity;
• Multiset member multiplicities and nested structure;
• unreduced Fraction and Power exponent structure, and unevaluated Sum, Product, and Quotient constituents;
• floating-point bit identity, including signed zero;
• Array type, shape, and portable element values; and
• ExternalFile and ExternalArray relative paths, metadata, and SHA-256 digests.

A reader MUST NOT change an ExternalArray into an Array, or an ExternalFile into another DCT, merely because the referenced bytes are available.

Contained byte payloads

Bytes DCIs contain their entire byte sequences. Writers MUST preserve those bytes directly in the selected representation: raw length-delimited bytes in .dcb, hexadecimal Shorthand in .dcs, or kind = "bytes" datasets in .h5. No external-resource namespace or companion file is required for Bytes.

Reading a Bytes DCI reconstructs its byte sequence. Applications may attach interpretation metadata using a containing Record or another suitable DCT; persistence does not interpret or execute byte contents.

External references are not eagerly loaded

Reading a DCI MUST NOT require external resources to exist, be opened, or be verified. An ExternalFile or ExternalArray DCI is valid independently of the availability of its referenced payload.

Thus, a metadata read of a .dcs, .dcb, or .h5 representation containing external DCIs MUST be possible without first materializing those external resources, provided the DCI representation itself is valid.

For HDF5, reading the DCI hierarchy does not require reading the /_external/*/data payload datasets. An implementation MAY read nonsemantic metadata needed to establish future source context.

A reader MAY expose a separate operation that verifies or materializes external payloads. Such an operation is not part of ordinary metadata reading.

Source context

A reader SHOULD retain sufficient source context to resolve external resources later without requiring the application to resupply the location of the representation. This context MAY be associated with the returned DCI tree, with its external nodes, or with a separate source handle.

Source context is nonsemantic. It MUST NOT participate in DCI equality, canonical encoding, or fingerprinting. Attaching, replacing, or discarding source context MUST NOT change the DCI.

The source context may identify a directory containing external files, an HDF5 file containing embedded payloads, or another implementation-supported resource namespace.

The representation’s source context does not modify the relative path stored in any external DCI.

External resource namespaces

Abstract relative paths

ExternalFile and ExternalArray DCIs contain DataCanon relative paths, defined by Core Version 1 as nonempty ordered sequences of path components. A path is not a host operating-system pathname and does not include an absolute location or a source identifier.

Two equal external DCIs retain the same identity even when resolved against different source namespaces. The choice of namespace determines which bytes may be found; it does not determine DCI equality.

A resource is accepted as satisfying an external DCI only after the required digest and, for ExternalArray, element-stream checks succeed.

Directory-backed representations

For a .dcs or .dcb representation stored in a filesystem, the default resource namespace is the directory containing that representation file.

A DataCanon relative path with components ("inputs", "config.txt") is resolved by joining those components beneath the representation’s containing directory, using the host filesystem’s path conventions.

For example:

runs/run42/
├── calculation.dcs
└── inputs/
    └── config.txt

An ExternalFile whose DataCanon relative path is ("inputs", "config.txt") refers by default to runs/run42/inputs/config.txt when calculation.dcs is the source representation.

The same convention applies to a .dcb file in place of calculation.dcs. No particular name for the representation file or containing directory is required.

An implementation MAY accept an explicit directory source to override the default namespace. Such an override MUST NOT change the DCI’s stored relative path or its identity.

A conforming implementation MUST NOT interpret DataCanon relative path components . or .., absolute paths, or embedded separators as permissible alternate spellings. The Core Version 1 relative-path validation rules apply before resolution.

Host filesystem behavior, including permissions and symbolic links, is implementation-specific. Implementations SHOULD avoid unintended resolution outside an explicitly trusted resource namespace, particularly when handling untrusted input.

HDF5-backed representations

For a DCI read from a DataCanon HDF5 file, the default external resource namespace is that HDF5 container’s embedded /_external collection.

The reader resolves a DataCanon relative path by matching its complete ordered component sequence against the path stored with an embedded resource entry. The numeric name of the /_external entry is not semantic and MUST NOT be used as the resource identifier.

The HDF5 Version 1 specification defines the precise layout, uniqueness rules, and payload datasets. Persistence MUST NOT reinterpret those structures.

A source-aware implementation MAY support an explicitly supplied alternative resource namespace, but MUST NOT silently substitute a different payload for a missing or invalid embedded resource.

No source available

A DCI constructed directly in memory, or decoded from canonical bytes without an associated representation location, may have no source context.

Such a DCI remains valid. Materializing an uncached external resource requires an appropriate source to be supplied or otherwise made available. Failure to locate a resource MUST NOT mutate the DCI or change its fingerprint.

External payload materialization and verification

General requirements

Before returning or otherwise accepting retrieved external bytes as the content identified by an ExternalFile or ExternalArray DCI, an implementation MUST compute SHA-256 over the retrieved bytes and compare the resulting 32-byte digest to the DCI’s declared digest.

For ExternalArray, retrieved payload bytes mean the logical canonical element stream after any storage decompression. For ExternalFile, they mean the exact opaque stored bytes.

If the digest differs, the operation MUST fail. It MUST NOT return the mismatching bytes as verified content.

A missing, inaccessible, truncated, or corrupt resource MUST cause the attempted materialization to fail. Materialization failures do not invalidate the metadata DCI.

The cryptographic limitations of SHA-256 are those described in Core Version 1: digest equality is a practical integrity check based on collision resistance, not a proof of byte-sequence identity.

ExternalFile

An ExternalFile payload is an opaque byte sequence. Materialization MUST preserve its exact bytes. DataCanon Persistence does not mandate any text decoding, YAML parsing, or application-specific interpretation of those bytes.

An implementation MAY provide extension-based or user-supplied content loaders as a convenience. Such interpretation occurs after digest verification and is not part of DCI identity or portable persistence semantics.

ExternalArray

An ExternalArray payload is the canonical Array element byte stream, not the complete canonical encoding of an Array DCI. It contains no Array type, shape, or other framing metadata.

After any storage decompression, an implementation MUST validate the digest and the payload against the external DCI’s portable Array type and shape, including:

• the exact required byte length;
• canonical Boolean element bytes (00 and 01 only);
• finite floating-point and complex elements; and
• the element ordering and representation prescribed by Core Version 1.

A materialized Array value MAY be provided as an instance of a conventional numerical container type, such as a NumPy array in Python. This does not change the original ExternalArray DCI into an Array DCI.

ExternalArray storage compression

ExternalArray logical payloads are canonical Array element streams. Directory storage MUST use a relative path whose final component ends in .bin for an uncompressed stream or .bin.gz for a gzip-compressed stream. Suffixes are case-sensitive. Other suffixes are not supported by this directory storage profile. Core relative-path validity is unchanged: metadata decoding does not need to retrieve a resource or apply this storage profile.

The digest ALWAYS covers the uncompressed canonical element bytes. A reader MUST decompress .bin.gz resources before accepting their digest and Array contents. It MUST enforce the exact logical byte length determined by the Array type and shape; it SHOULD reject excess decompressed output while reading, rather than allocating an unbounded decompressed buffer. Missing resources, malformed or truncated gzip, digest mismatch, invalid Boolean bytes, nonfinite elements, and incorrect logical lengths cause materialization to fail.

Compressed and uncompressed paths identify distinct DCIs even if their logical payloads agree. A transfer MUST preserve the stored path and digest. In particular, HDF5-to-directory export MUST recreate a gzip file when the path ends in .bin.gz. A directory-to-HDF5 transfer embeds the verified logical stream, rather than a gzip envelope. HDF5 dataset filters handle storage compression transparently; readers MUST NOT decompress the returned logical bytes a second time because a resource path ends in .gz.

ExternalFile payloads remain opaque. A .gz ExternalFile is verified over its stored bytes and is not implicitly decompressed. One path MUST NOT be shared by an opaque ExternalFile and a compressed ExternalArray, even when their declared digests agree, because their directory storage requirements conflict. Uncompressed ExternalArrays may share paths with ExternalFiles when all byte and validity requirements agree. Writers MUST reject conflicting references before producing a complete package.

The Python conventional constructor accepts compression=False or True. A suffixless filename receives .bin or .bin.gz; an omitted filename is the logical digest in hexadecimal plus the selected suffix. Explicit suffixes must agree with the selected compression setting. The constructor finalizes the path before DCI construction. There is no additional semantic compression field. The reference writer uses gzip level 6 for directory streams and embedded HDF5 ExternalArray datasets selected by .bin.gz. Compression is off by default. Native Array datasets are unaffected by this option.

Cached data

An implementation MAY retain external payloads in memory, including data supplied when an external DCI is constructed. Such caches are nonsemantic.

A cache MUST NOT be trusted merely because it is associated with a DCI. Cached bytes MUST satisfy the same digest and, where applicable, ExternalArray validation requirements before they are returned as verified content or persisted as a valid external resource.

An implementation MAY reuse the result of a prior successful verification when it can ensure that the verified bytes have not changed.

Writing and packaging

Representation writing

A conforming writer MUST emit the selected representation of the DCI according to its governing format specification. It MUST NOT change semantic constituents, simplify mathematical structures, or replace external DCTs with their materialized values.

A writer MAY accept an instance of a conventional type and first convert it to a DCI under implementation-defined conversion rules. That conversion is outside this specification; the persistence requirements apply to the resulting DCI.

Filesystem-backed complete packages

When asked to produce a complete .dcs or .dcb package, a writer MUST make each referenced external payload available beneath the destination representation’s containing directory at the DataCanon relative path specified by its DCI.

The writer MUST verify each payload before accepting it for output. It MUST preserve the relative path, digest, Array type, and shape constituents of every external DCI.

A writer MAY obtain the payload from a verified cache, an existing filesystem source, or a supported container source. Copying the bytes to a new namespace MUST NOT change the DCI.

A writer MUST NOT silently omit required external payloads while claiming to have produced a complete package.

Writing only a .dcs or .dcb DCI representation without copying its external payloads MAY be supported as a metadata-only export. Such an export can be a valid representation of the DCI, but it is not necessarily a complete package. An implementation SHOULD make this distinction clear to its caller.

HDF5 complete packages

A conforming HDF5 writer MUST follow the HDF5 Version 1 rules for collecting, verifying, and embedding referenced external bytes in /_external. In particular, it MUST reject incompatible references to the same relative path.

A complete HDF5 package contains the DCI hierarchy and all required embedded external payloads. The semantic external DCI paths and digests remain unchanged by embedding.

A reader MAY reconstruct the DCI without accessing those embedded bytes. Completeness of the package and eagerness of the reader are separate concerns.

Multiset traversal

External-resource discovery MUST include every member occurrence of a Multiset, recursively through nested constituent structures. Retained source context MUST also reach external DCIs inside Multisets.

Persistence MUST preserve member multiplicity and nesting. Multiple occurrences of the same external reference do not require duplicate payload files or embedded payload entries: one resource location satisfies all consistent references to its relative path. This resource sharing MUST NOT remove or merge member occurrences in the represented DCI.

Repeated paths and conflicting references

Within one complete package, two external DCIs with the same relative path refer to one resource location. A conforming writer MUST ensure that all references to that path are consistent with one shared byte sequence.

In particular, if their declared digests differ, the writer MUST reject complete-package creation. Even when digests agree, every ExternalArray reference to that path MUST be compatible with the shared payload’s type, shape, and element-stream validity.

Identical payload bytes referenced by different relative paths remain different resource locations. A writer MUST preserve the paths; it MUST NOT silently merge their semantic references into one path.

Overwriting and failure safety

Writers SHOULD verify all required source payloads and detect conflicting references before overwriting a destination representation or external resource. A writer SHOULD use temporary files, atomic replacement where supported, or equivalent safeguards to avoid leaving partially written packages.

The exact transaction mechanism is implementation-specific. A failed write MUST NOT be reported as a successful complete-package write.

A writer SHOULD avoid destructive overwrites of unrelated files in the destination resource namespace. Applications are responsible for choosing appropriate destination directories and overwrite policies.

Identity-preserving transfers

A transfer between supported representations consists conceptually of:

1. reconstructing a DCI from the source representation;
2. retaining or supplying a source namespace for any external resources;
3. verifying required payloads when preparing a complete destination package; and
4. writing an identity-preserving destination representation and its required resources.

For example:

runs/run42/calculation.dcs
              |
              v
      DCI + source context
              |
              v
archive/calculation.h5

If the source DCI contains an ExternalArray, the destination HDF5 file embeds its verified element bytes but retains the ExternalArray DCT, relative path, Array type, shape, and digest.

The reverse transfer from HDF5 to .dcs or .dcb materializes verified embedded resources into files under the destination directory when a complete package is requested. Their relative paths are preserved.

For equal source and destination DCIs x and y:

E(x) = E(y)
F(x) = F(y)

where E and F are the canonical encoding and fingerprint defined by Core Version 1. The surrounding file bytes need not match across formats.

A transfer MUST NOT silently resolve conflicts by changing paths, digests, or DCTs. If an identity-preserving complete transfer cannot be performed, it MUST fail or explicitly produce a metadata-only export when the caller has selected that behavior.

Fingerprints and integrity metadata

A persistence format MAY carry a DCI fingerprint as nonsemantic metadata. The fingerprint is always SHA-256 of the DCI’s canonical encoding, not of the bytes of the .dcs or .h5 file.

The HDF5 Version 1 specification defines its optional file-root fingerprint attribute. This Persistence specification does not introduce an additional required fingerprint field for .dcs or .dcb.

A reader MAY check a stored fingerprint. It MUST NOT treat the fingerprint as an additional DCI constituent or use it to replace the DCT-defined equality relation.

The fingerprint of an ExternalFile or ExternalArray DCI covers its path and declared content digest, not the external payload bytes directly. External payload verification remains a separate operation.

Errors and resource limits

A conforming reader MUST reject invalid representation syntax, invalid canonical bytes, or invalid HDF5 DCI structure according to the governing format specification. It MUST NOT silently coerce malformed data into a different DCI.

An implementation MAY enforce practical limits on input size, nesting depth, Array allocation, number of external references, or payload size. Such limits do not redefine DataCanon identity.

External content MUST be treated as untrusted until verified. A verified digest does not make an external file’s application-level contents safe to execute or interpret. DataCanon Persistence does not require execution of external content or of any code contained in Shorthand or YAML input.

Error types, diagnostic wording, caching strategy, concurrency controls, and source-handle representation are implementation details.

Interoperability examples

Directory-backed Shorthand

experiment/
├── structure.dcs
└── data/
    └── forces.bin

structure.dcs contains an ExternalArray with relative path ("data", "forces.bin"), a portable Array type and shape, and the SHA-256 digest of forces.bin’s canonical element bytes.

Reading structure.dcs reconstructs the ExternalArray metadata without reading forces.bin. Materialization resolves the relative path under experiment/, verifies its digest and element validity, and produces the numerical data.

Raw canonical bytes

A .dcb file containing the canonical encoding of ExternalFileDC is valid even if the external file is absent. Decoding recovers the relative path and digest. The missing payload prevents materialization, not decoding or fingerprinting.

HDF5 embedded resources

experiment.h5
  /root                  DCI hierarchy
  /_external/0/path      Relative-path components
  /_external/0/data      Verified payload bytes

The HDF5 reader recovers external DCIs from /root without eagerly loading /_external/0/data. When materialization is requested, the resource is located by its stored path components and its digest is verified.

Cross-format transfer

experiment/structure.dcs
             |
             v
        experiment.h5
             |
             v
archive/structure.dcb
archive/data/forces.bin

A complete transfer preserves the DCI’s canonical encoding and fingerprint, despite differences in the storage representation and external-payload location.

Cross-implementation conformance

Independent implementations, including Python and Julia, SHOULD test:

1. Round trips of all 24 DCTs through .dcs, .dcb, and .h5 where supported by the corresponding representation specification.
2. Preservation of DCI identity and canonical fingerprints across format changes.
3. Metadata-only reads of external DCIs when referenced payloads are missing.
4. Directory-relative resolution for .dcs and .dcb sources.
5. HDF5 path-based lookup of embedded payloads, independent of entry enumeration order.
6. ExternalFile digest verification and ExternalArray digest, length, and element validation.
7. Cached-data verification and explicit source overrides.
8. Complete transfer between directory-backed and HDF5-backed representations.
9. Detection of shared-path conflicts and rejection of incomplete complete-package writes.
10. Distinction between valid metadata-only exports and complete packages.
11. Multiset multiplicity and nesting across all formats, including deferred retrieval, shared resources, and conflicting external references within Multiset members.
12. Preservation of Sum/Product multiplicity and nesting, unreduced Power exponent constituents, Quotient constituent roles, and Complex components using all permitted numerical DCTs.

Cross-language fixtures SHOULD compare reconstructed DCIs and canonical encodings, not the physical bytes of HDF5 files or the whitespace of Shorthand documents.

Version 1 finalization notes

The following points should be confirmed against the current Python reference reader.py, writer.py, and external.py before this draft is declared final:

• Format dispatch: verify exactly which suffix aliases and explicit-format overrides are supported. This draft standardizes .dcs, .dcb, and .h5 as the portable convention.
• Directory-backed packaging: verify whether the current generic writer always copies external payloads, offers metadata-only export, or supports both. The distinction in this draft is normative for implementations claiming complete-package support; it does not assert that the Python API currently exposes both modes.
• Default source precedence: verify how cached bytes, retained _source context, explicitly supplied sources, and destination directories are selected. Source precedence is not specified as a universal rule here; every accepted payload must still be verified.
• HDF5 source fallback: confirm whether an explicit alternate source can override an unavailable embedded payload, and whether that should be a portable option or remain application policy.
• Filesystem transactionality: test that conflicts and missing resources are detected before destructive destination writes, especially for directory-backed output.
• Source metadata propagation: confirm that external nodes read from any supported format retain enough nonsemantic source context for deferred materialization and later complete-package transfer.
• Source namespace security: define implementation guidance for symlinks and path traversal through filesystem indirection without changing the Core relative-path DCT.
• YAML persistence: decide whether .yaml/.yml should be added to generic persistence dispatch or remain explicit YAML interchange operations.

The finalization notes identify focused interoperability and implementation-audit items, not reasons to modify DCI identity or canonical bytes.

Draft encoding revision

The current draft encodes fixed integer and Unicode information directly, without redundant Integer or String DCT identifiers. Canonical bytes and fingerprints of affected DCIs differ from earlier drafts, including DCIs that contain affected values. No automatic detection or migration of previous draft .dcb encodings is defined. Regenerate those files from the corresponding DCI or another identity-preserving representation. HDF5 fixed fields retain their payload storage types and no longer require kind attributes.