HDF5

DataCanon HDF5 — Version 1 Draft

Status: Draft for independent implementation review
Depends on: DataCanon Core Version 1
Scope: Identity-preserving HDF5 representation of DataCanon Instances (DCIs), including embedded external resources
Out of scope: Canonical byte encoding, Shorthand and YAML syntax, filesystem packaging conventions, and application-specific source-resolution APIs

Introduction

DataCanon HDF5 defines a portable representation of DataCanon Instances (DCIs) in HDF5 files. Its purpose is to permit independent implementations, including Python and Julia, to exchange structured scientific data while preserving DataCanon identity.

This specification defines the logical HDF5 objects, required names, attributes, data types, ordering conventions, and handling of externally referenced resources. It does not define new DataCanon Types or change their equality relations. Those are defined by DataCanon Core Version 1.

A conforming HDF5 writer MUST produce a file from which a conforming reader reconstructs an equal DCI. Files representing equal DCIs need not have identical HDF5 bytes. Compression, chunking, storage order, object addresses, and other physical details MAY differ without changing DCI identity.

This specification describes the interoperable layout used by the Python reference implementation (h5py and NumPy). It is written in terms of HDF5 objects and values, not Python classes.

Conformance

The terms MUST, MUST NOT, SHOULD, and MAY indicate normative requirements and recommendations.

Let H(x) be a conforming HDF5 representation of DCI x, and P_H its interpretation as a DCI. The primary requirement is:

P_H(H(x)) = x

Equality here is DataCanon equality, including DCT distinctions, structural numerical constituents, signed floating-point zeros, Array types and shapes, ordering of ordered collections, and external-resource metadata.

A reader MUST NOT change a DCI merely to fit the conventions of its implementation language. A reader MAY reject malformed or unsupported HDF5 input; it MUST NOT silently substitute a different DCI.

File layout

A DataCanon HDF5 file contains a required top-level object named root and may contain a top-level group named _external.

/
├── root                 required; representation of one DCI
└── _external            optional; embedded external byte resources
    ├── 0
    │   ├── path
    │   │   ├── 0
    │   │   └── ...
    │   └── data
    └── ...

The HDF5 file root group MAY have a fingerprint attribute as described below.

The root object MUST represent exactly one DCI, which may contain arbitrarily nested DCIs. It may be an HDF5 group or dataset, depending on its DCT.

Except for the structural helper groups explicitly identified in this specification, each HDF5 object representing a DCI MUST have a scalar string attribute named kind, with the exact value specified for its DCT. Structural helper groups (for example, mapping-entry groups and path groups) do not represent DCIs and do not require kind.

Object names and DCT kind strings are case-sensitive.

General storage conventions

Ordered children

Where a group represents an ordered sequence, its children are named by consecutive zero-based decimal indices:

0, 1, 2, ..., N-1

Indices MUST be written without leading zeros, except for 0. Semantic order is numeric index order, not HDF5 group iteration order or lexicographic name order. Empty sequences have no indexed children.

The same convention applies to List and Tuple items, Set and Multiset member occurrences, Mapping and UnorderedMapping entries, Record fields, path components, and Array shape dimensions stored as children.

Strings

A String DCI is represented as a one-dimensional HDF5 dataset of unsigned 8-bit integers containing the exact UTF-8 bytes of the string, with kind = "string".

No terminating NUL is added. Embedded NUL bytes are permitted as part of the UTF-8 encoding of a valid String DCI. An empty String is represented by an empty dataset of shape (0,).

A reader MUST decode the bytes as valid UTF-8 and apply the String invariants from Core Version 1. No Unicode normalization is performed.

This byte-array representation is intentional: it avoids restrictions of HDF5 variable-length string interfaces on embedded NUL characters.

Bytes

A Bytes DCI is represented as a rank-one HDF5 dataset of unsigned 8-bit integers containing the exact byte sequence, with kind = "bytes". An empty Bytes DCI has dataset shape (0,). No text decoding, terminating NUL, digest, or framing metadata is applied to its contents.

A reader MUST reject Bytes objects that are groups, have a null dataspace, have rank other than one, or use a datatype other than unsigned 8-bit integers. Enumeration datatypes, including Boolean enums, are not valid Bytes datasets.

The kind distinguishes Bytes from String and uint8 Array even when their stored element bytes happen to agree. Bytes payloads are stored directly in the DCI hierarchy, not in the external-resource collection.

Arbitrary-precision integers

An Integer DCI is represented as a scalar HDF5 UTF-8 string dataset containing its value in base-ten notation, with kind = "integer".

A conforming writer MUST use ordinary decimal notation: optional leading - for negative values, followed by ASCII digits; no leading +, exponent, separators, or unnecessary leading zeros. Zero is written as 0. The value is not restricted to a fixed machine-integer width.

A reader MUST recover the mathematical integer exactly. This representation is deliberately different from an HDF5 native integer dataset.

Fixed-width numerical values

Float DCIs use scalar HDF5 IEEE-754 binary64 datasets. The exact finite binary64 value, including the sign of zero, MUST be preserved.

For native numerical datasets, HDF5 datatype endianness and storage layout MAY differ, provided the logical values and their required IEEE-754 bit identities are recovered exactly. Nonfinite values are not valid Version 1 DCIs.

Fixed-domain information fields

Prescribed integer and Unicode fields are not DCI objects and carry no required kind attribute. Writers MUST omit kind on those fields. Integer information uses scalar UTF-8 decimal datasets with the same payload rules as Integer; string information uses rank-one uint8 UTF-8 datasets with the same payload rules as String. Readers validate the payload directly rather than recursively interpreting the field as a DCI. Additional nonsemantic attributes may be ignored.

This applies to Fraction numerator/denominator, Power exponent fields, Record and field names, path components (including embedded-resource paths), external Array type names, and external Array shape dimensions. The Power base, Record field values, Complex components, Quotient operands, and collection members remain full DCI representations with required kind attributes.

Required structure

The tables below define the layout produced by a conforming writer. A conforming reader MUST recognize these layouts and MUST reject input that cannot be interpreted as a valid DCI without loss of identity. Additional application-specific HDF5 metadata MAY be present, but MUST NOT change the interpretation of the specified DataCanon objects.

Primitive DCTs

The Boolean dataset MUST represent exactly false or true; arbitrary nonzero integers are not a valid substitute for the Boolean DCT. Implementations using the HDF5 Boolean enumeration convention MUST recognize FALSE and TRUE.

Constant

A Constant is a group with kind = "constant" and exactly one child dataset, name. The dataset contains direct String information, stored as rank-one uint8 UTF-8 bytes using the String information rules; it is not a String DCI. Its decoded value MUST be exactly pi or e. Unknown names, missing fields, extra child objects, and incorrect dataset layouts MUST be rejected.

Fraction

A Fraction is a group with kind = "fraction" and two fixed integer fields:

fraction/
├── numerator       Decimal integer information
└── denominator     Decimal integer information

The denominator MUST be positive. Neither constituent is reduced or normalized.

Atomic DCTs

Power

A Power is a group with kind = "power", one base DCI, and two fixed integer fields:

power/
├── base               Scalar Numerical DCI
├── numerator          Positive decimal integer information
└── denominator        Positive decimal integer information

All children MUST be present, including denominator one. A reader MUST verify the base DCT and exponent positivity. Exponent constituents MUST NOT be reduced and nested base structure MUST be preserved.

Quotient

A Quotient is a group with kind = "quotient":

quotient/
├── numerator          Scalar Numerical DCI
└── denominator        Scalar Numerical DCI

Both children MUST be present and have permitted numerical DCTs. Readers MUST preserve their roles and structure without division or simplification. The Core Quotient invariants determine validity; HDF5 introduces no additional mathematical denominator check.

Complex

A Complex is a group with kind = "complex":

complex/
├── real            Scalar Numerical DCI
└── imaginary       Scalar Numerical DCI

The real and imaginary components are themselves complete DCI representations. A Complex DCI is distinct from an Array element of a complex Array type.

Composite collections

List, Tuple, and Set

List, Tuple, and Set DCIs use groups whose kind values are respectively list, tuple, and set. Each contains indexed child DCIs named 0, 1, and so forth.

collection/            kind = "list", "tuple", or "set"
├── 0                  DCI
├── 1                  DCI
└── ...

For List and Tuple, numeric child order is semantic and MUST be preserved. For Set, member order is not semantic; a conforming writer MAY choose any indexed member order, and a reader MUST construct a Set DCI with distinct members according to Core Version 1.

The HDF5 representation of a Set does not need to use canonical byte ordering. Canonical ordering belongs to the canonical binary encoding, not to HDF5 storage.

Sum and Product

Sum and Product are groups with kind = "sum" and kind = "product", respectively. Each operand occurrence is a complete DCI child named by its consecutive zero-based index.

An empty instance has no indexed children. Each duplicate occurrence MUST be stored separately. Child order is not semantic and need not match canonical byte order. Every Sum operand MUST be a Numerical DCI. Every Product operand MUST be a Scalar Numerical DCI, including the recursive restrictions on Sum and OrderedProduct. Array rank and shape are not Sum invariants. Readers MUST preserve nested nodes and MUST NOT deduplicate or simplify them.

OrderedProduct

An OrderedProduct is a group with kind = "ordered_product". Each operand is stored as a complete indexed DCI child named 0, 1, and so forth. The consecutive index order is semantic and MUST be retained. Every operand MUST be a Numerical DCI. Empty, repeated, scalar-only, and nested operands are permitted. No sorting or evaluation is performed. No Array rank or shape compatibility condition is imposed on this representation.

Multiset

A Multiset is a group with kind = "multiset". Each indexed child represents one complete member DCI:

multiset/              kind = "multiset"
├── 0                  DCI
├── 1                  DCI
└── ...

Child names follow the consecutive numeric-index convention. An empty Multiset is a group with no indexed children. Each repeated occurrence MUST have its own indexed child; readers MUST NOT deduplicate equal members.

Numeric child order does not participate in Multiset identity. A conforming writer MAY choose any member order, and canonical byte ordering is not required for HDF5 storage. Nested Multisets MUST retain their DCT and structure.

External DCIs within Multiset members MUST be discovered for payload collection and source-context attachment, including recursively nested members. Repeated references to one relative path preserve all member occurrences while sharing one embedded payload under the existing path-consistency rules.

Mapping and UnorderedMapping

Mapping and UnorderedMapping DCIs use groups with kind = "mapping" and kind = "unordered_mapping", respectively. Each indexed child is a structural entry group containing two complete DCIs:

mapping/               kind = "mapping" or "unordered_mapping"
├── 0/
│   ├── key            DCI
│   └── value          DCI
├── 1/
│   ├── key            DCI
│   └── value          DCI
└── ...

For Mapping, numeric entry order is semantic. For UnorderedMapping, entry order is not semantic. Keys MUST be distinct according to DataCanon equality, which need not coincide with the equality rules for keys of a conventional dictionary type in the implementation language.

Entry groups do not require a kind attribute.

Record

A Record is a group with kind = "record":

record/
├── name               UTF-8 byte information
└── fields/
    ├── 0/
    │   ├── name       UTF-8 byte information
    │   └── value      DCI
    ├── 1/
    │   ├── name       UTF-8 byte information
    │   └── value      DCI
    └── ...

The Record name is semantic. Fields are ordered numerically; their names MUST be unique according to String equality. The fields group and its indexed entry groups are structural helpers without required kind attributes.

Array

An Array DCI is represented as a native HDF5 numerical dataset with kind = "array". Its HDF5 datatype, dataspace shape, and element values together determine the DataCanon Array type, shape, and elements.

A conforming reader and writer MUST support the 14 portable Array types of Core Version 1:

For the complex types, the compound fields r and i represent real and imaginary components respectively, in that order. The field names and component widths match the Python h5py reference representation and are part of the portable interchange layout. A reader MUST reconstruct the corresponding DataCanon complex Array type, not a Record or a structured Array of unrelated fields.

The Array rank and shape are taken from the HDF5 dataspace. Rank-zero Arrays use scalar dataspaces. Zero-sized dimensions are permitted and MUST be preserved. A rank-zero Array is not the same DCI as a rank-one Array containing one element.

The element ordering is the logical ordering of the HDF5 dataspace, consistent with the Core Version 1 Array definition. An implementation MUST preserve the exact logical numerical elements, including signed floating-point zeros and the real and imaginary components of complex elements. HDF5 chunking, compression, and physical data ordering do not participate in DCI identity.

A conforming writer MUST NOT store a portable Array as a nested collection of DCI groups merely because that is easier for its implementation. The native numerical dataset is a defining feature of this representation.

External DCIs

General principle

An ExternalFile or ExternalArray remains an external DCT even when its bytes are embedded in an HDF5 file. Embedding payloads MUST NOT transform ExternalArray into Array or ExternalFile into String or Array.

The semantic constituents are represented in the root DCI tree. The external payload bytes are stored separately in the top-level _external group.

Relative path representation

A DataCanon relative path is represented as a structural group named path, containing indexed UTF-8 byte datasets for its path components:

path/
├── 0                 UTF-8 byte information
├── 1                 UTF-8 byte information
└── ...

A path MUST have at least one component and MUST satisfy the Core Version 1 relative-path invariants. The path group itself has no kind attribute. The path is not stored as a host-operating-system path string.

Digest representation

A SHA-256 content digest is stored in a dataset named digest containing exactly 32 unsigned 8-bit integers, in digest byte order. The digest is not stored as hexadecimal text.

ExternalFile

An ExternalFile is a group with kind = "external_file":

external_file/
├── path/              Relative-path component group
└── digest             uint8 dataset, shape (32,)

The content digest is the SHA-256 digest of the exact external file bytes.

ExternalArray

An ExternalArray is a group with kind = "external_array":

external_array/
├── path/              Relative-path component group
├── array_type         UTF-8 byte information
├── shape/             Indexed decimal integer datasets
└── digest             uint8 dataset, shape (32,)

The array_type UTF-8 byte dataset contains one of the 14 portable Array type names. The shape group contains indexed decimal integer datasets, one per dimension; an empty group denotes rank zero.

The content digest is the SHA-256 digest of the canonical Array element byte stream, not the complete canonical encoding of an Array DCI. The logical element bytes contain no dtype, shape, header, or other framing information. Directory gzip envelopes are not embedded as the logical payload.

Embedded external payloads

Structure

When external resources are present, a conforming HDF5 writer embeds their verified bytes under the top-level group _external:

_external/
├── 0/
│   ├── path/           Indexed UTF-8 byte datasets
│   └── data            Rank-one uint8 dataset
├── 1/
│   ├── path/
│   └── data
└── ...

Each indexed _external child is a structural group, not a DCI. Its path group uses the same relative-path component encoding as above. Its data dataset contains the logical resource bytes as a rank-one uint8 dataset, including empty payloads when applicable. ExternalFiles contribute their opaque bytes; ExternalArrays contribute their verified, uncompressed canonical element streams.

The numeric index of an embedded payload is not semantic. A reader locates an embedded resource by matching the full sequence of relative-path components, not by assuming that a particular DCI points to a particular numeric index.

ExternalArray compression

The reference writer applies HDF5 gzip compression (level 6, chunked storage) to embedded ExternalArray payload datasets whose logical path ends in .bin.gz. It does not store a gzip-file envelope in the dataset. Dataset reads return the canonical element bytes directly; the content digest covers those bytes. Scalar Arrays and empty Arrays still use rank-one byte payload datasets, so the same storage rule applies. .bin ExternalArray payloads and native Array DCIs remain uncompressed by default.

Compression filters and physical layout are nonsemantic. Readers reconstruct the logical bytes through HDF5 and validate them normally, regardless of the physical filters in a valid input dataset. Readers MUST NOT perform another gzip decompression based on the logical resource path. Directory export uses the path’s .bin/.bin.gz suffix to recreate the required storage representation.

Uniqueness and consistency

A conforming writer MUST embed each distinct referenced relative path at most once. If multiple external DCIs refer to the same relative path, their digests MUST agree and the referenced bytes MUST be consistent with each DCI’s requirements.

If two references use the same path but require conflicting bytes or digests, a conforming writer MUST reject the attempted file creation rather than silently select one payload.

An embedded ExternalArray logical payload MUST satisfy its declared Array type and shape, including element-byte length, canonical Boolean bytes, and finite floating-point/complex elements. A compressed ExternalArray and an opaque ExternalFile MUST NOT share a relative path; their directory storage requirements conflict even if their digests agree.

Writing and verification

Before embedding external bytes, a writer MUST verify that their SHA-256 digest equals the digest stored in the corresponding DCI. It MUST NOT embed unverified bytes as though they satisfy the reference.

The Python reference writer collects and verifies all referenced external payloads before creating or overwriting the destination HDF5 file. This preflight behavior is recommended because it prevents missing or conflicting resources from destroying an existing file. Other implementations MAY use different transactional mechanisms providing equivalent integrity safeguards.

A writer may obtain bytes from a verified in-memory cache, an existing filesystem resource, or another supported representation containing the resource. The choice of source does not change the resulting DCI.

Reading and deferred access

Reading the HDF5 DCI hierarchy MUST reconstruct ExternalFile and ExternalArray DCIs from their semantic metadata without automatically materializing their external payload bytes.

A conforming reader SHOULD retain sufficient nonsemantic source context to resolve embedded resources on later request. Such context does not participate in DCI equality or canonical encoding.

When an implementation does retrieve an embedded payload, it MUST verify its digest against the DCI before accepting the bytes. For ExternalArray it MUST additionally validate the declared Array type, shape, byte length, and element validity.

A missing or corrupt embedded payload does not change the identity of the metadata DCI; it prevents successful retrieval or verification of the referenced content.

The general semantics of resolving external resources across different storage representations belong to the separate DataCanon Persistence specification.

Optional fingerprint attribute

A writer MAY store a scalar file-root attribute named fingerprint. Its value is the 64-character lowercase hexadecimal textual representation of the SHA-256 fingerprint of the DCI stored at /root, as defined by Core Version 1.

The attribute is metadata, not an additional constituent of the DCI. Its presence or absence does not affect identity.

The Python reference reader does not currently use this attribute to validate the file. A conforming reader MAY verify it and MUST NOT treat a mismatching fingerprint as evidence that two unequal DCIs are equal.

Reading, writing, and validity

A conforming writer MUST produce the DCI layouts and semantic types specified above. A conforming reader MUST support every required Version 1 layout and reconstruct the same DCI identity.

Readers MUST reject absent /root, unknown kind values, missing required constituents, invalid constituent DCTs, invalid relative paths, and numerical or structural data that cannot represent a valid DCI. They MUST NOT silently repair or simplify invalid DCIs.

A reader MAY ignore additional nonsemantic attributes or groups, provided they do not create ambiguity about the required DCI structure. The strictness of rejecting all extraneous structural objects is not yet fully specified; conforming writers SHOULD emit only the defined structural children.

External payload verification is distinct from decoding the DCI hierarchy. A reader need not access embedded payload bytes merely to construct a metadata-only external DCI.

Illustrative HDF5 trees

Scientific Mapping

/
└── root/                        kind = "mapping"
    ├── 0/
    │   ├── key                  kind = "string", UTF-8 bytes "lattice"
    │   └── value                kind = "array", shape (3, 3), float64
    └── 1/
        ├── key                  kind = "string", UTF-8 bytes "scale"
        └── value/               kind = "fraction"
            ├── numerator        decimal integer text "1"
            └── denominator      decimal integer text "2"

ExternalArray

/
├── root/                        kind = "external_array"
│   ├── path/
│   │   ├── 0                   UTF-8 bytes "external_dc"
│   │   └── 1                   UTF-8 bytes "forces.bin"
│   ├── array_type              UTF-8 bytes "float64"
│   ├── shape/
│   │   ├── 0                   decimal integer text "100"
│   │   └── 1                   decimal integer text "3"
│   └── digest                  uint8[32]
└── _external/
    └── 0/
        ├── path/
        │   ├── 0               UTF-8 bytes "external_dc"
        │   └── 1               UTF-8 bytes "forces.bin"
        └── data                uint8[2400]

The example assumes float64 elements (8 bytes each), shape (100, 3), and therefore 2,400 canonical element bytes.

Cross-implementation conformance

Implementations SHOULD test at least the following cases:

1. All eight primitive DCTs, including both named Constants, unreduced Fractions, empty and arbitrary Bytes sequences, arbitrary-precision Integers, Unicode Strings, embedded NULs, and positive versus negative Float zero.
2. All three atomic DCTs: Power with a Scalar Numerical base and positive integer exponent fields, Quotient with Scalar Numerical operands, and Complex with components of different DCTs.
3. All collection DCTs, including empty collections, ordered versus unordered semantics, Multiset and numerical Sum/Product multiplicities and nesting, and keys that would collide under ordinary programming-language equality. OrderedProduct tests MUST distinguish reversed operand order and preserve Array operands without rank checks.
4. Record names and field names containing Unicode and spaces; preservation of field order.
5. All 14 portable Array types; scalar Arrays; empty dimensions; signed zeros; complex components; native HDF5 dataset interoperability.
6. ExternalFile and ExternalArray metadata round trips without eager payload loading.
7. Embedded payload retrieval, digest verification, shared paths, and rejection of conflicting paths.
8. Optional fingerprint metadata, without making it part of identity.
9. Rejection of malformed files, including missing /root and unknown kind attributes.

Cross-language test fixtures SHOULD include files written independently by Python and Julia. HDF5 file bytes need not match; reconstructed DCIs and their canonical encodings MUST match.

Version 1 finalization notes

The following points warrant focused interoperability testing before declaring this document final; they are not reasons to redesign the representation:

• Native HDF5 datatypes: confirm that Julia’s HDF5 implementation reads and writes the h5py Boolean enum and r/i complex compound layouts with the intended semantics, especially across endianness and floating-point widths.
• Reader strictness: the Python reference reader currently relies on HDF5 conversion and DCI constructors for some validation and does not explicitly reject every possible malformed HDF5 datatype or extra child. The normative reader requirements may therefore require additional validation tests or a narrower portable-input statement.
• Embedded payload lookup: confirm that source-aware loading resolves the /_external entry by complete relative path, independently of enumeration order, and verifies digest and Array element validity on retrieval.
• Version metadata: the current layout contains no mandatory HDF5 format-version attribute. If one is introduced later, it should be nonsemantic and backward-compatible with existing Version 1 files.
• Exact string datatype details: the Integer DCI uses an HDF5 scalar UTF-8 variable-length string, while String DCI uses a uint8 byte dataset. Independent implementations must not confuse the two.

The finalization items should be settled by conformance tests and targeted reference-implementation changes, rather than by altering the DataCanon core identity model.