Persistence and External Resources

A scientific dataset often combines a small description with a large numerical Array or an existing file. DataCanon can store the data together or preserve an explicit reference to external content. The DCI retains its identity as a complete dataset moves between a directory and an HDF5 container.

This guide constructs a dataset with an external Array, writes a complete package, retrieves its numerical contents, and transfers the package. The Identity and Fingerprinting guide explains what its identifier covers.

Decide what belongs inside the DCI

An Array DCI contains its numerical elements. An ExternalArray DCI contains the information needed to identify and interpret an external element stream: a relative path, portable Array type, shape, and SHA-256 content digest.

Array and ExternalArray are distinct DCTs. Converting an Array into an external reference changes the DCI. Moving an existing ExternalArray’s payload between supported storage representations can preserve its identity.

Construct a dataset with an external Array

Force-constant matrices provide a useful example: a calculation may produce a large matrix with many zeros, accompanied by relatively small descriptive data. Use a small matrix here to keep the example easy to run:

import numpy as np

from data_canon import ExternalArray, as_dc_instance, fingerprint

forces = np.zeros((32, 32), dtype="float64")
forces[0, 0] = 2.5
forces[0, 1] = -0.0

dataset = as_dc_instance({
    "description": "Force-constant example",
    "force_constants": ExternalArray(
        forces,
        filename="forces",
        compression=True,
    ),
})
dataset_id = fingerprint(dataset)

The conventional ExternalArray object supplies the numerical data during conversion. The resulting ExternalArray DCI records the path ("external_dc", "forces.bin.gz"), type float64, shape (32, 32), and content digest. The implementation retains the supplied data as a nonsemantic cache so the writer can persist it.

The content digest covers the uncompressed canonical Array element bytes. Those bytes contain only elements: no type, shape, or DCI header. A .bin or .bin.gz payload is therefore different from a .dcb file, which contains one complete canonical DCI encoding.

Write a complete directory package

from pathlib import Path
from data_canon import read, write

source = Path("force_example") / "dataset.dcs"
write(dataset, filename=source)

The writer creates the Shorthand representation at force_example/dataset.dcs and the compressed payload at force_example/external_dc/forces.bin.gz. Together, the representation and all referenced payloads form a complete package.

Use dataset.dcb instead to store the DCI as canonical binary bytes. Both directory-backed formats resolve relative paths beneath the representation’s containing directory by default.

The current Python write() verifies and persists required resources. It does not expose a metadata-only export option. Missing resources or conflicting contents can cause a write to fail. An existing external file with conflicting contents is not silently overwritten.

Read metadata before retrieving content

restored = read(source)
assert restored == dataset
assert fingerprint(restored) == dataset_id

Reading reconstructs the DCI hierarchy without materializing external payloads. The dataset’s metadata can be read even if the payload is unavailable. The reader retains the representation’s source location as nonsemantic context for later retrieval.

Converting the restored dataset to conventional objects requests the external Array’s numerical contents:

from data_canon import dc_instance_to_python

conventional = dc_instance_to_python(restored)
loaded_forces = conventional["force_constants"]
assert np.array_equal(loaded_forces, forces)
assert np.signbit(loaded_forces[0, 1])

Materialization decompresses the payload, verifies its digest, and validates its type, shape, required byte length, and element validity. It returns a NumPy array; the original DCI remains an ExternalArray DCI.

A missing file, malformed compressed stream, incorrect digest, or invalid element stream causes retrieval to fail. The metadata DCI and its fingerprint remain unchanged.

Transfer the package to HDF5

container = Path("force_example") / "dataset.h5"
write(restored, filename=container)

embedded = read(container)
assert embedded == dataset
assert fingerprint(embedded) == dataset_id

The HDF5 writer retrieves and verifies the source payload before embedding it. The container holds the DCI hierarchy and an embedded payload collection, so the complete package can travel as one file. External references retain their paths and digests; they are not converted into Array DCIs.

Reading the HDF5 hierarchy still does not materialize its external Arrays. Subsequent conversion retrieves the embedded payload by its complete relative path and verifies it.

Export the package back to a directory:

destination = Path("force_export") / "dataset.dcb"
write(embedded, filename=destination)

assert (destination.parent / "external_dc" / "forces.bin.gz").exists()
assert read(destination) == dataset

The writer recreates the compressed external file using the same relative path. The Shorthand, HDF5, and binary representations recover equal DCIs and therefore share a fingerprint. Their physical file bytes need not agree.

Understand compression and identity

ExternalArray(..., compression=True) selects .bin.gz storage. With compression=False, the default, a filename stem receives .bin instead. Explicit .bin and .bin.gz suffixes must match the compression setting.

In a directory, .bin.gz stores a gzip stream. In HDF5, the same reference uses a gzip-compressed dataset containing the logical element bytes. The HDF5 library handles dataset decompression; the implementation does not interpret those returned bytes as another gzip file.

Changing forces.bin to forces.bin.gz changes the path and therefore changes DCI identity. Once the path is chosen, transferring that reference between directory and HDF5 storage preserves it. Compression does not change the content digest’s definition.

The option applies to ExternalArrays. It does not automatically compress every Array or ExternalFile, or introduce support for .dcb.gz representations.

Keep opaque files opaque

Use an ExternalFile when the payload is a file whose interpretation belongs to the application:

from data_canon import ExternalFile

notes = as_dc_instance(ExternalFile.from_bytes(
    b"Calibration instrument: A17\n",
    filename="calibration.txt",
))
write(notes, filename=Path("force_example") / "calibration.dcb")

An ExternalFile digest covers the file’s exact bytes. A .gz filename does not cause DataCanon to reinterpret that digest as a digest of decompressed content. The Python conversion interface can apply a filename-based content loader after verification; for example, .txt content becomes text. Such interpretation is separate from the external DCI’s identity.

Move resources without changing references

Move a directory package with its relative resource tree intact. Moving only dataset.dcs or dataset.dcb may leave a valid metadata representation whose payloads cannot be found in the new directory.

If a DCI was decoded without a source location, supply the namespace explicitly when converting it:

from data_canon import canonical_decode, canonical_encode

detached = canonical_decode(canonical_encode(dataset))
recovered = dc_instance_to_python(detached, source=source.parent)
assert np.array_equal(recovered["force_constants"], forces)

The source selects where to look. The DCI still determines which bytes are acceptable through its digest and Array metadata. Source context never changes equality or fingerprinting.

Within a complete package, references to the same path must agree on one shared payload. Different paths remain distinct locations, even when their payloads agree. Writers reject conflicting references rather than rename paths or select one payload silently.

For method signatures, see the Python API. The Persistence specification and HDF5 specification define the portable requirements for independent implementations.