Identity and Fingerprinting

A calculation’s inputs may appear in a readable text file, a binary archive, and several programming languages. Comparing the files tells you whether their bytes agree. DataCanon lets you ask a different question: do the representations describe the same structured data?

This guide identifies a calculation’s inputs, distinguishes changes in data from changes in presentation, and uses fingerprints to associate results with their inputs. Start with Getting Started for installation and the first persistence workflow.

Identify the inputs to a calculation

Consider an application that evaluates a response at several sampling points:

import numpy as np

from data_canon import as_dc_instance, canonical_encode, fingerprint

inputs = {
    "model": "response-v1",
    "temperature": 300.0,
    "sampling_points": np.array(
        [[0.0, 0.0, 0.0], [0.5, 0.0, 0.0]],
        dtype="float64",
    ),
    ("solver", "tolerance"): 1e-8,
}
instance = as_dc_instance(inputs)
input_id = fingerprint(instance)

The DCI identifies the whole input structure: its keys, values, types, Array shape, and Mapping entry order. The fingerprint is SHA-256 of the canonical encoding, returned as 64 lowercase hexadecimal characters.

Canonical encoding preserves all distinctions in DCI identity. Equal DCIs have identical canonical bytes; unequal DCIs have different canonical bytes. SHA-256 compresses those bytes to a fixed-size identifier. Fingerprint equality is a practical identification mechanism based on collision resistance, not the definition of DCI equality.

Change the presentation without changing the data

Render the input using different layouts, then parse both versions:

from data_canon import parse_shorthand, render_shorthand

readable = render_shorthand(instance)
single_line = render_shorthand(instance, layout="inline")

first = parse_shorthand(readable)
second = parse_shorthand(single_line)

assert first == second == instance
assert canonical_encode(first) == canonical_encode(second)
assert fingerprint(first) == fingerprint(second) == input_id

Whitespace and layout belong to the text representation. They do not change the DCI. The same principle applies when the DCI is stored in HDF5 rather than Shorthand: the physical file bytes can differ while the reconstructed data has the same identity.

Hash a file’s bytes when you need to identify that exact file. Fingerprint its DCI when you need to identify its DataCanon data across representations.

Recognize changes that matter

A change in type can reveal an unintended change in a calculation pipeline:

changed_inputs = dict(inputs)
changed_inputs["sampling_points"] = inputs["sampling_points"].astype("float32")
changed = as_dc_instance(changed_inputs)

assert changed != instance
assert fingerprint(changed) != input_id

All coordinates in this example are exactly representable in both Array types. The type change still changes the DCI. DataCanon retains a distinction that a comparison of numerical values alone could miss.

Other identity rules follow each DCT’s contract:

Choose the DCT that expresses the application’s intended distinctions. If association order should not matter, an UnorderedMapping expresses that choice explicitly. The Reference summarizes all DCT identity rules.

Choose what the identifier covers

An input identifier covers exactly the information placed in its DCI. Adding a run timestamp changes that DCI even when the calculation inputs stay the same. Keep operational metadata separate when it should not affect input identity:

run_record = {
    "input_fingerprint": input_id,
    "inputs": instance,
    "operator_note": "Convergence study, second run",
}
run_instance = as_dc_instance(run_record)

The input DCI remains available unchanged inside the run record. The run record has its own identity, which also covers the note.

Application conventions determine what is required to reproduce a calculation. Include a model version, solver settings, or other dependencies when they affect the result. DataCanon cannot infer missing information or decide that two different parameterizations describe the same physical system. An application can perform its own normalization before constructing a DCI when it wants such equivalences to share an identifier.

Records provide named structures for application conventions. See Custom Conversion Rules for mapping an application’s classes to explicit Record structures.

Associate results with their inputs

A fingerprint is useful as a lookup key for a result cache or provenance record:

results_by_input = {
    input_id: {"response": 12.5, "converged": True},
}
result = results_by_input[fingerprint(instance)]
assert result["converged"]

For a cache to be valid, its key must cover every dependency that can affect the cached result. If a code version or random seed matters, include it in the identified inputs or in a larger cache-key DCI.

Within one Python process, DCIs can also be dictionary keys directly:

point = as_dc_instance(np.array([0.5, 0.0, 0.0], dtype="float64"))
responses = {point: 12.5}
same_point = as_dc_instance(np.array([0.5, 0.0, 0.0], dtype="float64"))
assert responses[same_point] == 12.5

All DCIs are Python-hashable, including Array and List DCIs. Python’s hash() supports dictionary lookup; use fingerprint() for a portable identifier to store or communicate between processes and implementations. Preserve DCI immutability when using a DCI as a key.

External content has a separate integrity check

An ExternalFile or ExternalArray DCI includes a relative path and a declared content digest. Its fingerprint covers that metadata. Constructing the DCI or computing its fingerprint does not retrieve the external bytes.

Materialization verifies the retrieved content against the declared digest. Missing or corrupt content prevents materialization without changing the metadata DCI’s identity. The Persistence and External Resources guide develops that workflow, including complete packages and cross-format transfer.

For the formal encoding and fingerprint contract, see DataCanon Core.