This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Getting Started

    Scientific data moves between programs, readable input files, and storage formats. A different file layout need not mean different data; a changed Array type or coordinate structure can mean different data even when the numbers look the same.

    DataCanon gives structured scientific data a precisely defined identity, independent of its programming language or storage representation. Each DataCanon Instance (DCI) has one unique canonical byte encoding. A fingerprint derived from those bytes lets you identify the data across representations.

    This guide constructs a dataset, identifies it, stores and inspects it, then shows how to author scientific input directly.

    Installation

    Clone the DataCanon code repository, then run these commands from its root:

    python3 -m venv DC
source DC/bin/activate
python -m pip install .

    The Makefile also provides environment setup, editable installation, and tests.

    Construct and identify a DCI

    Start with a Python dictionary containing a Tuple key, a List of species, a coordinate Tuple, and a numerical Array:

    import numpy as np

from data_canon import as_dc_instance, fingerprint

data = {
    ("Eg", 0): 2.37,
    "species": ["Sr", "Ti", "O"],
    "origin": (0.0, 0.0, 0.0),
    "lattice": np.eye(3, dtype="float64"),
}
instance = as_dc_instance(data)
identifier = fingerprint(instance)
print(identifier)

    Output:

    eb8f306c801ca4452e283f052b51a6834a846399cc3aac43720fb7cf48faeca4

    The whole dataset is one DCI: a Mapping containing other DCIs. Mapping keys can be structured values, including the Tuple ("Eg", 0).

    Python’s dict, tuple, and NumPy arrays are conventional types used by the application. Conversion constructs a DCI whose information and equality rules are defined by DataCanon, independently of those Python types.

    The fingerprint is SHA-256 of the DCI’s canonical bytes, written as 64 lowercase hexadecimal characters. You can record it alongside a calculation to identify its input data. It is derived from the data’s defined identity, not from the memory address of a Python object or the layout of a text file.

    Type participates in identity. Replacing the origin Tuple with a List changes the DCI even though its coordinates have the same numerical values:

    changed_data = dict(data)
changed_data["origin"] = [0.0, 0.0, 0.0]
changed_instance = as_dc_instance(changed_data)

assert changed_instance != instance
assert fingerprint(changed_instance) != identifier

    Array type and shape also participate in identity. Equal DCIs have identical canonical bytes and fingerprints; DCI equality itself is defined by the data model, not by comparing fingerprints.

    Write a DCI to disk as binary data

    Use .dcb to store the canonical bytes directly:

    from data_canon import read, write

write(instance, filename="output_data.dcb")
restored = read("output_data.dcb")

assert restored == instance
assert fingerprint(restored) == identifier

    The file is binary, not text. Reading reconstructs the same DCI, including its types, Array shape, and element values. DataCanon decoding reconstructs DCIs without invoking application-defined code.

    Render a DCI as Shorthand

    Shorthand is a human-readable representation of a DCI:

    from data_canon import parse_shorthand, render_shorthand

text = render_shorthand(instance)
print(text)

    Output:

    {
  ( "Eg", 0 ): 2.37,
  "species": [ "Sr", "Ti", "O" ],
  "origin": ( 0.0, 0.0, 0.0 ),
  "lattice":
    array(
      [[1.0, 0.0, 0.0],
       [0.0, 1.0, 0.0],
       [0.0, 0.0, 1.0]]
    )
}

    Parsing the text recovers the same identity:

    parsed = parse_shorthand(text)
assert parsed == instance
assert fingerprint(parsed) == identifier

    Text layout is presentation. Canonical bytes determine the fingerprint, so spacing does not become part of the data’s identity.

    Inspect a DCI via the command line

    Installing the package also installs dc. With the environment active, display the binary file as the Shorthand shown above:

    dc output_data.dcb

    Inspect particular values without writing a Python script:

    dc output_data.dcb --get '("Eg", 0)'
dc output_data.dcb --get species --index 0

    Output:

    2.37
"Sr"

    Print the same fingerprint calculated in Python:

    dc output_data.dcb --fingerprint

    The CLI guide covers searches, selection, and format transfers in more detail.

    Preserve identity through YAML and HDF5

    The same DCI can be represented in other formats without changing its identity. YAML uses native values where faithful and a $ marker for embedded strict Shorthand, including Tuple keys and Array values:

    from data_canon import parse_yaml, render_yaml

yaml_text = render_yaml(instance)
print(yaml_text)

    Output:

    $ ( "Eg", 0 ): 2.37
species:
- Sr
- Ti
- O
origin: $ ( 0.0, 0.0, 0.0 )
lattice: |-
  $
  array(
    [[1.0, 0.0, 0.0],
     [0.0, 1.0, 0.0],
     [0.0, 0.0, 1.0]]
  )

    from_yaml = parse_yaml(yaml_text)
assert from_yaml == instance
assert fingerprint(from_yaml) == identifier

    For numerical storage, HDF5 represents Arrays as native numerical datasets:

    write(instance, filename="output_data.h5")
from_hdf5 = read("output_data.h5")
assert from_hdf5 == instance
assert fingerprint(from_hdf5) == identifier

    Use .dcs with the same write() and read() functions to store Shorthand text. The .dcb, Shorthand, YAML, and HDF5 representations recover equal DCIs and therefore the same canonical bytes and fingerprint. Hashing the YAML or HDF5 file bytes directly would instead identify that particular file representation.

    Author scientific data directly

    So far, we have constructed data in Python. DataCanon also lets us author structured scientific data directly in Shorthand:

    scientific_input = parse_shorthand("""
{
  "position": (1/2, 1/4, 0),
  "phase": complex(1/2, 1/2 * sqrt(3))
}
""")

    The position contains exact Fractions. The Product 1/2 * sqrt(3) exactly represents an irrational quantity, and the Complex retains the structure of its real and imaginary components.

    DataCanon does not do math. Parsing constructs and preserves Fraction, Power, Product, and Complex DCIs. Applications can convert DCIs to instances of conventional types for computation; DC identity belongs to the stored data structure.

    Make scientific input easier to maintain

    Optional Shorthand preprocessing lets an input file use named parameters. An excerpt of a crystal-structure input illustrates the idea: a parameter controls several coordinates while exact reference positions remain visible.

    source = """
u = 0.305 |
{
  "basis_atoms": {
    "O": [
      (u,        u,        0),
      (1/2 +  u, 1/2 + -u, 1/2)
    ]
  }
}
"""

authored = parse_shorthand(source, preprocess=True)
write(authored, filename="structure.dcb")

    Preprocessing substitutes u before strict parsing. The resulting DCI stores the coordinate values and their numerical structure; the parameter definition is an input convenience. The same identity and persistence operations now apply to this authored data.

    The CLI enables preprocessing for .dcs input files. The Shorthand Preprocessing guide develops substitutions, compact Lists, and explicit eval(...) separately, and includes a complete crystal-structure example.

    Next steps