DataCanon

DataCanon is a data model for structured scientific data with deterministic, language-independent identity.

Scientific data often outlives the software that created it. A calculation may move between programming languages, be stored in several formats, and remain in use through years of development. DataCanon defines what the data contains and when two instances are equal, independently of how it is stored or displayed.

Each DataCanon Instance (DCI) belongs to exactly one DataCanon Type (DCT). Its type and information determine its identity. Every valid DCI has one canonical byte encoding; equal DCIs have identical encodings across conforming implementations. SHA-256 of those bytes provides a deterministic fingerprint.

Store, recover, and compare data

The Python reference implementation constructs DCIs from instances of supported conventional types, writes the DCIs to disk, and reconstructs them with their DataCanon identity intact. Conventional types include built-in types, library types, and user-defined types used directly for data and computation. Canonical binary files (.dcb) contain the complete canonical encoding.

Human-readable Shorthand, YAML interchange, and HDF5 provide other representations of the same data. Shorthand can mix concise mathematical and collection notation with explicit constructors. Its longhand form uses constructors for structured values while retaining ordinary literals such as 1, 1.0, and "hello". HDF5 stores Arrays as native numerical datasets and can embed externally referenced payloads.

Text layout and storage choices do not determine DCI identity. A change from a List to a Tuple does: those are different DCTs even when their items agree.

Rigidly structured scientific data

DataCanon supports exact numerical structure alongside finite machine numbers. Fractions, Sums, Products, OrderedProducts, Powers, Quotients, and Complex values can represent quantities such as

1/2 * sqrt(3)
quotient(1, sqrt(3) + sqrt(2))
complex(1/2, 1/2 * sqrt(3))

A Fraction and a Power within a Product can exactly represent the irrational number in the first expression. The second preserves a structured numerator and denominator. Complex components retain their individual DCTs.

Lists, Tuples, Sets, Multisets, ordered and unordered Mappings, and named Records provide different ways to organize data. Multisets preserve repeated members without assigning semantic order. Arrays retain their portable type, shape, and exact finite element representations, including signed zero.

Bytes contains an opaque byte sequence directly in a DCI. Applications can place interpretation metadata alongside it in a Record. ExternalFile and ExternalArray instead preserve a relative path and content digest. Their metadata can be read without retrieving the external bytes; retrieval and integrity verification are separate operations.

Identity and conventional types

DataCanon defines structured data, not mathematical equivalence. Integer 1 and Float 1.0 have different identities. Numerical nesting, repeated terms, and the constituent information prescribed by each DCT are preserved.

Conversion interfaces connect DCIs with instances of conventional types in a given programming language. Python examples include list, Fraction, and NumPy arrays. A List DCI represented by a ListDC object and a conventional list object belong to different types with different equality and operation rules. Both are Python objects.

Where suitable conventional types are missing, the Python implementation supplies additional types such as Power, Sum, and Multiset. The numerical conventional types support arithmetic independently of DCI construction and persistence. Their operations can simplify structure; conversion through conventional objects is not guaranteed to retain an arbitrary DCI’s identity. The DataCanon data model defines no such arithmetic operations.

Optional Shorthand preprocessing supplies compact Lists, textual substitutions, and explicit scalar eval(...) before strict parsing. It is an input facility, not an extension of DCI identity.

Using this documentation

• Getting Started: construct data, write binary files, and explore the other representations.
• Guides: conventional types, command-line operations, and Shorthand preprocessing, explained step by step.
• Python API: conversion tables, conventional types, rendering, and persistence.
• Reference: the 24 DCTs, their structure, and representation choices.
• Specifications: seven Version 1 documents, including the optional Conventional Types companion.

Version 1 remains a draft. Outstanding interoperability and conformance items are recorded in the individual specifications.