Guides on Data Canon

Identity and Fingerprinting

Mon, 01 Jan 0001 00:00:00 +0000

A calculation’s inputs may appear in a readable text file, a binary archive, and several programming languages. Comparing the files tells you whether their bytes agree. DataCanon lets you ask a different question: do the representations describe the same structured data?

This guide identifies a calculation’s inputs, distinguishes changes in data from changes in presentation, and uses fingerprints to associate results with their inputs. Start with Getting Started for installation and the first persistence workflow.

Persistence and External Resources

Mon, 01 Jan 0001 00:00:00 +0000

A scientific dataset often combines a small description with a large numerical Array or an existing file. DataCanon can store the data together or preserve an explicit reference to external content. The DCI retains its identity as a complete dataset moves between a directory and an HDF5 container.

This guide constructs a dataset with an external Array, writes a complete package, retrieves its numerical contents, and transfers the package. The Identity and Fingerprinting guide explains what its identifier covers.

Conventional Types

Mon, 01 Jan 0001 00:00:00 +0000

DataCanon implementations connect DataCanon Instances (DCIs) with instances of conventional types in a programming language. Conventional types include built-in types, types provided by libraries, and user-defined types used directly for data and computation. Python examples include list, fractions.Fraction, and numpy.ndarray.

A DataCanon Type (DCT) defines information, invariants, and equality. A DCI is an instance of a DCT. The Python implementation represents DCIs using classes such as ListDC and FractionDC. A ListDC object and a conventional list object are both Python objects, but belong to different types with different rules. Conversion connects the types; it does not make their semantics identical.

Shorthand Preprocessing

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Shorthand preprocessing provides three conveniences for entering scientific data by hand: compact List notation, named textual substitutions, and scalar mathematical evaluation with eval(...).

Preprocessing transforms text into strict Shorthand. Parsing that text then constructs a DataCanon Instance (DCI). This guide introduces each feature separately before showing how they work together.

Start with one compact List

Ordinary Shorthand Lists use brackets and commas:

[1, 2, 3]

Compact notation lets you enter the same List using angle brackets and spaces:

Custom Conversion Rules

Mon, 01 Jan 0001 00:00:00 +0000

DataCanon Records let applications describe structured values without adding new DataCanon Types. An application can define a Record convention and supply explicit rules connecting that Record to a conventional type.

A calendar date is a useful example. Python provides datetime.date, but DC need not prescribe a calendar, year range, or date arithmetic. The application can make those choices and store the necessary information in a Record.

This guide defines an illustrative "example.Date" convention. The name is application data, not a built-in DC type or a universally prescribed date format.

Command-Line Interface

Mon, 01 Jan 0001 00:00:00 +0000

The dc command loads a DataCanon file and displays, converts, searches, or selects items from its DCI. It works with Shorthand (.dcs), canonical binary bytes (.dcb), and DataCanon HDF5 files (.h5 or .hdf5), and YAML interchange files (.yml or .yaml).

Put an input filename first, or omit it to read Shorthand from standard input. With no action option, dc prints Shorthand:

dc structure.dcb

Installation and completion

Installing the Python package installs the dc command in the same environment. Activate that environment before using it. For a development installation from the code repository: