Command-Line Interface

The dc command loads a DataCanon file and displays, converts, searches, or selects items from its DCI. It works with Shorthand (.dcs), canonical binary bytes (.dcb), and DataCanon HDF5 files (.h5 or .hdf5), and YAML interchange files (.yml or .yaml).

Put an input filename first, or omit it to read Shorthand from standard input. With no action option, dc prints Shorthand:

dc structure.dcb

Installation and completion

Installing the Python package installs the dc command in the same environment. Activate that environment before using it. For a development installation from the code repository:

source DC/bin/activate
make install
dc --help

Enable shell completion in the active environment:

eval "$(register-python-argcomplete dc)"

The Makefile also provides make completion, which prints that command for you to execute. Add the registration to your shell initialization after environment activation if you want it applied automatically. Completion suggests command-line options, filenames, and choices such as rendering styles. It does not inspect file contents to suggest mapping keys. Option names appear when you type -; completion does not fill an empty argument position with every CLI option. After --get, further bare arguments are keys until another option begins.

Read Shorthand from standard input

Omit the filename to try a DCI directly at the command line:

echo 'power(8, 1, 3)' | dc

Output:

8**(1/3)

The command preserves the Power DCI and displays it in Shorthand. Constructor names are lowercase. All existing actions and rendering options also work with piped input:

echo 'power(8, 1, 3)' | dc --longhand
echo '{"outer": {"x": 3}}' | dc --get x
echo '[1, 2, 1]' | dc --count 1
echo 'power(8, 1, 3)' | dc --fingerprint
echo 'power(8, 1, 3)' | dc --dump root.dcb

Use - to select standard input explicitly, including with shell redirection:

dc - --layout standard --align left < structure.dcs

The CLI preprocesses Shorthand input by default, so compact Lists, substitutions, and explicit eval(...) expressions need no extra option:

echo '<1 2, 3 4>' | dc --dump matrix.dcb

The resulting DCI is a List of Lists. Standard input defaults to Shorthand; select YAML with --input-format yaml. Use a filename for canonical binary or HDF5 input. A displayed DCI can feed another dc command:

dc matrix.dcb | dc --longhand

Input may span multiple lines and must contain exactly one complete expression. The command reads through end-of-file; when typing directly in a terminal, finish with Ctrl-D on Linux or macOS. Empty input and malformed expressions are errors.

External-resource paths in standard input resolve from the current working directory. Piped Shorthand carries the DCI’s metadata; it does not carry the original file’s resource namespace or its external payloads.

Example input file

A crystal structure illustrates how named parameters keep a scientific input file readable. The example describes rutile, a form of titanium dioxide (TiO₂). Three lattice vectors define a repeating cell; atom positions are given as fractions of those vectors and grouped by chemical element.

Save the following input as rutile.dcs:

a        = 4.593
c_over_a = 0.644
c        = a * c_over_a
u        = 0.305 |
{
  "lattice_vectors": [
    (a, 0, 0),
    (0, a, 0),
    (0, 0, c)
  ],

  "basis_atoms": {
    "Ti": [
      (0,   0,   0),
      (1/2, 1/2, 1/2)
    ],

    "O": [
      (u,        u,        0),
      (1   + -u, 1   + -u, 0),
      (1/2 +  u, 1/2 + -u, 1/2),
      (1/2 + -u, 1/2 +  u, 1/2)
    ]
  }
}

The lattice_vectors and basis_atoms keys follow the Principia Materia (PM) conventions. Lattice vectors and atom positions are written explicitly as Lists of Tuples. The exact halves remain Fractions, and the parameter u makes the relationships between oxygen positions visible in the input.

Load the input file directly:

dc rutile.dcs

Definitions are resolved in source order, so c can use a and c_over_a. The definition c = a * c_over_a retains a Product in the third lattice vector. No evaluation is needed to load or store that structure. Use c = eval(a * c_over_a) if a computed numerical lattice parameter is wanted instead.

The CLI preprocesses .dcs and standard-input Shorthand by default. Use --strict to disable compact Lists, substitutions, and eval(...) and require strict Shorthand:

dc structure.dcs --strict
echo '<1 2, 3 4>' | dc --strict

The second command reports a syntax error. --strict controls input parsing; it does not change rendering or mapping-key interpretation. Canonical binary and HDF5 files use their existing readers in either mode. The Python parse_shorthand() function, read() function, and YAML airlocks retain their strict defaults. See Shorthand Preprocessing for the individual input features.

Dump to another format

Write the resolved DCI as canonical binary bytes:

dc rutile.dcs --dump rutile.dcb

The .dcb file is binary. It stores the DCI, including its numerical structure; substitution definitions and source formatting are not stored. Subsequent reads of this file do not need preprocessing.

The destination suffix selects the output format:

For example:

dc rutile.dcb --dump rutile.h5
dc rutile.h5 --dump restored.dcs
dc rutile.dcb --dump rutile.yaml

A successful dump prints nothing. The resulting representation preserves the DCI’s identity. For DCIs with external references, dumping produces a complete package: resources are verified and copied beside .dcs, .dcb, or YAML output, or embedded in HDF5. Missing or invalid required resources cause the dump to fail.

ExternalArray paths ending in .bin.gz select gzip storage for directory packages and compressed embedded datasets for HDF5. The digest covers the uncompressed canonical element bytes. Transfers retain the same path and digest; an HDF5-to-directory dump recreates the gzip file. Configure compression when constructing the conventional ExternalArray; no additional CLI flag is needed. ExternalFile payloads remain opaque even when their paths end in .gz.

Display YAML

Use --yaml to print the DCI as YAML instead of Shorthand:

dc structure.dcb --yaml
dc structure.dcb --get basis_atoms --yaml
echo '{("Eg", 0): 2.37, "position": (1/2, 0, 0)}' | dc --yaml

The existing YAML interchange rules apply: native YAML values are used where faithful, and $ introduces strict Shorthand for other DCTs, including Tuple keys and values. The $ marker must be followed by whitespace. $100 and ${HOME} remain literal Strings. The marker performs no substitution or evaluation; quoting alone does not escape it. A literal reserved prefix uses a backslash, such as \$ 1/2. The writer handles literal-prefix escaping.

YAML output uses block-style Lists and Mappings, with one List member per - entry; empty collections remain [] or {}. Single-line $ values are not wrapped to a fixed width. YAML quotes scalars only when needed by its syntax or scalar-resolution rules, and multiline Shorthand uses block scalars.

A .yml or .yaml dump target selects YAML without requiring --yaml:

dc structure.dcb --dump structure.yml
dc structure.dcb --yaml --dump structure

The second command creates structure.yaml. Combining --yaml with an explicit non-YAML suffix is an error. YAML export preserves the DCI and does not convert it to conventional Python objects. Terminal output needs only external metadata; a YAML file dump verifies and copies referenced payloads beside the output file.

--yaml works with display and --get, but not --count or --fingerprint. YAML output accepts --align left, --align right, --align none, and --padding N:

dc rutile.dcs --yaml --align left --padding 0
dc rutile.dcb --dump rutile.yaml --align left --padding 1

Alignment is disabled by default. Within a native YAML List, rectangular groups of nonempty, flat Tuple rows share column widths. Each List aligns independently. Ragged rows, mixed List/Tuple rows, or cells requiring multiline Shorthand fall back to normal rendering. Native YAML Lists remain Lists; alignment does not replace them with Tuple escapes. Embedded Shorthand uses the same alignment rules as render_shorthand().

Padding defaults to 1 and controls spaces beside same-line delimiters inside $ escapes, including Tuple keys and numerical constructors. --padding 0 produces $ (1, 2) instead of $ ( 1, 2 ). Padding does not change YAML indentation or the space separating $ from its value. Fraction literals and other fixed forms keep their usual spelling. --longhand, --layout, --custom-layout, and --indent remain Shorthand-only options and are rejected for YAML output.

Read or write YAML

A .yaml or .yml input suffix selects YAML automatically. The default output remains Shorthand, so conversion needs no output option:

dc structure.yaml
dc structure.yaml --dump structure.dcb
dc structure.yaml --yaml --align left

For standard input, choose YAML explicitly:

cat structure.yaml | dc --input-format yaml
cat structure.yaml | dc --input-format yaml --yaml --align left --padding 0

--input-format yaml and --input-format shorthand override text input format selection for stdin or a named file. Input and output choices are independent; --yaml always selects output, never input. The CLI does not guess format from text contents. Canonical binary and HDF5 input still require filenames and their existing readers.

YAML output uses block-style Lists and Mappings. Native multiline Strings use YAML literal block scalars where supported by PyYAML, with quoted fallback when necessary to preserve their contents. Multiline $ escapes likewise use literal block scalars when possible. The --fold-string N option enables width-controlled folded scalars for eligible long single-line Strings. YAML’s native folding and chomping rules apply; these differ from DataCanon Shorthand block-string rules. YAML $ escapes always use strict Shorthand; they never run preprocessing. --strict applies only to Shorthand input and is rejected with YAML input. External paths in a YAML file resolve relative to its containing directory; YAML received on stdin uses the current working directory. Metadata reads do not retrieve external payloads. Python YAML import remains parse_yaml(); this CLI support does not change generic Python read()/write() dispatch.

Choose the displayed notation and layout

Choose delimiter layout, column alignment, and indentation independently:

dc rutile.dcb --layout standard --align left
dc rutile.dcb --layout compact --align left --indent 4
dc rutile.dcb --longhand

The default is standard layout with shorter notation where appropriate. --layout standard puts multiline delimiters on separate lines; --layout semi-compact shares the opening line with the first item or entry; --layout compact also shares the closing line with the final content. --layout inline forces the selected value and its whole subtree onto one line.

dc rutile.dcb --layout inline --padding 0
dc rutile.dcb --custom-layout '{"list":"inline"}'

A List forced inline also renders nested Mappings, Arrays, and other values inline. Child layout overrides yield to that enclosing inline requirement; siblings outside the List retain their own layouts. Column alignment and structural indentation have no effect inside an inline subtree.

--align left, --align right, and --align none control column padding. --align without a value selects left alignment. Alignment is disabled unless requested. --indent N controls structural nesting on new lines, defaulting to 2. It does not add spaces after a same-line opener. Compact continuation items align beneath the first item; indentation moves nested blocks below keys rather than changing that alignment.

--padding N controls spaces beside same-line collection and constructor delimiters, defaulting to 1. It applies to flat values too: [ 1, 2 ] becomes [1, 2] with --padding 0. Standalone delimiters receive no padding. Both options accept nonnegative integers, including zero.

echo '{1, 2, [0, 1]}' | dc --layout compact --indent 0 --padding 0

Output:

{1,
 2,
 [0, 1]}

Use --custom-layout for per-type overrides:

dc rutile.dcb --layout standard --custom-layout '{"mapping":"semi-compact","list":"compact","tuple":"compact","set":"compact"}' --align left

The argument is a strict DataCanon Mapping or UnorderedMapping with String keys and String values. Quote the entire argument for the shell. Constructor form, such as mapping("set": "compact"), is also accepted. The CLI does not use JSON, Python evaluation, or preprocessing to read this configuration.

Every structured type accepts the same four layout contracts. For example, "set", "multiset", "unordered_mapping", "record", and "array" can be configured alongside Lists, Tuples, and Mappings. The Python API lists all accepted type names. An override affects only its named DCT; unspecified types use --layout (default standard). No separate per-type command-line flags are needed. Unknown names, invalid values, and duplicate keys are errors before input loading. Multiline Mapping values start below their keys unless an enclosing inline layout suppresses the line breaks.

--longhand selects explicit constructors for structured values. Null, Boolean, Integer, Float, and String remain literals: sum(1, 1.0, fraction(1, 2)) is longhand. Fraction uses fraction(...), Bytes keeps bytes(...), and Array elements retain their specialized syntax. Layout and notation are independent:

dc rutile.dcb --longhand --layout semi-compact --align left

Lists, Tuples, Sets, and Multisets use one row per item when they contain a List or Tuple, or an item that already spans multiple lines, unless forced inline. Coordinate Tuples remain inline, so the rutile lattice vectors and atom positions appear as separate rows. Alignment adds column padding for rectangular rows, including coordinates containing Sums or Products. For example:

echo '[(1, 0), (0, -1)]' | dc --layout semi-compact --align left

Output:

[ ( 1,  0 ),
  ( 0, -1 )
]

Using --layout compact also puts the final ] on the last row.

The same options apply when dumping Shorthand:

dc rutile.dcb --dump readable.dcs --layout standard --align left
dc rutile.dcb --dump constructors.dcs --longhand

Rendering options affect textual representation, not DCI identity. The options also apply to values printed by --get. They are rejected with --count, --fingerprint, and binary or HDF5 output. YAML accepts alignment and padding as described above; Shorthand layout options do not apply to YAML.

Render multiline and folded Strings

By default, standard Shorthand layout renders Strings containing LF characters as literal blocks when possible:

{
  "message":
    """
    First line
    Second line
    """
}

Literal blocks preserve the String’s actual line breaks and whitespace. Strings that cannot safely use block syntax retain their ordinary quoted representation.

For long Strings without semantic line breaks, use --fold-string N to request physical wrapping at a preferred width of N characters:

dc input_data.dcs --fold-string 80
dc input_data.dcs --dump readable.dcs --fold-string 80

The renderer uses folded blocks when it can do so without changing the String DCI:

{
  "description":
    """>
    This is a long description that can be wrapped across
    physical lines without introducing semantic newlines
    into the underlying String.
    """
}

The requested width is a preference, not a strict maximum. Long words and significant whitespace may require longer lines or fallback to quoted rendering.

Block Strings are emitted only under standard Shorthand layout. The semi-compact, compact, and inline policies retain quoted Strings with escapes. Enclosing inline layouts take precedence over child rendering preferences.

The same --fold-string option is available for YAML output:

dc input_data.dcs --yaml --fold-string 80
dc input_data.dcs --dump readable.yaml --fold-string 80

YAML uses its native literal (|) and folded (>) scalar syntax. PyYAML may choose quoted scalars when block formatting is unsuitable, particularly when significant trailing whitespace is present. Width-controlled folding is applied selectively to eligible Strings, not globally to all YAML output.

These options affect textual presentation only. Parsing the output recovers the same DCI and therefore the same canonical bytes and fingerprint.

Find a mapping key

Supply a bare name for a String key. For example, find the titanium positions in the nested basis_atoms Mapping:

dc rutile.dcb --get Ti

The search visits Mappings and UnorderedMappings throughout the DCI, so it finds "Ti" inside "basis_atoms". It also visits mappings nested in collections, Record field values, and mapping keys. It does not require a Mapping at the root.

One matching entry prints its value. If the same key occurs in multiple mappings, the command prints a List containing all matching values, in traversal order. Equal repeated values remain separate occurrences. It continues searching inside a matched value, so deeper matches are included too. Traversal follows the recovered constituent order; unordered collections do not promise an application-level ordering of results.

For example, save this strict Shorthand as repeated.dcs:

{"left": {"x": 1}, "right": {"x": 2}}

Then:

dc repeated.dcs --get x

prints a List containing 1 and 2. A single match whose value is itself a List is printed directly; multiple List-valued matches form an outer List.

Keys can belong to any DCT. For example, when a file contains a Tuple key:

dc results.dcb --get '(1/2, 0, 0)'

Valid Shorthand keys retain their DCTs. To select String keys whose text would otherwise parse as another DCT, include explicit Shorthand quotes:

dc results.dcb --get 1          # Integer key
dc results.dcb --get true       # Boolean key
dc results.dcb --get '"1"'      # String key
dc results.dcb --get '"true"'   # String key

Implicit String handling accepts bare identifiers, including Unicode letters and names such as basis_atoms or site_1. Names with spaces or punctuation need explicit Shorthand quotes, for example --get '"site label"'. Malformed structured keys such as power(1, remain errors; they do not become Strings.

Key comparison uses DCI equality. Integer 1, Float 1.0, and Boolean true remain distinct keys. Record names and field names are fixed metadata, so --get does not treat them as mapping keys. A missing key is an error.

Follow a chain of mapping keys

Supply several keys to --get to search for the first and then follow direct Mapping entries:

dc rutile.dcb --get basis_atoms O
dc results.dcb --get measurements '("Eg", 0)'

The first key is searched recursively using the existing single-key behavior. Every subsequent key is looked up directly in each matching Mapping or UnorderedMapping value. Later steps do not search recursively or descend through Lists. Each key uses the same strict Shorthand or bare-String-name rules. Negative integer keys work within the chain. For negative expressions that argument parsing could mistake for options, use constructor form, such as --get measurements 'fraction(-1, 2)'. Dots are not separators; a String key containing a dot needs Shorthand quotes, for example --get '"basis.atoms"'.

Branches missing a subsequent key, or whose intermediate value is not a Mapping, are skipped. If no branch completes the chain, the command reports an error. Results stay separate during traversal: one final match is printed directly, while multiple final matches become a List in traversal order. Equal repeated matches remain separate, and List-valued matches are not flattened.

For example, this input has two complete chains and one incomplete branch:

echo '[{"a": {"b": 1}}, {"a": {"b": 2}}, {"a": 3}]' | dc --get a b

Output:

[ 1, 2 ]

Put an input filename before --get; subsequent bare arguments are consumed as keys. For stdin, omit the filename or use -. Another option ends the key sequence:

dc rutile.dcb --get basis_atoms O --yaml --align

Pipelines remain useful, but a second dc --get searches recursively again. A multi-key --get follows direct entries after the first recursive search.

Select or slice a List, Tuple, or Array

--index SELECTOR [SELECTOR ...] applies integers or slices successively. Integer indices are zero-based and accept negative values from the end:

echo '[(1, 2), (3, 4)]' | dc --index 0
echo '[(1, 2), (3, 4)]' | dc --index=-1
echo '[(1, 2), (3, 4)]' | dc --index 1 0

Without --get, indexing applies to the root DCI. With --get, it applies to the final result after the entire key chain:

dc rutile.dcb --get basis_atoms O --index 2
dc rutile.dcb --get basis_atoms O --index=-1 --yaml

If there are multiple final matches, --index selects one member of the aggregate List; it does not index each match individually. A single List-valued match is indexed directly. Selection order is always --get, then --index, then rendering, independently of where the options occur on the command line.

For Lists and Tuples, integer indexing selects a contained DCI. For Arrays, integer indexing selects along the first dimension. If the result is an Array, it is returned as an Array DCI; if the result is an Array element, that value is converted to a DCI. Strings, Sets, Multisets, and Mappings are not indexed. An incompatible target or an out-of-range integer index reports an error; malformed selectors are command-line usage errors. Empty sequences have no valid indices.

Slicing operates along the first dimension and preserves the DCT: a List slice is a List, a Tuple slice is a Tuple, and an Array slice is an Array. Out-of-range bounds are clipped, and slices may produce an empty result. A zero step or malformed slice is a usage error before input is loaded. Subsequent selectors operate on the slice result itself, without distributing across its members.

Shorthand formatting and YAML output options apply to the selected DCI. --index cannot be combined with --dump, --count, or --fingerprint. Each index is applied to the preceding result: --index 1 0 selects item 1, then item 0 from that value. Every intermediate target must be a List or Tuple; otherwise the command fails without printing a partial result. Another option ends the index sequence. Put the filename before --index as well as --get. Use --index -1 -1 for several negative integer indices.

Slices use start:stop[:step], with Python’s exclusive stop bound, omitted bounds, negative bounds, and negative steps:

dc rutile.dcb --get basis_atoms O --index 0:2
echo '(0, 1, 2, 3, 4)' | dc --index ::2
echo '[0, 1, 2, 3, 4]' | dc --index ::-1

Integer indexing selects a contained DCI for Lists and Tuples. For Arrays, integer indexing selects along the first dimension. If the result is an Array, it is returned as an Array DCI; if the result is an Array element, that value is converted to a DCI. Slicing operates along the first dimension and preserves the DCT: a List slice is a List, a Tuple slice is a Tuple, and an Array slice is an Array. Out-of-range bounds are clipped, and slices may produce an empty result. A zero step or malformed slice is a usage error before input is loaded. Subsequent selectors operate on the slice result itself, without distributing across its members:

echo '[(0, 1), (2, 3), (4, 5)]' | dc --index 1: 0 1

That command selects the suffix List, its first Tuple, then the Tuple’s second item, producing 3.

A slice beginning with - can look like an option to argparse, so attach it with =. Repeat --index to continue the same selection chain:

echo '[0, 1, 2, 3, 4]' | dc --index=-2:
echo '[[0, 1, 2], [3, 4, 5]]' | dc --index 1 --index=-2: --index 0

Repeated --index options concatenate their selectors in order. Integers and slices are the entire selection syntax; there is no expression evaluation or filter language. Put the input filename before the selection arguments.

Approximate and round numerical values

Use --round N to round floating-point values to N decimal places, including floating-point and complex Array elements. Negative N rounds to powers of ten. Array type and shape are preserved. Rounding uses Python’s round rule on binary floating-point values, not exact decimal arithmetic.

dc results.dcb --get reciprocal_vectors --round 3 --align
dc results.dcb --round 3 --yaml

Use --to-float to collapse a complete scalar Fraction, Constant, Sum, Product, Power, or Quotient value to a Float. Integer and Boolean DCIs remain unchanged; Complex components are processed individually. Existing Arrays retain their types. Array-valued Sums and OrderedProducts retain their structure; scalar subexpressions inside them are approximated without evaluating Array arithmetic. Lists, Tuples, Records, and other collections are processed recursively. For example:

echo '2 * Pi + 1/2' | dc --to-float --round 3

produces:

6.783

Sum and Product operands are evaluated in canonical order so equal DCIs have equal approximations, independent of their stored operand order.

Selection (--get and --index) happens first, then conversion, then rounding, regardless of the order of flags. Mapping keys and external references are unchanged, and no external payload is loaded. A Set transformation that would produce duplicate members fails explicitly; Multisets retain occurrences.

Both options create a transformed DCI. They work with Shorthand, YAML, and --dump, including binary or HDF5 output. The original DCI is unchanged; a transformed DCI can have a different fingerprint. Neither option can be combined with --count or --fingerprint. Omit --to-float to round Float leaves while retaining the surrounding numerical structures and exact fields.

Count occurrences of a DCI

--count searches for a complete DCI using its type-defined equality:

dc rutile.dcb --count '1/2'

It prints an integer count. Zero matches is a successful search that prints 0.

The search includes the root and all constituent DCIs: collection members, mapping keys and values, Record field values, Complex components, Power bases, and Quotient constituents. Repeated occurrences count separately, including repeated members of Multisets, Sums, and Products.

Fixed information prescribed directly by a DCT is not a separate DCI. Searching for Integer 1 does not match a Fraction numerator or a Power exponent field. Likewise, Array elements, Record names, relative-path components, and byte payload contents are not searched as constituent DCIs.

Queries are never preprocessed. --count requires strict Shorthand; --get additionally accepts bare String names as described above. For a negative numeric query, use an equals sign so argument parsing keeps the value with its option:

dc results.dcb --count=-1

The same convention works for negative keys: --get=-1.

Print the fingerprint

dc rutile.dcb --fingerprint

The output is the 64-character lowercase hexadecimal SHA-256 fingerprint of the DCI’s canonical bytes. It is independent of Shorthand layout and HDF5 storage layout. It is not a hash of the input file’s representation bytes, except when that file is itself the raw canonical .dcb encoding.

Displaying, searching, and fingerprinting external DCIs use their metadata. Metadata operations do not retrieve external payloads. Dumping a complete package does require those resources.

Options and errors

Choose at most one action: --dump, --get, --count, or --fingerprint. --get accepts one or more keys. --index optionally selects from the root or the final --get result; it is not valid with the other actions. With none supplied, the action is to display Shorthand, or YAML with --yaml. Omitting the filename, or supplying -, reads text from standard input (Shorthand by default; --input-format yaml selects YAML). Help and version output do not read input:

dc --help
dc --version

Successful output goes to standard output and diagnostics go to standard error. Displayed Shorthand and YAML are suitable for shell redirection, but use --dump when creating a complete package with external resources.

A malformed Shorthand query, invalid input file, or failed resource verification produces a diagnostic and a nonzero exit status.