Shorthand Preprocessing

Overview

Shorthand preprocessing provides three conveniences for entering scientific data by hand: compact List notation, named textual substitutions, and scalar mathematical evaluation with eval(...).

Preprocessing transforms text into strict Shorthand. Parsing that text then constructs a DataCanon Instance (DCI). This guide introduces each feature separately before showing how they work together.

Start with one compact List

Ordinary Shorthand Lists use brackets and commas:

[1, 2, 3]

Compact notation lets you enter the same List using angle brackets and spaces:

<1 2 3>

Use preprocess_shorthand() to expand it:

from data_canon import preprocess_shorthand

source = "<1 2 3>"
prepared = preprocess_shorthand(source)
print(prepared)

The resulting text is:

[1, 2, 3]

No DataCanon Instance has been constructed yet. prepared is a Python String. Parse that String to construct the List DCI:

from data_canon import parse_shorthand

instance = parse_shorthand(prepared)

Preprocessing and strict parsing are separate steps: preprocessing expands input notation; strict parsing interprets the resulting Shorthand and validates the data.

A compact group always produces a List, even when it contains only one item:

<6>  -> [6]

Empty compact groups are invalid. Use ordinary [] for an empty List.

Enter nested Lists

A comma can separate rows whose items are separated by spaces:

<1 2 3, 4 5 6>

expands to:

[
  [1, 2, 3],
  [4, 5, 6]
]

A semicolon can introduce another level:

<1 2, 3 4; 5 6, 7 8>

expands to:

[
  [
    [1, 2],
    [3, 4]
  ],
  [
    [5, 6],
    [7, 8]
  ]
]

At each level, the preprocessor looks for separators in this order:

1. Newline.
2. Semicolon.
3. Comma.
4. Horizontal whitespace: space, tab, or carriage return.

The first separator that splits the contents into multiple nonempty parts establishes the outer level. Each part is then processed in the same way. Newlines therefore introduce a level outside semicolons, commas, and spaces. Explicitly nested angle-bracket groups are also supported.

The layouts above illustrate the resulting structure; exact output spacing is not important to strict parsing. Avoid repeated or trailing separators while their treatment remains a Version 1 draft finalization item.

Existing structures remain intact

Separators inside quoted Strings or balanced (), [], and {} do not split items. For example:

<[1,2] 3>

becomes:

[[1,2], 3]

The inner [1,2] is one item. Likewise,

<"Sr Ti O" 3>

becomes:

["Sr Ti O", 3]

The spaces inside the String do not introduce List items.

Construct an Array when needed

Compact notation alone creates Lists. To enter an Array, place it inside an Array expression:

source = 'array(<1.0 0.0, 0.0 1.0>, dtype="float64")'
prepared = preprocess_shorthand(source)
instance = parse_shorthand(prepared)

The prepared text represents:

array([[1.0, 0.0], [0.0, 1.0]], dtype="float64")

The strict Array parser constructs the Array. Compact notation supplies its numerical body without itself performing Array inference.

Substitute one name

A substitution associates a name with replacement text. Define the name before |, then use it in the document body:

species="Sr" | {"species": species}

Preprocessing produces:

{"species": "Sr"}

In Python:

source = 'species="Sr" | {"species": species}'
prepared = preprocess_shorthand(source)
instance = parse_shorthand(prepared)

The name species is an input convenience. After substitution, the parser receives an ordinary Shorthand Mapping.

Define several names

Separate definitions with whitespace. Whitespace around = is permitted:

species = "Sr"
position = [0.5, 0.5, 0.5] |
{
  "species": species,
  "position": position
}

becomes:

{
  "species": "Sr",
  "position": [0.5, 0.5, 0.5]
}

Definitions can contain Strings, Tuples, Arrays, Mappings, or other text:

label = "oxygen"
site = (1/2, 1/2, 0)
lattice = array([[1.0, 0.0], [0.0, 1.0]], dtype="float64") |
{
  "label": label,
  "site": site,
  "lattice": lattice
}

Replacement text is inserted literally. It is not separately parsed as a DCI; only the final document must be valid strict Shorthand. Replacement text can therefore also be an expression fragment used later inside eval(...).

Name rules and protected Strings

A name begins with an ASCII letter or underscore, followed by ASCII letters, digits, or underscores. Names are case-sensitive and must be unique.

Substitution replaces complete identifier-like tokens. A definition for a does not replace part of alpha or a2, and nothing is substituted inside quoted Strings:

a=3 | {"value": a, "text": "a"}

becomes:

{"value": 3, "text": "a"}

Build on earlier definitions

Definitions are processed from left to right. Each definition may use externally supplied names and any earlier local definitions:

a        = 4.593
c_over_a = 0.644
c        = a * c_over_a |
[c]

This produces [4.593 * 0.644], which represents a List containing a Product. To compute the lattice parameter for a machine Array, request evaluation:

source = """a=4.593 c_over_a=0.644 c=eval(a * c_over_a) |
array(<a 0 0, 0 a 0, 0 0 c>)"""
instance = parse_shorthand(source, preprocess=True)

The third diagonal entry is 2.957892. An evaluated definition is resolved before later definitions use it, so d=eval(c * 2) would also work. Self-references and references to later definitions are rejected; arrange parameters before the definitions that depend on them.

Replacements remain literal text. For example, a=1 + 2 b=3 * a | b becomes 3 * 1 + 2, without implicit grouping. Use constructors when a nested numerical structure matters: a=sum(1, 2) b=product(3, a) | b preserves the Sum as one Product constituent. Ordinary Shorthand parentheses create Tuples, not groups. Inside eval(...), arithmetic parentheses provide ordinary grouping.

Readable input files with named parameters

Named parameters make an input file easier to read and revise: repeated values have one definition, and relationships remain visible in the data.

A crystal structure provides a useful example. Atomic positions are often specified as fractions of the repeating cell, with several positions sharing one displacement parameter. Define delta once and write each coordinate in terms of its reference position and displacement:

from data_canon import parse_shorthand, preprocess_shorthand, render_shorthand

source = """
delta = 0.199 |
{
  "basis_atoms": [
    (delta,         1/4 + delta,  1/8),
    (1/2 + -delta,  3/4 + -delta, 5/8),
    (-delta,        3/4 + delta,  3/8),
    (1/2 + delta,   1/4 + -delta, 7/8)
  ]
}
"""

prepared = preprocess_shorthand(source)
instance = parse_shorthand(prepared)
print(render_shorthand(instance, layout="standard", align="left"))

After substitution, the strict parser receives the following structure (whitespace may differ):

{
  "basis_atoms": [
    (0.199,         1/4 + 0.199,  1/8),
    (1/2 + -0.199,  3/4 + -0.199, 5/8),
    (-0.199,        3/4 + 0.199,  3/8),
    (1/2 + 0.199,   1/4 + -0.199, 7/8)
  ]
}

The List contains four Tuple DCIs. Coordinates such as 1/4 + 0.199 are Sum DCIs whose constituents retain the exact Fraction reference position and the Float displacement separately. The resulting DCI no longer contains the substitution name; the source text remains useful for editing the parameter.

The ASCII name delta is used because substitution names currently follow the ASCII identifier rules. With the positive value defined here, -delta becomes the negative Float literal -0.199 by textual substitution. This example needs neither Compact List notation nor eval(...).

Supply substitutions from Python

Instead of placing definitions in the input, supply a mapping of names to replacement Strings:

prepared = preprocess_shorthand(
    '{"species": species, "position": position}',
    substitutions={
        "species": '"Sr"',
        "position": "[0.5, 0.5, 0.5]",
    },
)
instance = parse_shorthand(prepared)

The result is the same Mapping as in the inline example. The values in substitutions must be text. The quotation marks in '"Sr"' are intentional: they make the replacement a Shorthand String literal.

External substitutions may also be used in inline definitions:

prepared = preprocess_shorthand(
    'site=(x, x, 0) | {"position": site}',
    substitutions={"x": "1/2"},
)

This produces:

{"position": (1/2, 1/2, 0)}

Each inline definition expands external names and earlier local names in one pass, then resolves its explicit eval(...) expressions. After definitions are resolved, external names are substituted once in the body, followed by local names once. The external mapping itself is not recursively expanded. An inline name must not conflict with an externally supplied name.

Array presets and linear combinations

Substitutions can name Arrays, making input files concise without changing DCI identity. For example, the Pauli matrices provide a small linear-operator example:

Sigma_x = array([[0, 1], [1, 0]])
Sigma_y = array([[0, 0-1j], [0+1j, 0]]) |
{
  "operator": 2 * Sigma_x + 3 * Sigma_y
}

After substitution, * constructs OrderedProduct because an Array is present; + constructs Sum. DC preserves those structures without multiplying or adding anything. Their conventional counterparts can evaluate the linear combination. The same mechanism supports 2 * I when I is defined as an identity Array.

Evaluate one scalar expression

Use eval(...) when you want to compute a numerical result before parsing:

prepared = preprocess_shorthand("eval(2 + 3)")
print(prepared)

Output:

5

Parsing prepared constructs Integer 5:

instance = parse_shorthand(prepared)

Evaluation is explicit. The preprocessor does not perform arithmetic on expressions outside eval(...).

Supported arithmetic

The evaluator supports real integer and floating-point literals, parentheses, unary + and -, binary +, -, *, /, and **, and real sqrt(...):

Operator precedence follows Python. Division / performs true division; exponentiation uses **, not ^.

Intermediate values use arbitrary-precision integers and binary64 floating-point arithmetic. If a final Float has an exactly integral value, it is emitted as an Integer. Otherwise, a finite floating-point result is emitted as a Float literal. No tolerance is used to decide whether a value is integral.

Evaluation versus stored structure

Strict Shorthand can store an irrational value as a Product containing a Fraction and a Power:

1/2 * sqrt(3)

That expression retains its Product and constituent structure when preprocessed. To compute a floating-point approximation instead, request evaluation:

eval(1/2 * sqrt(3))

This distinction lets you choose between preserving scientific structure and performing numerical arithmetic before constructing the DCI.

Strict structure and preprocessing arithmetic

Outside eval(...), 1 + 2 constructs a Sum DCI and 2 * 3 constructs a Product DCI. Inside eval(...), those operators perform scalar arithmetic before the result is parsed:

assert parse_shorthand("1 + 2") != parse_shorthand("3")
assert parse_shorthand(preprocess_shorthand("eval(1 + 2)")) == parse_shorthand("3")

The evaluator follows Python operator precedence. For example, eval(1/2**3) produces 0.125; strict Shorthand rejects 1/2**3 because a Fraction base requires an explicit power(...) constructor. They are different input grammars used at different stages.

Evaluation limits

The evaluator supports real scalar arithmetic only. It excludes unresolved names, arbitrary function calls, Arrays, indexing, attribute access, and general code execution.

Division by zero, negative real square roots, complex results, and nonfinite floating-point results are rejected. Implementations may limit expression length, nesting, and intermediate integer size. Cross-language numerical conformance details remain recorded in the Version 1 draft specification.

Combine the features

Once each feature is familiar, they can be used together. Consider entering a symmetric matrix with repeated numerical parameters:

a=1.1 b=0.2 |
<a 0 b, 0 a 0, b 0 eval(-a)>

First, compact List notation expands:

[
  [a, 0, b],
  [0, a, 0],
  [b, 0, eval(-a)]
]

Next, substitutions are inserted:

[
  [1.1, 0, 0.2],
  [0, 1.1, 0],
  [0.2, 0, eval(-1.1)]
]

Finally, eval(...) is evaluated:

[
  [1.1, 0, 0.2],
  [0, 1.1, 0],
  [0.2, 0, -1.1]
]

Strict parsing now constructs a List of Lists. To construct an Array, wrap the compact body in array(..., dtype="float64").

The order also matters when replacement text contains spaces. For example:

term=1/2 * sqrt(3) | <term 0>

Compact List expansion happens first:

[term, 0]

Substitution then produces:

[1/2 * sqrt(3), 0]

The spaces within the replacement Product do not split it into additional List items because compact List expansion has already finished.

Longhand and preprocessing

Longhand is the explicit-constructor subset of strict Shorthand. Preprocessing can insert constructors just as it can insert shorter forms:

prepared = preprocess_shorthand('values=list(1, 2) | tuple(values, fraction(1, 3))')
assert parse_shorthand(prepared) == parse_shorthand('([1, 2], 1/3)')

The longhand renderer flag has no effect on preprocessing. Compact List notation refers specifically to angle-bracket input groups; it is unrelated to the renderer’s layout="compact" delimiter policy.

Python parsing and CLI defaults

Call parse_shorthand() directly for strict Shorthand. When preprocessing is wanted, call preprocess_shorthand() first and parse its result.

Quoted Strings are protected during every stage. Text such as "<1 2>", "eval(2 + 3)", or "a | b" remains ordinary String content.

Preprocessing does not change DataCanon’s types or identity rules. YAML $ values and .dcs persistence files use strict Shorthand and do not implicitly perform preprocessing when read through the ordinary Python persistence API. The CLI is a human-input frontend and enables preprocessing by default:

echo '<1 2, 3 4>' | dc
dc rutile.dcs --dump rutile.dcb

Use --strict to require strict Shorthand input in the CLI. The option does not change the Python API defaults or enable preprocessing in key/count queries.

For the detailed rules, see the Shorthand Preprocessing specification.