Shorthand Preprocessing

DataCanon Shorthand Preprocessing — Version 1 Draft

Status: Draft for independent implementation review
Depends on: DataCanon Shorthand Version 1; DataCanon Core Version 1
Scope: Optional textual transformation of augmented Shorthand into strict Shorthand
Out of scope: DCI construction, canonical encoding, persistence, application-specific mathematical aliases

Introduction

DataCanon Shorthand preprocessing is an optional text-to-text transformation intended for human-authored Shorthand. It supports three conveniences:

1. compact List notation;
2. named textual substitutions; and
3. explicitly requested scalar arithmetic with eval(...).

The preprocessor does not construct, inspect, or validate DataCanon Instances (DCIs). Its output is intended to be parsed by a strict Shorthand parser, which remains responsible for all syntax and DCT validity checks. Preprocessing does not change DataCanon Core identity or canonical encoding.

An implementation MAY offer a combined parsing interface. A strict Shorthand parser MUST NOT implicitly enable preprocessing. A human-facing command-line interface MAY preprocess Shorthand input by default, provided it documents that behavior and offers an explicit mode requiring strict Shorthand without preprocessing. Such a frontend does not change the strict grammar or the requirements on persisted strict Shorthand representations.

Conformance and terminology

MUST, MUST NOT, SHOULD, and MAY indicate normative requirements and recommendations.

A conforming preprocessor accepts Unicode text and optionally a mapping from substitution names to replacement text. For inputs within its supported language and resource limits, it MUST apply the transformations in this order:

1. Expand compact Lists, throughout the document, including substitution definitions.
2. Resolve substitutions, processing inline definitions in source order using external names and earlier inline definitions. Evaluate explicit eval(...) expressions in each resolved definition before making it available to later definitions, then substitute names in the document body.
3. Evaluate explicit eval(...) expressions remaining in the body, replacing each with an ordinary Shorthand numeric literal.

The result is text, not a DCI. An output that fails strict Shorthand parsing is not necessarily a preprocessing error: textual substitution values are not individually required to be valid Shorthand.

The preprocessor MUST NOT silently perform arbitrary mathematical multiplication, interpret application-defined symbols such as I or 2I, or evaluate expressions that are not explicitly enclosed in eval(...).

Lexical conventions

Strings and protected regions

A double-quoted string is protected during all three transformations. Compact List delimiters, substitution names, |, and eval(...) appearing inside a quoted string are treated as ordinary text.

The preprocessor recognizes a backslash as escaping the immediately following character for the purpose of locating the end of a quoted string. Validation of permitted String escapes belongs to the strict Shorthand parser.

For example:

"<1 2>"
"eval(1+2)"
"a | b"

are left unchanged by preprocessing.

Bytes literals

The hexadecimal String argument of bytes("...") is protected like every other quoted String. Preprocessing MUST NOT substitute names, expand Compact Lists, or evaluate text inside that String. Bytes parsing and hexadecimal validation belong to the strict Shorthand parser.

Substitution identifiers

A substitution name consists of an ASCII letter or underscore, followed by zero or more ASCII letters, digits, or underscores:

[A-Za-z_][A-Za-z0-9_]*

Names are case-sensitive. The preprocessor does not attach DataCanon meaning to them.

Delimiters and nesting

The preprocessor recognizes quoted strings and balanced (), [], and {} when locating top-level delimiters and splitting compact List contents. It recognizes matching parentheses when locating the end of an eval(...) expression.

No comments are defined by this preprocessing specification.

Compact List notation

General form

A compact List is enclosed in angle brackets:

<contents>

It expands to ordinary Shorthand List syntax using square brackets. A single item remains a one-element List:

<6>              -> [6]
<1 2 3>          -> [1, 2, 3]
<1 2; 3 4>       -> [[1, 2], [3, 4]]
<1 2, 3 4>       -> [[1, 2], [3, 4]]

Compact Lists may occur anywhere outside quoted strings, including inside Array syntax, Mapping values, and inline substitution definitions. Nested angle-bracket groups are expanded from the inside outward.

An empty compact List (<> or a whitespace-only group) is invalid. Use ordinary [] for an empty List.

Splitting rules

Within each compact List group, delimiters are considered at the top level of that group: separators inside quoted strings or balanced (), [], and {} do not split an item.

The reference transformation applies the following separator precedence, from highest to lowest:

1. line feed (\n);
2. semicolon (;);
3. comma (,);
4. horizontal whitespace (space, tab, carriage return).

At the first level having more than one nonempty part, each part is recursively transformed and the results are enclosed in [...]. If no level splits the contents, the contents form a single item. The outermost compact group always yields a List.

This is a textual convenience, not Array inference. In particular, <1 2; 3 4> expands to a List of Lists. array(<1 2; 3 4>) may subsequently be parsed as an Array by strict Shorthand.

Examples:

<[1,2] 3>             -> [[1,2], 3]
<eval(1 + 2) 0>       -> [eval(1 + 2), 0]
<"a > b" 3>           -> ["a > b", 3]

The eval(...) in the second example is evaluated only in the third preprocessing stage.

Compatibility note: the current reference implementation discards empty parts when splitting compact Lists. The treatment of repeated/trailing separators should be fixed explicitly before Version 1 is finalized; independent implementations SHOULD not rely on such inputs in the meantime.

Textual substitutions

External substitutions

An application MAY supply a mapping of substitution names to replacement text. Every name MUST satisfy the identifier rule above, and every replacement MUST be text.

For example, with external substitutions S = '"Sr"' and P = '[0.5, 0.5, 0.5]':

{"species": S, "position": P}

becomes:

{"species": "Sr", "position": [0.5, 0.5, 0.5]}

Substitution replaces complete identifier-like tokens outside quoted strings; it MUST NOT replace a substring of a longer ASCII identifier. For example, a substitution for a does not change alpha or a2.

Replacement text is inserted literally. It is not parsed or rendered as a DCI, and its whitespace is not normalized.

Inline substitutions

A document MAY begin with inline definitions separated from the document body by a single top-level vertical bar (|):

a=1/2 b=sqrt(3) | [a, b]

becomes:

[1/2, sqrt(3)]

A top-level | is one that is outside quoted strings and outside balanced (), [], and {}. A vertical bar within a quoted string is not a delimiter.

The definition region consists of one or more whitespace-delimited assignments of the form name = replacement-text. Each subsequent top-level name = begins another definition. Definitions may span lines and may contain bracketed structures or quoted strings. Replacement text extends up to the next definition or the | delimiter, with surrounding whitespace removed.

Definition names MUST be unique. An inline name MUST NOT conflict with an externally supplied name. Empty replacement values, text before the first definition, and more than one top-level | are invalid.

If no top-level | is present, the entire text is the document body; only external substitutions are applied.

Expansion order and dependencies

External substitutions are supplied as literal replacement text; the external mapping is not itself processed as a sequence of dependent definitions.

Inline definitions MUST be processed in source order. For each definition:

1. Replace external names and earlier inline names in a single textual pass.
2. Evaluate explicit eval(...) expressions in the resulting replacement text.
3. Store that resolved text for use by later definitions and the document body.

An unquoted reference to the current or a later inline name MUST be rejected, even in an unused definition. This includes such a reference exposed by an external replacement. Quoted occurrences are ordinary String contents, not dependencies. Names not defined anywhere in the document are left as text; the strict parser remains responsible for rejecting unknown constructs.

For example:

a        = 4.593
c_over_a = 0.644
c        = a * c_over_a |
[c]

produces:

[4.593 * 0.644]

No multiplication is performed. To obtain a machine numerical value instead, write c = eval(a * c_over_a). The stored replacement for c is then 2.957892. A later definition such as d = eval(c * 2) uses that numerical result, rather than creating a nested eval(...) call. Explicit evaluation in an unused inline definition is also performed and MUST reject invalid arithmetic.

After all inline definitions have been resolved, external names are replaced once in the body, followed by resolved inline names once. Explicit eval(...) expressions remaining in the body are evaluated in the final stage. Replacement text inserted during one pass is not recursively scanned during that pass. There is no dependency graph resolution or repeated expansion to convergence.

Compact Lists are expanded before definitions are resolved. Consequently, a=1/2 * sqrt(3) row=<a 0> | row produces [1/2 * sqrt(3), 0] without splitting the spaces inside the replacement into additional List items.

Expression boundaries

Substitution is literal text replacement. It does not add parentheses or constructor calls, and it does not guarantee that an inserted infix fragment will form one constituent in the surrounding expression. For example:

a=1 + 2 b=3 * a | b
    -> 3 * 1 + 2

To preserve a Sum as one Product constituent, use explicit constructor syntax:

a=sum(1, 2) b=product(3, a) | b
    -> product(3, sum(1, 2))

Within eval(...), ordinary arithmetic grouping parentheses may be included in the replacement text. Outside eval(...), strict Shorthand parentheses denote Tuples, not grouping. Applications SHOULD use substitutions at syntactic value boundaries rather than relying on token concatenation.

Multiset input

Multiset constructor syntax belongs to strict Shorthand and requires no preprocessing. Compact angle-bracket notation still expands to Lists, not Multisets.

Substitutions may be used within a Multiset constructor without removing repeated occurrences:

a=3 | multiset(a, a, 2)
    -> multiset(3, 3, 2)

The preprocessor does not inspect Multiset membership or equality. The strict parser constructs and validates the resulting Multiset DCI.

Scalar eval(...)

Purpose and placement

After resolving the names in an inline definition, and again after substitution in the document body, the preprocessor locates unquoted eval(...) expressions, allowing whitespace between eval and (. It finds the matching closing parenthesis, including nested parentheses, and replaces the complete expression with a numeric literal.

The evaluator is not a general Python evaluator. It supports only real scalar arithmetic. Arrays, DCIs, symbols, arbitrary functions, attribute access, indexing, comprehensions, Boolean expressions, comparisons, and arbitrary code execution are excluded.

Permitted expression forms

Within eval(...), the following Python-like forms are supported:

• integer and floating-point numeric literals;
• parentheses;
• unary + and -;
• binary +, -, *, /, and **;
• sqrt(expression) with exactly one positional argument.

Operator precedence and associativity follow Python expression semantics. In particular, exponentiation binds more tightly than unary minus on its left (-2**2 is -4), and / performs true division (1/2 is 0.5). The caret ^ is not exponentiation and is unsupported.

The sqrt operation has the semantics of Python’s real-valued math.sqrt, including rejection of negative arguments. No names other than the permitted sqrt call are evaluated.

Results

Intermediate arithmetic uses Python-compatible arbitrary-precision integers and IEEE-754 binary64 floating-point behavior. Every intermediate and final value MUST be a real int or float (not Boolean, complex, or another type). Nonfinite floating-point values are invalid.

If the final value is a finite Float with exactly integral value, it is converted to an Integer before rendering. This conversion uses no numerical tolerance. Consequently:

eval(1/2)                       -> 0.5
eval(1/2 + 1/2)                 -> 1
eval(2**10)                     -> 1024
eval(sqrt(9) + 1)               -> 4
eval(-2**2)                     -> -4
eval(5+(1.89-2.53)*6/3.72)      -> 3.967741935483871

The replacement MUST be a valid strict Shorthand Integer or Float literal representing the computed result. A Python-compatible shortest round-tripping decimal representation is RECOMMENDED for finite binary64 results; an implementation MUST preserve the intended binary64 value when subsequently parsed as strict Shorthand.

The evaluator MUST NOT use unrestricted host-language eval on untrusted input. Implementations MAY use an expression parser or a restricted abstract syntax tree evaluator.

Resource limits

Implementations MAY reject expressions that exceed reasonable limits on length, nesting depth, intermediate integer size, or exponentiation magnitude. The limits are resource-protection policies, not extensions to the permitted arithmetic language.

The current Python reference implementation uses a maximum expression length of 1,000 characters, maximum AST recursion depth of 64, and a 4,096-bit intermediate integer limit, with additional exponentiation checks. The thresholds are reference-implementation limits, not required values for other languages.

Portability note: fully identical last-bit results for every floating-point arithmetic expression and sqrt across languages/platforms are not yet guaranteed by this draft. A Version 1 conformance profile should fix the reference numerical semantics and test vectors before claiming universal byte-identical evaluated results.

Error handling and integration with strict Shorthand

A conforming preprocessor MUST reject malformed compact List delimiters, invalid or duplicate substitution definitions, conflicting external/inline names, self-references or forward references between inline definitions, unterminated quoted strings encountered during scanning, malformed eval(...) syntax, and unsupported or invalid evaluated operations.

A preprocessor MAY use implementation-specific exception classes and messages. It does not need to detect ordinary strict Shorthand errors such as unknown DCT identifiers or invalid Fraction denominators. Those errors belong to the strict parser.

For a conforming combined interface, preprocessing followed by strict parsing MUST be equivalent to calling the two operations separately:

parse_shorthand(text, preprocess=True, substitutions=S)
    = parse_shorthand(preprocess_shorthand(text, substitutions=S))

The notation above describes behavior, not mandatory function names in other languages.

Strict parsing without preprocessing MUST remain available and MUST reject preprocessor-only syntax that is not valid strict Shorthand.

Relationship to other DataCanon specifications

• DataCanon Core Version 1 defines DCIs, equality, canonical encoding, and fingerprinting.
• DataCanon Shorthand Version 1 defines the strict textual grammar and identity-preserving rendering.
• This specification defines optional text transformations performed before strict Shorthand parsing.
• YAML interchange, HDF5 persistence, and other representations are not required to perform preprocessing automatically.

The Python reference implementation provides preprocess_shorthand(...) and the explicit parse_shorthand(..., preprocess=True) option. The Python parser and ordinary .dcs persistence reader remain strict by default. The dc command-line frontend preprocesses .dcs and standard-input Shorthand by default; --strict disables preprocessing. Queries are not preprocessed. Other implementations MAY expose different interfaces while preserving the same transformations and access to strict parsing.

Appendix A — Preprocessing conformance examples

The output strings below illustrate the intended transformations. Where leading/trailing whitespace is not material to strict Shorthand parsing, it is omitted from the examples.

With external substitutions S='"Sr"' and P='[0.5, 0.5, 0.5]':

{"species": S, "position": P}
    -> {"species": "Sr", "position": [0.5, 0.5, 0.5]}

Invalid examples include:

<>                               # empty compact List
<1 2                           # unterminated compact List
a=1 | b=2 | [a]                # multiple top-level delimiters
a=1 a=2 | [a]                  # duplicate inline name
a=a | a                        # self-reference
a=b b=2 | a                    # forward reference
eval(1/0)                       # division by zero
eval(2^3)                       # unsupported operator
eval([1,2])                     # non-scalar expression
eval(sqrt(-1))                  # invalid real square root
eval(__import__("os"))         # arbitrary code is not allowed

Appendix B — Outstanding Version 1 finalization items

The following details should be resolved deliberately rather than inferred from implementation accidents:

1. Compact List empty-part behavior: whether repeated or trailing separators are rejected, ignored, or permitted as empty items.
2. Inline definition boundaries: precise treatment of unusual top-level name = sequences within replacement text.
3. Numerical conformance: a designated reference Python version/semantics for eval(...), including float formatting and sqrt edge cases, plus cross-language numerical test vectors.
4. Identifier scanning consistency: align the preprocessor’s eval token recognition with the ASCII identifier convention adopted for strict Shorthand and substitution names.

The finalization items do not affect DataCanon Core identity or canonical bytes. They affect only the optional preprocessing language and can be settled independently.