Operations
DataCanon Operations — Version 1 Draft
Status: Draft for independent implementation review
Depends on: DataCanon Core Version 1
Related: DataCanon Conventional Types Version 1; DataCanon Persistence Version 1
Scope: Explicit queries, transformations, composition, resource resolution, and ordered operation pipelines on DCIs
Out of scope: DCI type definitions, canonical encoding, textual syntax, CLI option spelling, and general symbolic algebra
Purpose and boundaries
DataCanon Core defines data and identity. This optional companion defines operations on DataCanon instances (DCIs). An operation receives a DCI, and either returns a DCI or reports a failure. Operations MUST NOT be performed implicitly by parsing, canonical encoding, fingerprinting, or persistence.
A transformation result can have a different DCT, value, or fingerprint from its input. Operations do not redefine DCI equality. A conforming implementation MUST NOT silently alter the input DCI. It MAY return an existing immutable object when no change is needed.
The terms MUST, MUST NOT, SHOULD, and MAY express requirements, recommendations, and permissions. Conformance to Core does not require implementing this companion. Implementations MUST identify which operations they support. Operations relying on a conventional-language binding or external resources MUST document those dependencies.
Operation categories
Operations are grouped by their effect, not by their CLI spelling:
- Queries select, count, or inspect existing DCIs.
- Structural transformations construct DCIs with changed containers or entries.
- Conventional conversions explicitly pass through a language binding and may normalize or approximate data.
- Numerical and array transformations apply specified numerical computations.
- Resource-aware transformations load referenced DCIs and combine them with local data.
An operation MUST report unsupported input types and invalid arguments rather than silently applying unrelated conversions. In particular, an implementation MUST NOT assume that a DCI numerical expression has already been reduced or approximated.
Traversal and composition
An operation MAY support recursive and non-recursive modes. Non-recursive means that the operation examines or transforms the current DCI only, except for any explicitly specified key chain, index chain, or resource-resolution semantics. Recursive means that the operation may visit eligible descendant DCIs according to its documented traversal contract. Recursive support does not imply that mapping keys, external payloads, or every DCT are transformed: each operation specifies its applicable descendants.
An operation that supports only one traversal policy MUST reject an explicit unsupported traversal modifier. The absence of a modifier selects the operation’s documented default.
Operations compose in order: the result of one is the input to the next. A failure MUST stop execution rather than substitute a partially computed result. An operation pipeline is not a transaction over external files or other side effects; immutability concerns its DCI inputs.
Queries and selection
Get
get selects values from mappings by a nonempty chain of DCI keys. In recursive mode, the first key is searched through eligible descendants; subsequent keys are followed directly from the selected values. One match returns that value; multiple matches return a List DCI. No matches cause an error. Direct mode searches only the current DCI for the first key.
Mapping containment
map_contains accepts a key and value DCI and returns a List DCI of mappings containing an equal key/value entry. Direct mode examines the current DCI; recursive mode searches eligible descendants.
Count
count returns an Integer DCI counting occurrences equal to the query DCI. In direct mode it returns zero or one for the current DCI. In recursive mode it counts eligible occurrences in traversal, including the root where applicable.
Index
index selects from a List, Tuple, or Array DCI using integer or slice selectors. A chain of indices is applied successively to the preceding result. Indexing is direct, not distributed across query matches. Invalid selectors or out-of-range indices are errors.
Structural transformations
Container conversion
to_list and to_tuple convert eligible sequence and array representations while retaining their element structure according to the language binding. Their recursive modes may transform eligible descendants. to_array converts an eligible rectangular numerical structure to an Array DCI; incompatible shapes or unsupported element types cause an error. These operations MUST NOT silently reinterpret arbitrary mappings as sequences.
Mapping operations
drop_key removes entries with specified equal DCI keys from mappings; recursive mode additionally processes eligible descendants. Missing keys are ignored. map_values returns a List DCI containing a Mapping DCI’s values in entry order.
Pure merge
merge accepts two or more DCIs of the same container DCT, limited to Mapping, List, and Tuple. Mapping entries are concatenated in input order; duplicate equal keys MUST cause an error. List and Tuple items are concatenated in input order. Merge is shallow: duplicate mapping keys are never recursively combined, overwritten, or coerced. The inputs MUST remain unchanged.
Conventional-language transformations
The following operations explicitly invoke conversion or computation supplied by a conventional-language binding. They do not define new mathematical identities for conventional objects. An implementation MUST document unsupported conversions and failures.
Roundtrip
roundtrip converts the entire input DCI into its conventional representation and then converts the result back to a DCI. It does not explicitly invoke numerical reduction. Constructor normalization may change the resulting DCI, so roundtrip is not required to preserve identity or fingerprint. A conversion that cannot be represented as a DCI MUST fail rather than silently invent a new DCT.
Reduce
reduce invokes the binding’s supported reduction method on eligible conventional numerical objects and converts the returned object to a DCI. The reduction method, not the DCI transformation, determines what simplifications are valid. Reduction may change DCTs, preserve unevaluated expressions, or produce a result that cannot be represented as a portable DCI. The latter MUST be reported as a conversion failure. Recursive mode applies the transformation to eligible descendants according to the implementation’s documented traversal.
In the Python binding, the conventional method is .reduce(). In particular, reduction of an array-valued expression may produce a NumPy object array containing conventional scalar objects; such an object array is not itself a portable DataCanon Array.
To float
to_float requests floating-point conversion of eligible numerical values using the conventional binding’s conversion protocol. In the Python binding, scalar conversion delegates to float() where supported. Existing Float DCIs remain unchanged. Boolean, string, and other nonnumerical DCIs MUST NOT be converted merely because the host language permits a coercion. Failed numerical conversion MUST NOT be silently replaced by a different mathematical result.
The conversion may approximate, and is not required to preserve DCI equality for unordered expressions whose conventional numerical evaluation depends on operand order. to_float MUST NOT reorder a conventional Sum or Product merely to force equal floating-point results.
Rounding
round takes an integer number of decimal digits and explicitly rounds eligible Float DCIs and floating or complex Array elements. It preserves the applicable container and array type/shape contracts. Rounding changes numerical data and therefore may change fingerprints; it MUST NOT be implicit in to_float.
Numerical and array operations
transpose, eigenvalues, determinant, and trace operate on eligible Array DCIs and report incompatible shapes or numerical domains. real and imaginary extract supported components, with recursive traversal where offered. Their numerical algorithms and supported dtypes MUST be documented by the binding; this companion does not mandate a particular linear-algebra library or algorithm.
A conforming implementation MUST preserve the distinction between an ordered numerical expression and an unordered one. It MUST NOT silently replace an ordered product with a commutative product or change matrix multiplication into elementwise multiplication.
File-backed merge resolution
Directive forms
resolve_merges is an explicit, resource-aware transformation. It recognizes the reserved String key merge_from in a Mapping DCI, or a two-element Tuple DCI of the form ("merge_from", specification) when that tuple appears as a direct element of a List or Tuple DCI.
A specification is one of:
- a String DCI naming a file;
- a two-element Tuple DCI containing a filename String and an expected fingerprint String; or
- a nonempty List DCI containing any combination of the preceding two reference forms.
The list form is a collection of references, not a reference to a List DCI merely because it is a List. Malformed specifications MUST fail.
Resolution and insertion
Each referenced file is read as a DCI and its own merge directives are resolved recursively before insertion. A Mapping directive requires referenced Mapping DCIs. The directive entry is replaced at its position with the referenced mappings’ entries in reference order; equal duplicate keys anywhere in the resulting mapping MUST cause an error.
For a List or Tuple, each directive element is replaced at its position by the items of referenced containers of the same DCT. A List reference cannot be spliced into a Tuple, or vice versa. Other elements and nested eligible containers are visited without changing their relative order. Resolved directives do not remain in the output.
Paths, fingerprints, and cycles
A relative reference is interpreted relative to the file containing its directive. When the root DCI has no source-file context, the binding MUST document the fallback base directory; the Python implementation uses the current working directory. Each referenced file establishes its own source context for its nested references.
If a reference supplies a fingerprint, the implementation MUST compare it against the canonical fingerprint of the fully resolved referenced DCI, before insertion into its parent. A mismatch MUST fail. Fingerprints are optional: an omitted fingerprint imposes no verification requirement beyond successful loading.
Circular file references MUST fail. A missing file, unreadable resource, malformed referenced DCI, wrong container DCT, or duplicate mapping key MUST fail. Resolution MUST NOT occur automatically as part of reading an ordinary DCI.
Illustrative mapping
A file may contain the following DCI, expressed here in Shorthand:
{
"merge_from": [
"../../crystal_structures/SrTiO3.dcs",
("../metadata.dcs", "expected-canonical-fingerprint")
],
"quad_irr_deriv": ...
}
The ellipsis is explanatory, not a literal valid DCI value. This example illustrates the reference shape; it is not an executable fixture.
Structural operation pipelines
An execution pipeline is a List DCI of three-element Tuple DCIs. Each tuple contains, in order:
- a String DCI operation name;
- a DCI argument value; and
- a Null DCI (default traversal) or Boolean DCI (explicit recursive setting).
Steps execute in list order. Operation names and argument types MUST be validated; unknown names, malformed steps, unsupported traversal modifiers, and invalid argument counts MUST fail. A binding MAY provide a preparation interface accepting higher-level requests and converting them into this structural form. Such request syntax is distinct from the prepared execution pipeline.
In the Python implementation, prepare_pipeline() accepts (name, arguments) requests or a List DCI of single-entry Mapping requests and produces the prepared three-element Tuple steps used by apply_pipeline(). The Python execution API also accepts optional source-file context for resource-aware resolution.
Output directives such as render and dump are handled by an output layer, not as ordinary DCI-to-DCI transformations. Where supported, an output directive MUST be terminal and MUST NOT be interpreted as a change to DCI identity.
Conformance and compatibility
An implementation claiming an operation MUST document its supported DCTs, default traversal, accepted arguments, error conditions, and any dependence on conventional-language behavior or external resources. A binding MAY expose additional operations, but MUST NOT silently change Core DCI equality or canonical encoding to accommodate them.
This specification does not define DCT CLI flags, terminal output formatting, shell completion, or application-specific automatic invocation of operations. Those belong to tools and applications built on DataCanon.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.