Reference

• 1: Terminology
• 2: Behavioral rules
• 3: Python API
• 4: Errors

1 - Terminology

DataCanon (DC) defines a data model, equality, canonical bytes, and fingerprinting. DataSchemer (DS) applies application schemas to values in that model. SchemaCommandLine (SCL) is DS’s command-line interface builder.

Data types and values

A DataCanon Type (DCT) defines a kind of data, the information constituting its values, its validity rules, and equality. Integer, List, and Array are DCTs.

A DataCanon Instance (DCI) is a value of exactly one DCT. For example, IntegerDC(3) is a DCI of the Integer DCT. A DCI has canonical bytes and can be represented in Shorthand, YAML interchange, or supported persistence formats. The representation is separate from the value’s identity.

Conventional types are the usual types used in a programming language: builtins, library types, and user-defined classes. Python’s list, int, fractions.Fraction, and NumPy arrays are examples. A ListDC instance and a list are both Python objects but have different roles. DS preserves the DCI and converts it to conventional objects when requested.

Schemas and data

A schema definition is metadata describing permitted fields, optional DCT constraints, defaults, and other input rules. In the Python implementation, definitions are conventional dictionaries. The data supplied to the schema processor is a Mapping DCI with String field keys.

A field is a named entry governed by a schema. The Python schema API uses variables as the key containing field definitions.

A base schema supplies definitions through inheritance. A derived schema inherits and may override those definitions.

A child schema validates a nested Mapping entry in input data. Child schemas and inheritance are separate relationships. A supplied empty child Mapping is empty data for a defined schema; it is not an empty schema definition.

A projection is the validated result for one input DCI, after schema defaults have been applied. A compiled SchemaProjector can produce multiple independent SchemaProjection results. Each result retains the original input separately from processed output.

2 - Behavioral rules

This reference records the schema-processing contract for DataCanon Instances (DCIs), the values defined by DataCanon’s data model. A schema definition is conventional Python metadata. A projection is the validated result for one input DCI. A child schema describes nested data; a derived schema inherits definitions from bases.

Schema definitions

The single-schema form contains a top-level variables Mapping and has default name "default". Otherwise definitions map schema names to schema dictionaries, and schema_name is required. A schema with no local variables can use an empty variables dictionary in the single-schema form or a named definition in the multiple-schema form.

substitutions, help, metavar, alias, nargs, const, store_true, and store_false provide SchemaCommandLine (SCL) interface metadata. They do not change DCI equality or turn DCT constraints into conversion instructions.

Field names must be nonempty strings. Constructor aliases must be nonempty strings and unique at the schema level. A local field name cannot also be a declared child-schema name.

Schema definitions and named schema entries must be dictionaries. Schema names must be nonempty strings, and inheritance must list nonempty schema names. Malformed field choices, dependency names, and conflict groups are rejected with DS user-facing errors. Composition and removal still precede final field validation.

DCT constraints

The allowed type names are:

Null Boolean Integer Float String Bytes Constant Fraction
Power Quotient Complex
List Tuple Set Multiset Sum Product OrderedProduct Mapping UnorderedMapping Record
Array ExternalFile ExternalArray

Constraints use exact DCT identity. Integer does not satisfy Float, Float does not satisfy Integer, and Boolean does not satisfy Integer. DataSchemer (DS) performs no implicit numerical coercion. An omitted type accepts any DCI that satisfies the field’s other constraints.

A default has its own DCT but does not imply a field constraint. Defaults and choices may be conventional objects or DCIs. They are converted to DCIs at compilation using the supplied conversion registry, if any. Declared type and choice constraints apply to defaults as well as supplied values.

Array element-type constraint

An Array field may declare array_type as one of bool, int8, int16, int32, int64, uint8, uint16, uint32, uint64, float16, float32, float64, complex64, or complex128. Names are the exact portable DC Array type names.

array_type requires an explicit type: "Array". It validates the existing element type of supplied values and defaults; Array type, shape, and contents are preserved. It does not accept a List or an ExternalArray. Without array_type, the Array DCT constraint accepts every portable Array element type.

Field composition resolves this property like any other property. Validation of the constraint follows overrides and removals. A mismatch raises DSInvalidArrayTypeError with the field’s data path and required/received portable Array type names.

Field composition

Field definitions are resolved with these precedence rules:

1. Resolve listed bases recursively. For each field property, the first base explicitly defining it wins; later bases fill absent properties.
2. Apply local variables property overrides.
3. Resolve copy_variables destinations from their source field definitions. A copied definition replaces an inherited destination definition. Local destination properties override the copy.
4. Apply delete_properties, then delete_variables.
5. Fill implicit property defaults and validate the final field definitions.

A property value is replaced as a whole, including a choices list or Mapping default. Property values are not recursively merged. Implicit defaults are not inserted into base definitions before composition.

Copy references use "schema@field". Cross-schema sources include their resolved inheritance and removals. Same-schema copies can reference another copy declared later, and source fields before current-schema removals. Cyclic inheritance or copy references are errors. Missing copy sources or removal targets are errors.

delete_properties maps field names to lists of property names. delete_variables lists fields to remove. Final validation follows removals, allowing an invalid inherited definition to be removed before validation. update_variables is rejected, including an empty occurrence; local property overrides belong under variables.

Field order follows first appearance through the bases, then new local fields; new copy destinations follow their insertion order. Composition does not mutate the caller’s field definitions.

Effective data and validation

Each projected schema level receives a Mapping DCI with String field keys. Values of ordinary fields may contain arbitrary nested DCIs and Mapping keys.

Projection retains the original complete input DCI and computes processed data:

1. Separate declared child entries from local fields and reject unknown local fields.
2. Fill absent local fields from schema defaults.
3. Check required fields, DCT constraints, choices, requires, and conflicts.
4. Project supplied child Mappings recursively.
5. Assemble the complete processed Mapping.

Required fields cannot declare defaults. Optional fields without defaults remain absent when not supplied. Explicit Null and false are present values. Defaults satisfy requires and participate in conflicts. A conflict group permits at most one present member, regardless of the members’ truth values.

input_dci is unchanged. data_dci contains local processed fields, while data_tree_dci also includes processed child entries. Supplied Mapping order is preserved; added defaults append in schema order.

Conventional output is converted on access. Processing alone does not retrieve external payloads or invoke Record-to-conventional conversion handlers. Those operations can occur when a conventional view or selected constructor argument is requested, using DataCanon (DC)’s conversion and integrity rules.

Child schemas

child_schemas may name one child or list child names. Inheritance accumulates base declarations first, then local declarations, removing duplicates while preserving first occurrence. Every declared child must exist. Child-schema cycles and field/child-name collisions are rejected. A schema used as a child cannot declare non-None required_file metadata.

A child is projected only if its entry is present in the data. An absent child stays absent. An explicitly empty child Mapping receives defaults and validation. Null and other non-Mapping values are rejected. A supplied parent Mapping does not implicitly activate its children.

child_schema_mode belongs to SCL navigation and is not interpreted or exposed by SchemaProjector. Compiling or caching a child projector does not activate child data in any projection. Projector searches cover already compiled definitions and preserve distinct paths for reused schema names.

SCL input boundaries

SCL uses the same schema processor after parsing its input into DCIs.

• Option names use kebab-case; aliases add command-line spellings.
• Ordinary option values use Shorthand with preprocessing enabled.
• Declared String fields accept ordinary text unless the argument begins with a double quote, in which case it is parsed as a DC literal.
• After failed Shorthand parsing, simple bare identifiers and paths can become String DCIs. Malformed structured expressions are rejected.
• Boolean flags and nargs determine interface arity; supplied values still undergo schema validation. Omitted options remain absent until defaults apply.
• Required option values and the required positional file are consumed before command-token recognition. Variable arity can end at a command once its minimum number of values has been consumed.
• Structured files use .dcs with preprocessing, canonical .dcb, DC HDF5, or DC YAML according to suffix. Every file root must be a String-keyed Mapping DCI.
• Stdin uses DC YAML unless the constructor selects Shorthand. Blank structured text denotes an empty Mapping. Stdin cannot be consumed twice.
• Loaded external references retain nonsemantic source context for deferred access.

The input_files control is enabled only when declared in the schema, with default nargs: "+". Its paths are literal strings. A constructor argument can select another control name. SCL control fields remain in its processed local data and are subject to normal constructor argument selection.

SCL source merging

Files selected through the input-file control merge in order. A structured required positional file overlays them, then command-line values overlay both. Schema defaults fill remaining absent fields after merging.

Only declared child-schema blocks merge recursively. Ordinary Mapping-valued fields replace earlier values as a whole. Existing field positions remain; new fields append. The merged DCI becomes input_dci, before defaults. It is not a collection of separate snapshots of the original source files.

SCL navigation

Selecting a child command supplies its Mapping, even without options. Selecting a nested command supplies Mappings along that path. File-supplied child entries can also be projected without selecting their commands.

child_schema_mode: "exclusive" is the default: navigation descends into child commands. "inclusive" also permits a one-level sibling jump between that schema’s children. Navigation does not search upward across multiple ancestors, and neither mode activates absent data. Invalid navigation settings raise DSUserError in SCL.

SCL rejects child schemas directly declared under multiple parents. The standalone projector can reuse a child definition across branches.

required_file is root SCL metadata and is not inherited. With apply_as_schema_data: True, structured loading selects the format by suffix and required_file_content stays None. Otherwise read_mode selects path, text, or binary access.

Required-file switches are Booleans. Invalid modes, malformed metadata, and unknown required-file keys are rejected before access. Application-specific text mode decodes UTF-8; read failures are DS user errors. Path mode retains the path without opening its contents.

Application construction

get_instance(callable) selects local processed fields by constructor argument name after code_alias mapping. It passes keyword arguments, supports keyword-only parameters, and supplies additional local fields when the callable accepts **kwargs. Child projections are constructed separately.

A schema default is processed data and takes precedence over a callable default. An absent optional field leaves a callable default intact. Callable defaults and annotations do not define DCT constraints or satisfy required schema fields.

Signature binding happens before conventional conversion. Only selected argument values are converted. A callable without an inspectable signature, or one needing required positional-only arguments, can be adapted with a Python wrapper. Application constructor exceptions and conventional conversion failures propagate.

SCL’s target_class uses this same construction contract. SCL calls _operations() after construction, then prints requested attributes.

Print attribute configuration

print_attribute is disabled when absent or false; true enables automatic discovery of public properties. A configuration dictionary can control:

Attribute aliases must be one-to-one and disjoint from schema aliases in both external names and internal targets. print_attribute is a reserved injected SCL field; applications must not also declare it in variables.

Output configuration has separate inheritance semantics from field properties. Bases are applied in listed order, then local configuration. A false value disables output; true preserves an inherited configuration dictionary. Dictionaries merge alias maps with later values overriding earlier ones, extend and deduplicate choices and exclude lists, and replace other properties. Explicit choices: None clears manual choices; exclude: None clears exclusions. Later-base dictionary settings can therefore override earlier-base settings in this output configuration.

Attribute output always uses DC YAML interchange, including with omit_key. Numerical report formatting belongs to application code. The removed precision_from and array_style_from settings raise DSUserError, including when inherited from a base schema.

Errors and scope

Input constraint failures use DS user-facing errors, including structured DSInvalidTypeError for DCT mismatches. SCL’s main() catches DSUserError and normally returns status 2. Application and conversion errors remain separate.

The error reference gives exact type diagnostics. DCT identity, canonical bytes, resource integrity, and the syntax of DC representations remain defined by DataCanon.

Application-defined CLI substitutions

A field’s optional substitutions property is a dictionary mapping DC substitution identifiers to self-contained Shorthand text. Schema compilation validates names and text, prepares optional preprocessing in each value, and retains the prepared replacements in the compiled field metadata. Original schema definitions are not modified.

SCL supplies the replacements to DC preprocessing for that field’s command-line values, including individual nargs items. String fields retain their existing literal-input behavior. Local input definitions may use preset names but cannot redefine them. Field constraints apply to the resulting DCI.

Substitutions do not apply to defaults, input files, or programmatically supplied DCIs. Inheritance, copying, overriding, and deletion use ordinary field-property composition; overriding substitutions replaces the entire mapping.

3 - Python API

Import the public classes from their defining modules:

from data_schemer.schema_projector import SchemaProjector, SchemaProjection
from data_schemer.schema_command_line import SchemaCommandLine

SchemaProjector

SchemaProjector(schema_definitions, *, schema_name=None, input_data=None,
                registry=None, source=None)

schema_definitions is a conventional Python dictionary in single-schema or named-schema form. schema_name selects a definition in named-schema form. registry supplies a DataCanon (DC) conversion registry. source supplies external-resource context for conventional conversion. An optional input_data DataCanon Instance (DCI) creates an initial projection.

Accessing initial-projection members without an initial input raises DSUserError. Reusable projectors and per-input projections have distinct roles; passing a new input to get_projection() does not replace the initial projection.

For a field declared type: "Array", optional array_type constrains its portable element type, for example float64. It requires an exact match and preserves the supplied Array. This metadata is resolved during schema composition and validated before projection; defaults are checked as well.

SchemaProjection

Obtain a projection through projector.get_projection(input_dci).

Conventional views are computed on request. get_instance() selects fields before converting, so unrelated external references need not be materialized. See Application construction.

SchemaCommandLine

SchemaCommandLine(schema_definitions, target_class=None, schema_name=None,
                  argv=None, input_files_tag="input_files",
                  prog=None, registry=None, stdin_format="yaml")

Construction parses sources, validates the merged DCI, optionally constructs the target, calls _operations(), and handles requested attribute output. Omit argv to use sys.argv[1:]. prog selects the displayed executable name. stdin_format accepts "yaml" or "shorthand" for structured stdin.

SchemaCommandLine (SCL) exposes input_dci, data_dci, data, data_tree_dci, data_tree, child_projections, and child_data from its projection. Additional members:

main() forwards arguments to the constructor. debug=True reraises caught DS user errors. Argparse help can exit through normal SystemExit(0) behavior.

Schema metadata helpers

The following helpers are available from data_schemer.schema_projector:

get_schema_variables(schema_name, schema_definitions, allowed_types=None, defaults=None)
get_schema_child_schemas(schema_name, schema_definitions)
get_schema_conflicts(schema_name, schema_definitions)
get_schema_print_attribute(schema_name, schema_definitions)

get_schema_variables() resolves metadata. SchemaProjector performs additional compilation checks and converts defaults and choices to DCIs; use the projector when processing input.

For SCL tooling, compute_print_attribute_choices() is available from data_schemer.schema_command_line:

compute_print_attribute_choices(print_attribute, *, target_class, schema_vars)

It returns (choices, external_to_internal) for attribute validation and completion. Callers should reuse it when they need SCL’s naming rules.

Output formatting

SCL attribute output uses DC YAML interchange. Applications handle their own numerical reports. render_data.py, numpy_print.py, and SCL’s numerical report controls have been removed; see Output and application reports.

4 - Errors

DataSchemer (DS) validates DataCanon Instances (DCIs), the values in DataCanon’s data model. A DataCanon Type (DCT) constraint requires a particular data type. Catch DSUserError at the application’s user-input boundary:

from data_schemer.errors import DSUserError

try:
    projection = projector.get_projection(input_dci)
except DSUserError as error:
    print(error.user_message())

The base class provides user_message() and a default exit_code of 2. SchemaCommandLine.main() prints caught DS user errors to stderr and returns their exit code. Calling the constructor directly allows the caller to handle exceptions itself.

DCT mismatch

DSInvalidTypeError records a field whose DCT differs from its declared constraint. For a nested Float supplied where Integer is required:

Invalid type for field "solver.iterations".
Expected: Integer
Received: Float
Value: 1.0

The exception exposes:

Paths describe the input data, independently of schema names and constructor aliases. A literal field name containing a dot uses bracket notation:

Invalid type for field ["solver.iterations"].

That path is ("solver.iterations",), distinct from the nested path above. Diagnostic summaries describe numerical arrays by dtype and shape, bound long values, and do not retrieve external payloads or invoke conventional conversion.

Array element-type mismatch

DSInvalidArrayTypeError reports an Array DCI that does not satisfy its field’s array_type constraint:

Invalid Array type for field "solver.matrix".
Expected: float64
Received: float32
Value: Array(type=float32, shape=(2, 2))

It is a subclass of DSInvalidTypeError and DSUserError, retaining path, expected, received, and value_summary. For this exception, expected and received name portable Array element types. Nested invalid defaults preserve the same exception class and data path. A non-Array value fails the DCT check first and raises the ordinary DSInvalidTypeError.

Other input errors

The unused legacy DSCoercionError and DSInvalidChoiceError classes have been removed. Failed field choices are reported through DSUserError.

Invalid configuration and required files

Malformed schema dictionaries, inheritance lists, field choices or dependencies, conflict groups, and attribute-output settings produce DSUserError diagnostics. SCL also reports invalid option metadata and conflicting aliases as DS user errors rather than exposing argparse construction errors.

required_file metadata is checked before file access. Its switches must be Booleans, read_mode must be path, text, or binary, and unknown metadata keys are rejected. Missing or unreadable files and invalid UTF-8 text produce user-facing messages. Path-only mode does not open or check a file’s contents.

Constructor and conversion failures

Missing constructor arguments, non-callables, and uninspectable callable signatures produce DSUserError during get_instance(). Signature binding runs before conventional conversion.

Once selected arguments are converted and application construction starts, conversion failures and application exceptions propagate. For example, a constructor’s own ValueError remains a ValueError. A successful DCI projection does not guarantee that an external resource can later be retrieved or that an application constructor will accept its conventional arguments.

Validation of schema and SCL metadata is separate from application execution. DS does not broadly wrap arbitrary implementation failures or exceptions from registered conventional conversion handlers into input errors.