Pipeline schemas

An ordinary DataSchemer (DS) schema processes a Mapping of named variables. A pipeline schema, declared with pipeline: true, processes an ordered List of steps instead. Each step is a Mapping with one operation name and its value. The same operation can appear more than once.

Define and project a pipeline

from data_canon import as_dc_instance
from data_schemer.schema_projector import SchemaProjector

definitions = {
    "steps": {
        "pipeline": True,
        "variables": {
            "get": {"type": "String", "optional": True},
            "index": {"type": "Integer", "optional": True},
        },
    },
}

projector = SchemaProjector(definitions, schema_name="steps")
projection = projector.get_projection(as_dc_instance([
    {"get": "results"},
    {"index": 0},
    {"get": "energy"},
]))

assert projection.data == [
    {"get": "results"}, {"index": 0}, {"get": "energy"},
]

A PipelineProjection preserves the order and repetition of steps. A step must contain exactly one recognized operation. Invalid steps are rejected. Unlike ordinary Mapping projections, repeated operation names are meaningful.

The pipeline schema describes and validates operations; it does not itself execute the scientific or application-specific transformations they name.

Defaults and multiple input sources

Pipeline variables can declare a default and default_position to insert a default step at the beginning (0) or end (-1) of a pipeline. A supplied occurrence of that operation suppresses its default.

When multiple input sources supply the same pipeline, their steps are concatenated in source order, rather than merged as Mapping fields. For ordinary schema input, Mapping merge rules apply instead.

Command-line order

SchemaCommandLine preserves operation order and repeated options for pipeline schemas. For example, a parent schema can declare a pipeline child:

from data_schemer.schema_command_line import SchemaCommandLine

definitions = {
    "root": {"child_schemas": ["steps"]},
    "steps": {
        "pipeline": True,
        "variables": {
            "get": {"type": "List", "nargs": "+", "optional": True},
            "index": {"type": "List", "nargs": "+", "optional": True},
        },
    },
}

command = SchemaCommandLine(
    definitions, schema_name="root",
    argv=["steps", "--get", "results", "--index", "0", "--get", "energy"],
)
assert command.child_data["steps"] == [
    {"get": ["results"]},
    {"index": [0]},
    {"get": ["energy"]},
]

These are ordinary DS schema variables named get and index; DS does not assign DataCanon transformation semantics to them. An application can interpret the validated pipeline as it chooses.

Nested children

A pipeline may contain child-schema steps, including children that are ordinary Mapping schemas or other pipeline schemas. Child projections are constructed separately, and the pipeline retains their positions among ordinary operations. A nested child may itself contain further children.

This is different from inheritance: inherit reuses schema definitions; child_schemas defines the permitted nested structure. See Child schemas and SchemaCommandLine.