Pipeline schemas
An ordinary DataSchemer (DS) schema processes a Mapping of named variables.
A pipeline schema, declared with pipeline: true, processes an ordered
List of steps instead. Each step is a Mapping with one operation name and its
value. The same operation can appear more than once.
Define and project a pipeline
from data_canon import as_dc_instance
from data_schemer.schema_projector import SchemaProjector
definitions = {
"steps": {
"pipeline": True,
"variables": {
"get": {"type": "String", "optional": True},
"index": {"type": "Integer", "optional": True},
},
},
}
projector = SchemaProjector(definitions, schema_name="steps")
projection = projector.get_projection(as_dc_instance([
{"get": "results"},
{"index": 0},
{"get": "energy"},
]))
assert projection.data == [
{"get": "results"}, {"index": 0}, {"get": "energy"},
]
A PipelineProjection preserves the order and repetition of steps. A step
must contain exactly one recognized operation. Invalid steps are rejected.
Unlike ordinary Mapping projections, repeated operation names are meaningful.
The pipeline schema describes and validates operations; it does not itself execute the scientific or application-specific transformations they name.
Defaults and multiple input sources
Pipeline variables can declare a default and default_position to insert
a default step at the beginning (0) or end (-1) of a pipeline. A supplied
occurrence of that operation suppresses its default.
When multiple input sources supply the same pipeline, their steps are concatenated in source order, rather than merged as Mapping fields. For ordinary schema input, Mapping merge rules apply instead.
Command-line order
SchemaCommandLine preserves operation order and repeated options for
pipeline schemas. For example, a parent schema can declare a pipeline child:
from data_schemer.schema_command_line import SchemaCommandLine
definitions = {
"root": {"child_schemas": ["steps"]},
"steps": {
"pipeline": True,
"variables": {
"get": {"type": "List", "nargs": "+", "optional": True},
"index": {"type": "List", "nargs": "+", "optional": True},
},
},
}
command = SchemaCommandLine(
definitions, schema_name="root",
argv=["steps", "--get", "results", "--index", "0", "--get", "energy"],
)
assert command.child_data["steps"] == [
{"get": ["results"]},
{"index": [0]},
{"get": ["energy"]},
]
These are ordinary DS schema variables named get and index; DS does not
assign DataCanon transformation semantics to them. An application can interpret
the validated pipeline as it chooses.
Nested children
A pipeline may contain child-schema steps, including children that are ordinary Mapping schemas or other pipeline schemas. Child projections are constructed separately, and the pipeline retains their positions among ordinary operations. A nested child may itself contain further children.
This is different from inheritance: inherit reuses schema definitions;
child_schemas defines the permitted nested structure. See
Child schemas and
SchemaCommandLine.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.