Conventional Types

DataCanon implementations connect DataCanon Instances (DCIs) with instances of conventional types in a programming language. Conventional types include built-in types, types provided by libraries, and user-defined types used directly for data and computation. Python examples include list, fractions.Fraction, and numpy.ndarray.

A DataCanon Type (DCT) defines information, invariants, and equality. A DCI is an instance of a DCT. The Python implementation represents DCIs using classes such as ListDC and FractionDC. A ListDC object and a conventional list object are both Python objects, but belong to different types with different rules. Conversion connects the types; it does not make their semantics identical.

The examples use Python. Implementations in Julia, Rust, C, or another language can define conversions involving conventional types from their own language and libraries. The DataCanon DCTs and DCI equality rules remain independent of those choices.

The optional Conventional Types specification defines common construction, arithmetic, comparison, and collection requirements for independent implementations. Support is declared per conventional type; Core conformance does not require conventional arithmetic. The guide introduces the Python binding, including its interfaces and conversion choices.

Distinguish DCTs, their implementations, and conventional types

The table identifies conversion relationships, not type equivalences or a universal one-to-one correspondence. For example, both built-in complex and ComplexNumber objects can be converted to Complex DCIs, but their component information differs. The Python API specifies the conversion directions and resulting types.

Exact named mathematical constants

The conventional Constant type preserves a named mathematical constant. Pi and E are instances representing pi and Euler’s number; neither stores a decimal approximation. Conventional arithmetic retains the constant:

import math
from data_canon import Constant, Pi, E, Sum, Product, as_dc_instance, render_shorthand

assert Pi == Constant("pi")
assert Sum(Pi, Pi) == Product(2, Pi)
assert Pi / Pi == 1
assert Pi != math.pi
assert float(E) == math.e

instance = as_dc_instance(2 * Pi)
assert render_shorthand(instance, padding=0) == "2 * Pi"

float(Pi) explicitly requests an approximation. The corresponding Constant DCI preserves its name, and its shorthand spelling is Pi (E for Euler’s number). Longhand uses constant("pi") or constant("e"). Physical constants and user-defined names are not included.

Begin with a List DCI and a conventional list

Construct a List DCI from a conventional list, then convert the DCI to a new list object:

from data_canon import as_dc_instance, dc_instance_to_python
from data_canon.instance import ListDC

source = [1, 2, 3]
instance = as_dc_instance(source)
converted = dc_instance_to_python(instance)

assert isinstance(instance, ListDC)
assert type(converted) is list
assert converted == source

Both instance and converted are Python objects. The function constructs an instance of the conventional list type from the List DCI represented by instance.

The difference affects equality:

assert [1] == [1.0]
assert as_dc_instance([1]) != as_dc_instance([1.0])

Conventional lists use the equality rules of their members. List DCIs compare the identities of their constituent DCIs, so Integer and Float constituents remain distinct. Conversion must be understood in terms of both types’ rules.

Use existing conventional types

Most scientific data already has useful conventional types in Python. Use built-in scalars and containers, Fraction for rational numbers, and NumPy arrays for machine data:

from fractions import Fraction
import numpy as np
from data_canon.instance import MappingDC

measurements = {
    ("Eg", 0): 2.37,
    "position": (Fraction(1, 2), Fraction(1, 4), 0.199),
    "lattice_vectors": np.eye(3, dtype="float64"),
}
measurement_instance = as_dc_instance(measurements)
recovered = dc_instance_to_python(measurement_instance)

assert isinstance(measurement_instance, MappingDC)
assert type(recovered) is dict
assert recovered[("Eg", 0)] == 2.37
assert type(recovered["position"]) is tuple
assert isinstance(recovered["position"][0], Fraction)
assert isinstance(recovered["lattice_vectors"], np.ndarray)

The Tuple key is a DCI within the Mapping DCI and a conventional tuple within the recovered dictionary. Array type and shape are preserved by the specified Array conversion. The returned NumPy array is read-only; .copy() creates an independently mutable array. Mutable buffers or unsupported numerical types may require explicit conversion before they can be represented by a DCI.

as_dc_instance() accepts an existing DCI too, returning that instance unchanged. The identity case does not eliminate the distinction between a DCI and a conventional object of another type.

NumPy scalar values

NumPy scalars convert by value to the corresponding scalar DCTs:

For example:

from data_canon.instance import FloatDC, IntegerDC

assert as_dc_instance(np.uint64(2**64 - 1)) == IntegerDC(2**64 - 1)
assert as_dc_instance(np.float32(0.5)) == FloatDC(0.5)

Binary16 and binary32 values widen exactly to binary64. Scalar conversions do not preserve the original NumPy width as identity metadata. For a numerical element whose type must be preserved, use a rank-zero Array:

from data_canon.instance import ArrayDC

scalar = as_dc_instance(np.float32(0.5))
typed_scalar = as_dc_instance(np.array(0.5, dtype="float32"))
assert isinstance(scalar, FloatDC)
assert isinstance(typed_scalar, ArrayDC)
assert typed_scalar.data.shape == ()
assert typed_scalar.data.dtype == np.dtype("float32")

NaN and infinities remain invalid numerical DCIs. Extended-precision scalar formats, such as a platform’s wider longdouble or clongdouble, require an explicit application conversion or custom registry rule. A platform alias of a supported portable dtype follows that portable dtype’s rule.

NumPy datetime64, timedelta64, and structured or opaque void scalars have no default conversion. Their units or field structure must be handled explicitly, for example by a registered conversion to a Record. A duration never implicitly becomes a unitless Integer. Exact-type registry rules take precedence over the default NumPy conversions and apply recursively in collections and Mapping keys.

Supply conventional types where suitable ones are missing

A language may lack a conventional type for a structure that DataCanon can represent. The Python implementation supplies Power, Sum, Product, Quotient, ComplexNumber, Multiset, and UnorderedDict for such cases. The additional classes are conventional types, just like types provided by other libraries; they are distinct from PowerDC, SumDC, and the other classes implementing DCTs.

DataCanon defines structured data and its identity; it does not define mathematical operations on that data. Working with structured numbers naturally creates a need to calculate with them. The additional numerical conventional types therefore support arithmetic under their own construction, equality, and simplification rules, similar in spirit to Python’s Fraction.

The arithmetic examples below construct conventional objects directly. No Shorthand parser, DCI construction, or persistence operation is needed to use Power, Sum, or the other numerical conventional types. Arithmetic operations on those objects do not evaluate or simplify stored DCIs.

Arithmetic with structured numbers

Retain a radical exactly

Construct a conventional Power object with a base and a positive rational exponent:

from data_canon import Power

sqrt_two = Power(2, 1, 2)
sqrt_three = Power(3, 1, 2)
cube_root_five = Power(5, 1, 3)

Power(3, 1, 2) represents three raised to the power one half. Its base remains a conventional int object here. Omitting the denominator gives an integer exponent: Power(3, 2) means three squared.

Conventional Power reduces the exponent fraction and extracts an exact root of the base when that makes the exponent an integer. Construction retains the Power type and the remaining integer exponent:

assert Power(5, 2, 4) == Power(5, 1, 2)
root_four = Power(4, 1, 2)
assert type(root_four) is Power
assert (root_four.base, root_four.numerator, root_four.denominator) == (2, 1, 1)
assert root_four == 2
assert Power(8, 1, 3) == 2

cubed = Power(16, 3, 2)
assert (cubed.base, cubed.numerator, cubed.denominator) == (4, 3, 1)

integer_power = Power(4, 8)
assert (integer_power.base, integer_power.numerator, integer_power.denominator) == (4, 8, 1)
assert integer_power == 65536
assert int(integer_power) == 65536

An integer exponent is preserved, rather than expanded merely to initialize the object. Equality and explicit int() or float() conversion can still compute its value. If the requested root is not exact, the constructor retains the base and reduced exponent: Power(8, 1, 2) keeps those fields.

Float bases follow the same rule, with an exactness check on the replacement float. For example, Power(4.0, 1, 2) becomes Power(2.0, 1, 1) and Power(16.0, 3, 2) becomes Power(4.0, 3, 1). Integer powers such as Power(0.1, 2) remain powers; construction introduces no floating-point rounding.

The behavior follows the model of Fraction: construction retains the requested conventional type, while equality can identify an equivalent scalar. A Power DCI has different rules and retains its original base and exponent constituents.

Calculate with exact geometric quantities

For an equilateral triangle of side length one, retain the exact height and area using conventional Fraction and Power objects:

from fractions import Fraction

height = Fraction(1, 2) * sqrt_three
area = height / 2

assert area == Fraction(1, 4) * sqrt_three
assert height + height == sqrt_three

The multiplication produces a conventional Product object. Division by a conventional int adjusts the rational factor; adding the two identical heights combines their factors. No decimal approximation of the square root is needed.

Add terms and combine matching contributions

Python operators construct conventional Sum and Product objects:

from data_canon import Sum

value = sqrt_two + sqrt_three + 3 * sqrt_three
assert value == Sum(sqrt_two, 4 * sqrt_three)
assert sqrt_three - sqrt_three == 0

Conventional Sum combines matching terms and rational constants, and flattens nested Sum objects. Extractable radical factors are simplified before terms are collected:

assert Sum(Power(12, 1, 2), sqrt_three).items == (3 * sqrt_three,)
assert Sum(1, Power(4, 1, 2)).items == (3,)

Explicit constructors are useful when assembling a collection of terms:

terms = [sqrt_two, sqrt_three, 3 * sqrt_three]
assembled = Sum(*terms)
assert assembled == value

The constructor retains a conventional Sum object even if simplification leaves one term. Arithmetic operators may return that remaining term directly:

assert Sum(sqrt_three, 3 * sqrt_three) == Sum(4 * sqrt_three)
assert sqrt_three + 3 * sqrt_three == 4 * sqrt_three

Sum(*items) and Product(*items) take separate operands, rather than one iterable argument. Conventional Sum and Product objects are immutable and unhashable, independently of the hashable Sum and Product DCIs.

Multiply factors and combine powers with a common base

Conventional Product combines rational factors and combines exponents for matching bases:

from data_canon import Product

assert Product(Fraction(1, 2), Fraction(2, 3)) == Product(Fraction(1, 3))
assert Power(5, 1, 2) * Power(5, 1, 3) == Power(5, 5, 6)
assert sqrt_two * sqrt_three == Product(sqrt_two, sqrt_three)

The last product can retain its two factors while comparing equal to the square root of six. Surrounding arithmetic can evaluate an integer power and combine the result with other terms or factors. Retaining the exponent in a standalone Power does not prevent that simplification:

assert sqrt_two * sqrt_three == Power(6, 1, 2)
assert sqrt_three * sqrt_three == 3
assert Product(3, Power(4, 1, 2)).items == (6,)

Multiplication distributes over conventional Sum objects:

assert 2 * Sum(sqrt_two, sqrt_three) == Sum(2 * sqrt_two, 2 * sqrt_three)

The simplification rules support exact structured calculations while keeping the scope of simplification limited and predictable.

Divide by a structured denominator

A reciprocal of a sum needs more than a conventional Fraction. Quotient retains a structured numerator and denominator:

from data_canon import Quotient

denominator = sqrt_three + sqrt_two
reciprocal = 1 / denominator
assert reciprocal == Quotient(1, Sum(sqrt_three, sqrt_two))

The denominator remains a conventional Sum; construction does not rationalize it. Equality can nevertheless recognize the rationalized equivalent by exact cross multiplication:

assert reciprocal == sqrt_three - sqrt_two
assert Quotient(4, 6) == Fraction(2, 3)
assert Quotient(2 * sqrt_three, 4 * sqrt_three) == Fraction(1, 2)

The Quotient constructor reduces rational coefficients and cancels common factors. It retains the Quotient type even when the result equals a scalar. Division rejects denominators recognized as zero.

Division by a conventional int or Fraction adjusts exact rational factors, as in the triangle example. Division by a conventional float follows floating-point arithmetic.

Keep floating-point contributions separate

Exact rational and floating-point contributions retain their separate roles:

mixed = Sum(Fraction(1, 4), 0.125)
assert mixed.items == (Fraction(1, 4), 0.125)

with_offset = Sum(sqrt_three, 0.25, 0.5)
assert with_offset == Sum(sqrt_three, 0.75)

Conventional float contributions can combine with one another using Python floating-point arithmetic, with its usual rounding. Float contributions remain separate from exact rational coefficients and are not converted to large rational numbers. The numerical conventional types accept finite float values and reject bool values as numerical operands.

Compare numerical values exactly

Conventional numerical equality uses exact normalization and cross multiplication. Equality can hold across Sum, Product, Power, Quotient, int, Fraction, and float; it is independent of presentation order and does not require matching conventional types.

Float comparison uses the exact binary value already stored by the float, following Python’s Fraction behavior:

assert Quotient(1, 2) == 0.5
assert Quotient(1, 10) != 0.1
assert Sum(2**53 + 1) != float(2**53 + 1)
assert mixed == Fraction(3, 8)

Keeping float and rational contributions separate during construction does not require them to compare unequal. No rounded floating-point evaluation is used as an equality test. Exact comparison forms are cached when needed without changing the object’s displayed operands or its conversion to a DCI.

Normalization handles rational arithmetic, distributive expressions, rational radicals, and quotient cross multiplication. It is not a complete procedure for recognizing every mathematical equivalence. Fractional powers with nonrational bases remain symbolic. Integer factorization uses trial divisors through 1000 and extracts residual perfect powers through degree 64; a large unresolved factor stays intact. Comparisons can therefore miss equivalences requiring further factorization or algebraic identities. No numerical tolerance or floating-point fallback is used to guess equality.

Choose when to request a machine number

Use float() at the point where a numerical routine or display needs a machine floating-point result:

import math

assert math.isclose(float(area), math.sqrt(3) / 4)
assert math.isclose(float(reciprocal), 1 / (math.sqrt(3) + math.sqrt(2)))

The conventional numerical object remains unchanged. Fractional powers use the real-valued domain and rounding behavior of math.pow during explicit evaluation.

int() also evaluates, then truncates toward zero, following Python conventions:

assert float(Sum(1, 1.0)) == 2.0
assert int(Quotient(-7, 3)) == -2

Explicit numerical conversions can raise errors for undefined operations or overflow. Construction rejects an even root of a known negative scalar; odd roots of negative scalars use the real branch. A symbolic expression with a more complicated base may require further evaluation to determine its domain.

Use structured complex components

Use conventional ComplexNumber when real or imaginary components should retain Fraction, Power, Sum, or other conventional numerical objects:

from data_canon import ComplexNumber

phase = ComplexNumber(Fraction(1, 2), Fraction(1, 2) * sqrt_three)
shifted = phase + 1
assert shifted.real == Fraction(3, 2)
assert shifted.imaginary == phase.imaginary

Component equality follows the same numerical rules. For example, ComplexNumber(Power(4, 1, 2), 0) compares equal to 2. Comparisons with built-in complex values compare the actual component values. Conventional ComplexNumber objects are unhashable.

Addition, subtraction, multiplication, division, negation, and conjugation are available. For example, multiplication by the imaginary unit rotates the components:

rotated = phase * ComplexNumber(0, 1)
assert rotated.real == -phase.imaginary
assert rotated.imaginary == phase.real
assert phase.conjugate() == ComplexNumber(phase.real, -phase.imaginary)

Request a built-in Python complex number explicitly when needed:

machine_phase = complex(phase)
assert math.isclose(machine_phase.real, 0.5)
assert math.isclose(machine_phase.imag, math.sqrt(3) / 2)

Component operations follow their underlying Python types. In particular, division of two conventional int values or float operations can introduce floating-point results; use Fraction or structured numerical operands when exact rational division is required.

Array expressions and ordered multiplication

The conventional Sum class supports scalar terms and Array expressions. The conventional OrderedProduct class preserves factor order, while Product remains a scalar type. Construction preserves the expression; .evaluate() performs conventional arithmetic. For example, combine the Pauli matrices:

import numpy as np
from data_canon import Sum, OrderedProduct, Power, as_dc_instance

sigma_x = np.array([[0, 1], [1, 0]])
sigma_y = np.array([[0, -1j], [1j, 0]])
operator = Sum(OrderedProduct(2, sigma_x), OrderedProduct(3, sigma_y))
np.testing.assert_array_equal(np.asarray(operator, dtype=complex),
                              2 * sigma_x + 3 * sigma_y)
instance = as_dc_instance(operator)

Factor order matters in conventional matrix multiplication as well as DCI identity. OrderedProduct(A, B) and OrderedProduct(B, A) can give different results. Scalar factors may occur anywhere; NumPy array factors currently must have rank two when an OrderedProduct is evaluated. Conventional Sum supports NumPy arrays of any rank with exactly matching shapes. Neither operation broadcasts or reshapes data. Nonzero scalar terms mixed into an Array Sum require an explicit identity Array.

Exact coefficients are retained in object-array entries:

exact = OrderedProduct(Power(2, 1, 2), np.eye(2, dtype=int)).evaluate()
assert exact.dtype == object
assert exact[0, 0] == Power(2, 1, 2)

from fractions import Fraction
exact_y = OrderedProduct(Fraction(1, 3), sigma_y).evaluate()
assert exact_y[1, 0].imaginary == Fraction(1, 3)

Use an explicit dtype when a machine approximation is wanted. An exact object Array cannot itself be encoded as a portable Array DCI; use a supported machine dtype or nested Lists to preserve exact per-element DCTs. Scalar-valued fields such as Power bases and Complex components do not accept Array-valued Sums.

Additional conventional collection types

Retain repeated collection members

A conventional Multiset preserves the number of occurrences without assigning semantic order. For example, it can describe a composition:

from data_canon import Multiset

composition = Multiset(["Ti", "O", "O"])
assert composition == Multiset(["O", "Ti", "O"])
assert composition != Multiset(["Ti", "O"])

composition.add("O")
assert composition.count("O") == 3

Conventional Multiset is mutable. It accepts an iterable and supports list-like operations. Its members need not be hashable:

samples = Multiset([[1, 2], [1, 2]])
assert samples == Multiset([[1, 2], [1, 2]])

Member comparison follows Python equality, so members must provide usable Boolean equality results. This differs from DCI equality; for example, Python Boolean and integer equality can identify values that DataCanon distinguishes.

Mark an unordered dictionary

Python dict already compares equal dictionaries independently of insertion order. Its insertion order is nevertheless retained when converting it to a DataCanon Mapping. Use UnorderedDict to request an UnorderedMapping instead:

from data_canon import UnorderedDict

settings = UnorderedDict({"temperature": 300, "pressure": 0})
settings["temperature"] = 350
assert settings["temperature"] == 350

Conventional UnorderedDict behaves as a Python dictionary, with the same key hashability and equality rules. Its additional role is to express the intended DataCanon collection type during conversion.

Construct DCIs from the additional conventional objects

The arithmetic and collection examples above created instances of conventional types. Use the conversion interface when their results need DataCanon identity:

from data_canon.instance import ProductDC, MultisetDC, UnorderedMappingDC

area_instance = as_dc_instance(area)
assert isinstance(area_instance, ProductDC)
assert dc_instance_to_python(area_instance) == area
assert isinstance(as_dc_instance(composition), MultisetDC)
assert isinstance(as_dc_instance(settings), UnorderedMappingDC)

A conventional Power object follows the arithmetic implementation’s rules. A Power DCI retains its base DCI and prescribed exponent fields. Likewise, conventional Sum objects simplify matching terms, while Sum DCIs retain the specified numerical structure and multiplicity.

The conventional objects can be placed in a dictionary with Tuple keys and converted together:

from data_canon import render_shorthand

results = {
    ("triangle", "height"): height,
    ("triangle", "area"): area,
    "phase": phase,
    "composition": composition,
    "settings": settings,
}
result_instance = as_dc_instance(results)
print(render_shorthand(result_instance))
assert render_shorthand(results) == render_shorthand(result_instance)

render_shorthand() accepts conventional objects by constructing a DCI through the same conversion interface before rendering. The DCI’s identity is governed by its DCT, rather than by the original object’s equality rules.

Preserve the distinction during conversion

Conventional types have their own construction, equality, and operation rules. Conversion can simplify structure, lose distinctions, or fail when a target type cannot hold the required information. A round trip through conventional objects is not guaranteed to preserve an arbitrary DCI’s identity.

For example, converting a Sum DCI invokes the conventional Sum constructor:

from data_canon import parse_shorthand, Sum

original = parse_shorthand("sum(1, 1)")
calculated_sum = dc_instance_to_python(original)
assert calculated_sum == Sum(2)
assert as_dc_instance(calculated_sum) != original

No original DCI is modified by the conversion. The newly constructed Sum DCI has the structure of the conventional result, rather than that of original. Python’s Fraction reduces ratios, and conventional Power reduces its rational exponent; their conversion rules can also change DCI structure.

Collection conversion has similar consequences:

first = {"temperature": 300, "pressure": 0}
second = {"pressure": 0, "temperature": 300}

assert first == second
assert as_dc_instance(first) != as_dc_instance(second)
assert as_dc_instance(UnorderedDict(first)) == as_dc_instance(UnorderedDict(second))

Python dictionary equality ignores insertion order, while Mapping DCI equality includes entry order. UnorderedDict requests an UnorderedMapping DCI instead. A conventional Multiset uses its members’ conventional equality rules, which need not coincide with DCI equality.

All DCIs in the reference implementation can be used as Python dictionary keys. Converting a DCI key to a conventional object can nevertheless produce an unhashable list or NumPy array. Distinct Boolean, Integer, or Float DCI keys can also collide under conventional dictionary-key equality. Conversion to a conventional dictionary is therefore not always a suitable way to retain a DataCanon Mapping. See the Python API for conversion details.

Store and recover the results

Writers accept conventional objects through as_dc_instance(). Readers reconstruct DCIs from the stored representation:

from data_canon import read, write

write(results, filename="exact_results.dcb")
restored = read("exact_results.dcb")
assert restored == result_instance

recovered_results = dc_instance_to_python(restored)
assert recovered_results[("triangle", "area")] == area

The storage round trip preserves DCI identity. The subsequent conversion from the Mapping DCI to a conventional dict is a separate operation, subject to the target types’ rules. Keep the DCI when its complete structural identity is needed; use conventional objects when their collection or arithmetic operations are wanted.

ExternalFile and ExternalArray are conventional types for constructing external references. Their reverse conversions are different operations: dc_instance_to_python() retrieves and verifies an ExternalFile payload, returning bytes or a content loader’s result, and materializes an ExternalArray as a NumPy array. The conversion does not return another ExternalFile or ExternalArray construction object. Metadata-only work can use the external DCIs without retrieving the payloads.

For constructor signatures, complete conversion tables, and resource handling, use the Python API. For numerical DCT structure, nesting, and identity, use the Reference and Core specification. For the language-independent conventional behavior and its required comparison cases, use the Conventional Types specification.

Application-defined Record conventions

An application can connect its conventional types to ordinary Records through an explicit ConversionRegistry. For example, a conventional date can become a Record holding year, month, and day fields, while another convention might also store calendar metadata. DC preserves that structure without choosing a calendar or defining date operations.

Rules apply recursively through supported collections and Record fields, using as_dc_instance(obj, registry=registry) and dc_instance_to_python(node, registry=registry). Reading a stored DCI does not execute those rules. The Custom Conversion Rules guide works through a complete date example and explains validation and defaults.

Compressed external Arrays

Use ExternalArray(array, filename="forces", compression=True) to select gzip storage for zero-heavy numerical data such as force constants. The conventional constructor finalizes the name as forces.bin.gz; compression defaults to off, which produces forces.bin. Explicit suffixes must agree with the setting.

Convert to a DCI before writing. Directory-backed packages contain the gzip file, while HDF5 packages apply dataset compression to the logical element bytes. The digest covers uncompressed canonical bytes in both representations. The DCI’s finalized path and identity are preserved through transfers. A name ending in .bin identifies a different DCI from the corresponding .bin.gz name. ExternalFiles retain opaque-byte semantics.

See the Python API for filename examples, validation, and compression behavior.