Publication Outline

This outline proposes a paper for the Journal of Open Source Software (JOSS) describing DataCanon and its Python reference implementation.

The paper should explain the research problem, the software’s contribution, and the evidence supporting its use. The seven specifications and documentation carry the detailed type definitions, encoding rules, and API reference.

Working title

DataCanon: Deterministic Identity and Persistence for Structured Scientific Data

An alternative retaining the original emphasis is:

DataCanon: Deterministic Semantic Identity for Structured Scientific Data

A different choice which tries to capture the versatility for exploring data:

DataCanon: The Last Mile of Scientific Data

Canonical Identity Across Programming Languages and Storage Formats

Central thesis

Scientific software needs a precise way to say what structured data is, independently of how that data happens to be represented, stored, or manipulated. DataCanon provides a formal, language-independent data model for structured scientific data together with canonical identity.

DataCanon defines DataCanon Types (DCTs), their valid DataCanon Instances (DCIs), and their equality relations. Every DCI has a unique canonical byte encoding. Human-readable Shorthand, YAML interchange, and HDF5 represent that same data without defining its identity.

This distinction should be explicit throughout the paper. DataCanon is not a database, merely a serialization format, a schema system, or a programming language. Those systems answer related but different questions. DataCanon’s core question is: what data object is this? Representation asks how to communicate it; persistence asks how to store it; a database asks how to organize, locate, query, and update collections of it; computation asks what operations to perform on it.

The scientific motivation should lead: exact numerical structures built from Fractions, Products, Powers, Sums, and Quotients, structured Complex values, typed multidimensional Arrays, and reliable persistence of nested scientific data. The paper should then explain how canonical identity makes those structures portable and reproducibly identifiable.

JOSS format and scope

The current JOSS paper guidance sets a 750–1,750-word range and identifies sections for Summary, Statement of need, State of the field, Software design, Research impact statement, AI usage disclosure, Acknowledgements, and References.

Organize the manuscript around those sections. Aim for approximately 1,400–1,600 words of prose, leaving room to adjust the emphasis during writing. The word allocations below are planning targets, not journal requirements.

Section Approximate words

Summary 150 Statement of need 250 State of the field 325 Software design 400 Research impact statement 250 AI usage disclosure and Acknowledgements 100

The submitted paper concerns usable research software. The Python package, its tests, documentation, and scientific applications should therefore have a substantial role. Independent implementations strengthen the portability evidence, but should be described as completed work only when they have been implemented and validated.

1. Summary

Introduce DataCanon in language accessible to a researcher who has never used it. Describe it first as a formal, language-independent data model for structured scientific data with canonical identity, rather than as a serialization or persistence format.

Explain that DataCanon allows complex scientific data to be constructed, communicated, compared, fingerprinted, and persisted while retaining an identity independent of storage layout and implementation language.

Briefly identify the distinctive capabilities:

• Exact structured numerical data, including irrational Powers, structured Quotients, and Complex values with independently typed components.
• Portable typed Arrays alongside recursively structured collections.
• Deterministic binary identity and SHA-256 fingerprinting.
• Human-readable input and identity-preserving YAML and HDF5 representations.
• External-resource references with deferred retrieval and integrity checks.

Describe the Python package as the reference implementation of a language-independent specification. Avoid an API inventory or encoding tutorial in the summary.

2. Statement of need

Research problem and audience

Address developers of long-lived computational science and engineering software, especially workflows that combine exact numerical structure with large machine Arrays.

Explain that scientific software often has no difficulty storing data, but still lacks a representation-independent answer to more basic questions: What type of data object is this? When are two independently constructed objects the same? Which structural distinctions participate in identity? Can that identity survive a change of programming language or storage technology?

Use a concrete scientific object: a crystal structure or symmetry calculation containing lattice Arrays, fractional coordinates, exact complex phase factors, parameters, and externally stored numerical results.

Such an object may need to survive storage changes, support reproducible identification, and move between implementations while retaining distinctions that matter to the application.

Scientific example

Use a structured complex phase factor as the central numerical example:

complex(1/2, 1/2 * sqrt(3))

This preserves the exact real and irrational imaginary components instead of immediately replacing both with floating-point approximations. Discuss why this is useful for crystallographic or symmetry data, alongside ordinary floating-point Arrays required by numerical calculations.

Mention that DataCanon preserves type and constituent structure without mathematical simplification. Detailed normalization edge cases belong in the specifications rather than the motivation.

Practical outcome

Explain what the package enables: identity-preserving disk round trips, format transfers, stable fingerprints of scientifically meaningful data, and communication of the same DCI across independent implementations.

If a code fragment is needed, use one short persistence example; leave a full walkthrough in Getting Started.

3. State of the field

Use this section not only to compare features, but to establish what kind of thing DataCanon is. One of the easiest ways to understand DataCanon is to distinguish the data model from neighboring layers that solve different problems.

A useful organizing framework is:

DataCanon fundamentally addresses the first question. Its specifications and software also provide representations and persistence mechanisms, but those do not define DCI identity. DataCanon can therefore use or coexist with systems at the other layers rather than replacing them.

Comparison with generic data models

Compare DataCanon first with generic structured-data models such as JSON and CBOR, not only with their concrete serializations. This comparison establishes the conceptual neighborhood: recursively structured, language-independent data whose values can be realized in multiple programming languages.

A compact comparison table would be valuable in the paper. The exact entries must be verified against primary specifications during the literature review; the table below records the questions and intended comparison rather than final publication claims.

The point is not that other models are deficient. They make different choices for different purposes. The literature review should identify which DataCanon distinctions are unusual or useful for scientific software without claiming that canonicalization, portability, or structured data are unique inventions.

CBOR as a data model and binary encoding

Treat CBOR as a particularly important comparison, but distinguish its underlying data model from its binary encoding conventions. CBOR can represent additional scientific structures through tags and prescribed conventions. DataCanon’s contribution is not that CBOR is incapable of representing its DCTs; it is the explicit scientific type system, constituent structure, equality contract, and canonical identity.

A CBOR-based encoding of DCIs would remain possible without changing the DataCanon semantic model. However, it would require a fixed mapping of DCTs to CBOR values, rules for structured numerical types and Arrays, and a frozen deterministic encoding profile. Avoid claiming that CBOR cannot meet these requirements. Instead, distinguish the semantic need for DataCanon from the engineering choice of canonical byte encoding.

Comparison with document / NoSQL systems

Use document databases as a particularly intuitive comparison. DataCanon shares the document-model intuition that a scientifically meaningful object may naturally be one recursively structured object rather than something that must first be decomposed into relational tables.

The abstraction level is nevertheless different. A document database is primarily concerned with storing, retrieving, indexing, querying, and updating documents. DataCanon instead specifies the types, constituent structure, equality, and canonical identity of the data itself. A DCI could therefore be stored in a document database; the two systems need not compete.

Avoid treating “NoSQL” as one formal data model. Distinguish document databases from key-value, wide-column, graph, and other systems when preparing the final literature review. The document model is the relevant comparison here.

Comparison with representations and persistence formats

Then compare against technologies operating at neighboring layers: JSON canonicalization, deterministic CBOR, schema-driven serialization, YAML tooling, HDF5, and scientific data containers used by the intended audience. Select the closest alternatives rather than cataloguing every serialization format.

Evaluate the distinctions that motivate DataCanon:

Concern Question for the comparison

Identity What equality relation and canonical encoding are provided?

Scientific types How are structured exact numbers and typed Arrays represented?

Collections Are ordered and unordered structures distinguished?

Persistence Can storage layout change while retaining the same identity?

External data How are payload metadata, content verification, and retrieval separated?

Interoperability What agreement is required across implementations?

Give a concrete reason for developing DataCanon rather than extending a particular existing package. The proposed contribution is the combination of an explicit scientific data model, canonical identity, and multiple faithful representations. Existing formats may provide related capabilities or support extensions; comparisons must establish the relevant differences rather than claiming that canonicalization or portability is unique to DataCanon.

Explain that DataCanon deliberately uses mature facilities such as NumPy and HDF5 instead of replacing their numerical and storage infrastructure. The same principle applies more broadly: DataCanon defines identity at the data-model layer and can interoperate with representation, persistence, database, and computational systems built for other concerns.

4. Software design

Organize this section around the design decisions that make the abstract data model usable in scientific software. Because Section 3 establishes the data-model / representation / persistence distinction, avoid repeating that conceptual argument here; show how the software realizes it. Keep low-level byte tables in the specifications.

Identity is separate from representation

Define DCT and DCI briefly. Equality is specified by types and constituents; canonical encoding represents that equality injectively:

x = y  if and only if  E(x) = E(y)

Fingerprinting is:

F(x) = SHA-256(E(x))

Shorthand formatting does not determine identity. Distinguish exact canonical byte equality from digest comparisons, which rely on collision resistance.

Rationale for the canonical byte encoding

Explain why Core specifies its own canonical bytes instead of adopting CBOR’s binary encoding or a deterministic CBOR profile. The present encoding follows the DCT structure directly: a type identifier and a prescribed payload, with shared rules for lengths, constituents, and machine values. This favors a small, auditable specification and direct independent implementations over introducing another encoding layer.

Canonical bytes are normative to DataCanon identity, not an incidental serialization choice. Once Version 1 is stable, changing the encoding would change fingerprints even if DCI equality remained unchanged. An alternative CBOR-based encoding could preserve the same semantics, but would require an equally precise and version-stable byte contract.

Acknowledge the potential advantages of CBOR-style compact headers and established encoding tools, particularly for small nested objects. Do not claim that the current encoding is smaller or faster without measurements. The justification is presently architectural minimalism, clarity, and direct control, not demonstrated performance superiority.

Scientific types and machine Arrays

Version 1 defines the DCTs in the Core specification. In the manuscript, highlight composable numerical structures, Complex, Record, Multiset, Bytes, Array, and external references rather than explaining every type. Verify the final DCT count against the publication release rather than baking a draft count into the prose.

Numerical DCIs include Integer, Float, Fraction, Sum, Product, Power, and Quotient, together with any additional numerical DCTs present in the finalized Version 1 specification. This is a category for constituent restrictions, not an additional DCT. Type membership makes no assertion about mathematical evaluability.

Explain the composition rule with a concrete example. Fraction’s prescribed integers are direct information. A Complex component or Power base is a DCI, whose individual DCT and recursive structure matter. Encoding reuses shared payload rules for direct fields and complete DCI encodings for variable-type constituents.

Arrays retain portable type, shape, and element identity. Their elements are machine values, not individually tagged DCIs. Portable Array types support efficient numerical containers; canonical elements use logical C order and prescribed big-endian multibyte representations, independently of native strides or layout. Nested Lists remain available for individually typed constituent data.

Distinguish the fully specified Core Array identity from Shorthand inference and conversion rules where implementation-reference behavior remains relevant.

Order belongs to the data model

Mapping entry order participates in identity; UnorderedMapping entry order does not. Likewise, Lists and Tuples are ordered, while Sets are unordered. Multiset retains repeated members; numerical collection types likewise retain the multiplicity and nesting prescribed by Core. Canonical binary encoding orders unordered constituents according to the finalized Core rules. That encoding order is not a numerical comparison or a mandated text-rendering order.

Text and conventional-type conversion

One strict Shorthand grammar accepts explicit constructors and shorter forms. Longhand uses constructors for structured values and ordinary literals where specified by Shorthand. The boundary is syntactic rather than simply the Core type classification. Formatting choices preserve DCI identity.

Define conventional types as types used directly for data and computation in a programming language, including built-in, library-provided, and user-defined types. The conversion interface constructs instances of conventional types from DCIs and constructs DCIs from supported conventional objects. A ListDC object and a conventional list object are both Python objects, but their types and equality contracts differ.

Existing conventional types include list, Fraction, and NumPy arrays. The reference implementation supplies new conventional types where suitable types are missing. Arithmetic belongs to the numerical conventional types, not to the DCT identity rules. Conversion through conventional objects need not preserve an arbitrary DCI’s identity.

Present conventional arithmetic as an optional companion interface in the paper, without introducing a separately named arithmetic facility. The seventh specification, Conventional Types, defines common behavior for independent implementations, with conformance declared per type. Core does not require that support. Link the Conventional Types guide for worked examples and the companion specification for its behavioral requirements and comparison limits.

Contained and external payloads

Bytes stores an opaque byte sequence directly in the DCI. A Record can hold its interpretation metadata. This permits a canonical binary file to contain both structured scientific data and binary payloads without companion files. External references serve the distinct purpose described below.

External payloads and persistence

ExternalFile and ExternalArray retain relative paths and content digests as semantic metadata. ExternalArray also carries portable type and shape. External bytes are not needed to decode or fingerprint their metadata.

Source location, cached bytes, and retrieval state are nonsemantic. Retrieval verifies digests and, for ExternalArray, element-stream validity. Complete packages preserve those references while placing payloads in directory-backed storage or embedding them in HDF5.

Seven specifications and the Python implementation

Summarize the specification boundaries:

Specification Responsibility

Core DCTs, equality, canonical bytes, and fingerprints

Shorthand Strict text parsing and identity-preserving rendering

Shorthand Preprocessing Optional compact Lists, substitutions, and explicit eval(...)

YAML Interchange Native YAML conversion and the $ convention

HDF5 Portable object layout, numerical datasets, and embedded payloads

Persistence Format selection, resource namespaces, verification, and transfer

Conventional Types Optional conventional construction, arithmetic, comparison, collections, and conversion contracts

Connect the contracts to explicit conversion, encoding/decoding, parser/renderer, and persistence boundaries. The DCTs and DCI equality are language-independent. The optional Conventional Types companion specifies a common behavioral foundation. Language bindings document their supported types, interfaces, additional simplifications, and conversion choices. Python classes are examples of a binding rather than normative DCT definitions.

Core does not perform computation. Optional preprocessing performs explicitly requested arithmetic before strict parsing; it does not alter DCT identity.

5. Research impact statement

Build this section from specific evidence available at submission time. Distinguish actual use from proposed applications.

Possible evidence to collect includes:

• A documented Principia Materia integration showing which scientific objects use DataCanon and what workflow it supports.
• Reproducible crystallographic or symmetry examples using exact structured Complex values and typed Arrays.
• Verified format transfers preserving canonical bytes and fingerprints.
• Benchmarks for representative numerical workloads, if performance is a material reason to adopt the package.
• Independent users, integrations, or publications, if available.

The repository tree shows tests covering conversion, canonical encoding and vectors, parsing/rendering, YAML, HDF5, and external sources. Report test results and fixtures only after running and inspecting them; filenames alone are not validation evidence.

If independent language conformance is completed, include independently generated matching canonical bytes and fingerprints, all portable Array types, and bidirectional HDF5 fixtures. Text layout and HDF5 file bytes need not match. Report completed coverage and remaining limitations explicitly.

Deferred canonical-encoding evaluation

Record a possible future comparison of the current encoding with a CBOR-inspired alternative, holding the DCT model and equality rules fixed. Compare encoded size, encoding and decoding speed, compressed size, and specification/implementation complexity. Cover every DCT, small and deeply nested objects, contained Arrays across dtype, rank, and size, ExternalArrays, and representative scientific datasets. Do not assume that large Arrays are always externalized or that any one scientific workload is representative.

This evaluation is worthwhile but is not a prerequisite for completing the remaining Version 1 implementation work. Revisit the byte encoding before finalizing the canonical identity contract if evidence reveals a substantial benefit. Otherwise retain the simpler current design.

6. AI usage disclosure

Prepare an accurate account of AI assistance with software development, documentation, or manuscript preparation, together with the human review and verification performed. This documentation and outline have received AI assistance. The implementation, tests, specifications, and documentation have also received AI assistance during this redesign. Record the reviewed scope and the actual human checks performed before preparing submission text.

Record the tools and available version information when preparing the submission. Do not describe syntax-only checks as execution, interoperability validation, or scientific verification.

7. Acknowledgements

Identify contributors, institutional support, and funding actually associated with the software. Obtain the correct grant identifiers and attribution before submission.

8. References

Prepare software and primary-source citations for the closest alternatives, the numerical/storage facilities discussed, and scientific applications used as evidence. Link the versioned DataCanon specifications and archive the software release associated with the paper.

Version 1 draft revisions

Keep this section synchronized with the final Version 1 specification. The numerical model and DCT inventory have changed during development, so publication fixtures must use the finalized encodings and terminology rather than incompatible earlier drafts.

Do not mix results from incompatible draft revisions. Tests should cover the full finalized numerical structure, longhand and shorter text forms, byte payloads, multisets, and transfers preserving the current canonical bytes.

Supporting material and remaining work

The documentation should provide the material omitted from the short paper: full type definitions, byte encodings, fixed conformance vectors, API signatures, numerical conventional-type behavior, longhand rendering, preprocessing details, and persistence examples.

Before finalizing the manuscript, resolve or clearly delimit the portability items already recorded in the seven drafts, especially implementation-reference behavior, preprocessing numerical semantics, and HDF5 datatype validation. Confirm the package’s license, release, installation instructions, tests, contribution guidance, and cited scientific evidence against the actual repository. The checklist identifies items to inspect, not claims that they are currently missing.

A longer technical treatment could develop the full model and conformance results separately. The JOSS paper should stay centered on the research software, its design choices, and demonstrated usefulness.