RFC — Template libraries, cross-file imports, and load-time metaparameters

Status: Draft (proposed changeset for review) Bead: TBD (file on acceptance) Affects spec version: 0.8.0 (rides inside the in-progress clean break; no bump, no compat shims) Scope: esm-spec.md §1.1 / §2 / §4.7 / §9.6 / new §9.7, esm-schema.json (+ 4 binding mirrors), conformance fixtures, and follow-up work in all five bindings. Discretization content is explicitly out of scope — it lives in ../earthscidiscretizations. Depends on: the open-op-namespace-fixpoint-rewrite engine (§9.6.3 outermost-first + priority + bounded fixpoint + unlowered_operator gate) being present in a binding before that binding implements this RFC. Imports only widen the rule set handed to that engine; nothing here touches the engine itself.

Two post-acceptance retractions — see §15. (a) §5.4’s “round-trip emits the expanded, folded form; neither expression_template_imports nor metaparameters nor top-level expression_templates survives parse → emit” is RETRACTED: the declarations survive, only the import edge is consumed (esm-spec.md §9.6.4 rule 5). (b) The unconditional fold in §5.3 site 5 / §5.4 is AMENDED: a template-library file loaded standalone with no loader-API bindings keeps its metaparameters open and its structural integer sites symbolic. esm-spec.md is normative; §5.3 and §5.4 below carry the corrections inline.

1. Motivation

§9.6.1 declares templates component-local and defers “top-level templates and cross-component sharing … to a follow-up RFC”. This is that RFC. The forcing use case is the rewrite of EarthSciDiscretizations — the standard library of discretization rules that §9.6.8 promises — as a layered, pure-data library of .esm files:

1. a grid file declares the index sets and geometry constants of a grid family;
2. an interior-stencil file references the grid file and defines the interior finite-difference/finite-volume operator as a named (match-less) template;
3. a BC file references the stencil file and wraps it into the complete auto-applied match rule on spatial D — a makearray whose interior region invokes the imported stencil and whose boundary-face regions encode the boundary condition (§9.6.8);
4. a consuming model imports the rule it wants and writes D(c, wrt: lon).

Today the format cannot express any of the sharing edges above:

• expression_templates cannot appear outside a single model/reaction_system (§9.6.1), so there is no such thing as a rule file.
• A template body cannot reference another template (apply_expression_template_recursive_body, §9.6.3 constraint 2), so a BC layer cannot reuse an interior stencil; every BC variant would copy-paste the stencil aggregate.
• index_sets sizes, aggregate dense ranges, and makearray regions are literal integers, so every rule is welded to one concrete resolution. A library would need a generated file per (rule × resolution), and a convergence study (n = 16, 32, 64, 128) would need generated-fixture machinery in every consuming repo.

Three coupled mechanisms close these gaps: template-library files + imports (§3), registration-time body composition (§4), and load-time metaparameters (§5). All three resolve entirely at load, before validation and before the §9.6.3 fixpoint, so the determinism contract — byte-identical post-lowering ASTs across all five bindings — is preserved unchanged.

2. Summary of changes

Explicitly NOT done: import renaming/namespacing (as prefixes) — deferred until a real collision that only + explicit priority cannot resolve is exhibited; non-integer metaparameters — deferred (every motivating site is an index bound or size); cross-file sharing of anything other than templates, index sets, and metaparameters (variables, equations, loaders keep their existing §4.7 subsystem pathway).

3. Template-library files and imports

3.1 The library-file form (§9.7.1)

A template-library file is a valid ESM document (esm, metadata) whose payload is:

• expression_templates (required, non-empty) — top-level, same entry shape as §9.6.1;
• index_sets (optional) — the document-scoped registry, as today;
• metaparameters (optional, §5);
• expression_template_imports (optional) — libraries can layer on other libraries.

It MUST NOT declare models, reaction_systems, data_loaders, coupling, or domain. Purity keeps the two reference mechanisms disjoint: a §4.7 subsystem file is never importable as a library (template_import_not_library) and a library file is never includable as a subsystem (subsystem_ref_is_template_library). §2’s existence rule becomes “at least one of models, reaction_systems, data_loaders, or expression_templates”.

3.2 The expression_template_imports field (§9.7.2)

An ordered array appearing (i) inside a model or reaction_system, or (ii) at the top level of a library file. Each entry:

Import-graph cycles (over canonical paths) are rejected with template_import_cycle. A ref that fails to load or parse is template_import_unresolved.

3.3 Effective declaration order (§9.7.4)

The §9.6.3 engine tie-breaks equal-priority rules by declaration order. With imports, that order is pinned as the depth-first post-order over the import DAG:

the effective sequence of a component (or library file) = for each entry of its expression_template_imports, in array order: the imported file’s effective sequence; then the component’s own declarations, in declaration order.

Duplicates arriving via diamond imports deduplicate at first occurrence when the two definitions are deep-equal after resolution (same instantiated body, match, priority, params). A same-name collision with different definitions — import/import or import/local — is template_import_name_conflict. The engine never infers precedence across libraries: authors state inter-library precedence with explicit priority, and the effective sequence only breaks exact ties, exactly as within a single file today.

3.4 index_sets merge (§9.7.5)

An imported file’s top-level index_sets merge into the importing document’s document-scoped registry (after metaparameter instantiation, §5.4). Deep-equal redeclaration is idempotent — diamonds are fine. A non-equal collision is template_import_index_set_conflict. This is what lets a grid file own its axes: the consuming model’s variables are shaped over ["lon", "lat"] without redeclaring them.

4. Registration-time body composition (§9.7.3)

A template body MAY contain apply_expression_template nodes referencing other templates that are in scope (declared locally or imported) and match-less (a match rule is a rewrite step, not a named fragment; referencing one is apply_expression_template_unknown_template — it is not visible as an invocable name).

After the effective sequence is fixed, and before the §9.6.3 fixpoint runs, the loader:

1. builds the reference graph over apply_expression_template nodes in bodies;
2. rejects cycles (self or mutual) with apply_expression_template_recursive_body — this diagnostic is hereby redefined from “any nesting” to “cyclic reference”;
3. rejects reference chains deeper than MAX_TEMPLATE_EXPANSION_DEPTH = 32 with template_body_expansion_too_deep;
4. inlines in topological order by §9.6.3-constraint-5 pure substitution.

Substitution is confluent (no evaluation, no capture: parameters are the only free names and each inlining closes over its own bindings), so topological order cannot affect the result. By the time any match rule is considered by the engine, every rule body is a closed Expression AST containing zero apply_expression_template nodes — the §9.6.3 engine, its pass bound, its priority selection, and the unlowered_operator gate operate on exactly the same model of the world as today.

match patterns MUST NOT contain apply_expression_template nodes (unchanged).

5. Load-time metaparameters (§9.7.6)

5.1 Declaration

A top-level document field:

"metaparameters": {
  "NLON": { "type": "integer", "default": 144, "description": "zonal cell count" },
  "NLAT": { "type": "integer", "default": 91 }
}

type is required and MUST be "integer" (the only v1 type). default is optional. A metaparameter name MUST NOT collide with any variable, parameter, species, or index-set name visible in the document (metaparameter_name_conflict) — there is no shadowing.

5.2 Admissible sites

A metaparameter expression is: an integer literal; a declared metaparameter name; or {"op": <"+"|"-"|"*"|"/">, "args": [<metaparameter expressions>...]} (with unary - allowed). Metaparameter expressions are admissible wherever the schema previously required a bare integer in a structural position:

• index_sets.<name>.size (interval kind);
• aggregate dense ranges tuple entries ([start, stop] / [start, step, stop]);
• makearray regions bound pairs.

These sites are folded to concrete integers at load with exact integer arithmetic; / MUST divide exactly and folding MUST NOT overflow a 64-bit signed integer, else metaparameter_type_error.

In ordinary expression positions (template bodies, equations, index args, …) a metaparameter name is written as a bare string — the same surface syntax as a variable reference — and is substituted as an integer literal during load. No folding happens in expression positions: {"op": "/", "args": [360, "NLON"]} becomes {"op": "/", "args": [360, 144]} and stays an AST division. (Folding only where the schema demands an integer keeps the rule trivially deterministic; everything else is the evaluator’s job.)

Substitution is per-field, not per-node. A structural string field of an Expression node holds a name, not a reference to a value, and is copied verbatim even when a bound metaparameter spells it exactly — in particular the axis-naming scalars wrt, dim and integral’s var, which name a spatial coordinate (esm-spec §4.9.1, §4.2) and are the same three the §9.7.7 rename walk rewrites through isetmap. So with x bound to 3, {"op": "grad", "args": ["x"], "dim": "x"} becomes {"op": "grad", "args": [3], "dim": "x"} — the args occurrence folds, the dim occurrence does not.

5.3 Binding sites and value flow

Bindings flow down the reference DAG; open (unbound) metaparameters flow up:

1. Import edge: expression_template_imports[k].bindings closes the named metaparameters of the imported document — the imported subtree is instantiated with those values before its templates/index_sets enter the importer’s scope.
2. Re-export: metaparameters of an imported document left unbound at the edge are inherited into the importing document’s own metaparameter scope (deep-equal declarations dedupe; conflicting redeclarations are template_import_name_conflict). Binding NLON once at the top of a four-file chain therefore reaches the grid file at the bottom.
3. Subsystem edge: a §4.7 {"ref": …} subsystem reference MAY carry the same bindings field, closing the referenced document’s metaparameters (e.g. a ten-line convergence wrapper binding a problem file to n = 32).
4. Loader API: hosts MAY bind the root document’s open metaparameters at load (load(file, metaparameters = {...})) — the command-line -D of the mechanism.
5. Defaults, last: at the root document’s close, after API bindings, any still-open metaparameter takes its default; one with no default is metaparameter_unbound. (Amended — see §15.2.) This site does not run when the root document is a template-library file loaded with no loader-API bindings: there the metaparameters stay open by design, and an open one is not metaparameter_unbound whether or not it declares a default. An edge binding is likewise not required to be total: what an edge leaves unbound re-exports upward by site 2.

Because instantiation happens per import edge, a diamond whose two paths bind the same underlying file differently produces non-deep-equal templates or index sets and is rejected by the existing conflict diagnostics — no new machinery, no ambiguity.

5.4 Ordering within load

For each document, innermost-first: resolve imports (recursively, instantiating at each edge) → merge index_sets → close + fold this document’s metaparameters (conditionally — see the correction below) → then §4 body-composition and the §9.6.3 fixpoint on fully-concrete trees. Validators run on the folded form.

Corrected — see §15. The original text of this subsection continued: “Round-trip emits the expanded, folded form; neither expression_template_imports nor metaparameters nor top-level expression_templates survives parse → emit; source files remain the source of truth.” Both halves of that sentence are wrong and the corrected rules are:

• Declarations survive; only the call site is consumed. expression_template_imports is a call site and does not survive parse → emit. A top-level expression_templates registry and a top-level metaparameters block are declarations — peers of index_sets — and survive verbatim. (Retraction (a), §15.1.)
• The fold is conditional. When the root document is a template-library file (§3) and the load supplies no loader-API metaparameter bindings, the “close + fold this document’s metaparameters” step does not run: the metaparameters stay open, the structural integer sites (index_sets.<name>.size, dense ranges, makearray regions) stay symbolic exactly as authored, and an import-free library file round-trips to itself. Every other load — through an import edge, through a subsystem edge, with loader-API bindings, or of any file that is not a library — closes and folds as described above. (Amendment (b), §15.2.)
• Emit is the reference-preserving form of esm-spec.md §9.6.4 (Option B), not the expanded one; this RFC predates that change. Source files remain the source of truth for the consumed constructs only — import edges and eagerly-expanded call sites.

6. Worked example — the four-file layering

File 1 — grid_latlon.esm (grid: metaparameters, axes, geometry constant):

{
  "esm": "0.8.0",
  "metadata": { "name": "grid_latlon", "description": "Uniform lon-lat grid, NLON x NLAT" },
  "metaparameters": {
    "NLON": { "type": "integer", "default": 144 },
    "NLAT": { "type": "integer", "default": 91 }
  },
  "index_sets": {
    "lon": { "kind": "interval", "size": "NLON" },
    "lat": { "kind": "interval", "size": "NLAT" }
  },
  "expression_templates": {
    "dlon_deg": { "params": [], "body": { "op": "/", "args": [360, "NLON"] } }
  }
}

File 2 — central_D_lon_interior.esm (interior stencil; match-less):

{
  "esm": "0.8.0",
  "metadata": { "name": "central_D_lon_interior" },
  "expression_template_imports": [ { "ref": "./grid_latlon.esm" } ],
  "expression_templates": {
    "central_D_lon_interior": {
      "params": ["f"],
      "body": {
        "op": "faq", "output_idx": ["i", "j"], "args": ["f"],
        "ranges": { "i": [2, { "op": "-", "args": ["NLON", 1] }], "j": { "from": "lat" } },
        "expr": { "op": "/", "args": [
          { "op": "-", "args": [
            { "op": "index", "args": ["f", { "op": "+", "args": ["i", 1] }, "j"] },
            { "op": "index", "args": ["f", { "op": "-", "args": ["i", 1] }, "j"] } ] },
          { "op": "*", "args": [2, { "op": "apply_expression_template", "args": [],
                                     "name": "dlon_deg", "bindings": {} }] } ] }
      }
    }
  }
}

File 3 — central_D_lon_zero_grad_bc.esm (the complete auto-applied rule):

{
  "esm": "0.8.0",
  "metadata": { "name": "central_D_lon_zero_grad_bc" },
  "expression_template_imports": [ { "ref": "./central_D_lon_interior.esm" } ],
  "expression_templates": {
    "central_D_lon_zero_grad_bc": {
      "params": ["f"],
      "match": { "op": "D", "args": ["f"], "wrt": "lon" },
      "body": {
        "op": "makearray", "args": [],
        "regions": [
          [[2, { "op": "-", "args": ["NLON", 1] }], [1, "NLAT"]],
          [[1, 1], [1, "NLAT"]],
          [["NLON", "NLON"], [1, "NLAT"]]
        ],
        "values": [
          { "op": "apply_expression_template", "args": [],
            "name": "central_D_lon_interior", "bindings": { "f": "f" } },
          { "op": "faq", "output_idx": ["j"], "args": ["f"],
            "ranges": { "j": { "from": "lat" } },
            "expr": { "op": "/", "args": [
              { "op": "-", "args": [ { "op": "index", "args": ["f", 2, "j"] },
                                     { "op": "index", "args": ["f", 1, "j"] } ] },
              { "op": "apply_expression_template", "args": [], "name": "dlon_deg", "bindings": {} } ] } },
          { "op": "faq", "output_idx": ["j"], "args": ["f"],
            "ranges": { "j": { "from": "lat" } },
            "expr": { "op": "/", "args": [
              { "op": "-", "args": [ { "op": "index", "args": ["f", "NLON", "j"] },
                                     { "op": "index", "args": ["f", { "op": "-", "args": ["NLON", 1] }, "j"] } ] },
              { "op": "apply_expression_template", "args": [], "name": "dlon_deg", "bindings": {} } ] } }
        ]
      }
    }
  }
}

File 4 — a consuming model, at double resolution:

{
  "esm": "0.8.0",
  "metadata": { "name": "advection_lon" },
  "models": {
    "Advection": {
      "expression_template_imports": [
        { "ref": "https://earthsciml.org/lib/central_D_lon_zero_grad_bc.esm",
          "only": ["central_D_lon_zero_grad_bc"],
          "bindings": { "NLON": 288, "NLAT": 181 } }
      ],
      "variables": {
        "c": { "type": "state", "units": "kg/m^3", "shape": ["lon", "lat"], "description": "tracer" },
        "u": { "type": "parameter", "units": "m/s", "default": 5.0, "description": "zonal wind" }
      },
      "equations": [
        { "lhs": { "op": "D", "args": ["c"], "wrt": "t" },
          "rhs": { "op": "*", "args": [ { "op": "-", "args": ["u"] },
                                        { "op": "D", "args": ["c"], "wrt": "lon" } ] } }
      ]
    }
  }
}

At load: file 4’s import edge binds NLON/NLAT and instantiates files 3 → 2 → 1; lon/lat (sizes 288/181) merge into file 4’s registry; dlon_deg and the interior stencil inline into the rule body at registration; the unchanged §9.6.3 fixpoint lowers D(c, wrt: lon); the gate confirms nothing unlowered survives. Byte-identical across bindings. The same four files serve a convergence sweep by re-binding at the edge — no generated fixtures.

Reprojection fragments and regridding coupling expressions share by the same mechanism: a library exports match-less coordinate-transform or overlap-weight templates; a coupling transform invokes them.

7. Determinism

• Import resolution, instantiation, dedup, and conflict checks depend only on file content and the (ordered) import arrays — no filesystem enumeration, no clocks.
• Metaparameter folding is exact 64-bit integer arithmetic with mandatory divisibility — no floating point, no platform variance.
• Body inlining is confluent pure substitution over a statically-acyclic DAG.
• The rewrite engine’s inputs (rule sequence + concrete trees) are therefore identical across bindings, and §9.6.3 already guarantees the rest: all five bindings MUST produce byte-identical post-lowering canonical ASTs, or the same rejection diagnostic.

8. Diagnostics (new §9.7.8; mirrored into the §9.6.6 table)

9. Schema changes (esm-schema.json, 0.8.0 — in-progress clean break, no bump)

1. New $defs/TemplateImport: object, additionalProperties: false, required ["ref"]; ref string (minLength 1); only unique non-empty string array (minItems 1); bindings object of integers.
2. New $defs/Metaparameters: object; each value {type: "integer" (required const), default?: integer, description?: string}, additionalProperties: false.
3. New $defs/MetaparameterExpression (recursive): oneOf integer / string / {op: enum["+","-","*","/"], args: [MetaparameterExpression...] (minItems 1)}.
4. Top-level properties: add expression_templates (additionalProperties: {"$ref": "#/$defs/ExpressionTemplate"}), expression_template_imports (array of TemplateImport), metaparameters ($ref Metaparameters). Extend the root anyOf with {"required": ["expression_templates"]}.
5. $defs/Model, $defs/ReactionSystem: add expression_template_imports beside expression_templates.
6. $defs/SubsystemRef: add optional bindings (object of integers).
7. $defs/ExpressionTemplate: params.minItems 1 → 0; description updated for §9.7.3 (bodies may reference match-less templates; resolved at registration; acyclic; depth ≤ 32; match may not).
8. Metaparameter-expression widening: $defs/IndexSet.size → oneOf [integer, $ref MetaparameterExpression]; ExpressionNode.ranges dense-tuple items and ExpressionNode.regions bound-pair items likewise. (String metaparameter names are covered by the MetaparameterExpression ref.)

Library-file purity, DAG acyclicity, conflict detection, folding, and version gating are resolver-level checks (the resolver_only convention of tests/invalid/expected_errors.json). Run scripts/sync-schema.sh, then scripts/sync-schema.sh --check.

10. Spec changes (esm-spec.md)

Also: esm-libraries-spec.md §2.1b gains the import-resolution algorithm (parallel to the subsystem algorithm); ESM_COMPLIANCE_VALIDATION_MATRIX.md gains rows for each new diagnostic; CONFORMANCE_SPEC.md fixture-category text mentions the new fixtures.

11. Conformance fixtures

tests/conformance/expression_templates/:

• import_smoke/ — the §6 four-file example (grid_latlon.esm, central_D_lon_interior.esm, central_D_lon_zero_grad_bc.esm, fixture.esm) + expanded.esm. All five bindings: structurally-equal post-lowering model.
• import_diamond/ — two import paths reaching the same grid file with equal instantiation; dedupe; golden.
• import_order_determinism/ — two libraries with equal-priority rules matching the same node: winner fixed by import order; a variant flips the winner with explicit priority. Also entries in tests/conformance/determinism/manifest.json.
• metaparameter_resolutions/ — one problem file loaded at n = 4 and n = 8 via subsystem-ref bindings wrappers → two goldens.

tests/invalid/template_imports/ (+ expected_errors.json, resolver_only where appropriate): import_cycle_a/b.esm, import_not_library.esm, subsystem_ref_is_library.esm, import_name_conflict.esm, import_unknown_only.esm, index_set_conflict.esm, body_apply_cycle.esm, body_chain_too_deep.esm, metaparameter_unbound.esm, metaparameter_inexact_division.esm, metaparameter_name_conflict.esm, import_version_too_old.esm.

tests/valid/template_import_minimal.esm + sibling library files (library files are schema-valid documents and enter the valid suite; each binding’s valid-loop must tolerate a model-less document).

12. Binding implementation notes

Per binding, the work is: (a) extend the existing ref-loading module to load library files under the same visited-set cycle detection; (b) import-graph resolution, per-edge instantiation, effective-order construction, only filtering, dedup/conflict checks, index_sets merge; (c) metaparameter close + fold pass; (d) registration-time DAG inlining feeding the existing fixpoint engine; (e) the version gate and the §8 diagnostic codes; (f) loader-API metaparameter overrides.

Julia is the reference implementation and generates all goldens.

13. Phasing

1. Spec + schema (this RFC, esm-spec.md, esm-schema.json + mirrors, matrix/docs sweep) — one logical change per SCHEMA_CHANGE_PROCEDURE.md.
2. Julia reference + the §11 fixtures with Julia-derived goldens.
3. Python / TypeScript / Rust / Go ports (parallel; each lands behind the §9.6.3 engine for that binding).
4. EarthSciDiscretizations bootstrap consumes the mechanism (separate repo effort).

14. Open questions

• Import renaming (as prefixes) — deferred; revisit on first real collision.
• Non-integer metaparameters (e.g. a real-valued domain extent) — deferred; extents can live as ordinary parameters whose defaults are expressions over metaparameters.
• §6.6.5 inline-test reference expressions over unstructured grids (reference needs coordinate variables, not dimension names) — tracked separately; from_file references cover the gap meanwhile.

15. Post-acceptance corrections to §5.4 (round-trip and the fold)

esm-spec.md is normative. This section records the two places where §5.4 as originally written misled implementers, so a reader of this RFC is not misled the same way a third time. Both corrections are carried inline in §5.3 and §5.4 above.

15.1 RETRACTED — “nothing §9.7 survives parse → emit”

§5.4 asserted that neither expression_template_imports nor metaparameters nor a top-level expression_templates registry survives parse → emit. That conflated a declaration with a call site. A pure template-library file’s only payload is its top-level registry, so stripping it emitted {esm, metadata, index_sets} — a document carrying none of the top-level payload keys the schema’s top-level anyOf requires. A conforming library file was legal on disk and illegal the instant it was loaded and re-emitted. All five bindings implemented the retracted rule; all five had to be fixed.

Corrected rule (esm-spec.md §9.6.4 rule 5, §9.7.6 “Ordering within load”): the import edge is a call site and is expanded away; the top-level registry and metaparameters block are declarations, peers of index_sets, and survive verbatim.

15.2 AMENDED — the fold is conditional, not unconditional

The §15.1 patch fixed the declarations but left §5.4’s unconditional “close + fold” in place, which contradicted the round-trip requirement it had just introduced: a library whose index_sets.<name>.size is the metaparameter reference "N" cannot both fold to size: 8 and round-trip to itself. Two consequences reached the bindings:

1. A conforming library that declares a metaparameter with no default was unloadable in all five bindings — they applied §5.3 site 5 on the standalone path and raised metaparameter_unbound. But nothing requires a library to supply defaults, and site 2 (re-export) positively depends on names staying open: that is how binding once at the top of a chain reaches the grid file at the bottom.
2. Re-emitting a library folded its sizes at their defaults while leaving metaparameters in the file, so the emitted file still looked generic. Importing it afterwards with N = 16 silently produced size: 8, with no diagnostic.

Corrected rule (esm-spec.md §9.6.4 rule 5, “Two load modes”): a generic load — root is a template-library file, no loader-API bindings — leaves metaparameters open and structural integer sites symbolic, does not run site 5, and never raises metaparameter_unbound; an instantiating load — bindings supplied, or the file reached through an import or subsystem edge, or the file is not a library — closes, folds, and emits concrete. Both modes are idempotent. This mirrors the carve-out esm-spec.md §9.6.1 already states for where/shape constraint registration: “Loading or validating a library file standalone does not run this check (no component registers its rules).”

On the counter-argument that index-set sizes are not §9.7 machinery. Two bindings’ authors reasoned in writing that the index-set registry should be attached folded because “index sets are not §9.7 machinery and their sizes are genuinely resolved by it”. That holds for a size that is an integer literal, and for any size in a document that has actually been instantiated. It does not hold for a size that is a metaparameter expression in a library: that site’s binding site is the import edge, not the library’s own load, and §5.3 site 1 already says so — an edge binding value carrying open importer names “folds when the importing document closes … the same deferred fold that leaves an open index-set size symbolic until its binding site”. Folding at a standalone emit resolves the site against a binding site that has not happened yet, which is exactly the silent-size: 8 defect above.