EarthSciIO provider spec (v1)

Source of truth: spec/README.md .

The language-neutral contract the Python / Julia / Rust provider tracks implement against. This spec gates the cores (esio-9nb.{2,4,6}) and the providers (esio-9nb.{3,5,7}); the cross-language conformance harness (esio-9nb.9) and the S3/Zarr extensibility stubs (esio-9nb.8) consume it.

Cache-format version: v1. Bumping v1/ invalidates every cached blob.

What EarthSciIO is (the boundary)

EarthSciIO is the impure provider between “a parsed DataSource contract + a target time/grid” and “native-grid arrays in memory”. It owns: URL resolution at a time, download + content-addressed cache, and read → native arrays. It does not own the contract (ESM), the schema/types (ESS), regrid / reproject (ESD), or the solver (user). It returns raw native-grid arrays — variable-remap and regrid stay upstream/downstream.

Architecture (locked, epic esio-9nb): idiomatic per-language implementations against this shared contract, with a shared content-addressed cache spec so a file one language fetches is reused by the others — not a Rust+FFI core. Extensibility is by construction via the three registries below.

The deliverables of this spec

JSON Schemas: schemas/manifest.schema.json , schemas/cache-case.schema.json , schemas/native-field.schema.json , schemas/output-manifest.schema.json .

The Provider surface (context; detailed impl is the per-track beads)

This spec owns the cache/manifest format, the three registries, offline mode, and the conformance corpus. The Provider object that sits on top of them is specified per-language in the plan (§4.6); its surface is summarized here so the registries have a consumer to point at. The invariant this spec guarantees: the Provider depends only on the three registry interfaces — new backends register without touching it.

Provider(loader, *, window, offline=false, auth=None)   # bound to one loader
    materialize()      -> { file_variable: NativeField }       # CONST: call once
    refresh(t)         -> { file_variable: NativeField } | None # DISCRETE: at a cadence tstop; None if unchanged
    refresh_times()    -> [float]                              # the data_ingest schedule (solver tstops)
    prefetch(window)   -> None                                 # warm the cache offline-ready
    sample_at(t)       -> { file_variable: NativeField }       # refresh(t)'s data, no buffer touched; thread-safe
    with_window(w)     -> Provider                             # a provider with a named-dimension window baked in
    sample_windows_at(t, [w...]) -> [ { file_variable: NativeField } ]  # K windows, one decode of their union

PrefetchGroup({key: Provider}, *, depth, max_bytes, fetch_ahead, workers, windows, transform)
    boundaries() / start(order, from) / take(k) / ready(k) / cancel(wait) / close() / stats()

# component (a) seam drop-in (Phase-1 complement of ESS load_data):
cached_opener(loader, *, cache=None, offline=false, auth=None)  -> (url -> Dataset)
cached_fetcher(loader, *, cache=None, offline=false, auth=None) -> (url -> bytes)

NativeField = { array, coords } on the loader’s native grid (see native-field.schema.json ). CONST vs DISCRETE follows the loader’s temporal presence. Regrid is ESD/C4’s job.

refresh(t) is sample_at(t) followed by the buffer update, and a PrefetchGroup’s take(k) is bitwise equal to sample_at(boundaries()[k]) for every member: decoding ahead on background workers changes when a result is ready, never what it is. See prefetch.md .

Verifying this spec

python3 conformance/verify.py     # offline: schema-validate + run all golden cases
python3 conformance/generate.py   # deterministically (re)build the corpus