EarthSciIO extensibility registries (v1)

Source of truth: spec/registries.md .

Deliverable (b) of esio-9nb.1. Status: normative. Machine-readable companion: registries.json .

EarthSciIO is extensible by construction through three registries. Each registry is a name → implementation lookup. The single load-bearing rule:

A new backend registers under a new name WITHOUT touching the Provider API. The Provider depends only on the three interfaces below and resolves the concrete implementation by name at runtime. Adding S3 transport, a Zarr reader, or an object-store backend is a registration, never a Provider edit.

This is what lets the future S3-proxy + NetCDF→Zarr cloud path slot in later. S3 and Zarr are registered now as stubs (status:"stub" in registries.json) and exercised through the registries by esio-9nb.8; their real implementations land later — with zero change to Provider code. What those real implementations must deliver is charted in cloud-future.md (the esio-cloud epic).

The three registries are orthogonal: a single fetch composes one entry from each — transport gets the bytes, store holds them, format decodes them.

  resolved_url ──▶ [transport:scheme] ──▶ bytes ──▶ [store:name] ──▶ cached blob
                                                          │
                                          cache key = sha256(resolved_url)
                                                          ▼
                                              [format:name] ──▶ native arrays

Interfaces are given as language-neutral pseudo-signatures. Each language track binds them to its idiom (Python Protocol/ABC, Julia abstract type + methods, Rust trait); the per-language Provider signatures live in the plan (§4.6) and the per-track beads.

1. transport registry

Keyed by URL scheme. Fetches a resolved URL’s bytes into the cache. Bypassed entirely in offline mode (the transport is never constructed when offline=true).

interface Transport:
    schemes() -> [string]                         # e.g. ["http","https"]
    fetch(resolved_url: string,
          dest: WritablePath,                     # a tmp/<uuid>.part staging path
          conditional: {etag?, last_modified?},   # for revalidation; may be empty
          auth: AuthResolver?) -> FetchResult

FetchResult = {
    status: "downloaded" | "not_modified",        # not_modified ⇒ 304, reuse cache
    etag?: string, last_modified?: string,        # to persist into the manifest
    bytes_written: int
}

Registration key = URL scheme. The fetch layer reads the resolved URL’s scheme and looks up the transport; an unknown scheme is a registration gap, not a Provider change. Auth resolvers (CDS/FIRMS/OpenAQ/RDA/bearer) are a separate pluggable map injected as auth, never baked into a transport.

2. format registry

Keyed by format name (the “reader” registry). Opens a cached blob and returns CF-decoded native-grid arrays keyed by the on-disk file_variable name, plus native coordinates.

interface Reader:
    formats() -> [string]                         # e.g. ["netcdf"]
    extensions() -> [string]                      # sniff hints: ["nc","nc4"]
    open(blob_path: Path) -> Handle
    read_native(handle: Handle,
                variables: [string],              # file_variable names to read
                select: Selection,
                records: Records?) -> { string: NativeField }   # + coords
    dim_length(path: Path, dim: string) -> int?    # header only (§2.3)
    configured(options: {string: any}) -> Reader?  # loader-declared decode options

Records = { dim: string, indices: [int] }          # 0-based, file-local (§2.1)

NativeField = { dtype, dims: [string], shape: [int], data, fill_value? }

2.1 Reader options (reader_kwargs)

A loader carries format-specific decode options alongside its format name: Python/Julia spell them reader_kwargs on the DataSource and splat them into read_native; Rust spells them DataSource.reader_options and resolves them once, at Provider construction, through Reader::configured (its trait method signature takes no kwargs, so a configured reader instance is the Rust idiom). Either way the meaning is one thing:

How a format is decoded is a property of the DECLARATION, not of the caller. An .esm data loader that says {"member_glob": "*egu*", "skip_header_row": true} decodes correctly through any binding, without a caller hand-building a registry of pre-configured readers.

An option a reader does not recognise MUST be an error at construction, not a silently-ignored key — an ignored member_filter typo reads back as an empty selection or a mis-parsed table, arbitrarily far from its cause. Reader options never enter the cache key (the blob is the whole file; member selection and header handling are decode-side).

Per-track availability is machine-readable, not prose. registries.json carries a reader_options array on the format entry, and each option lists the tracks that really provide it (plus extensions for anything one track adds in process, and used_by_provider for whether each track’s Provider chooses to use it). This file’s tables are the explanation; that array is the contract. The reason is specific: an out-of-process caller reads registries.json to decide what to ask a binding for, so an option advertised there and implemented in one track only reads back as a TypeError/MethodError in the others — and no conformance case can catch that, because a corpus case pins decoded ARRAYS and a missing option produces no array to compare. Each track therefore asserts the file’s claims about itself against its own readers (tests/test_registry_dispatch.py, julia/test/test_registries.jl, rust/tests/registry_spec.rs); all three read the same file, which is what makes the three tracks’ claims about each other checkable.

2.2 Selection capability (supports_selection / store_backed)

A reader declares supports_selection when it can honour an orthogonal select without materialising the whole array. That is the whole promise; it says nothing about what is FETCHED, and the two are genuinely different:

Callers read the pair, and there is deliberately no third capability flag: supports_selection answers “may I push this down?”, store_backed answers “will the download shrink too?”. EarthSciAST’s provider_supports_selection is a single boolean, and a second capability would have to be threaded through it for a distinction only a caller optimising transfers can act on.

A selection never changes the cache key. The blob a whole-file reader windows is the same blob it would have read whole, keyed by sha256(resolved_url); the #bytes=<a>-<b> convention (cache-format.md §1) exists for the different case where the fetch narrows.

Hard boundary (Risk R3): the reader applies read/decode semantics only — CF scale_factor/add_offset, _FillValue → NaN, endianness, chunking. It does not remap file_variable → schema name and does not apply the loader’s unit_conversion. Those are ESS contract semantics and stay in ESS. The native array is keyed by the on-disk variable name.

Format is selected by the loader’s declared format (or a content-type / extension sniff), never by trusting the cache-blob suffix alone.

2.3 Metadata shape queries (array_shape / dim_length)

A caller decides WHAT to ask for before paying to decode anything, so both reader families expose a metadata-only shape query:

Both default to “cannot answer” (nothing/None) for a reader that does not implement them, and a caller must treat that as “read whole and slice on my own side” rather than as an error. dim_length is what makes a records pushdown (conformance.md, “NetCDF decode notes”) expressible out of process at all: the records a cadence owner wants are mod1(tick, len) of the file’s own record axis, so len has to be known before the decode that the selection is meant to narrow. All three tracks answer it for netcdf (registries.json format.netcdf.metadata_queries is the machine-readable list).

In process, that costs a second open of the blob — measured at 16.9 ms on a GEOS-FP A1 file against the 2.9 ms the record selection saves there — which is why the records option ALSO accepts a len -> indices callable resolved inside the decode’s own open in the Julia track, whose Provider pushes down on every sample and would otherwise pay that second open per sample. The list stays the wire form and the only portable one: Python and Rust refuse a callable by name, because a spelling that cannot cross a process boundary has no place in a contract an out-of-process caller reads.

3. store registry

Keyed by store name (the “backend” registry). Where the content-addressed cache physically lives. Realizes the layout in cache-format.md ; the cache key is store-independent.

interface Store:
    name() -> string                              # "local"
    exists(key: string) -> bool
    get_blob(key: string) -> Path | bytes | None  # None ⇒ cache miss
    put_blob(key: string, staged: Path) -> void   # atomic commit from tmp staging
    get_meta(key: string) -> Manifest | None
    put_meta(key: string, manifest: Manifest) -> void
    lock(key: string) -> Lock                      # advisory; scope = one blob fetch

Registration key = store name (config-selected). Swapping local→s3 changes where blobs live; the Provider, the key scheme, and every reader are untouched.

4. How the Provider stays unchanged (the invariant, restated)

Provider(loader, *, window, offline, auth)        # depends on the 3 INTERFACES only
   transport = transport_registry[scheme_of(url)] # resolved by name
   store     = store_registry[config.store]        # resolved by name
   reader    = format_registry[loader.format]      # resolved by name

Adding a backend = add one row to the relevant table in registries.json and register its implementation. No row in this document, and no line in the Provider, changes shape when S3/Zarr/object-store arrive. esio-9nb.8 proves this by registering and exercising the S3 + Zarr stubs through exactly these three lookups.