Reading an esm data source

Source of truth: spec/esm-sources.md .

Status: normative. Fixture: ../conformance/corpus/scenarios/esm-url-templates.json .

An .esm document names its inputs in data_sources (esm-spec §8). This page fixes how EarthSciIO reads the parts of such an entry that locate files, so that the Python, Julia and Rust tracks fetch the same file for the same entry at the same time.

1. URL templates (esm-spec §8.2)

A source.url_template, and every entry of source.mirrors, is an esm template: literal text with {…} tokens. EarthSciIO expands one with expand_esm_template(template, anchor, substitutions) (Python and Julia) and expand_esm_template(template, Some(anchor), &substitutions) (Rust), where anchor is a UTC instant (the start of the file the URL names) and substitutions maps names to strings.

1.1 Tokens

A token is {name} or {name:format}, where name matches [A-Za-z_][A-Za-z0-9_]* and format is any text without { or }. Any other brace is literal text, so the JSON in a cds:// request ({"variable": …}) passes through unchanged. Expansion is one left-to-right pass; inserted text is never expanded again.

1.2 Date directives

<format> is literal text and these directives. Every track implements this table itself rather than calling its platform strftime, whose locale and padding rules differ.

1.3 Errors

Each is an error whose message contains its code, names the template, and names the offending token or directive. No error is ever turned into a skipped mirror or a literal URL.

1.4 Custom substitutions

esm-spec §8.2 allows any name. A document that mounts a catalogue binds names at load through the mount edge’s url_substitutions, so a document that EarthSciAST has loaded may carry none. Whatever is left (reseact’s NEI 2016 source keeps {sector}) must be bound by the caller of EarthSciIO; an unbound one is esm_template_unbound rather than a fetch of a URL with braces in it. esm_template_names(template) returns the names a template needs, in order of first appearance and without date, so a caller can check them before any fetch.

1.5 Using a template in a provider

A loader whose URL is an esm template is built with EsmTemplate (Python EsmTemplate(template, substitutions), Julia EsmTemplate(template; substitutions), Rust DataSource::new(name, format, template).esm_template(substitutions)). The provider expands it at each file anchor, as it already does for the track-specific template forms. Mirrors given beside an esm template are esm templates too: each is expanded at the same anchor with the same substitutions and tried, in order, after the primary URL. Building the template refuses esm_template_reserved, and esm_template_unbound or esm_template_directive for any non-date token of the primary or of a mirror, by checking each token where it stands (the text around a {date:…} token never forms a new token: {{date:%Y}x} has one token). The {date:…} tokens are checked when expanded. The scenario’s construction cases pin this. The cache key is still sha256(primary URL) (cache-format.md §1), whichever candidate supplied the bytes.

2. Calendar cadences (esm-spec §8.3)

Fixture: ../conformance/corpus/scenarios/calendar-cadence.json .

A source’s temporal.frequency and temporal.file_period are ISO 8601 durations. A month or a year is not a fixed number of seconds, so EarthSciIO reads each as a calendar period (Python CalendarPeriod, Julia CalendarPeriod, Rust CalendarPeriod) and computes every anchor on the proleptic Gregorian calendar in UTC. No track approximates a month by a mean length.

2.1 Periods

P[nY][nM][nW][nD][T[nH][nM][nS]], each n an unsigned integer and the designators in this order, is the period of months = 12·Y + M and seconds = 604800·W + 86400·D + 3600·H + 60·M + S (UTC days have 86400 seconds). Anything else — a fraction, a sign, a missing P, an empty T, a designator out of place, an all-zero period — is an error with the code esm_period_invalid. A period may also be built from a fixed duration (a Python timedelta, a Rust time::Duration, a whole number of seconds), which is months = 0.

2.2 Anchors

For a cadence with epoch start and period p, anchor k (any integer) is

anchor(k) = add_months(start, k · p.months) + k · p.seconds

where add_months moves the calendar month, keeps the time of day, and keeps the day of the month, clamped to the last day of the target month. Every anchor is computed from start, never from the previous anchor, so a cadence starting on 31 January visits 29 February and then 31 March. floor(t) is the largest k with anchor(k) ≤ t, and ceil(t) the smallest with anchor(k) ≥ t.

2.3 What the provider derives

With f the frequency and F the file period, both anchored at start:

A fixed-length cadence (months = 0) gives exactly the anchors it gave before. The record index in its file is the cadence offset: it is what every track uses when the file has no time axis it can decode. When it has one, the record is found on that axis instead (§2.5).

2.4 Julia

The Julia Provider used to take only a caller-made list of times in the caller’s unit, and located a record as mod1(tick, file length), which is wrong when the list does not start at a file’s first record. It now also takes a SourceTemporal and a run window, as Python and Rust do: Provider(cache, url; format, temporal, window). Times are then Unix seconds (UTC). refresh_times returns §2.3’s refresh times over the window, upper being the window end (else stop) and never clipped to stop, as in Python and Rust. A sample at any time from start on, inside the window or not, is located by §2.3 on the calendar, its record in the covering file found as in §2.5, as in Python and Rust.

2.5 The file’s own time axis

Fixture: ../conformance/corpus/scenarios/decoded-time-axis.json , written by conformance/gen_time_axis.py.

The cadence offset of §2.3 is exact for whole files, but wrong for a file that lacks some records (a trimmed file) or whose records are not on the cadence. So every track finds the record of an anchor a in the file covering it this way (Python provider._record_index, Julia _axis_record, Rust time_axis::record_index):

1. Take the file’s coordinate named time_dim. If there is none, or it does not decode (below), or it is empty, use the cadence offset.
2. If a lies between the axis’s earliest and latest times (inclusive), the record is the latest one at or before a; of equal times, the first in file order. The axis need not be sorted.
3. Otherwise use the cadence offset. A cadence offset outside the file is an error; one inside it is read, even before a trimmed file’s first record.

The two-record bracket’s successor is the record of the next anchor in the file covering it (the same file or the next one), found as above, so a record and its time agree: in a file with a gap the successor of the last record before the gap is that record again, never the one after the gap. When the successor’s file cannot be read or its record is outside that file, the floor record is held ([r, r], equal times). The bracket’s time coordinate is always the two anchors, never the file’s own times.

Decoding. decode_time_axis(values, units, calendar) (Python and Julia; Rust decode_time_axis(&values, units, calendar, fill)) gives the axis as Unix seconds, floored, or nothing. It is the Python track’s decode (xarray’s decode_cf_datetime), which the other two reproduce:

Python accepts more reference spellings than the table lists, through pandas (2016/01/01, 20160101, a month name, …); Julia and Rust do not decode those, so on such a file they fall back to the cadence offset where Python does not. Python also reads a bare one-digit hour (2016-05-01 6) as midnight; Julia and Rust refuse it. A short year followed by another run of four digits (1-1-1 00:00:00.0000, 1-1-1 06:30 -0130) is not padded by xarray, so pandas reads it its own way (as 2001, when the values fit); Julia and Rust do not decode it. None of these differences is in the fixture.

In Julia the record is chosen inside the netcdf reader’s one open: the records option takes a two-argument resolver (len, axis) -> indices, which is handed the file’s raw time_dim coordinate with its attributes (registries.md §2, the netcdf reader’s records).

3. A provider from an esm data source

Fixture: ../conformance/corpus/scenarios/esm-sources.json .

Every track builds a provider straight from one data_sources entry:

entry is the entry’s JSON value as a document carries it after EarthSciAST has loaded it, so its locations are already absolute (esm-spec §8.2.1) and a mount’s url_substitutions already applied (§4.7). window is the run window in UTC.

3.1 What is read

url_override replaces url_template and drops the mirrors (a deployment’s local copy). It is an esm template too, so it cannot carry ${EARTHSCIDATADIR} (whose {…} would read as a substitution): a caller expands that itself. records_per_sample (1 or 2) is the consumer’s choice and is not read from the entry, since esm-spec §8 has no field for it yet.

3.2 What is not read here

select, codes, record_filter, extent and a parameter’s unit_conversion say what a source’s values mean to the model; the document binding (EarthSciAST’s providers_from_document) applies them to what this provider returns. kind needs nothing from EarthSciIO: a points source is read by a table reader (csv, parquet, ff10, shapefile) like any other. determinism, reference and the rest of metadata are not read.

3.3 Errors

The message contains the code and the source’s name, esm_template_* codes included: a template error of the entry is raised as the track’s esm source error (Python EsmSourceError, its __cause__ the EsmTemplateError; Julia EsmSourceError; Rust Error::EsmSource) carrying the template code.