native_dataset
Stable type · Native arrays and readers
Tier: stable · Kind: type · Topic: Native arrays and readers
| Python | Julia | Rust |
|---|---|---|
NativeDataset | NativeDataset | NativeDataset |
Python
class NativeDataset
NativeDataset(variables: Optional[Mapping[str, NativeField]] = None, coords: Optional[Mapping[str, NativeField]] = None)
NativeDataset.__getitem__(name: str) -> NativeField
NativeDataset.__contains__(name: object) -> bool
NativeDataset.variable_names() -> List[str]
NativeDataset.coord_names() -> List[str]
The native arrays from one blob: data variables + grid coords.
Both map a file_variable name → NativeField
. __getitem__
looks in variables then coords, so nds["t2m"] and nds["time"]
both resolve (mirrors the Julia/Rust NativeDataset). Coordinates are the
dimension-coordinate fields (e.g. latitude/longitude/time); the
data variables are everything else.
Members
variable_names— Names of the data variables (not coordinates), sorted.coord_names— Names of the coordinate fields, sorted.
Example (provider)
source = DataSource(
"stations",
"csv", # the format-registry key that selects the reader
(FIXTURES / "stations.csv").as_uri(),
variables=["value", "location_id"],
reader_kwargs={"numeric_columns": ["latitude", "longitude", "value"]},
)
data = Provider(source, cache).materialize()
assert isinstance(data, NativeDataset)
assert data.variable_names() == ["location_id", "value"]
assert data["value"].data.dtype == np.float64
assert list(data["value"].data) == [152.3, 168.7, 98.1, 110.4]
# Digit-only text columns stay strings: nothing is inferred.
assert data["location_id"].data == ["1", "1", "2", "2"]
Julia
struct NativeDataset
NativeDataset(variables::Dict{String, NativeField}, coords::Dict{String, NativeField})
NativeDataset(variables, coords)
NativeDataset()
NativeDataset(variables, coords)
The native arrays a reader returns from one blob: variables (data fields keyed
by file_variable) and coords (the dimension-coordinate fields — e.g. the
grid’s latitude/longitude/time). Both map String name → [NativeField].
getindex looks in variables then coords, so nds["t2m"] and nds["time"]
both resolve.
Example (provider)
url = "file://" * abspath(joinpath(FIXTURES, "stations.csv"))
p = const_provider(cache, url; format = "csv", source_loader = "stations",
variables = ["value", "location_id"],
reader_kwargs = (numeric_columns = ["latitude", "longitude", "value"],))
@assert is_const(p) && refresh_times(p) == Float64[] # a CONST provider never refreshes
data = materialize(p)
@assert data isa NativeDataset
@assert Set(variable_names(data)) == Set(["location_id", "value"])
@assert data["value"].data == [152.3, 168.7, 98.1, 110.4]
# Digit-only text columns stay strings: nothing is inferred.
@assert data["location_id"].data == ["1", "1", "2", "2"]