Python API reference

The primary domain objects, generated from their docstrings. SnowDb is the read/query surface; SnowDbManager wraps it with the admin/write surface. A Dataset is one gridded dataset within a snowdb, described by a DatasetSpec.

This is a curated starting selection of the public surface — extend it as more of the library becomes a supported entry point.

SnowDb

snowtool.snowdb.db.SnowDb

__init__

__init__(
    config,
    *,
    zone_layer_providers=DEFAULT_ZONE_LAYER_PROVIDERS,
)

Build a snowdb from a root config.

The single constructor: it takes a :class:~snowtool.snowdb.config.RootConfig -- loaded from a file (:meth:open) or built in code -- and resolves everything the config defines (the root for relative links, the pourpoint index/records locations, and each registered dataset). A dataset is either embedded inline (its config carried in the link) or referenced by a path link to a dataset.json; either way it is deserialized into a :class:DatasetSpec and bound to its data directory (see :meth:dataset_dir). The code follows the config rather than assuming paths.

dataset_dir

dataset_dir(
    name, dataset_config, *, base=None, default=None
)

Where name's data lives -- the single rule reads and writes share.

The config's data_dir (absolute -> anywhere; relative -> against base), else the convention default (data/<name>). base defaults to the root.

open classmethod

open(
    path,
    *,
    zone_layer_providers=DEFAULT_ZONE_LAYER_PROVIDERS,
)

Open a snowdb from its root config file -- the "from file" constructor.

path is the snowdb root directory (holding snowdb_conf.json) or the config file itself. The config is required: a root without one is not a snowdb this version understands, so this raises :class:~snowtool.exceptions.SnowDbConfigError pointing at snowtool init (the deliberate no-backwards-compat call -- there is no lenient un-initialized read path). The I/O half of construction: it reads + parses the root config, then hands it to the constructor.

available_zones

available_zones()

The query-able zone layers across this database's enabled providers.

Keyed '<provider>.<layer.key>' (e.g. 'terrain.elevation'); the union over every dataset's enabled providers, so a zone appears only if some dataset serves it. Only layers that declare a zoning scheme appear. The terrain aspect-orientation components are each their own banded axis (terrain.northness / terrain.eastness), so they are listed. The representation of a zone's valid values is its scheme's zones().

pourpoint_paths

pourpoint_paths()

The per-pourpoint record geojson under pourpoints/records/ (sorted).

pourpoints

pourpoints()

Parse and yield every stored pourpoint record.

pourpoint_triplets

pourpoint_triplets()

The station triplets of every stored pourpoint (parsed from the id).

pourpoint_record_path

pourpoint_record_path(triplet)

The canonical records/<triplet>.geojson path (: -> _).

load_pourpoint

load_pourpoint(triplet, *, index=None)

Parse the stored record for an indexed pourpoint triplet.

The index is the availability gate: only basin-bearing pourpoints are indexed (PourpointIndex.from_records skips point-only ones), so a triplet absent from the index -- a point-only record or anything dropped into records/ out of band without a pourpoint reindex -- is not served and raises :class:PourpointNotFoundError. Callers already holding the index (e.g. a listing loop) pass it in to avoid re-reading it.

pourpoint_index

pourpoint_index()

The persisted index.geojson manifest (empty if absent), mtime-cached.

Serves pourpoint list without parsing the (large) basin records. The index is maintained incrementally by import/sync/remove (an entry is reused as-is while its record and the registered-dataset set are unchanged); pourpoint reindex is the explicit full rebuild -- required after out-of-band records/ edits and after a grid change to an already-registered dataset name (the one change incremental maintenance cannot see; registering/removing a dataset self-heals). The result is cached and revalidated against the file's mtime (see :meth:_load_index), so repeated reads within one process are cheap yet still reflect an out-of-band rewrite.

pourpoint_dataset_coverage

pourpoint_dataset_coverage(triplet, dataset_name)

How fully dataset_name's grid covers pourpoint triplet's basin.

Read straight from the index's cached per-dataset coverage (computed at reindex/registration against each dataset's grid). A grid change is by definition a new dataset, so the cached value never goes stale -- and reading it avoids re-parsing the (large) basin record on every query.

A dataset registered after the index entry was written (a legacy out-of-order registration, or before a pourpoint reindex) has no key in the entry's coverage dict; that reads as :attr:~snowtool.snowdb.coverage.Coverage.NONE (no coverage) rather than an error, so a not-yet-recomputed dataset degrades to "off grid" instead of a 500. Raises only if the dataset is unknown or the pourpoint unindexed.

require_pourpoint_coverage

require_pourpoint_coverage(
    triplet, dataset_name, *, allow_partial=False
)

Query guard: raise unless dataset_name fully covers triplet.

The seam a stats/query call uses before reading rasters, closing the silent-partial-stats gap. allow_partial permits a knowingly-clipped query over a partially-covered pourpoint; a wholly off-grid one always raises. Returns the computed :class:Coverage for callers that want to log it.

dump_pourpoint

dump_pourpoint(triplet, dest_dir)

Copy a stored pourpoint record out to dest_dir (round-trip / archive).

A pure read/export -- it copies a record out without touching the database, so it lives on the read side even though the prune cascade (:class:~snowtool.snowdb.manager.SnowDbManager) also uses it.

SnowDbManager

snowtool.snowdb.manager.SnowDbManager

Owns every write against a held :class:SnowDb (its read/query surface).

Built around an already-constructed :class:SnowDb (reachable as :attr:db); :meth:open and :meth:initialize are the convenience constructors that build the read database (or its layout) and wrap it.

Concurrency: config and index writes are read-modify-write with no cross-process locking, so they assume a single writer at a time -- two admin commands mutating the root config or the pourpoint index concurrently can lose an update (last save wins). Ingest is different: it only writes per-date cogs/<date>/ directories, each committed by an atomic whole-directory swap, so bulk ingest parallelizes freely across distinct dates. Just avoid deliberately ingesting the same date from two processes at once.

open classmethod

open(
    path,
    *,
    zone_layer_providers=DEFAULT_ZONE_LAYER_PROVIDERS,
)

Open the read :class:SnowDb at path and wrap it in a manager.

initialize classmethod

initialize(
    path,
    specs=(),
    *,
    zone_layer_providers=DEFAULT_ZONE_LAYER_PROVIDERS,
)

Create the base snowdb layout + an empty root config at path.

The one entry point that creates the root structure -- the snowdb_conf.json root config (with no datasets registered; a dataset exists only once :meth:register_dataset links it, and is served only while its link is active), pourpoints/, data/, and a data/<name>/ directory per specs entry (a convenience for staging; the CLI init passes none). Idempotent: an existing config is loaded and left as is (its creation stamp and datasets preserved). Returns a manager over the root -- its read database is empty unless datasets were already registered.

register_dataset

register_dataset(
    name,
    dataset_config_path,
    *,
    link_type='path',
    active=True,
    coverage=None,
)

Commit a dataset registration: the root-config write is the commit point.

Writes datasets[name] -> a link at dataset_config_path, stored relative to the root when the config lives under the tree (a relocatable tree) and absolute otherwise (a staged-elsewhere dataset). Re-registering a name overwrites its link. active sets the link's visibility flag: registration makes a dataset exist (manageable by name); only an active one is served by readers (toggle later with :meth:set_dataset_active). Returns the updated config.

coverage (a triplet -> :class:Coverage map, produced by :meth:stage_dataset) is folded into every existing index entry under the new dataset's key before the config is written. The two writes are ordered index-first, config-second, and both are atomic (WS0), so every crash window is safe: a crash after the index write leaves only a harmless extra coverage key (readers still see the old dataset set from the config), and a crash before the config write leaves readers seeing exactly the old database. Without coverage (an out-of-band dataset register that skipped staging) only the config is written; the missing coverage key reads as Coverage.NONE until the next pourpoint reindex. Going live still needs a service restart -- the SnowDb is built once at startup.

name must be usable as a bare :meth:resolve_dataset token and a directory name, so a name containing a path separator or ending in .json (which that method's syntactic partition would read as a path) is rejected up front -- registration is the single choke point.

set_dataset_active

set_dataset_active(name, active)

Toggle dataset name's active flag in the root config.

The activation half of the register/activate split: registration says a dataset exists; this flips whether readers serve it. The config write is the commit point (atomic, like registration), and a running API server still needs a restart to see the change. Raises :class:~snowtool.exceptions.UnknownDatasetError for a name the root config does not register. Idempotent -- setting the current state re-saves harmlessly.

resolve_dataset

resolve_dataset(token)

Resolve a dataset NAME or a config path to a :class:Dataset.

The token is partitioned syntactically, so a name and a file can never shadow each other: a token containing a path separator or ending in .json is a PATH; anything else is a NAME. A path token never consults the catalog -- it must be an existing dataset config file (its NAME taken from the parent directory), else :class:~snowtool.exceptions.UnknownDatasetError. A name token never touches the filesystem -- it resolves only against the root config's registered datasets (active or not: management ops -- ingest, zone generation, diagnostics -- never care about reader visibility); an unregistered name raises the same error. To target an unregistered (staged) config, pass its path.

stage_dataset

stage_dataset(
    name, dataset_config_path, *, progress=NULL_PROGRESS
)

Build everything a new dataset needs, all invisible to readers.

The staging half of the register split: it builds the dataset from its config (:meth:_build_staged_dataset, so it works before the dataset is in self.db.datasets) and, entirely under data/<name>/ -- a directory a reader ignores because datasets come only from the root config -- creates the skeleton, rasterizes every indexed (basin-bearing) pourpoint's basin onto the new grid, and computes each pourpoint's geometric coverage of that grid. Zone layers are never generated here -- that is a separate explicit operation (:meth:generate_zone_layers_for, which shares one source read across datasets). Nothing here touches the root config or the index, so a fresh SnowDb.open still does not see the dataset until :meth:register_dataset commits it (passing back :attr:StagedDataset.coverage).

progress reports each slow phase as a sequential tracked task: parsing the pourpoint records, the per-pourpoint coverage computation, and the AOI rasterize pass. Coverage is computed first and only basins the new grid can serve (PARTIAL/FULL) are rasterized -- an off-grid basin has no window to burn, though its NONE coverage is still recorded so the index reports it as off-grid. Converge-by-default, like ingest: an existing skeleton is tolerated, and rasterization rebuilds an AOI raster only when it is absent or its provenance tag reads stale (a changed basin polygon or a format-version bump). A byte-level forced rebuild is :meth:rasterize_aois with rebuild=True (the pourpoint rasterize --rebuild command).

create_dataset

create_dataset(name, config, *, progress=NULL_PROGRESS)

Stamp a brand-new dataset name from config: stage it, then register it inactive -- the whole lifecycle the dataset create command used to orchestrate step-by-step in the CLI.

Resolves the dataset's data directory the way a later SnowDb.open will (:meth:~snowtool.snowdb.db.SnowDb.dataset_dir), writes config beside its data as data/<name>/dataset.json so :meth:stage_dataset can build from it and :meth:register_dataset can link it, stages every artifact (skeleton, AOI rasters, coverage -- but never zone layers; those are the separate :meth:generate_zone_layers_for pass), and registers the staged dataset. Converge-by-default like ingest and staging: the directory mkdir and the config write are idempotent overwrites, and staging rebuilds an AOI raster only when its provenance tag reads stale.

The one real invariant it enforces: an existing registration is never clobbered. Registration happens only when name is not already in the root config -- so a re-create of a live dataset never deactivates it or relinks its config out from under readers (its active state and link survive verbatim). A fresh registration is committed inactive (active=False) with the staged coverage folded into the index, so the dataset exists (manageable by name) but stays invisible to readers until an explicit :meth:set_dataset_active. Returns a :class:CreatedDataset carrying the staging result and whether this call registered the dataset.

rasterize_aoi

rasterize_aoi(aoi, force=False)

Rasterize a pourpoint's basin onto every registered dataset's grid.

Pourpoints are shared across datasets, but each dataset has its own grid, so an AOI must be burned once per dataset (different grids -> different tile windows and masks). Covers inactive datasets too, so activating one later is instant -- its AOI rasters already exist. Returns the resulting AOI raster keyed by dataset name.

generate_zone_layers

generate_zone_layers(
    provider_name,
    datasets,
    *,
    source=None,
    force=False,
    options=None,
    progress=NULL_PROGRESS,
)

Generate a provider's zone layers for several datasets in one pass.

Reads source (default: this database's resolved source for provider_name) once over the combined extent of datasets' grids and bins it into all of them -- e.g. terrain's aspect must be computed at the source resolution, so sharing the read is the whole point. datasets are passed as objects (registered or merely staged), so activation is irrelevant here: zone layers live under data/<name>/ regardless of whether the root config links the dataset. Only the datasets that enable provider_name are targeted (the rest have no such zone layer). options carries engine knobs (e.g. terrain's workers/ block_size). Returns each generated dataset's provenance hash, keyed by name.

generate_zone_layers_for

generate_zone_layers_for(
    datasets,
    provider_names=None,
    *,
    source_overrides=None,
    force=False,
    options=None,
    progress_factory=None,
)

Generate zone layers across datasets with one shared read per provider.

The many-datasets orchestrator over :meth:generate_zone_layers: for each selected provider it resolves the source once (an override from source_overrides, else the configured default) and reads it a single time over the combined extent of every dataset that enables that provider -- so standing up N datasets that share a provider pays that provider's expensive source read once, not N times. provider_names limits the providers (default: the union of every dataset's enabled providers); an unknown name -- selected or overridden -- raises :class:~snowtool.exceptions.UnknownZoneLayerProviderError. progress_factory builds a per-provider reporter (default: silent). Returns {provider_name: {dataset_name: hash}}, with provider keys that targeted no dataset omitted.

reindex_pourpoints

reindex_pourpoints(*, progress=NULL_PROGRESS)

Rebuild index.geojson from the records/ dir and persist it.

The explicit FULL rebuild: every record is re-parsed and the persisted index is ignored -- the recovery path for out-of-band records/ edits and for a grid change to an already-registered dataset (the one change the incremental :meth:_update_index cannot see). Coverage is re-derived against every registered dataset's current grid (active or not -- an inactive dataset carries real coverage the moment it is activated), so the manifest always reflects the live grids.

import_pourpoints

import_pourpoints(
    src, *, dry_run=False, progress=NULL_PROGRESS
)

Additively import Pourpoint(s) from a file or directory into records/.

Imports only polygon-bearing pourpoints (skips point-only ones, reports unparseable ones); never removes anything. Idempotent: re-importing a triplet overwrites its record. Updates the index incrementally (:meth:_update_index -- untouched entries are reused, not re-parsed) unless dry_run. progress reports the parse and index phases.

sync_pourpoints

sync_pourpoints(
    src,
    *,
    prune_to=None,
    dry_run=False,
    progress=NULL_PROGRESS,
)

Mirror a directory into storage: import it, then prune absent records.

Imports src (directory only), then removes every stored pourpoint whose triplet is not present in src -- dumping each to prune_to first. Removal is gated: if any pourpoint would be pruned and prune_to is None (and not a dry run), raises :class:PourpointPruneDestinationRequiredError before writing anything, so the destructive step is never silent. The index is updated incrementally (:meth:_update_index; pruned triplets simply fall out) unless dry_run. progress reports the parse and index phases.

remove_pourpoint

remove_pourpoint(
    triplet, *, dry_run=False, progress=NULL_PROGRESS
)

Remove a stored pourpoint and its per-dataset rasters; True if it existed.

Cascade-deletes the record plus every aoi-rasters/<triplet>.tif and updates the index incrementally (:meth:_update_index -- surviving entries are reused, the removed one falls out). Idempotent: removing an absent pourpoint is a no-op success.

rasterize_aois

rasterize_aois(
    pourpoints,
    datasets,
    *,
    rebuild=False,
    progress=NULL_PROGRESS,
)

Burn each pourpoint's basin onto each dataset's grid when missing or stale.

Builds the cartesian product of pourpoints x datasets, (re)building a raster only when absent or its :attr:Pourpoint.geometry_hash tag no longer matches (rebuild=True forces all). A pair whose basin does not intersect the dataset's coverage domain at all is skipped (an off-grid basin has no tile window to burn), so a batch over mixed-extent grids never trips :class:~snowtool.exceptions.GeometryOutsideGridError. progress reports the pass, advancing once per pourpoint-dataset pair (built or skipped) -- the same seam zone-layer generation uses. Returns the built vs. skipped (triplet, dataset_name) pairs.

Dataset

snowtool.snowdb.dataset.Dataset

A :class:DatasetSpec bound to its data/<name>/ directory.

Owns the per-dataset filesystem layout (aoi-rasters/, the per-provider zone-layer subdirs, cogs/) and the operations on it; grid/variables are reached through self.spec.

coverage_domain property

coverage_domain

The static region this dataset can serve (for AOI coverage).

create classmethod

create(spec, path, force=False)

Create the dataset's directory skeleton.

Zone layers (terrain, land cover, ...) are not built here: each needs a source and is generated separately by :meth:generate_zone_layers (so generation can share one source read across every dataset -- see :meth:SnowDb.generate_zone_layers).

zone_target

zone_target(provider)

This dataset's grid as a target for provider's generation engine.

generate_zone_layers

generate_zone_layers(
    provider,
    source,
    *,
    force=False,
    options=None,
    progress=NULL_PROGRESS,
)

Generate this dataset's zone-layer set for provider from source.

A single-grid pass over the source (binning only into this grid); for the multi-grid shared-source pass, see :meth:SnowDb.generate_zone_layers. options carries engine knobs (e.g. terrain's workers/ block_size); progress reports the long step. Returns the set's provenance hash.

rasterize_aoi

rasterize_aoi(aoi, force=False)

Burn aoi's basin onto this dataset's grid as an AOI raster.

The tile window is clamped to the grid (see :func:~snowtool.snowdb.grid.bounding_tiles), so a basin straddling a grid edge burns only its in-grid portion; a basin entirely outside the grid raises :class:~snowtool.exceptions.GeometryOutsideGridError (the batch paths pre-filter those by coverage instead of calling this).

aoi_raster_hash

aoi_raster_hash(station_triplet)

The AOI-geometry hash an existing AOI raster was burned from.

Reads only the COG's tags (no array decode); returns None if the raster does not exist or predates the AOI_HASH_TAG tagging.

aoi_raster_is_current

aoi_raster_is_current(aoi)

Whether a burned AOI raster exists AND matches aoi's geometry AND the current burned-raster format version.

False means missing or stale (changed geometry or an old format version) -- either way :meth:rasterize_aoi should (re)build it.

rasterize_aoi_if_needed

rasterize_aoi_if_needed(aoi, *, rebuild=False)

Build the AOI raster when missing or stale; True if it was (re)built.

rebuild=True forces a rebuild regardless of current state. The converge-by-default path (rebuild=False) skips a raster only when it is already current (a matching :attr:Pourpoint.geometry_hash tag).

remove_aoi_raster

remove_aoi_raster(station_triplet)

Delete this dataset's burned AOI raster for triplet; True if present.

ingest

ingest(source, *, force=False, progress=NULL_PROGRESS)

Ingest a source artifact into per-date COGs, via this dataset's ingester.

Delegates to spec.ingester (the dataset-kind-specific parser); raises if the dataset has no configured ingester. Returns an :class:~snowtool.snowdb.ingest.IngestResult splitting the dates written from those skipped as already current. progress reports each date's per-variable COG writes (see :meth:write_date_cogs).

write_date_cogs

write_date_cogs(
    date,
    rasters,
    *,
    source_hash,
    force=False,
    progress=NULL_PROGRESS,
)

Write a date's already-on-grid rasters into cogs/<YYYYMMDD>/ atomically.

The dataset-agnostic write side of ingest: it owns the date directory; the rasters (produced by an :class:~snowtool.snowdb.ingest.Ingester) know how to write themselves as COGs into it. source_hash is the versioned hash of the source artifact this date came from (see :data:INGEST_FORMAT_VERSION); it is both stamped on every COG (via the ingester's SOURCE_HASH tag) and used by the skip check below. Returns True if the date dir was (re)built, False if it was skipped as already current.

The whole per-date directory is the unit of commit. Writes stage into a temp dir beside the target (:func:~snowtool.snowdb.atomic.staged_dir) and are swapped in wholesale, so (a) a crash mid-ingest never leaves a partial date on disk -- a reader sees the wholly-old dir or the wholly-new one -- and (b) stale COGs from a prior, differently-named source vanish by construction rather than lingering beside the new ones and making a variable unresolvable (the finding-5 duplicate-__swe.tif bug).

Completeness is enforced at date granularity. Before any filesystem work the supplied rasters must cover every spec variable, so a source that is short a required input variable raises :class:IncompleteDatasetDataError up front; after writing, every spec variable must resolve to exactly one COG in the staged dir or the swap is abandoned and the existing date dir left untouched.

Idempotent-skip granularity is likewise per-date, not per-file: without force a date is skipped only when its dir already holds exactly the COGs this call would write (complete and free of stale members) and their stored SOURCE_HASH equals source_hash. The filename set alone is not enough: source filenames embed provenance, so a renamed re-release is caught by a name mismatch, but a re-release under the same filename with different bytes would keep the names identical -- the hash equality catches that, forcing a rebuild. A missing tag (a date dir written before hashing) also reads as stale. Any divergence rebuilds the whole date dir; force always rebuilds.

load_aoi_raster

load_aoi_raster(station_triplet)

Open the burned AOI raster for triplet (the stats read input).

Raises :class:FileNotFoundError (pointing at pourpoint rasterize) when the raster has not been built for this dataset, so a stats query surfaces a clean missing-prerequisite error rather than a bare open failure.

date_dir

date_dir(d)

The cogs/<YYYYMMDD>/ directory for date d (may not exist).

available_dates

available_dates(*, start=None, end=None)

Every date with an ingested cogs/<YYYYMMDD>/ directory, ascending.

start/end bound the result inclusively; either may be omitted for an open end.

variable_path

variable_path(d, variable)

The single COG for variable on date d, or None if absent.

missing_variables

missing_variables(d)

Spec variables whose glob matches no file in date d's cogs dir.

An absent date directory yields every variable (nothing is present).

aoi_raster_paths

aoi_raster_paths()

The burned aoi-rasters/*.tif files, sorted by path.

aoi_raster_triplets

aoi_raster_triplets()

Station triplets that have a burned aoi-rasters/<triplet>.tif.

artifact_status

artifact_status()

Which of this dataset's on-disk artifacts currently exist.

remove_date

remove_date(d)

Delete a date's cogs/<YYYYMMDD>/ directory; True if it existed.

DatasetSpec

snowtool.snowdb.spec.DatasetSpec

crs cached property

crs

The grid's CRS (pyproj), the single source for every CRS-derived value -- is_geographic, cell_area, and the dataset's rasterio write CRS (:attr:Dataset.grid_crs).

coverage_domain cached property

coverage_domain

The static region this dataset can serve.

Used by AOI coverage classification: the dataset's footprint when it declares one (e.g. a MODIS block minus a never-ingested tile), else the full grid-extent rectangle -- so a basin over a permanently-empty hole is not reported as fully covered.

is_geographic cached property

is_geographic

Whether cell area varies across the grid (geographic CRS) or is constant (projected/linear CRS). Drives whether an AOI raster burns per-row geodesic area or the constant cell_area.

cell_area cached property

cell_area

The constant per-cell area, in square metres. Only meaningful on a projected grid; raises on a geographic grid, where area varies by latitude and the AOI raster burns per-row geodesic area instead.

griffine reports a projected grid's planar cell area in the CRS's own linear units squared, so it is converted to m^2 here -- every area we emit (the area_m2 field, the CSV column) is metres regardless of the grid's units.

model_prefix cached property

model_prefix

CamelCase prefix for this dataset's generated response models (e.g. snodas -> Snodas -> SnodasZonalStat).

Names that differ only by case or -/_ collapse to the same prefix (foo-bar and foo_bar both -> FooBar), so SnowDb enforces prefix uniqueness across its specs to avoid OpenAPI schema-name collisions between datasets.

zonal_stat_model cached property

zonal_stat_model

The generated per-elevation-band response model for this dataset.

zonal_stats_model cached property

zonal_stats_model

The generated per-date response model (a date plus its zones).

enables

enables(provider_name)

Whether this dataset enables (generates + serves) provider_name.

zone_params

zone_params(provider_name, layer_key)

The configured default query params for one zone layer.

None when the provider/layer is not configured, or is configured with no params -- either way the scheme's own defaults apply, so callers pass the result to :meth:ZoneScheme.configured uniformly.

from_config classmethod

from_config(config, name)

Deserialize a :class:~snowtool.snowdb.config.DatasetConfig into a spec.

A trivial pass-through (no merge, no runtime kind): the config's grid, variables, zones and footprint are already the domain types, so they carry straight over; only the ingester name is resolved to the concrete ingester from the registry (None for a read-only/derived dataset). name is supplied separately because the config does not carry one -- it comes from where the config is registered.