Provenance and staleness

Every derived artifact a snowdb writes is expensive to produce and cheap to get wrong: a burned AOI raster, a terrain or land-cover layer set, a date's ingested COGs each cost a reprojection, a large source read, or a multi-hundred-megabyte download. Rebuilding them unconditionally on every command would be wasteful; trusting whatever is already on disk would be wrong the moment an input changes. snowtool splits the difference by stamping each artifact with a provenance tag that captures exactly what it was built from, and reducing "is this still current?" to a string comparison against what it would be built from now. The machinery is small, and each check is a cheap header read plus an equality test (the AOI raster's check bundles a few — hash, grid, tile-bbox — since it is shared with doctor; see below).

The tag is a versioned hash: v{format_version}:{sha256}, built by versioned_hash(version, digest) in snowdb/provenance.py. The digest covers the artifact's content or source geometry; the prefix is the on-disk format version of the artifact the hash guards. Storing the two together is the whole trick. A content change moves the digest, so the tag no longer matches — but a change to how the artifact is encoded on disk (a compression switch, a boolean-mask-to-cell-area redesign, a new band layout) moves nothing in the digest, yet is just as much a reason to rebuild. Folding the format version into the same string means one equality check catches both: bump the producer's version and every artifact written under the old one reads as stale, forcing a rebuild, even though its underlying digest is unchanged. All versions start at 1 — this is a greenfield database, so there is no legacy data to stay compatible with.

The format version is owned by whatever produces the artifact, never centralized. A zone-layer provider carries its own format_version (see ZoneLayerProvider in snowdb/zones/zone_layer.py); the AOI-raster writer owns AOI_RASTER_FORMAT_VERSION in snowdb/aoi_raster.py, because an AOI raster has no ingester or provider — the Dataset burns it generically; and ingested COGs carry INGEST_FORMAT_VERSION from snowdb/ingest.py. Each producer bumps its own version on a material change to its output format, and only artifacts of that kind go stale.

The tags

Four hashes and one non-hash tag do the work. The SNOWTOOL_* provenance tags are declared with their rationale in snowdb/constants.py; the ingest source hash is spelled in snowdb/raster/cog.py because it is a source-record tag, not one of the geometry tags.

Tag	Digest of	Stamped on	Drives
SNOWTOOL_AOI_HASH	basin polygon WKB sha256	the AOI raster	pourpoint rasterize rebuild when missing or stale
SNOWTOOL_DEM_HASH	generated mean-elevation array sha256	every layer of a terrain set	terrain-set reconciliation; stale-format findings
SNOWTOOL_NLCD_HASH	generated percent-forest array sha256	every layer of a land-cover set	land-cover reconciliation; stale-format findings
SOURCE_HASH	the source artifact's bytes (sorted, multi-file)	every COG of an ingested date	per-date ingest skip
SNOWTOOL_TILE_BBOX	(not a hash — a tile bounding box)	the AOI raster	AOI window resolution (see below)

SNOWTOOL_AOI_HASH is the versioned hash of a pourpoint's basin geometry. Pourpoint.geometry_hash (snowdb/pourpoint.py) is the sha256 of the polygon's canonical little-endian WKB — only the basin polygon, never the point or the source properties, because only the polygon affects the burned raster — and aoi_provenance wraps it with AOI_RASTER_FORMAT_VERSION. When a Dataset rasterizes an AOI it decides whether to rebuild through the shared health check aoi_raster_issues (snowdb/aoi_raster.py) — the same check doctor reports with, so the writer's skip decision and the reader's report can never drift. Dataset.aoi_raster_is_current rebuilds when that check finds any actionable issue and skips otherwise. For the writer the check is header-only (a tags() read plus the transform/shape, no array decode): it compares the stored SNOWTOOL_AOI_HASH to the current aoi_provenance (a changed basin, a changed nodata mask, or a format bump), and also the raster's own transform against the grid and the presence of its tile-bbox tag — so a grid move or a corrupt/unreadable raster now forces a rebuild too, not just a hash mismatch. A clean check skips the work entirely; --rebuild forces it regardless.

SNOWTOOL_DEM_HASH is the versioned hash of a terrain generation's mean-elevation array, stamped identically on every layer the pass writes (elevation, aspect components, aspect-majority). Because it is the same on every layer, any present layer carries it, so ZoneLayerSet.provenance_hash reads it from just the first. It identifies the DEM a whole terrain set was derived from, so a set can be reconciled against its source. SNOWTOOL_NLCD_HASH is the exact land-cover analogue over the percent-forest array. The single generation digest is computed once for the whole streaming pass and then stamped — finalize_and_stamp in snowdb/zones/generate_common.py digests each target's name plus its finalized array in sorted order, turns that into a versioned hash, and writes it onto every output — so everything produced together reconciles as one set.

SOURCE_HASH guards ingest. hash_files (snowdb/provenance.py) computes a single streaming sha256 over the source artifact's bytes, reading each file in 1 MiB chunks (a SNODAS tar can be hundreds of megabytes) and digesting the files in sorted order so a date built from many tiles is independent of iteration order. Wrapped with INGEST_FORMAT_VERSION, it rides on every COG of the date. The per-date skip in Dataset.write_date_cogs reads it back (again header-only, via Dataset._date_source_hash) and rebuilds unless it matches. This closes a gap the filenames alone cannot: source filenames embed provenance, so a renamed re-release is already caught by a name mismatch, but a re-release under the same filename with different bytes would keep the names identical — the hash catches that. dataset ingest is therefore converge-by-default: a date whose COGs carry the current source hash is left untouched; --force bypasses the check and rebuilds regardless.

SNOWTOOL_TILE_BBOX is not provenance at all — it rides on AOI rasters to record the grid-tile bounding box the burned window spans (ul_row ul_col br_row br_col), which the reader uses to resolve the window's origin and tiles. It is documented with the AOI raster itself; see AOI rasters and rasterization.

Why AOI rasters carry area, not elevation

The most consequential provenance decision is what an AOI raster does not contain. A burned AOI raster holds per-pixel geographic cell area (m²) inside the basin and 0 outside — it is simultaneously the in/out-of-basin membership mask and the area weights the zonal reduction needs, with no separate area raster. It carries no elevation, aspect, or forest-cover values. Those are read live from the terrain and land-cover sets at query time and crossed against the AOI mask then, not baked in when the basin is burned.

That decoupling is deliberate, and its only AOI-side provenance axis is the geometry. The cell areas are a pure function of the fixed grid, so they add nothing to hash. Because elevation and land cover live in their own sets under their own tags, regenerating a DEM or the NLCD layer never invalidates a single AOI raster, and changing a basin polygon never forces a terrain or land-cover regeneration. The two sides move independently: a SNOWTOOL_DEM_HASH change reconciles the terrain set alone; a SNOWTOOL_AOI_HASH change re-burns the AOI raster alone. Query results reflect whichever terrain and land-cover sets are current at read time, decoupled from when any basin was last rasterized.

Checking staleness in practice

The staleness checks are header-only reads — rasterio's tags() (plus, for the AOI check, the transform and shape) without decoding the array — so the guard is orders of magnitude cheaper than the work it may avoid. The source-hash check runs in the ingest skip (Dataset.write_date_cogs) as one equality test; the AOI check runs in the rasterize path (Dataset.aoi_raster_is_current) and, being shared with doctor, is a small set of typed checks — hash, grid transform/shape, and tile-bbox presence — rather than a single comparison (the band-decoding emptiness check is skipped on this path, as it is non-actionable for the writer). The format-version check surfaces in the health checks. stale_format_zone_layers in snowdb/health_checks.py compares each built zone-layer set's stamped format version (via ZoneLayerSet.stored_format_version) against the provider's current one and emits a ZoneLayerFormat finding for any mismatch, so snowtool doctor flags a set that needs regenerating after a format bump. The shared AOI check (aoi_raster_issues) separately catches an AOI raster missing its SNOWTOOL_TILE_BBOX tag and points at a rebuild.

parse_format_version (snowdb/provenance.py) returns the integer version from a tag, or None for a missing value or one not in v{int}:{digest} form — an untagged or legacy artifact. It deliberately does not decide what that means: the caller does. ZoneLayerSet.format_is_current treats a built-but-untagged set as stale (stored None never equals a real version, so it is flagged for rebuild); a set that is not built at all reports None (nothing to check, and missing_artifacts already reports absence). The ingest and AOI reads treat a missing tag the same way — as stale — so a pre-tagging artifact always rebuilds rather than being trusted.

The payoff is convergence. Because the expensive operations — a ~1.5 GB NLCD read, a multi-dataset terrain reprojection, per-date COG builds — are each gated behind a tag comparison that costs a header read, re-running any command is safe and nearly free when nothing has changed, and does exactly the work required when something has. Re-run generate-zones, re-run ingest, re-run rasterize: each converges to the current inputs without redoing settled work.

For the artifacts these tags guard, see AOI rasters and rasterization, zone-layer generation, and the ingest pipeline; for where the versions and links are configured, see Configuration in depth and on-disk layout.