7  Releases

The integrated database is published as frozen releases: an immutable, versioned copy of every table, with a catalog that says exactly which bytes it holds and a DOI that names it. Nothing a consumer reads is ever edited in place. This chapter says what a release is, what ships with it, which versions are kept, and what is checked before one is cut.

7.1 A release in one line

The promoted release is v2026.09.06, cut 2026-09-06: 23 tables, 348,657,010 rows, 2.46 GB of parquet, DOI 10.5281/zenodo.22514953. A version is a date (vYYYY.MM.DD), and a release is cut whenever the data or the schema change enough to matter — every few weeks, in practice. latest.txt on the bucket names the promoted one, and it is promoted only after the consumer-contract suite has passed against the frozen files.

7.2 What ships

Everything lives under one public prefix, gs://calcofi-db/ducklake/releases/{version}/, read over HTTPS with no credentials:

  • catalog.json — the structural manifest and the data-access contract: the version, its DOIs and citation, and for every table its rows, its content_hash and its objects[] — one parquet object for a single-file table, one per partition for a partitioned one — each with a bucket-relative path, bytes, sha256 and since. Resolve a table through it (calcofi4r::cc_release_sources(), calcofi4py.release_sources()); never build a path by hand.
  • the parquet objects, content-addressed under gs://calcofi-db/ducklake/tables/{table}/…: immutable and shared between versions, so since on an object is the first release that shipped those exact bytes — the per-table changelog. A release’s own parquet/ copy exists only for the promoted version, its predecessor and the consolidated milestones (below).
  • the sidecars: metadata.json (every table and column, with units and descriptions), relationships.json (every key, Keys and integrity), integrity.json (the keys measured), datasets.json (one record per dataset — the record every portal reads), eml/ (one EML document per dataset), coverage.json and coverage_stations.json (what the Explorer paints first), grid.geojson, spatial.geojson, spatial_layers.json, a static stac/ catalog, erd.mmd, test_results.json, and RELEASE_NOTES.md.

Table 7.1 is that catalog rendered: every table of the promoted release, how many objects carry it, and the version its bytes last changed in.

Table 7.1: The tables of v2026.09.06 as the catalog describes them — objects, partitioning, size, and the release whose bytes each table last changed in.
table tier rows objects partitioned by bytes last changed
climatology core 768,880 71 measurement_type 9.9 MB v2026.09.06
cruise core 842 1 19.6 kB v2026.09.04
dataset core 16 1 14.7 kB v2026.09.04
dataset_taxon core 1,917 1 57.9 kB v2026.09.04
grid core 218 1 103 kB v2026.08.25
lookup core 26 1 1.6 kB v2026.08.25
measurement_type core 200 1 12 kB v2026.09.04
obs_attribute core 458,184 1 1.8 MB v2026.09.04
obs_bio core 1,258,665 1 25.3 MB v2026.09.04
obs_env core 25,006,583 84 measurement_type 317.2 MB v2026.09.04
region core 4 1 4 kB v2026.08.25
sample core 1,469,155 1 25.9 MB v2026.09.04
sample_measurement core 589,603 1 2 MB v2026.08.25
sample_spatial core 929,664 1 7.3 MB v2026.09.04
ship core 49 1 1.2 kB v2026.08.25
spatial core 13,206 1 47.8 MB v2026.08.25
spatial_attribute core 148,461 1 538 kB v2026.08.25
taxon core 2,614 1 87.8 kB v2026.09.06
taxon_group core 441 1 2.8 kB v2026.09.04
obs deprecated 26,265,248 16 dataset_key 400.6 MB v2026.09.04
obs_ctd_full supplemental 271,394,164 134 cruise_key 1.4 GB v2026.08.25
obs_mets_full supplemental 19,927,416 49 cruise_key 250.5 MB v2026.08.25
sample_root supplemental 421,454 1 10.5 MB v2026.09.04

7.3 Versions, and which are kept

Every version keeps its sidecars forever, so any release can still be browsed. Its parquet copy is kept for the promoted version, the one before it, and the consolidated milestones — versions a paper, a docs example or an app pins; the rest is thinned, and a thinned version’s retired marker names the consolidated one to read instead (metadata/release_policy.yml in CalCOFI/workflows decides; scripts/thin_releases.R applies it). The content-addressed store means a kept object is stored once however many versions share it. Table 7.2 is every version, newest first, with what is kept of each.

Table 7.2: Every release, newest first, from versions.json.
version date tables rows size (MB) DOI kept
v2026.09.06 2026-09-06 23 348,657,010 2,350 10.5281/zenodo.22514953 current
v2026.09.04 2026-09-04 23 348,657,010 2,351 10.5281/zenodo.22310858 current
v2026.08.25 2026-08-25 18 320,260,205 1,998
consolidated
v2026.08.14 2026-08-14 18 307,537,056 1,930
consolidated
v2026.08.11 2026-08-11 18 323,912,311 2,016
retired → v2026.08.14
v2026.08.10 2026-08-11 18 323,912,364 2,017
retired → v2026.08.14
v2026.08.08 2026-08-08 18 309,122,838 1,947
retired → v2026.08.14
v2026.08.07 2026-08-07 18 323,733,662 2,024
retired → v2026.08.14
v2026.08.06 2026-08-06 18 255,137,845 1,636
retired → v2026.08.14
v2026.08.05 2026-08-05 18 255,137,845 1,636
retired → v2026.08.14
v2026.08.04 2026-08-04 18 255,155,031 1,635
retired → v2026.08.14
v2026.08.03 2026-08-03 18 255,037,035 2,057
retired → v2026.08.14
v2026.08.02 2026-08-02 18 255,037,035 2,087
retired → v2026.08.14
v2026.07.30 2026-07-30 17 233,730,737 1,710
retired → v2026.08.14
v2026.07.17 2026-07-17 17 236,577,241 1,943
consolidated
v2026.07.16 2026-07-16 17 236,577,241 1,945
retired → v2026.07.17
v2026.07.15 2026-07-15 22 236,371,206 5,472
retired → v2026.07.17
v2026.06.26 2026-06-26 50 133,966,996 3,554
consolidated
v2026.06.08 2026-06-08 44 133,807,311 3,553
retired → v2026.06.26
v2026.06.07 2026-06-07 28 133,015,544 3,547
retired → v2026.06.26
v2026.05.20 2026-05-20 28 133,022,102 5,503
retired → v2026.06.26
v2026.05.19 2026-05-19 28 133,022,102 5,503
retired → v2026.06.26
v2026.05.15 2026-05-15 28 133,013,839 0
retired → v2026.06.26
v2026.05.14 2026-05-14 28 133,013,839 0
consolidated
v2026.04.08 2026-04-08 28 357,418,040 0
consolidated
v2026.04.06 2026-04-06 29 361,200,542 0
retired → v2026.04.08
v2026.04.03 2026-04-03 29 361,183,451 0
retired → v2026.04.08
v2026.04.02 2026-04-02 43 361,152,303 11,498
retired → v2026.04.08
v2026.03.25 2026-03-25 18 354,898,020 11,263
retired → v2026.04.08
v2026.03.14 2026-03-14 20 357,001,188 11,330
retired → v2026.04.08

Consolidated today: v2026.04.08, v2026.05.14, v2026.06.26, v2026.07.17, v2026.08.14, v2026.08.25.

To pin a version rather than follow the promoted one:

calcofi4r::cc_get_db(version = "v2026.08.25")          # R
calcofi4py.cc_get_db(version="v2026.08.25")             # Python
# any DuckDB, from any language: resolve a table's objects through the catalog and paste the
# read_parquet() fragment it hands back — never a typed releases/{v}/parquet path
calcofi4r::cc_read_parquet_sql(calcofi4r::cc_release_sources(calcofi4r::cc_catalog("v2026.08.25"), "sample"))

7.4 Two DOIs

Each release is tagged on GitHub, and Zenodo mints a version DOI for it; one concept DOI always resolves to the newest. Cite the version DOI in a paper — it names exactly the bytes you used — and the concept DOI where a living pointer is wanted. Both are in catalog.json, in the release notes’ How to cite, and on Cite this data:

7.5 The changelog

RELEASES.md is the database’s NEWS file: one section per release, newest first, saying what changed and why, with a Consumers: line wherever a change could break a reader, and an # Unreleased section collecting changes until the next cut. Each version’s RELEASE_NOTES.md is its section plus a generated appendix (tables and rows, datasets, the consumer-contract result, package versions). A release with no section for itself is refused before the freeze — the notes are written by the change that made them necessary, never reconstructed afterwards.

The most recent sections of RELEASES.md:

  • Unreleased
  • v2026.09.06 (2026-09-06)
  • v2026.09.04 (2026-09-04)
  • v2026.08.25 (2026-08-25)
  • v2026.08.14 (2026-08-14)
  • v2026.08.11 (2026-08-11)

7.6 What a release checks before it ships

release_database.qmd assembles every ingest’s parquet shards in memory, builds the release-only tables, and runs a series of gates before the freeze. Each is a function in calcofi4db with a test of its own, and the notebook’s rendered page shows every result. The ones that stop a release outright:

Table 7.3: The gates that stop a release outright.
gate what it measures on failure
check_release_relationships() every declared primary key unique and non-NULL; every non-NULL foreign key resolving; written to integrity.json stops the release
check_core_pk_unique() the core family’s primary keys, before the release-built tables exist (the gate added after v2026.08.25 shipped 4,855 duplicate sample_keys) stops the release
check_cruise_key_integrity() cruise_key format, its month against the NODC code, the sample/obs → cruise edge, cruise_uuid hygiene, every event’s date within its cruise’s span (± 31 days, named exceptions only) stops the release; three ratchets (span overlaps, derived cruises, NULL keys per dataset) may only ever go down
validate_for_release() NULLs in any *_id / *_key / *_uuid column; latitude, longitude and counts in range; no empty table NULLs stop the release; ranges warn
check_obs_pair_parity() obs_bio + obs_env reproduce obs — row counts, distinct ids, a hash of every column, per realm and dataset stops the release
test_release.qmd consumer contract every query shape the apps and packages depend on, run against the frozen release three ways (objects, view, pair), plus the presence of an all-ok integrity.json latest.txt is not promoted

And the ones that report and ratchet — a number that may only ever go down between releases, so a regression fails but a known backlog does not:

Table 7.4: The gates that report and ratchet rather than stop the release.
gate what it measures
check_depth_bounds() / check_depth_vs_seafloor() no depth is NaN, negative or beyond 6,500 m (fails); each root sample’s deepest depth against the GEBCO seafloor near its position (ratchet — the place is imprecise, not the measurement)
check_measurement_bounds() every measurement type has declared physical bounds and no value lies outside them (out of range fails; undeclared types ratchet)
check_taxon_ids() every taxon key is an accepted WoRMS or ITIS id, or on an explicit allowlist of dataset-local classes (fails)
check_dataset_citation() every dataset’s citation has a year and a locator, its licence is a registered id, its DOI resolves (fails unless a provider question is open on the field)
check_dataset_catalog() / check_eml_catalog() the record and each EML document validate against their schemas (fails)
observed_coverage() each dataset’s temporal and spatial extent, measured from the data — never typed (report)

A failure in Table 7.3 stops the release; a number in Table 7.4 that went up since the last release stops it too, and one that went down becomes the new ceiling.

For v2026.09.06 the consumer-contract suite ran 73 queries: 69 passed, 0 failed, 4 skipped (test_results.json).

A release that passes is uploaded, tested once more from the bucket by test_release.qmd, and only then promoted; the read-only consumers are refreshed afterwards by scripts/deploy_consumers.sh — the Explorer, ERDDAP and the Shiny apps on the CalCOFI server, and the ones hosted on GitHub Actions, which the script dispatches: db-viz-station, ctd-transects and this book, whose pre-render snapshots the promoted release’s catalog.json, integrity.json and sidecars and so describes the previous version until it re-renders (Architecture). A staging run exercises the whole path under a separate prefix without touching the promoted release or the shared object store.