16 Architecture

CalCOFI.io is a federated system: the data stay with the organizations that collect them, one pipeline integrates a copy into a versioned database, and everything people use — the explorer, the packages, the dataset pages, the portals — reads that database rather than each other. This chapter walks the system end to end (Figure 16.1), names where every URL is served from, and sets the 2022 design beside the 2026 one. It is the documentation of the federated architecture and its access paths that the data management plan asks for under Task 14.

Figure 16.1: The system along the phases of the plan: provider files are ingested by one notebook per dataset; the release assembles the shards, checks the keys and freezes a versioned copy with a DOI; the record and every product derive from the release. Everything a consumer reads is the release.

16.1 Sources

Providers’ files — spreadsheets, database exports, cast archives — live on a Google Shared Drive the program owns, in one folder per provider and dataset, and are mirrored by rclone to a bucket (gs://calcofi-files/) so a pipeline run never depends on a person’s Drive client. Drive keeps the archive as delivered; nothing is unzipped or edited there. Every ingest records which file, and which row of it, each value came from, and the record stamps when a source was last fetched (Metadata & the ingest loop).

16.2 The pipeline

CalCOFI/workflows holds one Quarto notebook per dataset — ingest_{provider}_{dataset}.qmd, sixteen datasets today, plus the boundary layers and the ship reference — orchestrated by targets from the calcofi: block in each notebook’s front matter, with the engine in the calcofi4db R package. Each notebook writes its tables as parquet shards to a staging directory outside the repository (mirrored to gs://calcofi-db/parquet/) and its JSON sidecars — the manifest, the metadata, the keys — into the repository, where they are the reviewable record. release_database.qmd assembles every shard in memory, completes the references, runs the gates and freezes a release (Releases); test_release.qmd tests the uploaded release and promotes it. The whole run is reproducible on a laptop or the server. → Ingesting a dataset

16.3 The release store

gs://calcofi-db/ducklake/ is public and read over HTTPS. releases/{version}/ holds each release’s sidecars; tables/{table}/{content_hash}/… holds the parquet objects, content-addressed and shared between versions; releases/latest.txt names the promoted version. Every reader resolves a table through its catalog.json; storage.calcofi.io is a browsable front door onto the same buckets that also redirects a legacy per-release path to its object. Each release is tagged and archived on Zenodo with its own DOI. → Access the data, Releases

16.4 The record and the portals

At release, one record per dataset (datasets.json) and one EML document per dataset are generated from the ingest’s front matter, the descriptive sidecar the provider edits, the registries and the measured coverage. From those two files come the dataset pages at calcofi.io/datasets with their JSON-LD and sitemap, the DCAT and STAC catalogs, the Darwin Core archives for OBIS, the EDI packages and ERDDAP’s own metadata — so a description written once reaches every portal, and a weekly observation asks each portal what it says now. → Portals, Cite this data

16.5 The products

Everything people use reads the release, and the same bytes:

the Explorer, a static site whose queries run in DuckDB inside the browser — six lenses, every view a URL;
the dataset pages, the schema browser and the query site, static sites over the release’s sidecars and objects;
calcofi4r and calcofi4py, which register every release table as a view and carry the matching, citation, quality and branding helpers;
erddap.calcofi.io, a CalCOFI-branded ERDDAP serving the release’s tables for the community that reads ERDDAP;
the Shiny applications on the CalCOFI server (app.calcofi.io/…), the first generation of viewers, now superseded by the Explorer and kept serving;
this book (calcofi.io/docs), which is a consumer like the rest: its pre-render snapshots the promoted release’s sidecars — catalog.json, integrity.json, metadata.json, relationships.json, datasets.json, the dataset table and the release notes — plus the registries in CalCOFI/workflows, and every table of fact in it is generated from that snapshot. Because the snapshot is of the promoted release, the book shows the previous version until it re-renders, so scripts/deploy_consumers.sh dispatches its workflow after latest.txt moves, a release-promoted dispatch triggers it, and it rebuilds weekly regardless.

→ Explore, Products, brand and uptime, Releases

16.6 The working store — available, not yet adopted

One piece of the system is built but not yet in use, and the figure above leaves it out for that reason. The CTD team’s PostgreSQL database on the CalCOFI server (PostgreSQL 18, PostGIS, pg_duckdb, reached over SSH) holds the cast archive verbatim with a flag ledger beside it, and a nightly snapshot of its accepted flags is written to the bucket for the next CTD ingest to apply. The accounts, the schema and the bridge all work; the team has not adopted it as its QA/QC practice, so today CTD flags still arrive with the provider’s files. Nothing in the pipeline reads the live database in either case. The figure with it in, and what adopting it would change, are in CTD QA/QC; the how-to is Server access.

16.7 Where every URL is served from

Table 16.1: Every calcofi.io address, what it serves and what serves it.

address	what	served by
`calcofi.io`	the landing page and the dataset catalog	GitHub Pages (Jekyll, `CalCOFI/CalCOFI.github.io`)
`calcofi.io/explore`	the Explorer	GitHub Pages (Vite + DuckDB-WASM, `CalCOFI/explore`)
`calcofi.io/docs`	this book	GitHub Pages (Quarto, `CalCOFI/docs`)
`calcofi.io/db-schema`, `/db-query`	the schema browser, the query site	GitHub Pages (Jekyll)
`calcofi.io/workflows`	every notebook, rendered	GitHub Pages (Quarto, `CalCOFI/workflows`)
`calcofi.io/calcofi4r`, `/calcofi4py`, `/calcofi4db`	the packages’ reference sites	GitHub Pages (pkgdown, MkDocs, pkgdown)
`calcofi.io/analytics`	usage per product	GitHub Pages (Hugo, `CalCOFI/analytics`)
`status.calcofi.io`	uptime per product	Upptime (`CalCOFI/uptime`)
`storage.googleapis.com/calcofi-db`	the release store, the parquet	Google Cloud Storage
`storage.calcofi.io`	a browsable front door onto the buckets, with the legacy-path redirects	Caddy on the CalCOFI server
`erddap.calcofi.io`	ERDDAP	the CalCOFI server (Docker)
`app.calcofi.io/…`	the Shiny applications	the CalCOFI server (Shiny Server)
`h3t.calcofi.io`	the cached hexagon-tile API behind the Hexagon Explorer	the CalCOFI server (FastAPI behind Varnish)
`rstudio.`, `pgadmin.`, `ssh.calcofi.io`	RStudio Server, pgAdmin, SSH and the PostgreSQL tunnel	the CalCOFI server
`tile.calcofi.io`, `file.calcofi.io`	vector tiles from PostGIS; public files on disk	the CalCOFI server

Table 16.1 is the whole surface area: everything on GitHub Pages rebuilds itself from a repository, and everything else is one machine. The server is one Docker Compose stack, configured in CalCOFI/server — the configuration of record for every *.calcofi.io service. Everything on GitHub Pages rebuilds itself on push and costs nothing to host.

16.8 Then and now

Figure 16.2 is the design this one replaced.

Figure 16.2: The 2022 design: sources ingested by scripts into a PostGIS database, an API feeding Shiny apps and reports, and a *publish* step to the portals that was intent rather than code.

The 2022 figure had the right boxes and the wrong plumbing. Its database was PostgreSQL behind an API, so every reader was a request to one server; its publish box was a plan. The 2026 system keeps the boxes and changes what connects them: the database is parquet on a public bucket that any DuckDB reads directly — the browser included, so the flagship application has no server at all; the API is retired; the publish box is generated from one record per dataset at every release; and the PostgreSQL that remains serves no consumer at all — it is a working store offered to one team, feeding the pipeline through a nightly file rather than answering queries. What did not change is the intent: one integrated, versioned, citable database that every product reads, so that a correction made once reaches everyone.