ingest_calcofi_bottle
CalCOFI Bottle Database
Hydrographic bottle data including temperature, salinity, oxygen, nutrients, pigments, and C14 from CalCOFI cruises.
Reproducible Quarto / R notebooks that ingest source datasets, freeze versioned releases, and publish the CalCOFI integrated database. Rendered from CalCOFI/workflows.
51 notebooks · 25 metadata-described workflows · generated 2026-08-17
Ingest 17
Acquire, standardize, and load a source dataset into the integrated database. Cards are grouped by data provider.
CalCOFI
ingest_calcofi_ctd-cast
CalCOFI CTD Cast Files
CTD downcast profiles with temperature, salinity, oxygen, fluorescence, and other sensors at standard depths. 1 m-binned final data for 1993-2002 come from the SIO-CalCOFI Technical Group's own archive (J. Wilkinson), which covers the decade calcofi.org does not publish finals for; 1998 and 2003 onward come from calcofi.org. Where both publish the same cruise, calcofi.org is used; the two agree on 130 of 140 comparable files, and the four cruises that differ are recorded in question calcofi_ctd-cast_20.
ingest_calcofi_dic
CalCOFI DIC
Discrete profile dissolved inorganic carbon (DIC) and total alkalinity (TA) from CalCOFI Niskin bottle samples, quality-controlled using WOCE flagging.
ingest_calcofi_mets
CalCOFI METS (Underway TSG/Meteorology)
Shipboard underway thermosalinograph (TSG) and meteorology data, ~1-minute resolution, recorded continuously along the cruise track.
ingest_calcofi_phyllosoma
CalCOFI Lobster Phyllosoma
Spiny lobster (Panulirus interruptus) phyllosoma larvae counts by developmental stage from CalCOFI net tows, 1951-2008. Source: EDI knb-lter-cce.188.4 (Koslow).
ingest_calcofi_phytoplankton
CalCOFI Phytoplankton (Venrick)
Abundance and species composition of phytoplankton (385 taxonomic categories) in the California Current, 1996-2022, by inverted-microscope counts (E. Venrick). Samples from the near-surface "second depth" are pooled across stations into four regions (NE, SE, Alley, Offshore; Hayward & Venrick 1998) before counting, so the grain is cruise x region, not per-station. Source: EDI knb-lter-cce.254.4.
ingest_spatial
SWFSC
ingest_swfsc_cufes
CalCOFI Underway CUFES Fish Eggs
Continuous Underway Fish Egg Sampler (CUFES) egg counts (sardine, anchovy, jack mackerel, hake, squid, other) with underway environmental conditions, from CalCOFI cruises (1996-present). Source: NOAA CoastWatch ERDDAP erdCalCOFIcufes.
ingest_swfsc_ichthyo
SWFSC Ichthyoplankton
Ichthyoplankton (fish eggs and larvae) collected by bongo and manta net tows on CalCOFI cruises, processed by SWFSC.
SIO
ingest_sio_mesopelagic-fish
UCSD SIO Mesopelagic Fish
Species-resolved mesopelagic fish counts (90 named species + 1 unidentified-fish catch-all) from MOHT (Matsuda-Oozeki-Hu Trawl) micronekton net tows, 7 CalCOFI cruises 2010-2012, all on RV New Horizon (NODC 32NM). Larval-fish counts in the same source file are a separate table, not included here.
ingest_sio_pic-zooplankton
SIO PIC Zooplankton Net Tows
Zooplankton net-tow sample registry from the SIO Pelagic Invertebrate Collection, scoped to CalCOFI-program expeditions. Position, time, depth, and gear metadata per tow. Includes the historical grid (1950s-80s inshore, Gulf of California and Baja lines), not just the modern pattern. Biovolume measurements are not in the source file and are pending from the provider (see Questions for Data Providers).
CCE-LTER
ingest_cce-lter_euphausiids
CCE-LTER Euphausiid Abundance
Species- and life-stage-resolved euphausiid (krill) abundance from CalCOFI / CCE-LTER net tows (BTEDB export), 1951-present, one row per tow x species x life stage. Supersedes the prior single-Abundance- column ingest, which had no taxonomic scope (Q02, now resolved) — units remain provisional pending provider confirmation (Q01).
ingest_cce-lter_picoplankton-bacteria
Picoplankton and Bacteria Abundance (CalCOFI Cruise)
Picophytoplankton (Prochlorococcus, Synechococcus, picoeukaryotes) and heterotrophic bacteria abundances analyzed by flow cytometry (FCM) from CCE-CalCOFI Augmented cruises in the California Current System, 2004-2023 (ongoing). Seawater collected from Niskin bottles at 3-8 depths per station; cells fixed shipboard with paraformaldehyde, stained with a DNA-specific dye, and enumerated on an Altra flow cytometer with dual argon-ion lasers.
ingest_cce-lter_zoodb
ZooDB Holoplankton Community
Holoplankton abundance and carbon biomass by 33 higher taxa (copepods to family, plus amphipods, chaetognaths, salps, pyrosomes, pteropods, ostracods, foraminifera, radiolaria and more) from CalCOFI net tows, 1951-2021, enumerated by microscopy (Ohman Lab / SIO Pelagic Invertebrate Collection). One row per (sample, taxon, measurement_type); unpooled tows are station-resolved, pooled samples are per-cruise regional composites. Raw (uncorrected) abundance per 1000 m3 and per m2, plus biomass mg C/m2.
ingest_cce-lter_zooscan
ZooScan PRPOOS Zooplankton
Zooplankton abundance, carbon biomass, and size (Feret diameter, individual carbon content) from ZooScan optical imaging of CalCOFI / CCE-LTER PRPOOS net tows on lines 80, 87 and 90, 2005-present, machine-classified into 23 bioclasses (copepod groups, euphausiids, chaetognaths, appendicularians, salps, doliolids, pteropods, rhizaria and more). One row per (sample, taxon, measurement_type); per-station tows. Source: SIO Ocean Informatics ZooScan portal (Ohman Lab; interface by Marina Frants).
CDFW
ingest_cdfw_dungeness-crab
CDFW Dungeness Crab Megalopae
Dungeness crab (Metacarcinus magister) megalopae and larval decapods picked from archived CalCOFI oblique-tow plankton samples, examined for the California Department of Fish and Wildlife. Two efforts: a 2008-2014 time series of 310 sorted samples (A. Klemmedson, 2015) yielding 24 M. magister megalopae from 14 samples, and a 1949-2009 master sorting log of 2,011 archived samples (E. Jones, as of 2012-02-15) recording which had been examined. Spring cruises, CalBOBL oblique tows, lines < 80 / stations < 90. Counts are occurrence in the examined aliquot, not standardized density. Sorting was performed at the SIO Pelagic Invertebrate Collection, where the removed megalopae are archived in vials.
Farallon Institute
ingest_farallon_bird-mammal
CalCOFI Bird & Mammal Census
Bird and mammal observations along CalCOFI (and NMFS, CPR) cruise transects, 1987-2021. Transect effort + counts + species/behavior lookups. Curated via CCE-LTER DataZoo 255 (PI Bill Sydeman).
Publish 3
Publish the released core to external services and formats — ERDDAP, CF NetCDF, OBIS. These are dataset-agnostic: one notebook covers every dataset in the release.
publish_ichthyo_to-obis
publish_to-erddap
Generates the ERDDAP configuration for every dataset in the frozen release, discovered from each dataset_key's presence in the core rather than a hand-maintained table list. Serves observations, sampling events, size/stage frequencies and the pre-thinning full series through DuckDB views, and executes every view against the real release before writing its config.
publish_to-netcdf
Publishes every dataset in the frozen release as a self-documenting CF NetCDF file. Dataset-agnostic: the sampling hierarchy in the core decides each file's shape — CF profile, trajectory, point, or netCDF-4 groups — so one notebook replaces the former per-dataset CTD and ichthyoplankton publishers.
Release & pipeline 7
Freeze a versioned release, validate it against the consumer contract, deploy it to the apps and services that read it, and the maintenance utilities that support the pipeline.
- clean_db
- deploy_consumersBrings the server-side consumers (db-viz-hex, db-viz-cruise, the h3t tile API) onto the promoted release, and reports which release each is actually serving before and after. Runs after test_release, so it can only ever deploy a version that passed the consumer-contract suite. Deploying is the default; CALCOFI_DEPLOY=false makes it a read-only drift report.
- load_views
- release_database
- test_release
- update_datasets-sitemap
- update_taxa
Reference & plans 3
Planning and reference notebooks. (Candidate to fold into calcofi.io/docs/.)
- bench_erddap_ctd
- ctd-cast_qa-qc-protocolEvery CTD quality check that runs, where its threshold came from, and what it cannot see — generated from metadata/qc_rules/ so it cannot drift from the rules. Also published as .docx for track-changes review.
- README_PLAN
Other notebooks 21
Exploratory analyses and legacy load scripts, kept for reference.
- explore_accdb_hydro-master
- explore_bird-mammal
- explore_bottle
- explore_ctd-cast
- explore_ctd-cast2
- explore_db
- explore_depth
- explore_dumping
- explore_gdatasets
- explore_grid-res
- explore_interpolate-larvae
- explore_join-larvae-bottle
- explore_pg_contour
- explore_pg_contour2
- explore_scripps.ucsd.edu_pelagic-invertebrate-collection
- explore_station-grid
- explore_station-vector
- ingest_ices.dk_ship-ices
- load_bottle-stations
- load_field-labels
- load_sanctuaries
Pipeline DAG
The targets dependency graph parsed from each notebook's
calcofi: front-matter (the same fields
calcofi4db::build_targets_list() wires up), grouped into
subgraphs by workflow type and color-coded to match the sections above.
graph LR deploy_consumers(["deploy_consumers"]):::release ingest_calcofi_bottle(["ingest_calcofi_bottle"]):::c_ingest_calcofi_bottle ingest_calcofi_ctd_cast(["ingest_calcofi_ctd_cast"]):::c_ingest_calcofi_ctd_cast ingest_calcofi_dic(["ingest_calcofi_dic"]):::c_ingest_calcofi_dic ingest_calcofi_mets(["ingest_calcofi_mets"]):::c_ingest_calcofi_mets ingest_calcofi_phyllosoma(["ingest_calcofi_phyllosoma"]):::c_ingest_calcofi_phyllosoma ingest_calcofi_phytoplankton(["ingest_calcofi_phytoplankton"]):::c_ingest_calcofi_phytoplankton ingest_cce_lter_euphausiids(["ingest_cce_lter_euphausiids"]):::c_ingest_cce_lter_euphausiids ingest_cce_lter_picoplankton_bacteria(["ingest_cce_lter_picoplankton_bacteria"]):::c_ingest_cce_lter_picoplankton_bacteria ingest_cce_lter_zoodb(["ingest_cce_lter_zoodb"]):::c_ingest_cce_lter_zoodb ingest_cce_lter_zooscan(["ingest_cce_lter_zooscan"]):::c_ingest_cce_lter_zooscan ingest_cdfw_dungeness_crab(["ingest_cdfw_dungeness_crab"]):::c_ingest_cdfw_dungeness_crab ingest_farallon_bird_mammal(["ingest_farallon_bird_mammal"]):::c_ingest_farallon_bird_mammal ingest_sio_mesopelagic_fish(["ingest_sio_mesopelagic_fish"]):::c_ingest_sio_mesopelagic_fish ingest_sio_pic_zooplankton(["ingest_sio_pic_zooplankton"]):::c_ingest_sio_pic_zooplankton ingest_spatial(["ingest_spatial"]):::ingest ingest_swfsc_cufes(["ingest_swfsc_cufes"]):::c_ingest_swfsc_cufes ingest_swfsc_ichthyo(["ingest_swfsc_ichthyo"]):::c_ingest_swfsc_ichthyo publish_ichthyo_to_obis(["publish_ichthyo_to_obis"]):::publish publish_to_erddap(["publish_to_erddap"]):::publish publish_to_netcdf(["publish_to_netcdf"]):::publish release_database(["release_database"]):::release test_release(["test_release"]):::release test_release --> deploy_consumers ingest_swfsc_ichthyo --> ingest_calcofi_bottle ingest_swfsc_ichthyo --> ingest_calcofi_ctd_cast ingest_calcofi_bottle --> ingest_calcofi_dic ingest_swfsc_ichthyo --> ingest_calcofi_mets ingest_swfsc_ichthyo --> ingest_calcofi_phyllosoma ingest_swfsc_ichthyo --> ingest_calcofi_phytoplankton ingest_swfsc_ichthyo --> ingest_cce_lter_euphausiids ingest_swfsc_ichthyo --> ingest_cce_lter_picoplankton_bacteria ingest_swfsc_ichthyo --> ingest_cce_lter_zoodb ingest_swfsc_ichthyo --> ingest_cce_lter_zooscan ingest_swfsc_ichthyo --> ingest_cdfw_dungeness_crab ingest_swfsc_ichthyo --> ingest_farallon_bird_mammal ingest_swfsc_ichthyo --> ingest_sio_mesopelagic_fish ingest_swfsc_ichthyo --> ingest_sio_pic_zooplankton ingest_swfsc_ichthyo --> ingest_swfsc_cufes release_database --> publish_ichthyo_to_obis release_database --> publish_to_erddap release_database --> publish_to_netcdf ingest_calcofi_bottle --> release_database ingest_calcofi_ctd_cast --> release_database ingest_calcofi_dic --> release_database ingest_calcofi_mets --> release_database ingest_calcofi_phyllosoma --> release_database ingest_calcofi_phytoplankton --> release_database ingest_cce_lter_euphausiids --> release_database ingest_cce_lter_picoplankton_bacteria --> release_database ingest_cce_lter_zoodb --> release_database ingest_cce_lter_zooscan --> release_database ingest_cdfw_dungeness_crab --> release_database ingest_farallon_bird_mammal --> release_database ingest_sio_mesopelagic_fish --> release_database ingest_sio_pic_zooplankton --> release_database ingest_spatial --> release_database ingest_swfsc_cufes --> release_database ingest_swfsc_ichthyo --> release_database release_database --> test_release classDef release fill:#f06595,stroke:#00000033,color:#10161c; classDef c_ingest_calcofi_bottle fill:#f5cad9,stroke:#00000033,color:#10161c; classDef c_ingest_calcofi_ctd_cast fill:#e0ccf0,stroke:#00000033,color:#10161c; classDef c_ingest_calcofi_dic fill:#ffd2bb,stroke:#00000033,color:#10161c; classDef c_ingest_calcofi_mets fill:#cce5ff,stroke:#00000033,color:#10161c; classDef c_ingest_calcofi_phyllosoma fill:#e0bbf0,stroke:#00000033,color:#10161c; classDef c_ingest_calcofi_phytoplankton fill:#b5e8b0,stroke:#00000033,color:#10161c; classDef c_ingest_cce_lter_euphausiids fill:#d2f0bb,stroke:#00000033,color:#10161c; classDef c_ingest_cce_lter_picoplankton_bacteria fill:#bbe0f0,stroke:#00000033,color:#10161c; classDef c_ingest_cce_lter_zoodb fill:#d9c2f0,stroke:#00000033,color:#10161c; classDef c_ingest_cce_lter_zooscan fill:#c2e8f0,stroke:#00000033,color:#10161c; classDef c_ingest_cdfw_dungeness_crab fill:#f5d5b8,stroke:#00000033,color:#10161c; classDef c_ingest_farallon_bird_mammal fill:#f0d2e8,stroke:#00000033,color:#10161c; classDef c_ingest_sio_mesopelagic_fish fill:#bbe0f0,stroke:#00000033,color:#10161c; classDef c_ingest_sio_pic_zooplankton fill:#bbe0f0,stroke:#00000033,color:#10161c; classDef ingest fill:#cdd9e5,stroke:#00000033,color:#10161c; classDef c_ingest_swfsc_cufes fill:#f0e8bb,stroke:#00000033,color:#10161c; classDef c_ingest_swfsc_ichthyo fill:#cdebc6,stroke:#00000033,color:#10161c; classDef publish fill:#20c997,stroke:#00000033,color:#10161c;