CalCOFI.io CalCOFI.io Workflows

Reproducible Quarto / R notebooks that ingest source datasets, freeze versioned releases, and publish the CalCOFI integrated database. Rendered from CalCOFI/workflows.

62 notebooks · 31 metadata-described workflows · generated 2026-10-14

Ingest 20

Acquire, standardize, and load a source dataset into the integrated database. Cards are grouped by data provider.

CalCOFI

ingest_calcofi_2022-edna

calcofi2022-10 to 2022-1033.8–36.3°N, 122.8–119.4°W
CalCOFI October 2022 Vertebrate eDNA

eDNA metabarcoding survey of vertebrates from water samples collected during the October 2022 CalCOFI cruise. Water filters from CalCOFI/GEMCAP stations off Southern California were run through one or two molecular assays: a mitochondrial D-loop assay (cetacean detections, incl. Delphinus and Megaptera novaeangliae) and a 12S rRNA "MiFish" assay (fish detections). Published as an intercalibration exercise (source datasetID "CalCOFI_Intercal") on GBIF/OBIS, linked to Patin et al. 2026 (Methods in Ecology and Evolution, https://doi.org/10.1111/2041-210x.70370). Sequence read counts are a semi-quantitative signal of PCR-amplified DNA, not abundance, and are not comparable between assays. Published as detections only: edna_presence = 1 and the read counts. The source archive holds positive detections only, so the absence of a row for a filter and taxon is not a non-detection for this dataset (a filter or assay run with no detection is simply missing, and absence cannot be inferred); this differs from sio_cetacean-edna, where 0 means not detected in a sequenced sample.

ingest_calcofi_ctd-cast

calcofi1993-08 to 2026-0729.8–37.8°N, 126.5–117.3°W
CalCOFI CTD Cast Files

CTD downcast profiles with temperature, salinity, oxygen, fluorescence, and other sensors at standard depths. 1 m-binned final data for 1993-2002 come from the SIO-CalCOFI Technical Group's own archive (J. Wilkinson), which covers the decade calcofi.org does not publish finals for; 1998 and 2003 onward come from calcofi.org. Where both publish the same cruise, calcofi.org is used; the two agree on 130 of 140 comparable files, and the four cruises that differ are recorded in question calcofi_ctd-cast_20.

ingest_calcofi_ctd-derived

calcofi1993-08 to 2026-0729.8–37.8°N, 126.5–117.3°W
CalCOFI CTD Derived Hydrographic Products

Hydrographic products DERIVED from the CalCOFI CTD cast profiles rather than measured: spice (spiciness at 0 dbar, TEOS-10) and the sensor-pair averaged sigma-theta at every published CTD depth; and, per cast, the mixed-layer depth (the CalCOFI legacy definition, with two alternative criteria beside it), the depth and value of the chlorophyll-a maximum and the chlorophyll-a integrated over the top 200 m, by the definitions of the CalCOFI CTD team. Computed from the full-resolution 1 m bins of the CalCOFI CTD Cast Files after the provider's quality flags are applied.

ingest_calcofi_dic

calcofi1983-03 to 2021-0728.1–37.0°N, 125.1–117.3°W
CalCOFI DIC

Discrete profile dissolved inorganic carbon (DIC) and total alkalinity (TA) from CalCOFI Niskin bottle samples, quality-controlled using WOCE flagging. Values on a Niskin the bottle database also carries are published once, on that bottle. Values on a sample that matches no bottle cast are published too, wherever the source gives the sample a depth, on the sample's own key: they are not linked to a bottle cast (the source names none), and the date of most of them is month-precision only (see questions calcofi_dic_01 and calcofi_dic_07).

ingest_calcofi_phyllosoma

calcofi1951-07 to 2009-0731.8–35.1°N, 121.5–117.3°W
CalCOFI Lobster Phyllosoma

Spiny lobster (Panulirus interruptus) phyllosoma larvae counts by developmental stage from CalCOFI net tows, 1951-2008. Source: EDI knb-lter-cce.188.4 (Koslow).

ingest_calcofi_phytoplankton

calcofi1996 to 202232.4–33.6°N, 121.7–118.8°W
CalCOFI Phytoplankton (Venrick)

Abundance and species composition of phytoplankton (385 taxonomic categories) in the California Current, 1996-2022, by inverted-microscope counts (E. Venrick). Samples from the near-surface "second depth" are pooled across stations into four regions (NE, SE, Alley, Offshore; Hayward & Venrick 1998) before counting, so the grain is cruise x region, not per-station. Source: EDI knb-lter-cce.254.4.

NOAA SWFSC

ingest_swfsc_cufes

swfsc1996-03 to 2022-0429.8–54.7°N, 134.1–117.2°W
CalCOFI Underway CUFES Fish Eggs

Continuous Underway Fish Egg Sampler (CUFES) egg counts (sardine, anchovy, jack mackerel, hake, squid, other) and eggs per cubic metre pumped, from CalCOFI cruises (1996-2022). Source: SWFSC's CSV export of its CalCOFI database (Cufes.csv, linked to the cruise list by CruiseId); until release v2026.10.01 the same records were read from NOAA CoastWatch ERDDAP erdCalCOFIcufes.

ingest_swfsc_ichthyo

swfsc1951-01 to 2023-010.0–54.4°N, 179.8–77.2°W
SWFSC Ichthyoplankton

Ichthyoplankton (fish eggs and larvae) collected by bongo and manta net tows on CalCOFI cruises, processed by SWFSC. Counts are per net, with SWFSC's own effort (volume filtered, standard haul factor, proportion sorted). A tow with no egg, larva or invertebrate record is a true zero: it stays as a sample with its effort, and abundance rows are positive only. Counts of species codes that share a WoRMS AphiaID are summed per net; each code keeps its own name. A tow's and net's maximum depth is the maximum possible depth of the net (SWFSC NetDepth), not a measured depth: over a shallower sea floor the net did not reach it. Egg stages 12-15 are left out of the egg stage distribution. Fish grown past the larval stage are in their own table, fish. MOCNESS tows are not in the source.

SIO

ingest_sio_mesopelagic-fish

sio2010-01 to 2012-0229.8–35.0°N, 124.3–117.4°W
UCSD SIO Mesopelagic Fish

Species-resolved mesopelagic fish counts (90 named species + 1 unidentified-fish catch-all) from MOHT (Matsuda-Oozeki-Hu Trawl) micronekton net tows, 7 CalCOFI cruises 2010-2012, all on RV New Horizon (NODC 32NM). Larval-fish counts in the same source file are a separate table, not included here.

ingest_sio_pic-zooplankton

sio1939-05 to 2024-0419.9–51.0°N, 164.1–107.1°W
SIO PIC Zooplankton Net Tows

Zooplankton net-tow sample registry from the SIO Pelagic Invertebrate Collection, scoped to CalCOFI-program expeditions. Position, time, depth, and gear metadata per tow. Includes the historical grid (1950s-80s inshore, Gulf of California and Baja lines), not just the modern pattern. Biovolume measurements are not in the source file and are pending from the provider (see Questions for Data Providers).

CCE-LTER

ingest_cce-lter_euphausiids

cce-lter1951-01 to 2019-0428.0–46.6°N, 133.5–117.3°W
CCE-LTER Euphausiid Abundance

Species- and life-stage-resolved euphausiid (krill) abundance from CalCOFI / CCE-LTER net tows (BTEDB export), 1951-present, one row per tow x species x life stage. Supersedes the prior single-Abundance- column ingest, which had no taxonomic scope (Q02, now resolved) — units remain provisional pending provider confirmation (Q01).

ingest_cce-lter_iron

cce-lter2002-11 to 2004-0729.9–35.1°N, 124.3–117.3°W
CCE-LTER Dissolved Iron (CalCOFI Cruises)

Dissolved iron (Fe) concentrations from CalCOFI cruises in the southern California Current System, Nov. 2002 - July 2004 (completed). Surface (0 m) samples collected with a pole sampler, analyzed by FeLume flow-injection analysis. The source also carries total iron (no real values) and a shipboard iron-addition bioassay response (Chla response to Fe+, units unconfirmed); neither is released yet.

ingest_cce-lter_picoplankton-bacteria

cce-lter2004-11 to 2023-0729.8–35.6°N, 124.3–117.3°W
Picoplankton and Bacteria Abundance (CalCOFI Cruise)

Picophytoplankton (Prochlorococcus, Synechococcus, picoeukaryotes) and heterotrophic bacteria abundances analyzed by flow cytometry (FCM) from CCE-CalCOFI Augmented cruises in the California Current System, 2004-2023 (ongoing). Seawater collected from Niskin bottles at 3-8 depths per station; cells fixed shipboard with paraformaldehyde, stained with a DNA-specific dye, and enumerated on an Altra flow cytometer with dual argon-ion lasers.

ingest_cce-lter_zoodb

cce-lter1951-03 to 2015-0430.8–37.6°N, 125.8–117.3°W
ZooDB Holoplankton Community

Holoplankton abundance and carbon biomass by 33 higher taxa (copepods to family, plus amphipods, chaetognaths, salps, pyrosomes, pteropods, ostracods, foraminifera, radiolaria and more) from CalCOFI net tows, 1951-2021, enumerated by microscopy (Ohman Lab / SIO Pelagic Invertebrate Collection). One row per (sample, taxon, measurement_type); unpooled tows are station-resolved, pooled samples are per-cruise regional composites. Raw (uncorrected) abundance per 1000 m3 and per m2, plus biomass mg C/m2.

ingest_cce-lter_zooscan

cce-lter2005-07 to 2026-0430.4–34.5°N, 124.0–117.8°W
ZooScan PRPOOS Zooplankton

Zooplankton abundance, carbon biomass, and size (Feret diameter, individual carbon content) from ZooScan optical imaging of CalCOFI / CCE-LTER PRPOOS net tows on lines 80, 87 and 90, 2005-present, machine-classified into 23 bioclasses (copepod groups, euphausiids, chaetognaths, appendicularians, salps, doliolids, pteropods, rhizaria and more). One row per (sample, taxon, measurement_type); per-station tows. Source: SIO Ocean Informatics ZooScan portal (Ohman Lab; interface by Marina Frants).

CDFW

ingest_cdfw_dungeness-crab

cdfw1984-05 to 2014-0533.5–51.0°N, 132.2–120.7°W
CDFW Dungeness Crab Megalopae

Dungeness crab (Metacarcinus magister) megalopae and larval decapods picked from archived CalCOFI oblique-tow plankton samples, examined for the California Department of Fish and Wildlife. Two efforts: a 2008-2014 time series of 310 sorted samples (A. Klemmedson, 2015) yielding 24 M. magister megalopae from 14 samples, and 216 archived samples from the 1949-2009 sorting log that were examined and found free of M. magister (1984-2009, E. Jones, as of 2012-02-15). Only examined samples are published — the log's remaining unsorted rows are a fact of the archive's inventory, not a sample of this dataset. Spring cruises, CalBOBL oblique tows, lines < 80 / stations < 90. Counts are occurrence in the examined aliquot, not standardized density. Sorting was performed at the SIO Pelagic Invertebrate Collection, where the removed megalopae are archived in vials.

Farallon Institute

ingest_farallon_bird-mammal

farallon1987-05 to 2022-1029.8–37.8°N, 126.5–117.2°W
CalCOFI Bird & Mammal Census

Bird and mammal observations along CalCOFI (and NMFS, CPR) cruise transects, 1987-present. Transect effort, counts, and species live from NOAA ERDDAP (CAC_FI_SBAS_tr / CAC_FI_SBAS_obs / CAC_FI_SBAS_sp); behavior lookups curated via CCE-LTER DataZoo 255 (PI Bill Sydeman). On CalCOFI cruises the Scripps Whale Acoustics Lab runs its own cetacean line-transect survey (sio_cetacean-sightings) with an independent observer team: both datasets are kept, so a cetacean group seen by both teams appears once in each, and summing the two counts the same animals twice.

Publish 6

Publish the released core to external services and formats — ERDDAP, CF NetCDF, OBIS. These are dataset-agnostic: one notebook covers every dataset in the release.

publish_status

The caboose of the publishers: after publish_to-edi, -obis, -netcdf and -erddap have run against a promoted release, compares each dataset's built package, archive or file with the copy its portal holds, and lists what needs a fresh upload — never deposited, or changed since the last deposit — with what still blocks it and how to do it.

publish_to-edi

Publishes CalCOFI's own program datasets — the ones with no existing archive of record (bottle, CTD casts, underway TSG/meteorology) — as EDI data packages: the release's own EML 2.2 document (`build_eml()`) paired with each core table's rows for that dataset as CSV entities. Runs after every promoted release but rebuilds a package only when that dataset's rows, metadata or this code changed; evaluates against EDI's PASTA staging environment when credentials exist; creates/updates the real package only under an explicit flag, and says which packages EDI is behind on.

publish_to-erddap

Generates the ERDDAP configuration for every dataset in the frozen release, discovered from each dataset_key's presence in the core rather than a hand-maintained table list. Serves observations, sampling events, size/stage frequencies and the pre-thinning full series through DuckDB views, and executes every view against the real release before writing its config.

publish_to-netcdf

Publishes every dataset in the frozen release as a self-documenting CF NetCDF file. Dataset-agnostic: the sampling hierarchy in the core decides each file's shape — CF profile, trajectory, point, or netCDF-4 groups — so one notebook replaces the former per-dataset CTD and ichthyoplankton publishers.

publish_to-obis

Builds one Darwin Core Archive (Event core + Occurrence + eMoF + meta.xml + the release's own eml.xml) per biological dataset in the frozen release, generically from the core and the metadata registries. Replaces publish_ichthyo_to-obis.qmd, which read the swfsc_ichthyo source tables directly and left nine other biological datasets with no OBIS route. Writes files only: the upload to the OBIS-USA IPT stays a deliberate manual step.

Release & pipeline 8

Freeze a versioned release, validate it against the consumer contract, deploy it to the apps and services that read it, and the maintenance utilities that support the pipeline.

Reference & plans 3

Planning and reference notebooks. (Candidate to fold into calcofi.io/docs/.)

  • bench_erddap_ctd
  • ctd-cast_qa-qc-protocolEvery CTD quality check that runs, where its threshold came from, and what it cannot see — generated from metadata/qc_rules/ so it cannot drift from the rules. Also published as .docx for track-changes review.
  • README_PLAN

Other notebooks 25

Exploratory analyses and legacy load scripts, kept for reference.

Pipeline DAG

The targets dependency graph parsed from each notebook's calcofi: front-matter (the same fields calcofi4db::build_targets_list() wires up), grouped into subgraphs by workflow type and color-coded to match the sections above.

graph LR
  deploy_consumers(["deploy_consumers"]):::release
  ingest_calcofi_2022_edna(["ingest_calcofi_2022_edna"]):::c_ingest_calcofi_2022_edna
  ingest_calcofi_bottle(["ingest_calcofi_bottle"]):::c_ingest_calcofi_bottle
  ingest_calcofi_ctd_cast(["ingest_calcofi_ctd_cast"]):::c_ingest_calcofi_ctd_cast
  ingest_calcofi_ctd_derived(["ingest_calcofi_ctd_derived"]):::c_ingest_calcofi_ctd_derived
  ingest_calcofi_dic(["ingest_calcofi_dic"]):::c_ingest_calcofi_dic
  ingest_calcofi_mets(["ingest_calcofi_mets"]):::c_ingest_calcofi_mets
  ingest_calcofi_phyllosoma(["ingest_calcofi_phyllosoma"]):::c_ingest_calcofi_phyllosoma
  ingest_calcofi_phytoplankton(["ingest_calcofi_phytoplankton"]):::c_ingest_calcofi_phytoplankton
  ingest_cce_lter_euphausiids(["ingest_cce_lter_euphausiids"]):::c_ingest_cce_lter_euphausiids
  ingest_cce_lter_iron(["ingest_cce_lter_iron"]):::c_ingest_cce_lter_iron
  ingest_cce_lter_picoplankton_bacteria(["ingest_cce_lter_picoplankton_bacteria"]):::c_ingest_cce_lter_picoplankton_bacteria
  ingest_cce_lter_zoodb(["ingest_cce_lter_zoodb"]):::c_ingest_cce_lter_zoodb
  ingest_cce_lter_zooscan(["ingest_cce_lter_zooscan"]):::c_ingest_cce_lter_zooscan
  ingest_cdfw_dungeness_crab(["ingest_cdfw_dungeness_crab"]):::c_ingest_cdfw_dungeness_crab
  ingest_farallon_bird_mammal(["ingest_farallon_bird_mammal"]):::c_ingest_farallon_bird_mammal
  ingest_sio_mesopelagic_fish(["ingest_sio_mesopelagic_fish"]):::c_ingest_sio_mesopelagic_fish
  ingest_sio_pic_zooplankton(["ingest_sio_pic_zooplankton"]):::c_ingest_sio_pic_zooplankton
  ingest_spatial(["ingest_spatial"]):::ingest
  ingest_swfsc_cufes(["ingest_swfsc_cufes"]):::c_ingest_swfsc_cufes
  ingest_swfsc_ichthyo(["ingest_swfsc_ichthyo"]):::c_ingest_swfsc_ichthyo
  publish_ichthyo_to_obis(["publish_ichthyo_to_obis"]):::publish
  publish_status(["publish_status"]):::publish
  publish_to_edi(["publish_to_edi"]):::publish
  publish_to_erddap(["publish_to_erddap"]):::publish
  publish_to_netcdf(["publish_to_netcdf"]):::publish
  publish_to_obis(["publish_to_obis"]):::publish
  release_database(["release_database"]):::release
  test_release(["test_release"]):::release
  test_release --> deploy_consumers
  ingest_swfsc_ichthyo --> ingest_calcofi_2022_edna
  ingest_swfsc_ichthyo --> ingest_calcofi_bottle
  ingest_swfsc_ichthyo --> ingest_calcofi_ctd_cast
  ingest_calcofi_ctd_cast --> ingest_calcofi_ctd_derived
  ingest_calcofi_bottle --> ingest_calcofi_dic
  ingest_swfsc_ichthyo --> ingest_calcofi_mets
  ingest_swfsc_ichthyo --> ingest_calcofi_phyllosoma
  ingest_swfsc_ichthyo --> ingest_calcofi_phytoplankton
  ingest_swfsc_ichthyo --> ingest_cce_lter_euphausiids
  ingest_swfsc_ichthyo --> ingest_cce_lter_iron
  ingest_swfsc_ichthyo --> ingest_cce_lter_picoplankton_bacteria
  ingest_swfsc_ichthyo --> ingest_cce_lter_zoodb
  ingest_swfsc_ichthyo --> ingest_cce_lter_zooscan
  ingest_swfsc_ichthyo --> ingest_cdfw_dungeness_crab
  ingest_swfsc_ichthyo --> ingest_farallon_bird_mammal
  ingest_calcofi_ctd_cast --> ingest_farallon_bird_mammal
  ingest_swfsc_ichthyo --> ingest_sio_mesopelagic_fish
  ingest_swfsc_ichthyo --> ingest_sio_pic_zooplankton
  ingest_swfsc_ichthyo --> ingest_swfsc_cufes
  release_database --> publish_ichthyo_to_obis
  publish_to_edi --> publish_status
  publish_to_obis --> publish_status
  publish_to_netcdf --> publish_status
  publish_to_erddap --> publish_status
  test_release --> publish_to_edi
  test_release --> publish_to_erddap
  test_release --> publish_to_netcdf
  test_release --> publish_to_obis
  ingest_calcofi_2022_edna --> release_database
  ingest_calcofi_bottle --> release_database
  ingest_calcofi_ctd_cast --> release_database
  ingest_calcofi_ctd_derived --> release_database
  ingest_calcofi_dic --> release_database
  ingest_calcofi_mets --> release_database
  ingest_calcofi_phyllosoma --> release_database
  ingest_calcofi_phytoplankton --> release_database
  ingest_cce_lter_euphausiids --> release_database
  ingest_cce_lter_iron --> release_database
  ingest_cce_lter_picoplankton_bacteria --> release_database
  ingest_cce_lter_zoodb --> release_database
  ingest_cce_lter_zooscan --> release_database
  ingest_cdfw_dungeness_crab --> release_database
  ingest_farallon_bird_mammal --> release_database
  ingest_sio_mesopelagic_fish --> release_database
  ingest_sio_pic_zooplankton --> release_database
  ingest_spatial --> release_database
  ingest_swfsc_cufes --> release_database
  ingest_swfsc_ichthyo --> release_database
  release_database --> test_release
  classDef release fill:#f06595,stroke:#00000033,color:#10161c;
  classDef c_ingest_calcofi_2022_edna fill:#bbe0f0,stroke:#00000033,color:#10161c;
  classDef c_ingest_calcofi_bottle fill:#f5cad9,stroke:#00000033,color:#10161c;
  classDef c_ingest_calcofi_ctd_cast fill:#e0ccf0,stroke:#00000033,color:#10161c;
  classDef c_ingest_calcofi_ctd_derived fill:#cfe8dc,stroke:#00000033,color:#10161c;
  classDef c_ingest_calcofi_dic fill:#ffd2bb,stroke:#00000033,color:#10161c;
  classDef c_ingest_calcofi_mets fill:#cce5ff,stroke:#00000033,color:#10161c;
  classDef c_ingest_calcofi_phyllosoma fill:#e0bbf0,stroke:#00000033,color:#10161c;
  classDef c_ingest_calcofi_phytoplankton fill:#b5e8b0,stroke:#00000033,color:#10161c;
  classDef c_ingest_cce_lter_euphausiids fill:#d2f0bb,stroke:#00000033,color:#10161c;
  classDef c_ingest_cce_lter_iron fill:#f0e0c0,stroke:#00000033,color:#10161c;
  classDef c_ingest_cce_lter_picoplankton_bacteria fill:#bbe0f0,stroke:#00000033,color:#10161c;
  classDef c_ingest_cce_lter_zoodb fill:#d9c2f0,stroke:#00000033,color:#10161c;
  classDef c_ingest_cce_lter_zooscan fill:#c2e8f0,stroke:#00000033,color:#10161c;
  classDef c_ingest_cdfw_dungeness_crab fill:#f5d5b8,stroke:#00000033,color:#10161c;
  classDef c_ingest_farallon_bird_mammal fill:#f0d2e8,stroke:#00000033,color:#10161c;
  classDef c_ingest_sio_mesopelagic_fish fill:#bbe0f0,stroke:#00000033,color:#10161c;
  classDef c_ingest_sio_pic_zooplankton fill:#bbe0f0,stroke:#00000033,color:#10161c;
  classDef ingest fill:#cdd9e5,stroke:#00000033,color:#10161c;
  classDef c_ingest_swfsc_cufes fill:#f0e8bb,stroke:#00000033,color:#10161c;
  classDef c_ingest_swfsc_ichthyo fill:#cdebc6,stroke:#00000033,color:#10161c;
  classDef publish fill:#20c997,stroke:#00000033,color:#10161c;