Ingest CCE-LTER ZooDB Holoplankton

Published

2026-06-26

1 Overview

Source: ZooDB — the SIO Ocean Informatics zooplankton database (Mark Ohman Lab, SIO Pelagic Invertebrate Collection). Per-taxon CSVs are scripted from the portal’s save.php endpoint (public access) by libs/download_zoodb.R and consolidated to zoodb_holoplankton.csv (see the Acquire step below + by_taxon/_PROVENANCE.md).

  • Provider: cce-lter (curated by CCE-LTER / Ohman Lab; sibling of cce-lter_euphausiids, which the ZooDB methods note as a separate database)
  • Grain: one row per (sample × taxon × measurement_type). A measurement row means the taxon was analyzed for that sample; measurement_value = 0 means analyzed-but-absent. Coverage varies by taxon/cruise — a missing (sample, taxon) is not analyzed, so we never impute zeros.
  • Samples (506): 351 unpooled net tows (station-resolved) + 155 pooled per-cruise regional composites (Lavaniegos & Ohman 2007; no per-station position; a cruise/region may hold >1 composite, e.g. shallow vs deep strata).
  • Taxa (33): higher taxa — all genera/species and phases/stages combined — standardized to WoRMS (taxon_worms.csv).
  • Measurements: zooplankton_abundance (count/1000 m³), zooplankton_abundance_areal (count/m²), zooplankton_biomass_carbon (mg C/m²). Raw (uncorrected for net type).

This ingest proceeds with documented provisional decisions (cruise/ship resolution, timezone, raw vs corrected, pooled positioning); each is tracked in the Questions for Data Providers section.

Code
graph LR
  A[zoodb_holoplankton.csv<br/>10,316 sample x taxon rows] --> B[zoodb_sample<br/>503 samples + keys]
  A --> C[zoodb_measurement<br/>long: abundance x2 + biomass]
  A --> D[zoodb_taxon<br/>33 higher taxa + WoRMS]
  C -.sample_id.-> B
  C -.taxon_id.-> D
  B -.ship/cruise/grid.-> E[(shared refs)]

2 Setup

Code
devtools::load_all(here::here("../calcofi4db"))
devtools::load_all(here::here("../calcofi4r"))

librarian::shelf(
  CalCOFI/calcofi4db, CalCOFI/calcofi4r,
  DBI, dplyr, DT, fs, glue, here, janitor, jsonlite, knitr, lubridate, purrr,
  readr, sf, stringr, tibble, tidyr, units, quiet = T)
options(readr.show_col_types = F)
options(DT.options = list(scrollX = TRUE))

source(here("libs/ingest.R"))   # overwrite, dir_data

cc           <- read_calcofi_meta(here("ingest_cce-lter_zoodb.qmd"))
provider     <- cc$provider
dataset      <- cc$dataset
tables_owned <- cc$tables_owned
dir_label    <- glue("{provider}_{dataset}")

dir_parquet  <- here(glue("data/parquet/{dir_label}"))
db_path      <- here(glue("data/wrangling/{dir_label}.duckdb"))

if (overwrite) {
  if (file_exists(db_path))                 file_delete(db_path)
  if (file_exists(paste0(db_path, ".wal"))) file_delete(paste0(db_path, ".wal"))
  if (dir_exists(paste0(db_path, ".tmp")))  dir_delete(paste0(db_path, ".tmp"))
}
dir_create(dirname(db_path))
con <- get_duckdb_con(db_path)
load_duckdb_extension(con, "spatial")

meas_type_csv <- here("metadata/measurement_type.csv")
d_meas_type   <- read_csv(meas_type_csv)

3 Acquire Source Data

ZooDB is an interactive portal, not a file download. libs/download_zoodb.R scripts its public login + save.php CSV export — the 33 higher taxa × 2 abundance units (pooled=combined, raw/uncorrected, all years/months/times/ regions) — and merges the two unit files per taxon (on the source RowNumber) into zoodb_holoplankton.csv. It runs once then caches; set overwrite_all = TRUE in libs/ingest.R to re-scrape. Exact query parameters are recorded in by_taxon/_PROVENANCE.md.

Code
source(here("libs/download_zoodb.R"))
zoo_dir <- path_expand(glue("{dir_data}/cce-lter/ZooDB"))
zoo_csv <- download_zoodb(zoo_dir, overwrite = overwrite_all)
ZooDB: using cached zoodb_holoplankton.csv 

4 Read Source Data

The consolidated long table (one row per taxon × sample) carries N/A for the per-station fields on pooled rows, read as NA.

Code
stopifnot("zoodb_holoplankton.csv not found" = file_exists(zoo_csv))

# archive source (consolidated + per-taxon exports + portal method PDFs) to GCS
sync_to_gcs(
  local_dir  = zoo_dir,
  gcs_prefix = glue("archive/{provider}/{dataset}"),
  bucket     = "calcofi-files-public",
  exclude    = c(".DS_Store", "*.tmp", "*.gdoc"))
# A tibble: 81 × 4
   file                                      action   size reason        
   <chr>                                     <chr>   <dbl> <chr>         
 1 by_taxon/_all_taxa__1000m3.csv            skipped 44108 checksum match
 2 by_taxon/_all_taxa__m2.csv                skipped 43502 checksum match
 3 by_taxon/_manifest.csv                    skipped  5867 checksum match
 4 by_taxon/_PROVENANCE.md                   skipped  2393 checksum match
 5 by_taxon/amphipoda_gammaridea__1000m3.csv skipped 41943 checksum match
 6 by_taxon/amphipoda_gammaridea__m2.csv     skipped 41572 checksum match
 7 by_taxon/amphipoda_hyperiidea__1000m3.csv skipped 42494 checksum match
 8 by_taxon/amphipoda_hyperiidea__m2.csv     skipped 41898 checksum match
 9 by_taxon/appendicularia__1000m3.csv       skipped 42757 checksum match
10 by_taxon/appendicularia__m2.csv           skipped 42170 checksum match
# ℹ 71 more rows
Code
d_raw <- read_csv(zoo_csv, na = c("", "NA", "N/A"))
cat(glue("Read {format(nrow(d_raw), big.mark=',')} rows, {ncol(d_raw)} columns ",
         "from {basename(zoo_csv)}"), "\n")
Read 10,316 rows, 19 columns from zoodb_holoplankton.csv 

5 Clean and Type-Cast

Canonical field names follow metadata/field_dictionary.csv. A sample_key uniquely identifies each sample: U|cruise|towbegin|line|station (unpooled) or P|cruise|date|region (pooled).

Code
d <- d_raw |>
  transmute(
    taxon_slug           = taxon,
    source               = str_to_lower(source),
    cruise_orig          = cruise,
    ship_name            = ship,
    date                 = as.Date(date),
    line                 = suppressWarnings(as.numeric(line)),
    station              = suppressWarnings(as.numeric(station)),
    region               = region,
    datetime_start_utc   = as_datetime(tow_begin),   # Q01: tz assumed local/ship
    datetime_end_utc     = as_datetime(tow_end),
    latitude             = suppressWarnings(as.numeric(latitude)),
    longitude            = suppressWarnings(as.numeric(longitude)),
    max_depth_m          = suppressWarnings(as.numeric(max_depth_m)),
    min_depth_m          = suppressWarnings(as.numeric(min_depth_m)),
    net_type             = net_type,
    n_tows_pooled        = suppressWarnings(as.integer(n_tows_pooled)),
    abundance_per_1000m3 = suppressWarnings(as.numeric(abundance_per_1000m3)),
    abundance_per_m2     = suppressWarnings(as.numeric(abundance_per_m2)),
    biomass_mgC_per_m2   = suppressWarnings(as.numeric(biomass_mgC_per_m2))) |>
  mutate(
    site_key = if_else(
      is.na(line) | is.na(station), NA_character_,
      sprintf("%05.1f %05.1f", line, station)),
    # pooled composites can co-occur per cruise/region (e.g. shallow vs deep
    # strata) — n_tows_pooled disambiguates the otherwise-identical key
    sample_key = if_else(
      source == "unpooled",
      paste("U", cruise_orig, datetime_start_utc, line, station, sep = "|"),
      paste("P", cruise_orig, date, region, n_tows_pooled, sep = "|")))

cat(glue(
  "Cleaned {format(nrow(d), big.mark=',')} (sample x taxon) rows; ",
  "{n_distinct(d$sample_key)} samples ",
  "({n_distinct(d$sample_key[d$source=='unpooled'])} unpooled, ",
  "{n_distinct(d$sample_key[d$source=='pooled'])} pooled); ",
  "{n_distinct(d$taxon_slug)} taxa"), "\n")
Cleaned 10,316 (sample x taxon) rows; 506 samples (351 unpooled, 155 pooled); 33 taxa 

6 Build Taxon Table (WoRMS)

zoodb_taxon is the 33 higher taxa with WoRMS AphiaIDs + classification, resolved offline and cached in metadata/cce-lter/zoodb/taxon_worms.csv (33/33 matched; 2 WoRMS-unaccepted concepts flagged, Q07).

Code
d_worms <- read_csv(here("metadata/cce-lter/zoodb/taxon_worms.csv"))

zoodb_taxon <- d_worms |>
  arrange(taxon_zoodb) |>
  transmute(
    taxon_id        = row_number(),
    taxon_zoodb, taxon_slug,
    aphia_id        = as.integer(aphia_id),
    scientific_name = scientific_name_accepted,
    rank, taxon_status = status,
    kingdom, phylum, class, order_taxon = order, family)   # 'order' is a SQL reserved word

stopifnot("taxon_slug mismatch source vs WoRMS map" =
            all(unique(d$taxon_slug) %in% zoodb_taxon$taxon_slug))
dbWriteTable(con, "zoodb_taxon", zoodb_taxon, overwrite = TRUE)
cat(glue("zoodb_taxon: {nrow(zoodb_taxon)} taxa ",
         "({sum(!is.na(zoodb_taxon$aphia_id))} WoRMS-matched, ",
         "{sum(zoodb_taxon$taxon_status=='unaccepted')} unaccepted [Q07])"), "\n")
zoodb_taxon: 33 taxa (33 WoRMS-matched, 2 unaccepted [Q07]) 

7 Build Sample Table + Resolve Keys

One row per distinct sample. Resolve ship_key/ship_nodc against the shared ship registry, derive cruise_key as YYYY-MM-{ship_nodc} (validated against cruise), and fall back to a unique year-month match for pooled samples (which have no ship). Q01 (timezone) and Q02 (cruise mapping) are tracked.

Code
# distinct sample-level attributes (constant across taxa within a sample)
d_sample0 <- d |>
  distinct(
    sample_key, source, cruise_orig, ship_name, date,
    datetime_start_utc, datetime_end_utc, line, station, site_key, region,
    latitude, longitude, max_depth_m, min_depth_m, net_type, n_tows_pooled)
stopifnot("sample_key not unique over sample attributes" =
            n_distinct(d_sample0$sample_key) == nrow(d_sample0))

# shared reference tables from the ichthyo ingest
load_prior_tables(
  con, parquet_dir = here("data/parquet/swfsc_ichthyo"),
  tables = c("ship", "cruise", "grid"), geom_tables = c("grid"), as_view = TRUE)
# A tibble: 3 × 3
  table   rows has_geom
  <chr>  <dbl> <lgl>   
1 cruise   691 FALSE   
2 grid     218 TRUE    
3 ship      48 FALSE   
Code
d_ship <- dbGetQuery(con, "SELECT ship_key, ship_name, ship_nodc FROM ship") |>
  mutate(ship_name_norm = str_squish(str_to_upper(ship_name))) |>
  distinct(ship_name_norm, .keep_all = TRUE)

# year-month -> unique cruise_key (pooled samples + fallback)
cr_ym <- dbGetQuery(con, "SELECT cruise_key, date_ym FROM cruise") |>
  mutate(ym = format(as.Date(date_ym), "%Y%m")) |>
  group_by(ym) |>
  summarize(cruise_key_ym = if (n() == 1) cruise_key[1] else NA_character_,
            .groups = "drop")
valid_ck <- dbGetQuery(con, "SELECT DISTINCT cruise_key FROM cruise")$cruise_key

d_sample <- d_sample0 |>
  mutate(ship_name_norm = ship_name |>
           str_replace("^R/?V\\.?\\s+", "") |> str_to_upper() |> str_squish()) |>
  left_join(d_ship |> select(ship_name_norm, ship_key, ship_nodc),
            by = "ship_name_norm") |>
  mutate(
    ym = format(date, "%Y%m"),
    cruise_key_ship = if_else(
      is.na(ship_nodc), NA_character_,
      as.character(glue("{format(date, '%Y-%m')}-{ship_nodc}"))),
    cruise_key_ship = if_else(cruise_key_ship %in% valid_ck,
                              cruise_key_ship, NA_character_)) |>
  left_join(cr_ym, by = "ym") |>
  mutate(cruise_key = coalesce(cruise_key_ship, cruise_key_ym)) |>
  arrange(source, date, cruise_orig, region, line, station, sample_key) |>
  mutate(sample_id = row_number(), .before = 1)

# id <-> key map for the measurement join, then drop the key from the table
sample_map   <- d_sample |> select(sample_id, sample_key)
zoodb_sample <- d_sample |>
  select(sample_id, source, cruise_orig, cruise_key, ship_name, ship_key,
         date, datetime_start_utc, datetime_end_utc, line, station, site_key,
         region, latitude, longitude, max_depth_m, min_depth_m, net_type,
         n_tows_pooled)
dbWriteTable(con, "zoodb_sample", zoodb_sample, overwrite = TRUE)

n_ship   <- sum(!is.na(d_sample$ship_key))
n_cruise <- sum(!is.na(d_sample$cruise_key))
cat(glue(
  "zoodb_sample: {nrow(zoodb_sample)} samples; ",
  "ship_key {n_ship}/{nrow(d_sample)} ({round(100*n_ship/nrow(d_sample),1)}%), ",
  "cruise_key {n_cruise}/{nrow(d_sample)} ({round(100*n_cruise/nrow(d_sample),1)}%)"), "\n")
zoodb_sample: 506 samples; ship_key 351/506 (69.4%), cruise_key 384/506 (75.9%) 

8 Add Spatial

Unpooled tows get a point geometry and a CalCOFI grid_key; pooled samples have no position (NULL geometry / grid, Q05).

Code
add_point_geom(con, "zoodb_sample", lon_col = "longitude", lat_col = "latitude")
grid_stats <- assign_grid_key(con, "zoodb_sample")
grid_stats |> datatable(caption = "Grid assignment (unpooled tows)")

9 Pivot Measurements to Long Format

The three measured quantities pivot into long form, joined to sample_id (via sample_key) and taxon_id (via taxon_slug). Explicit zeros (analyzed- but-absent) are retained; only NA / non-finite values are dropped.

Code
meas_recode <- c(
  abundance_per_1000m3 = "zooplankton_abundance",
  abundance_per_m2     = "zooplankton_abundance_areal",
  biomass_mgC_per_m2   = "zooplankton_biomass_carbon")

zoodb_measurement <- d |>
  left_join(sample_map, by = "sample_key") |>
  left_join(zoodb_taxon |> select(taxon_id, taxon_slug), by = "taxon_slug") |>
  select(sample_id, taxon_id,
         abundance_per_1000m3, abundance_per_m2, biomass_mgC_per_m2) |>
  pivot_longer(
    cols = c(abundance_per_1000m3, abundance_per_m2, biomass_mgC_per_m2),
    names_to = "src_col", values_to = "measurement_value") |>
  filter(!is.na(measurement_value), is.finite(measurement_value)) |>
  mutate(measurement_type = unname(meas_recode[src_col])) |>
  arrange(sample_id, taxon_id, measurement_type) |>
  transmute(measurement_id = row_number(),
            sample_id, taxon_id, measurement_type, measurement_value)

dbWriteTable(con, "zoodb_measurement", zoodb_measurement, overwrite = TRUE)
cat(glue(
  "zoodb_measurement: {format(nrow(zoodb_measurement), big.mark=',')} rows ",
  "({format(sum(zoodb_measurement$measurement_value==0), big.mark=',')} explicit zeros = ",
  "analyzed-but-absent, Q04)"), "\n")
zoodb_measurement: 30,948 rows (13,152 explicit zeros = analyzed-but-absent, Q04) 

10 Add Measurement Types

Code
zoo_types <- tibble(
  measurement_type = c("zooplankton_abundance", "zooplankton_abundance_areal",
                       "zooplankton_biomass_carbon"),
  description = c(
    "Zooplankton numerical abundance (volumetric density) per net tow, by higher taxon. Raw (uncorrected for net type).",
    "Zooplankton areal abundance (depth-integrated density) per net tow, by higher taxon. Raw (uncorrected).",
    "Zooplankton carbon biomass per net tow, by higher taxon. Raw (uncorrected for net type)."),
  units = c("count/1000m3", "count/m2", "mgC/m2"),
  is_canonical = c(TRUE, NA, TRUE),
  `_source_column` = c("abundance_per_1000m3", "abundance_per_m2", "biomass_mgC_per_m2"),
  `_source_table` = "zoodb_measurement",
  `_source_datasets` = "cce-lter_zoodb",
  `_qual_column` = NA_character_, `_prec_column` = NA_character_)

new_types <- zoo_types |> filter(!measurement_type %in% d_meas_type$measurement_type)
if (nrow(new_types) > 0) {
  d_meas_type <- bind_rows(d_meas_type, new_types)
  write_csv(d_meas_type, meas_type_csv)
  cat(glue("Added measurement type(s): {paste(new_types$measurement_type, collapse=', ')}"), "\n")
} else cat("zooplankton measurement types already registered\n")
zooplankton measurement types already registered
Code
dbWriteTable(con, "measurement_type", d_meas_type, overwrite = TRUE)

11 Load Dataset Metadata

Code
d_dataset <- ingest_yaml_to_dataset_df(read_ingest_yaml(here()))
dbWriteTable(con, "dataset", d_dataset, overwrite = TRUE)
cat(glue("dataset: {nrow(d_dataset)} dataset(s) registered"), "\n")
dataset: 12 dataset(s) registered 

12 Schema Documentation

Code
zoodb_rels <- list(
  primary_keys = list(
    zoodb_sample      = "sample_id",
    zoodb_measurement = "measurement_id",
    zoodb_taxon       = "taxon_id",
    measurement_type  = "measurement_type"),
  foreign_keys = list(
    list(table = "zoodb_measurement", column = "sample_id",
         ref_table = "zoodb_sample", ref_column = "sample_id"),
    list(table = "zoodb_measurement", column = "taxon_id",
         ref_table = "zoodb_taxon", ref_column = "taxon_id"),
    list(table = "zoodb_measurement", column = "measurement_type",
         ref_table = "measurement_type", ref_column = "measurement_type")))

cc_erd(
  con,
  tables = c("zoodb_sample", "zoodb_measurement", "zoodb_taxon",
             "measurement_type", "dataset"),
  rels = zoodb_rels,
  colors = list(
    lightblue   = c("zoodb_sample", "zoodb_measurement"),
    lightgreen  = "zoodb_taxon",
    lightyellow = "measurement_type",
    white       = "dataset"))

Code
build_relationships_json(
  rels = zoodb_rels, output_dir = dir_parquet,
  provider = provider, dataset = dataset)
[1] "/Users/bbest/Github/CalCOFI/workflows/data/parquet/cce-lter_zoodb/relationships.json"

13 Validate

Code
results <- validate_for_release(con, checks = "all", strict = FALSE)
cat("Validation:", ifelse(results$passed, "PASSED", "FAILED"), "\n")
Validation: FAILED 
Code
if (length(results$errors) > 0)
  cat("Errors:\n", paste("-", results$errors, collapse = "\n"), "\n")
Errors:
 - Table 'zoodb_sample' has 122 NULL values in required column 'cruise_key'
- Table 'zoodb_sample' has 155 NULL values in required column 'ship_key'
- Table 'zoodb_sample' has 155 NULL values in required column 'site_key'
- Table 'zoodb_sample' has 156 NULL values in required column 'grid_key' 
Code
if (length(results$warnings) > 0)
  cat("Warnings:\n", paste("-", results$warnings, collapse = "\n"), "\n")
Warnings:
 - Missing expected tables: site, tow, net, larva, species 
Code
# NULLs in cross-dataset keys (cruise_key, ship_key, site_key, grid_key) are the
# EXPECTED unmatched / not-applicable remainder (pooled samples have no station;
# cruise/ship mapping is partial — Q02). Not a hard failure.
cov <- dbGetQuery(con,
  "SELECT
     AVG(CASE WHEN ship_key   IS NOT NULL THEN 1 ELSE 0 END) ship,
     AVG(CASE WHEN cruise_key IS NOT NULL THEN 1 ELSE 0 END) cruise,
     AVG(CASE WHEN grid_key   IS NOT NULL THEN 1 ELSE 0 END) grid
   FROM zoodb_sample")
cat(glue("Match coverage: ship_key {round(100*cov$ship,1)}%, ",
         "cruise_key {round(100*cov$cruise,1)}%, grid_key {round(100*cov$grid,1)}%"), "\n")
Match coverage: ship_key 69.4%, cruise_key 75.9%, grid_key 69.2% 
Code
# FK integrity: every measurement resolves to a sample and a taxon
orphan <- dbGetQuery(con,
  "SELECT
     SUM(CASE WHEN s.sample_id IS NULL THEN 1 ELSE 0 END) AS no_sample,
     SUM(CASE WHEN t.taxon_id  IS NULL THEN 1 ELSE 0 END) AS no_taxon
   FROM zoodb_measurement m
   LEFT JOIN zoodb_sample s USING (sample_id)
   LEFT JOIN zoodb_taxon  t USING (taxon_id)")
cat(glue("Orphan measurements: {orphan$no_sample} w/o sample, {orphan$no_taxon} w/o taxon"), "\n")
Orphan measurements: 0 w/o sample, 0 w/o taxon 

14 Data Preview

Code
samp_cols <- dbGetQuery(con,
  "SELECT column_name FROM information_schema.columns
   WHERE table_name='zoodb_sample' AND data_type NOT LIKE 'GEOMETRY%'")$column_name
dbGetQuery(con, glue("SELECT {paste(samp_cols, collapse=', ')} FROM zoodb_sample LIMIT 100")) |>
  datatable(caption = "zoodb_sample — first 100 rows", rownames = FALSE, filter = "top")
Code
dbGetQuery(con,
  "SELECT m.measurement_id, m.sample_id, x.taxon_zoodb, m.measurement_type, m.measurement_value
   FROM zoodb_measurement m JOIN zoodb_taxon x USING (taxon_id)
   ORDER BY m.measurement_id LIMIT 100") |>
  datatable(caption = "zoodb_measurement — first 100 rows", rownames = FALSE)
Code
dbGetQuery(con,
  "SELECT t.taxon_id, t.taxon_zoodb, t.scientific_name, t.aphia_id, t.rank,
          t.phylum, t.class, COUNT(DISTINCT m.sample_id) AS n_samples
   FROM zoodb_taxon t LEFT JOIN zoodb_measurement m USING (taxon_id)
   GROUP BY ALL ORDER BY t.taxon_id") |>
  datatable(caption = "zoodb_taxon — 33 higher taxa (samples analyzed)", rownames = FALSE)

15 Write Parquet Outputs

Code
dir_create(dir_parquet)

mismatches <- list(
  measurement_types = collect_measurement_type_mismatches(
    con, here("metadata/measurement_type.csv")),
  cruise_keys = collect_cruise_key_mismatches(con, "zoodb_sample"))

parquet_stats <- write_parquet_outputs(
  con        = con,
  output_dir = dir_parquet,
  tables     = c("zoodb_sample", "zoodb_measurement", "zoodb_taxon",
                 "measurement_type", "dataset"),
  sort_by    = list(zoodb_measurement = "sample_id"),
  strip_provenance = FALSE,
  mismatches = mismatches)

parquet_stats |> mutate(file = basename(path)) |> select(-path) |>
  datatable(caption = "Parquet export statistics")

16 Write Metadata

Code
d_tbls_rd <- read_csv(here("metadata/cce-lter/zoodb/tbls_redefine.csv"))
d_flds_rd <- read_csv(here("metadata/cce-lter/zoodb/flds_redefine.csv"))

metadata_path <- build_metadata_json(
  con        = con,
  d_tbls_rd  = d_tbls_rd,
  d_flds_rd  = d_flds_rd,
  metadata_derived_csv = here("metadata/cce-lter/zoodb/metadata_derived.csv"),
  output_dir = dir_parquet,
  tables     = c("zoodb_sample", "zoodb_measurement", "zoodb_taxon"),
  set_comments = TRUE,
  provider   = provider,
  dataset    = dataset,
  workflow_url = cc$workflow_url,
  tables_owned = tables_owned)

17 Upload to GCS

Code
sync_to_gcs(
  local_dir  = dir_parquet,
  gcs_prefix = glue("ingest/{dir_label}"),
  bucket     = "calcofi-db")
# A tibble: 8 × 4
  file                      action     size reason        
  <chr>                     <chr>     <dbl> <chr>         
1 dataset.parquet           uploaded   8904 crc32c changed
2 manifest.json             uploaded   2703 crc32c changed
3 measurement_type.parquet  uploaded   5793 crc32c changed
4 metadata.json             uploaded   8831 crc32c changed
5 relationships.json        skipped     727 checksum match
6 zoodb_measurement.parquet skipped  114501 checksum match
7 zoodb_sample.parquet      skipped   18704 checksum match
8 zoodb_taxon.parquet       skipped    3766 checksum match

18 Questions for Data Providers

Follow-up questions for CCE-LTER (Mark Ohman, Bertha Lavaniegos, Linsey Sala), ranked by importance. This ingest proceeded with the documented provisional decisions noted above; high answers may change cruise/ship matching or the timezone assumption on the next run. Tracked in metadata/cce-lter/zoodb/questions.csv.

Code
read_csv(here(glue("metadata/{provider}/{dataset}/questions.csv"))) |>
  arrange(factor(priority, c("blocker", "high", "normal", "low")), id) |>
  select(priority, question, context, related_field, status) |>
  datatable(
    caption = "Questions for data providers (ranked by importance)",
    options = list(dom = "t", pageLength = 20), rownames = FALSE)

19 Cleanup

Code
close_duckdb(con)
cat(glue("Parquet outputs written to: {dir_parquet}"), "\n")
Parquet outputs written to: /Users/bbest/Github/CalCOFI/workflows/data/parquet/cce-lter_zoodb 
Code
devtools::session_info()
─ Session info ───────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.5.2 (2025-10-31)
 os       macOS Sequoia 15.7.1
 system   aarch64, darwin20
 ui       X11
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       Europe/Rome
 date     2026-06-26
 pandoc   3.8.3 @ /opt/homebrew/bin/ (via rmarkdown)
 quarto   1.8.25 @ /usr/local/bin/quarto

─ Packages ───────────────────────────────────────────────────────────────────
 ! package             * version    date (UTC) lib source
   abind                 1.4-8      2024-09-12 [1] CRAN (R 4.5.0)
   arrow                 24.0.0     2026-04-29 [1] CRAN (R 4.5.2)
   askpass               1.2.1      2024-10-04 [1] CRAN (R 4.5.0)
   assertthat            0.2.1      2019-03-21 [1] CRAN (R 4.5.0)
   backports             1.5.1      2026-04-03 [1] CRAN (R 4.5.2)
   base64enc             0.1-6      2026-02-02 [1] CRAN (R 4.5.2)
   bit                   4.6.0      2025-03-06 [1] CRAN (R 4.5.0)
   bit64                 4.8.2      2026-05-19 [1] CRAN (R 4.5.2)
   blob                  1.3.0      2026-01-14 [1] CRAN (R 4.5.2)
   broom                 1.0.13     2026-05-14 [1] CRAN (R 4.5.2)
   bslib                 0.11.0     2026-05-16 [1] CRAN (R 4.5.2)
   cachem                1.1.0      2024-05-16 [1] CRAN (R 4.5.0)
 P calcofi4db          * 2.8.2      2026-06-07 [?] load_all()
 P calcofi4r           * 1.3.0      2026-06-05 [?] load_all()
   class                 7.3-23     2025-01-01 [1] CRAN (R 4.5.2)
   classInt              0.4-11     2025-01-08 [1] CRAN (R 4.5.0)
   cli                   3.6.6      2026-04-09 [1] CRAN (R 4.5.2)
   codetools             0.2-20     2024-03-31 [1] CRAN (R 4.5.2)
   crayon                1.5.3      2024-06-20 [1] CRAN (R 4.5.0)
   crosstalk             1.2.2      2025-08-26 [1] CRAN (R 4.5.0)
   curl                  7.1.0      2026-04-22 [1] CRAN (R 4.5.2)
   data.table            1.18.4     2026-05-06 [1] CRAN (R 4.5.2)
   DBI                 * 1.3.0      2026-02-25 [1] CRAN (R 4.5.2)
   dbplyr                2.5.2      2026-02-13 [1] CRAN (R 4.5.2)
   desc                  1.4.3      2023-12-10 [1] CRAN (R 4.5.0)
   devtools              2.5.0      2026-03-14 [1] CRAN (R 4.5.2)
   DiagrammeR            1.0.12     2026-04-27 [1] CRAN (R 4.5.2)
   DiagrammeRsvg         0.1        2016-02-04 [1] CRAN (R 4.5.0)
   digest                0.6.39     2025-11-19 [1] CRAN (R 4.5.2)
   dm                    1.1.2      2026-05-17 [1] CRAN (R 4.5.2)
   dplyr               * 1.2.1      2026-04-03 [1] CRAN (R 4.5.2)
   DT                  * 0.34.0     2025-09-02 [1] CRAN (R 4.5.0)
   duckdb                1.5.2      2026-04-13 [1] CRAN (R 4.5.2)
   dygraphs              1.1.1.6    2018-07-11 [1] CRAN (R 4.5.0)
   e1071                 1.7-17     2025-12-18 [1] CRAN (R 4.5.2)
   ellipsis              0.3.2      2021-04-29 [1] CRAN (R 4.5.0)
   evaluate              1.0.5      2025-08-27 [1] CRAN (R 4.5.0)
   farver                2.1.2      2024-05-13 [1] CRAN (R 4.5.0)
   fastmap               1.2.0      2024-05-15 [1] CRAN (R 4.5.0)
   fs                  * 2.1.0      2026-04-18 [1] CRAN (R 4.5.2)
   fuzzyjoin             0.1.8      2026-02-20 [1] CRAN (R 4.5.2)
   gargle                1.6.1      2026-01-29 [1] CRAN (R 4.5.2)
   generics              0.1.4      2025-05-09 [1] CRAN (R 4.5.0)
   geojsonsf             2.0.5      2025-11-26 [1] CRAN (R 4.5.2)
   ggplot2               4.0.3      2026-04-22 [1] CRAN (R 4.5.2)
   glue                * 1.8.1      2026-04-17 [1] CRAN (R 4.5.2)
   googleAuthR           2.0.2.1    2026-01-09 [1] CRAN (R 4.5.2)
   googleCloudStorageR   0.7.0      2021-12-16 [1] CRAN (R 4.5.0)
   googledrive           2.1.2      2025-09-10 [1] CRAN (R 4.5.0)
   gtable                0.3.6      2024-10-25 [1] CRAN (R 4.5.0)
   here                * 1.0.2      2025-09-15 [1] CRAN (R 4.5.0)
   highcharter           0.9.5      2026-04-22 [1] CRAN (R 4.5.2)
   hms                   1.1.4      2025-10-17 [1] CRAN (R 4.5.0)
   htmltools             0.5.9      2025-12-04 [1] CRAN (R 4.5.2)
   htmlwidgets           1.6.4      2023-12-06 [1] CRAN (R 4.5.0)
   httpuv                1.6.17     2026-03-18 [1] CRAN (R 4.5.2)
   httr                  1.4.8      2026-02-13 [1] CRAN (R 4.5.2)
   httr2                 1.2.2      2025-12-08 [1] CRAN (R 4.5.2)
   igraph                2.3.2      2026-05-29 [1] CRAN (R 4.5.2)
   isoband               0.3.0      2025-12-07 [1] CRAN (R 4.5.2)
   janitor             * 2.2.1      2024-12-22 [1] CRAN (R 4.5.0)
   jquerylib             0.1.4      2021-04-26 [1] CRAN (R 4.5.0)
   jsonlite            * 2.0.0      2025-03-27 [1] CRAN (R 4.5.0)
   KernSmooth            2.23-26    2025-01-01 [1] CRAN (R 4.5.2)
   knitr               * 1.51       2025-12-20 [1] CRAN (R 4.5.2)
   later                 1.4.8      2026-03-05 [1] CRAN (R 4.5.2)
   lattice               0.22-9     2026-02-09 [1] CRAN (R 4.5.2)
   lazyeval              0.2.3      2026-04-04 [1] CRAN (R 4.5.2)
   leafem                0.2.5      2025-08-28 [1] CRAN (R 4.5.0)
   leaflet               2.2.3      2025-09-04 [1] CRAN (R 4.5.0)
   librarian             1.8.1      2021-07-12 [1] CRAN (R 4.5.0)
   lifecycle             1.0.5      2026-01-08 [1] CRAN (R 4.5.2)
   lubridate           * 1.9.5      2026-02-04 [1] CRAN (R 4.5.2)
   magrittr              2.0.5      2026-04-04 [1] CRAN (R 4.5.2)
   mapgl                 0.5.0      2026-06-24 [1] Github (bbest/mapgl@1e52f60)
   mapview               2.11.4     2025-09-08 [1] CRAN (R 4.5.0)
   markdown              2.0        2025-03-23 [1] CRAN (R 4.5.0)
   Matrix                1.7-5      2026-03-21 [1] CRAN (R 4.5.2)
   memoise               2.0.1      2021-11-26 [1] CRAN (R 4.5.0)
   mgcv                  1.9-4      2025-11-07 [1] CRAN (R 4.5.0)
   mime                  0.13       2025-03-17 [1] CRAN (R 4.5.0)
   nlme                  3.1-169    2026-03-27 [1] CRAN (R 4.5.2)
   openssl               2.4.1      2026-05-14 [1] CRAN (R 4.5.2)
   otel                  0.2.0      2025-08-29 [1] CRAN (R 4.5.0)
   pillar                1.11.1     2025-09-17 [1] CRAN (R 4.5.0)
   pkgbuild              1.4.8      2025-05-26 [1] CRAN (R 4.5.0)
   pkgconfig             2.0.3      2019-09-22 [1] CRAN (R 4.5.0)
   pkgload               1.5.1      2026-04-01 [1] CRAN (R 4.5.2)
   plotly                4.12.0     2026-01-24 [1] CRAN (R 4.5.2)
   png                   0.1-9      2026-03-15 [1] CRAN (R 4.5.2)
   promises              1.5.0      2025-11-01 [1] CRAN (R 4.5.0)
   proxy                 0.4-29     2025-12-29 [1] CRAN (R 4.5.2)
   purrr               * 1.2.2      2026-04-10 [1] CRAN (R 4.5.2)
   quantmod              0.4.28     2025-06-19 [1] CRAN (R 4.5.0)
   R6                    2.6.1      2025-02-15 [1] CRAN (R 4.5.0)
   rappdirs              0.3.4      2026-01-17 [1] CRAN (R 4.5.2)
   raster                3.6-32     2025-03-28 [1] CRAN (R 4.5.0)
   RColorBrewer          1.1-3      2022-04-03 [1] CRAN (R 4.5.0)
   Rcpp                  1.1.1-1.1  2026-04-24 [1] CRAN (R 4.5.2)
   readr               * 2.2.0      2026-02-19 [1] CRAN (R 4.5.2)
   rlang                 1.2.0      2026-04-06 [1] CRAN (R 4.5.2)
   rlist                 0.4.6.2    2021-09-03 [1] CRAN (R 4.5.0)
   rmarkdown             2.31       2026-03-26 [1] CRAN (R 4.5.2)
   rnaturalearth         1.2.0      2026-01-19 [1] CRAN (R 4.5.2)
   rnaturalearthhires    1.0.0.9000 2025-10-02 [1] Github (ropensci/rnaturalearthhires@e4736f6)
   RPostgres             1.4.10     2026-02-16 [1] CRAN (R 4.5.2)
   rprojroot             2.1.1      2025-08-26 [1] CRAN (R 4.5.0)
   rstudioapi            0.18.0     2026-01-16 [1] CRAN (R 4.5.2)
   S7                    0.2.2      2026-04-22 [1] CRAN (R 4.5.2)
   sass                  0.4.10     2025-04-11 [1] CRAN (R 4.5.0)
   satellite             1.0.6      2025-08-21 [1] CRAN (R 4.5.0)
   scales                1.4.0      2025-04-24 [1] CRAN (R 4.5.0)
   sessioninfo           1.2.3      2025-02-05 [1] CRAN (R 4.5.0)
   sf                  * 1.1-1      2026-05-06 [1] CRAN (R 4.5.2)
   shiny                 1.14.0     2026-06-21 [1] CRAN (R 4.5.2)
   shinyWidgets          0.9.1      2026-03-09 [1] CRAN (R 4.5.2)
   snakecase             0.11.1     2023-08-27 [1] CRAN (R 4.5.0)
   sp                    2.2-1      2026-02-13 [1] CRAN (R 4.5.2)
   stars                 0.7-2      2026-04-03 [1] CRAN (R 4.5.2)
   stringi               1.8.7      2025-03-27 [1] CRAN (R 4.5.0)
   stringr             * 1.6.0      2025-11-04 [1] CRAN (R 4.5.0)
   terra                 1.9-34     2026-06-19 [1] CRAN (R 4.5.2)
   tibble              * 3.3.1      2026-01-11 [1] CRAN (R 4.5.2)
   tidyr               * 1.3.2      2025-12-19 [1] CRAN (R 4.5.2)
   tidyselect            1.2.1      2024-03-11 [1] CRAN (R 4.5.0)
   timechange            0.4.0      2026-01-29 [1] CRAN (R 4.5.2)
   TTR                   0.24.4     2023-11-28 [1] CRAN (R 4.5.0)
   tzdb                  0.5.0      2025-03-15 [1] CRAN (R 4.5.0)
   units               * 1.0-1      2026-03-11 [1] CRAN (R 4.5.2)
   usethis               3.2.1      2025-09-06 [1] CRAN (R 4.5.0)
   utf8                  1.2.6      2025-06-08 [1] CRAN (R 4.5.0)
   uuid                  1.2-2      2026-01-23 [1] CRAN (R 4.5.2)
   V8                    8.2.0      2026-04-21 [1] CRAN (R 4.5.2)
   vctrs                 0.7.3      2026-04-11 [1] CRAN (R 4.5.2)
   viridisLite           0.4.3      2026-02-04 [1] CRAN (R 4.5.2)
   visNetwork            2.1.4      2025-09-04 [1] CRAN (R 4.5.0)
   vroom                 1.7.1      2026-03-31 [1] CRAN (R 4.5.2)
   withr                 3.0.3      2026-06-19 [1] CRAN (R 4.5.2)
   xfun                  0.59       2026-06-19 [1] CRAN (R 4.5.2)
   xtable                1.8-8      2026-02-22 [1] CRAN (R 4.5.2)
   xts                   0.14.2     2026-02-28 [1] CRAN (R 4.5.2)
   yaml                  2.3.12     2025-12-10 [1] CRAN (R 4.5.2)
   zip                   2.3.3      2025-05-13 [1] CRAN (R 4.5.0)
   zoo                   1.8-15     2025-12-15 [1] CRAN (R 4.5.2)

 [1] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library

 * ── Packages attached to the search path.
 P ── Loaded and on-disk path mismatch.

──────────────────────────────────────────────────────────────────────────────