Package index

Cloud Storage (GCS)

functions for Google Cloud Storage operations

cc_stage_dir(): Local Staging Root for Bulk Data
cc_stage_path(): Path Within the Local Staging Root
cleanup_gcs_obsolete(): Clean up obsolete GCS directories from dataset renames
copy_gcs_file(): Server-side copy between GCS paths
create_gcs_manifest(): Create a manifest of current GCS files
delete_gcs_prefix(): Delete all objects under a GCS prefix
get_calcofi_file(): Get a CalCOFI file from the immutable archive
get_gcs_file(): Get a file from Google Cloud Storage
get_historical_file(): Get historical file from a specific date
get_manifest(): Get manifest for a specific date
list_calcofi_files(): List CalCOFI files from manifest
list_gcs_files(): List files in a GCS bucket/prefix
list_gcs_versions(): List versions of a file in GCS archive
put_gcs_file(): Upload a file to Google Cloud Storage
sync_to_gcs(): Sync local files to GCS, skipping unchanged files

Parquet

functions for Apache Parquet file operations

add_parquet_metadata(): Add metadata to Parquet file
csv_to_parquet(): Convert CSV file to Parquet format
export_parquet(): Export a DuckDB Table or Query to Parquet
get_parquet_metadata(): Get Parquet file metadata
read_parquet_table(): Read a Parquet table
upload_parquet(): Upload Parquet file to GCS
write_parquet_table(): Write data to Parquet format

DuckDB

basic DuckDB database operations

close_duckdb(): Disconnect from DuckDB and shutdown
create_duckdb_from_parquet(): Create DuckDB from Parquet files
create_duckdb_views(): Create views from a manifest
duckdb_to_parquet(): Export DuckDB table to Parquet
get_duckdb_con(): Get a DuckDB connection
get_duckdb_tables(): Get table information from DuckDB
load_duckdb_extension(): Install and load DuckDB extension
save_duckdb_to_gcs(): Save DuckDB to GCS
set_duckdb_comments(): Set table and column comments in DuckDB

Working DuckLake

functions for the internal Working DuckLake with provenance tracking

add_provenance_columns(): Add provenance columns to a data frame
get_working_ducklake(): Get Working DuckLake connection
ingest_dataset(): Ingest Dataset into Working DuckLake
ingest_to_working(): Ingest data to Working DuckLake
list_working_tables(): List tables with provenance in Working DuckLake
load_prior_tables(): Load Tables from a Prior Ingest's Parquet Directory
query_at_time(): Query Working DuckLake at a point in time
save_working_ducklake(): Save Working DuckLake to GCS
strip_provenance_columns(): Strip provenance columns from data

Frozen Releases

functions for creating and managing frozen DuckLake releases

compare_releases(): Compare two frozen releases
freeze_release(): Freeze a release of the DuckLake
get_release_metadata(): Get metadata for a frozen release
list_frozen_releases(): List available frozen releases
upload_frozen_release(): Upload Frozen Release to GCS
validate_for_release(): Validate Working DuckLake for release

Read

functions reading data, particularly from CSV files in the Google Drive CalCOFI data folder

create_redefinition_files(): Create Redefinition Files for Tables and Fields
determine_field_types(): Determine Field Types for Database
read_csv_files(): Read CSV Files and Their Metadata
read_csv_metadata(): Read CSV Files and Extract Metadata

Transform

functions to transform data, after reading data and before ingesting into the database

detect_csv_changes(): Detect Changes in CSV Files
display_csv_changes(): Display CSV Changes in a Formatted Table
print_csv_change_stats(): Print CSV Change Statistics
transform_data(): Transform Data for Database Ingestion

Check

functions for data validation and integrity checking

check_cruise_coverage(): Cruises that carry samples but no observations — the silent-loss guard
check_data_integrity(): Check Data Integrity for Ingestion
check_multiple_datasets(): Check Multiple Datasets for Integrity
check_taxon_ids(): The taxa that no authority resolved, per dataset — reported, and gated
render_integrity_message(): Render Data Integrity Check Message

Validate

functions for referential integrity validation and flagging invalid rows

bounds_datatable(): Render a bounds check as the standard notebook table
check_measurement_bounds(): Check measured values against the registry's declared bounds
delete_flagged_rows(): Delete Flagged Rows from Database
drop_out_of_bounds(): Delete values outside their declared bounds
flag_invalid_rows(): Flag and Export Invalid Rows
validate_dataset(): Run All Validations for a Dataset
validate_egg_stages(): Validate Egg Stage Values
validate_fk_references(): Validate Foreign Key References
validate_lookup_values(): Validate Lookup Values Exist

Ingest

functions for ingesting data into the database (PostgreSQL deprecated, use DuckLake)

changed_inputs(): Which inputs changed since a recorded fingerprint
ingest_csv_to_db() deprecated: Ingest CSV data to PostgreSQL database (DEPRECATED)
ingest_dataset_pg() deprecated: Ingest a Dataset to PostgreSQL (DEPRECATED)
input_fingerprint(): Fingerprint the inputs an ingest's outputs depend on
read_input_fingerprint(): Read a previously recorded input fingerprint
write_input_fingerprint(): Record an input fingerprint next to an ingest's outputs

Version

functions for schema and package versioning

get_schema_versions(): Get Schema Version History
init_schema_version_csv(): Initialize Schema Version CSV
record_schema_version(): Record Schema Version

Utilities

utility functions for database operations (PostgreSQL deprecated)

copy_schema(): Copy Database Schema
get_db_con() deprecated: Get a database connection to the CalCOFI PostgreSQL database (DEPRECATED)

Wrangle

functions for local DuckDB wrangling (keys, IDs, table consolidation)

apply_data_corrections(): Apply Data Corrections
assign_deterministic_uuids(): Assign deterministic UUIDs from composite key columns
assign_deterministic_uuids_md5(): Assign deterministic UUIDs using DuckDB-native md5
assign_sequential_ids(): Assign Sequential IDs with Deterministic Sort Order
build_metadata_json(): Build Metadata JSON for Parquet Outputs
build_relationships_json(): Build Relationships JSON from dm Object
collect_cruise_key_mismatches(): Collect Cruise Key Mismatches
collect_measurement_type_mismatches(): Collect Measurement Type Mismatches
collect_ship_mismatches(): Collect Ship Mismatches
consolidate_ichthyo_tables(): Consolidate Ichthyoplankton Tables into Tidy Format
convert_cruise_key_format(): Convert Old YYMMKK Cruise Key to YYYY-MM-NODC Format
create_cruise_key(): Create Cruise Key from Ship NODC Code and Date
create_lookup_table(): Create Lookup Table from Vocabulary Definitions
derive_measurement_type_datasets(): Derive measurement_type → contributing datasets from the data
enforce_column_types(): Enforce Column Types Before Export
ingest_yaml_to_dataset_df(): Build the dataset registry table from ingest YAML blocks
merge_metadata_json(): Merge Per-Ingest metadata.json into a Release-Level Sidecar
merge_relationships_json(): Merge Multiple Relationships JSON Files
propagate_natural_key(): Propagate Key from Parent to Child Table
read_calcofi_meta(): Read the calcofi YAML block from a single workflow file
read_ingest_yaml(): Read calcofi YAML blocks from ingest_*.qmd front matter
read_relationships_json(): Read Relationships JSON and Optionally Apply to dm
release_excluded_datasets(): Datasets held out of the release by calcofi.in_release: false
replace_uuid_with_id(): Replace UUIDs with Integer Foreign Keys
scan_metadata_gaps(): Scan a metadata sidecar for missing documentation
standardize_site_key(): Standardize Site Key from Line and Station Columns
write_parquet_outputs(): Write Tables to Parquet Files
write_spatial_manifest(): Write Spatial Manifest

Workflow

functions for orchestrating ingestion workflow steps

build_release_table_registry(): Build Release Table Registry from Ingest Manifests
build_targets_list(): Build Targets List from Quarto Frontmatter
finalize_ingest(): Finalize Ingest — Push Parquet Tables to Working DuckLake
integrate_to_working_ducklake(): Integrate Ingest Outputs into Working DuckLake
list_ingest_outputs(): List Available Ingest Outputs
parse_qmd_frontmatter(): Parse YAML Frontmatter from Quarto Notebooks
read_ingest_manifest(): Read Ingest Manifest from GCS
read_ingest_parquet(): Read Ingest Parquet Table from GCS
write_ingest_outputs(): Write Ingest Workflow Outputs to GCS

Display

helper functions for workflow display outputs (GitHub links, validation tables)

dt(): Create Interactive Data Table with CSV Export
github_file_link(): Create GitHub File Link
preview_tables(): Preview Tables with Head and Tail Rows
show_flagged_file(): Show Flagged File Result
show_validation_results(): Show Validation Results with GitHub Links

Version Sync

functions for synchronizing schema and package versions

commit_version_and_permalink(): Commit Version Changes and Get Permalink
complete_version_release(): Complete Version Release Workflow
get_package_version(): Get Current Package Version
suggest_next_version(): Suggest Next Version
update_package_version(): Synchronized Version Management for Package and Database

Visualize

functions visualizing diagnostic outputs, particularly color coded data tables

show_fields_redefine(): Show fields to redefine
show_source_files(): Show source files
show_tables_redefine(): Show tables to redefine

Other

check for other functions or datasets not captured by above categories

add_point_geom(): Add Point Geometry Column to a DuckDB Table
append_obs(): Append occurrence-headline rows into the core obs table
append_obs_attribute(): Append sub-occurrence attribute rows into the core obs_attribute table
append_sample(): Append event rows into the core sample dimension
append_sample_measurement(): Append event-level (effort) rows into the core sample_measurement table
assemble_core(): Assemble the whole consolidated core from the ingest shards
assemble_core_table(): Assemble one core table from its per-dataset shards
assign_grid_key(): Assign Grid Key via Spatial Join
build_dataset_taxon(): Build the dataset_taxon crosswalk (per-dataset vocabulary -> taxon)
build_grid_reference(): Build the shared grid reference table (deterministic, dataset-independent)
build_taxon_group(): Build the taxon_group grouping table (many taxa per group)
build_taxon_hierarchy(): Build Taxon Hierarchy from Local spp.duckdb via Recursive CTEs
build_taxon_reference(): Build the unified taxon reference table
build_taxon_table(): Build Taxonomic Hierarchy Table from WoRMS
check_registry_na_strings(): Reject sentinel strings that should have been empty cells
clean_taxon_name(): Normalize a source taxon name for an authority lookup
compat_event_sql(): Rebuild a per-dataset event table as a VIEW over the core sample
compat_measurement_sql(): Rebuild a per-dataset measurement table as a VIEW over obs
core_output_tables(): Core tables an ingest writes to parquet
core_relationships(): PK/FK spec for the consolidated core tables
core_shard_paths(): Find the per-dataset parquet shards for a core table
ctd_map_columns(): Map an uploaded file's columns onto measurement types
ctd_upload_to_core(): Project an uploaded file into the core obs / sample shape
declare_measurement_bounds(): Declare valid_min / valid_max on measurement types that already exist
derive_cruise_key_on_casts(): Derive Cruise Key on Bottle Casts via Ship Matching
discover_sample_levels(): Discover a dataset's sampling hierarchy from the core sample table
ensure_interim_ships(): Ensure Interim Ship Entries for Unmatched Ships
ensure_measurement_taxon(): Stage the _measurement_taxon crosswalk in a connection
ensure_taxon_lineage(): Materialize the WoRMS/ITIS lineage build_taxon_reference() reads
ensure_taxon_xref(): Materialize the authority cross-reference .taxon_norm_sources() reads
fetch_ship_ices(): Fetch Ship Codes from ICES Reference Codes API
fetch_taxon_lineage(): Fetch (and cache) the WoRMS/ITIS lineage for a set of taxon ids
fetch_taxon_xref(): Fetch (and cache) the WoRMS <-> ITIS cross-reference for a set of taxa
format_bbox(): Format a Bounding Box as a Human-Readable Extent
load_gcs_parquet_to_duckdb(): Load a GCS Parquet File into DuckDB
match_by_site_datetime(): Match Records to a Reference Table by Key + Datetime Window
match_cruise_by_track(): Match Records to a Cruise by Space-Time Proximity to an Occupied-Station Track
match_nearest_by_depth(): Match Records to the Nearest Reference Row Along a Continuous Axis
match_ships(): Match Ship Codes Across Datasets Using Multi-Source References
measurement_var_meta(): CF variable metadata from the measurement_type registry
merge_taxon_shards(): Merge the per-dataset taxon shards into one authoritative reference
nc_global_atts(): Build the CF/ACDD global attributes for a published dataset
nc_level_put(): Write the data and attributes for one level defined by nc_level_vars()
nc_level_vars(): Build the variable definitions for one level of a nested dataset
nc_profile_atts(): Write the CF Discrete-Sampling-Geometry attributes
nc_profile_def(): Define a CF Discrete-Sampling-Geometry file (profile or trajectory)
nc_profile_write(): Write one chunk of a CF profile file
ns_key(): Namespaced sample_key expression: dataset_key:sample_type:id
obs_wide_sql(): SQL that widens long obs rows into one column per measurement type
observed_coverage(): Measure Observed Temporal and Spatial Coverage per Dataset
plan_dataset_netcdf(): Plan the netCDF shape for a dataset
prune_taxon_shard(): Prune the taxa references to one dataset's shard
qc_cast_base(): Strip the direction suffix from a CTD cast sample_key
qc_cast_direction(): Direction of a CTD cast sample_key
qc_cast_profile(): Fetch one physical cast's profile, both directions
qc_parse_params(): Parse a rule's params cell into a named list
qc_present_types(): Which measurement types actually exist for a dataset
qc_read_rules(): Read the rule registry, attaching SQL text and parsed params
qc_render_sql(): Substitute {{param}} placeholders into a rule's SQL
qc_run_all(): Run every rule in a registry, one at a time
qc_run_rule(): Execute one rule, returning its findings
qc_stage_reference(): Stage the QC reference tables a rule registry expects
qc_summarize(): Collapse rule results into one row per rule
qc_upload_con(): A connection an uploaded cast can be QC'd on
question_statuses() question_priorities(): The controlled vocabulary of the question registry
questions_datatable(): Render a question registry as the standard notebook table
read_ctd_upload(): Read any supported CTD upload
read_measurement_type(): Read metadata/measurement_type.csv, refusing a corrupted registry
read_questions(): Read a dataset's questions.csv, validated and ranked
read_sbe_asc(): Read a Sea-Bird .asc ASCII export
read_sbe_btl(): Read a Sea-Bird .btl bottle summary
read_sbe_cnv(): Read a Sea-Bird .cnv converted data file
read_sbe_header(): Parse a Sea-Bird * / ** header block
register_measurement_types(): Append new measurement types to the shared registry, safely
report_ship_matches(): Report Ship Matching Status for a Dataset
sample_arm_self(): Build a sample arm for a single self-contained event table
sbe_split_header(): Recover the column names from a fixed-width Sea-Bird ASCII header
standardize_species(): Standardize Species Identifiers Using WoRMS/ITIS/GBIF APIs
standardize_species_local(): Standardize Species Using Local spp.duckdb Lookups
summarise_netcdf_plan(): Summarise a netCDF plan as one row
supplemental_core_tables(): Supplemental full-resolution tables declared by the ingests
taxa_rank_reference(): The canonical taxonomic rank ordering (taxa_rank)
taxon_key_of(): Encode an authority-prefixed taxon_key
upsert_measurement_types(): Replace a measurement type's definition while keeping its curated columns