8  Providing data to CalCOFI

Who this is for. Anyone who collects, curates or holds data from the CalCOFI region and would like it to be usable beside the rest — a program lab, a collaborating institution, a student project, an instrument group — whether the dataset already exists or is being designed. What it is for. To say, in one place, what a dataset needs to carry for the CalCOFI data team to integrate it: the few fields that are mandatory, the ones that make it more useful, and the practices that make it durable. It is guidance for a provider, not the database’s own rulebook — that is Naming conventions — and it asks for less than the database keeps.

8.1 The shape of a good dataset

A dataset CalCOFI can integrate answers three questions for every row: what was measured or counted, when and where. Everything else is context that makes the answer more useful. One row is one measurement or one count of one thing at one place and time; a spreadsheet with one column per variable is fine (the ingest turns it long), as long as each row is one event.

Names are not the hard part — the data team maps your columns onto the standard names at ingest and keeps a crosswalk of what you called them (Naming conventions). What cannot be recovered afterwards is a missing time zone, an undocumented unit, a count with no effort, or a species with no name that resolves. Those are the mandatory fields of Table 8.1.

8.2 Mandatory: what, when, where

Table 8.1: What every dataset must carry, in the form the ingest can read it.

field	what we need	example
the quantity	what was measured or counted, in words, with its unit — one column per quantity, or one variable / value / unit triple	temperature in degC; larvae as a count per tow
the time	date and time of the event in UTC (ISO 8601, 2019-02-07T23:48:00Z), or a local clock time and the offset that was in effect	2019-02-07T23:48:00Z
the place	latitude and longitude in decimal degrees, WGS 84, signed (south and west negative)	33.48066, -117.769
the depth, where it applies	metres below the surface, positive down; a range (depth_min, depth_max) for a tow or a bin	115.5; 0 – 210
the organism, for a biological count	a scientific name, and where you have it an identifier from WoRMS (AphiaID), ITIS or GBIF; a life stage if the count is stage-specific	Sardinops sagax, WoRMS 217452, larva
the effort, for a count	what the count is per: the volume filtered, the area or time surveyed, the fraction of the sample sorted	467.3 m³; 0.48 sorted
who made it	the organization that holds and can license the data, a contact, and how you want it cited	Scripps Institution of Oceanography

8.3 Optional: what makes it CalCOFI

These let the data team join your rows to everything else, rather than only place them on a map.

• The cruise — the cruise designation as your records carry it (1902RL, CalCOFI 2019-02, a NOAA cruise id). The database keys every event to a cruise by its date span, so a designation is a check, not a requirement.
• The station — CalCOFI line and station where the event was on the grid (93.3, 26.4), or your own station name where it was not.
• The gear and the tow — the net or instrument (a CalCOFI tow-type code such as CB for the standard bongo, or a description), the tow number within the cruise, the net side, and the net’s mesh size in millimetres where it applies — it sets the smallest organism the gear retains.
• A haul factor, if you already compute one — the factor you use to standardize a catch to a reference tow, so it’s comparable across gear. The database can derive this from volume filtered and tow geometry when you don’t supply it, but if you already have your own, send it rather than have it recomputed.
• Quality flags — your own vocabulary, as long as it is written down; the database keeps every flag exactly as you assign it and never re-interprets it.
• Precision and method — the significant digits of a measurement, the analytical method, the processing stage (preliminary, final).

8.4 Best practice: what makes it durable

• A stable identifier for every row you might refer to again — a cast number, a sample id, a net id — that stays the same across the versions of your data. If you run a database, a UUID for each cruise, station, tow or net is the best form of this, and the integrated database keeps it as a column beside its own keys; it is never required. A dataset with no identifiers at all is still ingestible: the database mints its own, from your rows.
• A licence — the terms under which the data may be reused (CC-BY-4.0 is the CalCOFI default; CC0, a public-domain statement or your institution’s terms are all fine as long as they are stated). An unstated licence is the single most common reason a dataset cannot be published onward.
• A citation and a DOI — how you want to be credited; if your data already has a DOI (at EDI, NCEI, Zenodo, a library repository), that is the citation the integrated database carries.
• A data dictionary — one line per column: what it is, its unit, how a missing value is written (NA, blank, -99; never a plausible number).
• Time zones and coordinate systems stated once, at the top, even when they are the standard ones.
• Versions, not edits — when the data change, a new file with a date, not a corrected copy under the same name, with the same columns in the same types: a column that turns from a number into text between versions stalls every automated update downstream.

8.5 An example

A net-tow dataset that carries everything above, as one row per taxon per tow:

Table 8.2: One row of a net-tow dataset that carries every mandatory field, and the optional ones that make it CalCOFI.

cruise	line	station	tow_number	tow_type	mesh_mm	datetime_utc	latitude	longitude	depth_max_m	volume_sampled_m3	std_haul_factor	prop_sorted	scientific_name	worms_id	life_stage	count	qual
1902RL	93.3	26.4	12	CB	0.505	2019-02-07T23:48:00Z	33.48066	-117.769	210	467.3	4.49	0.48	Sardinops sagax	217452	larva	42

From Table 8.2 the data team’s ingest would map cruise to cruise_key (2019-02-33UD after resolving the ship), line and station to site_key (093.3 026.4), the tow to a sample row with tow_type CB, each count to an obs_bio row with taxon_key worms:217452 and life_stage larva, volume_sampled_m3, std_haul_factor and prop_sorted to sample_measurement, and keep your tow_number and mesh_mm beside them (the gear registry already records the standard bongo’s 0.505 mm; a mesh that differs is what matters). Nothing in your file changes.

8.6 What happens next

1. Explore — the data team profiles the file and seeds a list of questions from what it cannot answer (a unit, a code, an ambiguous date).
2. You get one Sheet — one tab of questions per dataset, most already carrying a proposed answer for you to confirm, and a metadata tab for the abstract, methods, creators, licence and citation (Metadata & the ingest loop).
3. Ingest and validate — one reproducible notebook turns the file into the shared schema; every finding is fixed or becomes a question.
4. Release — the dataset joins the next frozen release with a DOI, gets its page at calcofi.io/datasets/{key}/, its citation on every figure drawn from it, and — where you want it — its way onward to ERDDAP, OBIS, EDI or the archives (Portals).

Write to data@calcofi.io to start.

8.7 Names you may see elsewhere

A handful of names are common in older CalCOFI files and documents but differ from the database’s registries. The registry is the record, so the ingest maps the older form onto it; Table 8.3 lists them so nobody reintroduces the older form.

Table 8.3: Older names and what the registry has.

often written as	the registry has	why
a ship as a two-letter code or an NODC code, in one column	a ship_code as your file has it, resolved to the ship row: ship_key, the two-letter CalCOFI code (RL), and ship_nodc, the four-character NODC code (33UD)	the NODC code is the suffix of every cruise_key (YYYY-MM-NODC, e.g. 2019-02-33UD, the form the older 1902RL becomes)
depth	depth_m for a point, depth_min_m / depth_max_m for a tow or a bin (on sample and obs); seafloor_depth_m for the seafloor under a sample	units are suffixes on every measured column; valid_depth_min_m / valid_depth_max_m are bounds on a measurement type, not a field on a sample
datetime_utc, datetime_local, utc_offset	datetime_start_utc / datetime_end_utc in the dictionary; datetime on the released sample	an event has a span; local time is recoverable from UTC and the position, never the reverse
larval stages as codes (YOLK, PREF, FLEX, TRNS, POST)	life_stage as words from the registry (egg, larva, …), with Darwin Core and NERC ids where exact	one vocabulary for every dataset; the ichthyoplankton stage codes map onto it at ingest
cruise_id in the long form 1949-03-01-C-31CR	an alias of cruise_key in the dictionary, not a field	legacy names are kept in the crosswalk, not in the schema
M2 as the MOCNESS 10 m frame	the registry names M2 MOCNESS (3 m²), which is not a standard MOCNESS size; being checked against the source with SWFSC (docs#14)	the gear registry is the record; if the source says otherwise the registry changes, not the guide
no OBLIQUE gear	OBLIQUE — an oblique tow whose gear was not recorded	the registry lists every code that appears in the data