---
title: "CalCOFI CTD QA/QC Protocol"
subtitle: "What every check does, where its threshold came from, and what it cannot see"
date: today
format:
html:
toc: true
toc-depth: 3
number-sections: false
code-fold: true
code-tools: true
# docx so the CTD team can suggest edits in track-changes. Code is echoed off
# rather than folded — Word has no fold, and a reviewer reading a protocol does
# not want R between the paragraphs. The DT tables degrade to nothing in docx,
# so anything a reviewer must SEE goes through the `tbl()` switch in `setup`.
docx:
toc: true
toc-depth: 3
number-sections: false
echo: false
calcofi:
target_name: ctd_cast_qa_qc_protocol
workflow_type: reference
# every pipeline notebook must declare its output: build_targets_list() uses it
# as the target's file, and a missing key fails the whole run at resolution time
output: _output/ctd-cast_qa-qc-protocol.html
workflow_url: https://calcofi.io/workflows/ctd-cast_qa-qc-protocol.html
description: >
Every CTD quality check that runs, where its threshold came from, and what
it cannot see — generated from metadata/qc_rules/ so it cannot drift from
the rules. Also published as .docx for track-changes review.
editor_options:
chunk_output_type: console
---
## What this document is
The reference for CalCOFI CTD quality control: every check that runs, stated
plainly enough that a data manager can agree or disagree with it without reading
SQL, and specific enough that a reviewer looking at a flagged cast knows what the
flag means.
**It is generated from the rules themselves.** The per-rule sections below are
assembled at render time from `metadata/qc_rules/` — `rules.csv` for the index,
`sql/*.sql` for the check and for the reasoning its author recorded in the file's
header comment. Nothing here is a separate description that could drift from the
code: change a threshold and this page changes with it, and a rule that runs
without a rationale [fails this document's build](#coverage).
```{r}
#| label: setup
devtools::load_all(here::here("../calcofi4db"))
librarian::shelf(
calcofi4db, dplyr, DT, glue, here, knitr, purrr, readr, stringr,
quiet = T)
source(here("libs/build_qc_protocol.R"))
dir_rules <- here("metadata/qc_rules")
rules_all <- qc_read_rules(dir_rules, active_only = FALSE)
rules_act <- filter(rules_all, active)
say <- function(...) cat(..., "\n", sep = "")
say(glue("rules: {nrow(rules_all)} registered, {nrow(rules_act)} active, ",
"{nrow(rules_all) - nrow(rules_act)} parked"))
say(glue("SQL files: {length(unique(na.omit(rules_all$sql_file)))}"))
# This document renders to HTML and to **docx** — the latter so the CTD team can
# suggest edits in track-changes. A DT::datatable() is an htmlwidget and becomes
# nothing at all in Word, so any table a reviewer must actually see goes through
# this switch rather than straight to datatable(). Sorting/paging is a luxury;
# a blank space where the quality-code vocabulary should be is a defect.
tbl <- function(d, caption, page_length = 12, ...) {
if (knitr::is_html_output())
datatable(d, caption = caption, rownames = FALSE,
options = list(pageLength = page_length, scrollX = TRUE, dom = "tip"),
...)
else kable(d, caption = caption)
}
```
### The pipeline these rules watch
Quality control here is not a stage bolted on at the end — it reads the same
tables consumers read, at the point they are published. So the shape of the
pipeline determines what a rule *can* see, and two steps in particular decide it:
**thinning**, which is why a rule targeting `obs` is blind to most scans, and the
**cast-direction choice**, which is why half of every physical cast carries no
published observation at all.
::: {.content-visible when-format="html"}
<!-- HTML ONLY, deliberately. Quarto renders mermaid for non-HTML formats by
shelling out to headless Chrome, which on this project hangs indefinitely
and unpredictably (see CLAUDE.md — a 60 KB diagram wedged a render for 3h15m
and ignored SIGTERM). The docx branch below states the same pipeline in
prose. Do not "simplify" by deleting this guard. -->
```{mermaid}
%%| label: fig-pipeline
%%| fig-cap: "From the source zips to the tables the rules read. Blue = source; grey = intermediate, never published; green = the published core; amber = the rule registry and the two places it runs. Dotted edges are what the rules read, not what the pipeline writes."
%% No `subgraph` on purpose: mermaid's default cluster fill is dark and renders
%% its own title illegibly against itself. Grouping is carried by classDef colour
%% and stated in the caption instead.
flowchart TD
Z["<b>CTDBTL CSVs</b> — calcofi.org, one zip per cruise<br/>82 cols · <b>1 m bin averages, not raw scans</b><br/>one file per cast per direction"]
D["<i>CTD-CSV-Format.pdf</i> · <i>PreliminaryDataReadme.txt</i><br/>ships in every zip"]
M["<i>span</i> · <i>DBcoeff</i> · <i>xmlcoeff</i><br/>per-cruise metadata, not yet read"]
Z --> R["<b>ctd_raw</b><br/>read · standardize · rename<br/>−99 sentinel deleted, not ingested<br/>distance filter · de-duplicate"]
R --> C["<b>ctd_cast</b> — cast metadata<br/>+ <i>data_stage</i> = final / preliminary"]
R --> ME["<b>ctd_measurement</b> — long<br/>type · value · qual · ~216 M scans"]
ME --> T["<b>ctd_thin</b> — adaptive thinning<br/>1. one direction per physical cast<br/>2. canonical types only<br/>3. ~10 m grid + RDP inflections<br/>4. + every bottle-trip depth"]
C --> S["<b>sample</b><br/>one row per <b>physical</b> cast"]
T --> O["<b>obs</b> — the headline<br/>~7 M rows · realm = env"]
ME --> F["<b>obs_ctd_full</b> — supplemental<br/>~212 M scans, opt-in"]
S --> REL["frozen release<br/>gs://calcofi-db/ducklake/releases/"]
O --> REL
F --> REL
S -.-> RULES["<b>metadata/qc_rules/</b><br/>rules.csv + sql/*.sql<br/><i>one engine, one registry</i>"]
O -.-> RULES
F -.->|"one cruise at a time"| RULES
D -.->|"thresholds · caveats"| RULES
M -.->|"question Q15"| RULES
RULES --> N["ingest notebook — <i>Findings</i>"]
RULES --> A["<b>ctd-qaqc</b> app<br/>+ ad-hoc .btl upload"]
classDef src fill:#eef3fb,stroke:#4a6fa5,color:#12243a
classDef core fill:#e7f6ec,stroke:#3f8f5a,color:#0f2a1a
classDef mid fill:#f6f7f9,stroke:#8b95a3,color:#1c222b
classDef qc fill:#fdf3e7,stroke:#b06000,color:#3a2400
class Z,D,M src
class R,C,ME,T mid
class S,O,F,REL core
class RULES,N,A qc
```
:::
::: {.content-visible unless-format="html"}
The pipeline, in the order a value travels it:
1. **Source.** One zip per cruise from calcofi.org. The data are the
`*CTDBTL*.csv` files — 82 columns, **1 m bin averages** rather than raw
scans, one file per cast per direction. Each zip also ships its own
documentation (`CTD-CSV-Format.pdf`, `PreliminaryDataReadme.txt`) and three
per-cruise metadata families (`span`, `DBcoeff`, `xmlcoeff`) that the
pipeline does not yet read.
2. **`ctd_raw`.** Read, standardized and renamed; the `-99` sentinel is deleted
rather than ingested as a value; points far from their station are filtered
and duplicates removed.
3. **Split.** `ctd_cast` carries the cast metadata (including `data_stage`,
`final` or `preliminary`); `ctd_measurement` carries every scan in long form
— roughly 216 million of them.
4. **`ctd_thin`.** The adaptive thinning that produces the headline series:
one direction per physical cast, canonical measurement types only, a ~10 m
depth grid with thermocline/halocline inflections preserved by
Ramer–Douglas–Peucker simplification, **plus every bottle-trip depth**.
5. **The published core.** `sample` (one row per *physical* cast), `obs` (the
thinned series, ~7 M rows, `realm = env`) and the supplemental
`obs_ctd_full` (~212 M scans) are frozen into a versioned release.
6. **The rules.** `metadata/qc_rules/` runs against `sample`, `obs` and
`obs_ctd_full` — from the ingest notebook's *Findings* section, from the
`ctd-qaqc` app, and against an ad-hoc `.btl` upload. One engine, one
registry.
:::
Three consequences of that shape run through everything below.
**A rule on `obs` cannot see what thinning removed.** `obs` is `ctd_thin`, not
`ctd_measurement`. Spikes, loop edits and down-vs-up disagreement are properties
of the dense scan set, so those rules target `obs_ctd_full` and are run one
cruise at a time — a sample, never a sweep.
**Half of every physical cast has no published `obs`.** Step ① keeps one
direction. `sample` is one row per *physical* cast, so a completeness rule that
expects an observation per cast direction would be wrong by construction.
**Nothing here is a filter.** Every rule returns a report. No rule drops,
rewrites or interpolates a value, and two blocker questions remain open with the
providers — acting on them before they are answered would be guessing with
released data.
### How to read a rule
Each rule declares four things that decide how a result should be read.
| field | what it changes |
|---|---|
| **severity** | `error` means the pipeline is broken and the finding is a defect in *our* processing. `warning` means the data looks wrong and a human must decide. No rule drops or rewrites a value. |
| **target** | `sample` (one row per cast), `obs` (the thinned, published profile) or `obs_ctd_full` (the full-resolution scans, supplemental). A rule on `obs` cannot see what thinning removed. |
| **scope** | `all` runs over the whole dataset. `cruise` rules read the 212M-row `obs_ctd_full` and are run one cruise at a time — they are a *sample*, never a sweep, and the app is where the remaining cruises get covered. |
| **requires** | measurement types that must be present. If one is absent the rule reports **`skip`**, never `pass`. |
That last row is the single most important convention here.
Every rule returns at least `subject_key` (what is flagged — the unit of review)
and `detail` (one sentence naming the problem). A rule whose finding concerns a
particular **scan** rather than a whole cast also returns `depth_min_m` and
`measurement_type`. Those two are a contract rather than a habit: they are what
lets a reviewer click a finding in `ctd-qaqc` and land on the right profile at the
right depth, without the app needing to know anything about the rule that produced
it. The two cast-level rules below (`ctd_cast_deeper_than_seafloor`,
`ctd_cast_position_vs_station`) correctly return neither — they are about the cast,
not about a reading.
::: {.callout-important title="`skip` is not `pass`, and this is not hypothetical"}
A rule whose input is missing returns zero rows — which is indistinguishable from
"the data is clean" unless something says otherwise. The three bottle-vs-sensor
calibration rules did exactly that against release `v2026.07.30`, which predates
the change making the bottle reference types canonical: they reported no findings
while comparing nothing at all.
The engine therefore treats an unmet precondition as `skip` with a stated reason,
and the same principle governs reference data — `calcofi4db::qc_stage_reference()`
leaves a missing input as a **missing table** so its rules error visibly, rather
than creating an empty one whose rules would quietly pass.
:::
Results carry one of five statuses:
| status | meaning |
|---|---|
| `pass` | ran, found nothing |
| `flag` | ran, found something a human should look at (`severity = warning`) |
| `FAIL` | ran, found something that should not be possible (`severity = error`) |
| `skip` | **did not run** — a precondition was unmet |
| `ERROR` | could not run (missing reference table, bad SQL) |
## Where these rules came from
Two sources, deliberately combined: the checks CalCOFI has been running for
decades, and the checks that only make sense on CTD data.
### Mining the CalCOFI hydrographic master
The program's working database is an MS Access file carrying the bottle record
from 1949, and — more valuably — **155 saved queries**, a self-documenting field
dictionary, an instrument/accuracy registry, a 280-row dated change log, and a
fitted seasonal climatology. Three decades of institutional QA logic lived there
and nowhere else.
It was extracted reproducibly on macOS/Linux (`libs/extract_accdb.R`,
`scripts/extract_accdb.sh`) with **two engines on purpose**: `mdbtools` for bulk
table export, and Jackcess for the saved-query SQL, which is authoritative.
`mdb-queries` reconstructs the stored parse tree only partially — it silently drops
JOIN clauses, GROUP BY, HAVING and aliases. The output looks plausible and is
wrong: a LEFT JOIN find-unmatched query degrades into a cross join, which inverts
the meaning of the QA check it encodes. Both engines are run and diffed so a
regression in either surfaces instead of silently corrupting a ported rule.
Of the 155 queries, only one family is standing QA: the `TQ -` / `TR -` / `TV -`
tests and the `Anomalies` family. The rest are corrections, materializations and
derived products.
```{r}
#| label: triage
read_csv(here("metadata/calcofi/hydro-master/query_triage.csv"),
show_col_types = FALSE) |>
count(category, sort = TRUE) |>
mutate(`what it is` = recode(category,
validate = "QA checks — the family ported here",
`derived-product` = "computed products (MLD, nutricline, anomalies)",
`correction-history` = "one-off UPDATE/DELETE repairs, dated and initialled",
materialization = "make-table steps feeding other queries",
`export-or-info` = "extracts for outside consumers",
unclassified = "not confidently categorized")) |>
kable(caption = "The 155 Access saved queries, triaged")
```
**A faithful port is not always the right port.** Two of the Access checks could
not be carried over as written, and both are documented on the rules that replaced
them: `TQ - BottomDepth_Vs_AvgBottomDepth` compares the ship's echosounder to the
station average, but CTD casts carry no reported bottom depth at all
(`bottom_depth` exists in `sample_measurement` for 33,363 **bottle** casts and for
**0** of 14,336 CTD casts) — so the ported form tests position plausibility against
bathymetry instead, which is a different question. And
`TQ - StationNameChecker` compares `Rpt_Line > "76.6"` as a **string**, so line 100
sorts below line 76.6 and has been silently excluded from that check for twenty
years; whether to port the bug or the intent is question `hydro_master_09`.
### What the source documentation actually says
Every CTD source zip ships its own documentation — `CTD-CSV-Format.pdf`,
`CTD Data Files.pdf`, `FilenamingAndAscHeaderNotes.txt`, and a
`PreliminaryDataReadme.txt` on preliminary cruises. It answers several questions
this project had open, and it constrains how the released values may be read.
Combed 2026-08-01; what follows is from those files, not inferred.
#### The processing chain, and the three versions of every sensor value
Raw frames are processed with **SBE Data Processing** (Seasoft) tuned for the
CalCOFI 911/911+, then by SIO-CalCOFI's own **`BtlVsCTD`** against the cast's
bottle samples. The published CSV therefore carries the *same property three
times*, and the column suffix is the only thing that distinguishes them:
| suffix | what it is | fitted from |
|---|---|---|
| *(none)* | SBE-processed, **no bottle correction** | — |
| `_CruiseCorr` | corrected with one regression for the whole cruise | 4-second-averaged sensor data vs **all** the cruise's bottles (n ≈ 1400, fliers omitted) |
| `_StaCorr` | corrected with a regression fitted **per cast** | 1 m bin-averaged sensor data vs that cast's bottles (n ≈ 20, fliers omitted) |
The providers state that **station-corrected data are the best**, particularly for
estimated nitrate — which is the basis on which `*_sta_corr` types are the
canonical ones in this pipeline. That choice was previously undocumented here; it
follows the source's own recommendation.
Salinity is corrected differently again: offsets are derived from **bottle
salinities below 350 m** and applied to all depths, separately for the primary and
secondary sensors.
#### Four caveats that change how a value may be used
::: {.callout-warning title="Read these before drawing a conclusion from a corrected value"}
1. **Upcast bottles correct the downcast too.** The source says so explicitly, and
volunteers the objection: *"This practice may be debated due to internal waves
and hysteresis. If you disagree with bottle-correcting downcast sensor data,
use the SBE-processed data."* Any down-vs-up comparison — including the
`ctd_updown_disagreement` rule below — is therefore comparing two series that
share a correction derived from one of them.
2. **`_StaCorr` is missing by design, not by error.** It requires a 500 m cast with
roughly ten or more bottles; shallower stations need several. On winter and
spring cruises, lines north of 76.7 may be occupied with **0–12 bottles** and
get cruise-corrected values only. A completeness check that treats a null
`*_sta_corr` as a defect will be wrong for a whole class of casts.
3. **Estimated chlorophyll is only computed for 0–200 m.** `EstChl_*` comes from
applying the fluorometer regression to the 1 m bin-averaged voltage over that
range alone.
4. **Preliminary data are explicitly non-publication.** The readme is specific
about which variables move: *"Temperature and salinities may change very little
but oxygen, nitrate and chlorophyll data may change significantly after
post-cruise calibrations."* Final data are annotated `FinalQC`.
:::
#### Grain
The published values are **1 m bin averages**, not raw scans — so `obs_ctd_full`
is full *published* resolution, and the spike and loop-edit rules operate on
bin-averaged data. That is why the loop-edit threshold is a 1 m noise floor.
(Cruise-correction regressions use a 4-second average instead; the two grains are
not interchangeable.)
#### Documented but unused: three metadata families
Every cruise's `metadata/` folder carries files the pipeline does not read. They
are listed here because each answers a question this protocol currently has open:
| file | what is in it | what it would settle |
|---|---|---|
| `YYMM_span_###-###.csv` | per **cast**, the min and max of every variable, recorded by the processor | a per-cast range check grounded in the instrument rather than in our invented `valid_min`/`valid_max` (question `calcofi_ctd-cast_02`) |
| `YYMM_DBcoeff_###-###.csv` | per cast: `Salt1Offset`, `Salt1_SD`, the oxygen regression polynomials and their residual SDs | the **uncertainty of each correction**, which is exactly what the core's declared-but-empty `measurement_prec` is for |
| `YYMM_xmlcoeff_###-###.csv` | per cast, per channel: sensor serial number and calibration date | per-cast instrument provenance, far finer than the decade-scale eras in `measurement_method.csv` |
All three were surveyed across the 85 cruises in hand on 2026-08-02, and they are
**not equally ready to depend on** — which is the substance of the recommendation
now recorded on question `calcofi_ctd-cast_15`:
| file | files | rows | distinct headers | verdict |
|---|---|---|---|---|
| `DBcoeff` | 135 | 15,703 | **1** (43 columns, no variation at all) | **ingest first** — stable, and it carries the residual SDs `measurement_prec` wants |
| `span` | 136 | 16,254 | 70 (61–79 columns; the set follows each cruise's sensor suite) | **ingest second**, as a long `(cast, variable, min, max)` melt |
| `xmlcoeff` | 136 | 16,394 | 40 (1–198 columns; three files have a single column) | **not yet** — no rule needs it and the shape costs the most to make reliable |
They would land through a `libs/` builder called from the CTD ingest, **not** a
new `ingest_*.qmd`: these are per-cruise sidecars of `calcofi_ctd-cast`, not a
separate dataset, and a new notebook would mint a `dataset_key` and a release
shard for something that is not one. The per-cast scalars belong in
`sample_measurement`, the event-level grain that already holds cast conditions.
#### Smaller points
- **Time and position mean different things by direction.** For a downcast they are
the cast start; for an upcast they are the moment the **first bottle tripped**.
- **Voltage channels are relabelled.** SeaSave writes `V0`…`V7`; CalCOFI renames
them to `TRANSV`, `FLUORV`, `ISUSV`, `PARV`, `OX1V`, `PHV`, `ALTV`, `RINKOV` so
the sensor is identifiable. Unrelabelled originals are kept under
`original_asc-hdr-btl/`, and each cast's `.hdr` records the configuration.
- **Filename grammar**: `YY-YYMMSS_LLLLSSSS_###D`, where line and station are ×10
(`0767` = 76.7, `1000` = 100.0) and `D`/`U` is the direction.
- **The spec's `Comment` column does not exist in the files, and the files have a
column the spec does not document.** Settled by reading every `*CTDBTL*.csv`
shipped in the zips — **245 files across 85 cruises, 1998-02 to 2025-04** —
against the single `CTD-CSV-Format.pdf` version that all 84 copies of the spec
contain. The two lists agree **positionally through field 74 (`OxB`)**. The
files then carry **`OxBuM`** (bottle oxygen, µmol/kg) at 75, which the spec
never mentions; everything after shifts by one, so the file's 82nd column is
`SIL` (the spec's 81st) and the spec's `Comment` has no counterpart at all.
So this is **not** an off-by-one in the spec: it is one undocumented column
added and one documented column absent, which happen to cancel to the same
total of 82. Nothing is being dropped by this pipeline — `OxBuM` is ingested
as `oxygen_btl_umol_kg` — but a reader who trusts the spec's field numbering
will mis-read every column from 75 on. Raised as `calcofi_ctd-cast_18` so the
spec can be corrected at source.
One file departs from all of this: `20-1804SH_CTDFinalQC/Metadata/`
`20-1804SH_CTDBTL_001-103UResid.csv` has **84** columns, adding `BO2-SCO21`
and `BO2-SCO22` — per-sensor bottle-minus-CTD oxygen residuals. It is a
diagnostic left in a `Metadata/` folder, not a data file, and the ingest's
`data_stage` selector requires `Final|Prelim.*db[_-]csv` in the path, so it is
filtered out rather than pivoted. It is worth noticing for a different reason:
it is the residual that `measurement_prec` wants (question
`calcofi_ctd-cast_15`), already computed by the provider for one cruise.
### The climatology, and why two properties are excluded
`HarmCoeffBottle` holds a fitted annual harmonic per station × standard depth ×
property — mean, amplitude, frequency, phase, and a residual standard deviation.
That turns "is this value plausible" from a fixed global range into a *local,
seasonal* question, which nothing in the pipeline could ask before.
The providers have not confirmed the fitting procedure (question
`hydro_master_12`), so the reconstruction form was **determined empirically**:
candidate forms were scored against the 200,640 bottle observations the
coefficients were derived from.
| form | RMSE | vs mean-only |
|---|---|---|
| `Mean` (null) | 1.269 | — |
| **`Mean + Ampl·sin(Freq·(doy − Phase))`** | **0.949** | **+25.2%** |
| `Mean + Ampl·cos(Freq·(doy − Phase))` | 2.113 | −66.4% |
| `Mean + Ampl·cos(Freq·doy − Phase_deg)` | 1.800 | −41.8% |
| `Mean + Ampl·sin(2π(doy − Phase)/365)` | 1.474 | −16.1% |
Every cosine variant is *worse than using the mean alone*. Two independent checks
agree with the winner: physically, the seasonal signal is strongest at the surface
(42% RMSE reduction at 0 m) and decays monotonically to ~5% by 150–500 m, which is
what seasonal heating does and what a wrong form would not show; statistically,
residuals divided by `StDev` are near-standard-normal (mean −0.06, sd 1.14),
confirming `StDev` is the residual scale a z-score needs.
Each property was then scored the same way, and **only those that passed are used**:
| property | RMSE gain | sd(z) | verdict |
|---|---|---|---|
| Temp | 25.2% | 1.14 | use |
| SiO3 | 11.6% | 1.12 | use |
| NO3 | 10.3% | 1.40 | use |
| PO4 | 9.7% | 1.26 | use |
| O2 | 9.3% | 1.29 | use |
| Sig | 2.4% | 4.80 | **exclude** |
| Sal | 0.5% | 6.73 | **exclude** |
| NO2 | 5.0% | 5.4e12 | **exclude** |
| Chla | 4.3% | 1.2e11 | **exclude** |
Salinity and sigma-theta are excluded on evidence, not taste. A seasonal harmonic
explains almost none of their variance here — Southern California salinity is
advection-driven, not seasonally heated — and the tabulated `StDev` is 5–7× smaller
than the actual residual, so it is measuring something other than residual spread.
A z-score built on it would be wildly over-dispersed and would flag thousands of
ordinary values. NO2 and Chla fail differently: a handful of station × depth cells
carry a near-zero `StDev`, which makes z explode (one nitrite cell produced
z = 3.5e12), so they are left out rather than patched.
### Reference data the rules join against
None of this is part of the release. It is the QC engine's *input*, staged onto a
connection by `calcofi4db::qc_stage_reference()`.
```{r}
#| label: reference_inputs
tibble::tribble(
~table, ~source, ~role,
"measurement_type", "metadata/measurement_type.csv", "valid_min/valid_max per type — the registry, not a release, because it moves ahead of releases",
"measurement_qual", "metadata/measurement_qual.csv", "the quality-code vocabulary, recovered from the Access field documentation",
"climatology_harmonic", "hydro-master/reference/climatology_harmonic.csv", "fitted seasonal expectation per station x depth x property",
"station", "hydro-master/reference/station.csv", "the 75 current standard stations and their average bottom depth",
"standard_depth", "hydro-master/reference/standard_depth.csv", "the standard depth grid",
"sample_seafloor", "GEBCO 2025 crop (apps/ctd-viz)", "seafloor depth at each cast position — the only bottom-depth reference CTD casts have") |>
kable(caption = "QC reference inputs, none of which ship in the release")
```
### The quality-code vocabulary
`measurement_qual` was a verbatim pass-through for years: carried through every
table, never interpreted, with no vocabulary anywhere in the repo.
There are now **two**, and conflating them was a real error. The first was
recovered from the hydrographic master — the **bottle** database. The CTD files
use a different set, documented all along in the `CTD-CSV-Format.pdf` that ships
inside every source zip:
| code | CTD meaning |
|---|---|
| `0` or blank | **good** — "if no data code is displayed then sensors were operating normally" |
| `1` | **use the primary sensor** of the dual pair |
| `2` | **use the secondary sensor** — the primary is failing or the pump is bio-fouled |
| `8` | data questionable |
| `9` | **bad or missing** sensor data |
Two consequences. `1` and `2` are **not quality grades at all** — they are
sensor-selection instructions, so ranking them somewhere between "good" and
"suspect" is a category error. And `9` means bad *or* missing, which resolves what
looked like a contradiction: a row flagged `9` that still carries a number is bad
data, correctly flagged.
`metadata/measurement_qual.csv` therefore carries a `code_set` column
(`ctd` / `bottle`) and the rule below filters to the CTD set — joining the whole
file would let a bottle-only code mask an unrecognized CTD one. The **bottle**
vocabulary remains incomplete, and those gaps are still blockers.
```{r}
#| label: qual_vocab
d_qual <- read_csv(here("metadata/measurement_qual.csv"),
col_types = cols(qual_code = col_character(), .default = col_guess())) |>
select(set = code_set, code = qual_code, label, description,
documented = is_documented, n_observed) |>
arrange(set, code)
w <- tbl(d_qual, "metadata/measurement_qual.csv — the controlled vocabulary")
if (knitr::is_html_output())
w <- formatStyle(w, "documented",
color = styleEqual(c(TRUE, FALSE), c("inherit", "#d03b3b")))
w
```
Questions remain open on the **bottle** side, two of them blockers: what the
undocumented single-digit codes mean there (`hydro_master_02` — the CTD
documentation now supplies `0`/`1`/`2`, but whether the bottle master uses them
the same way is unconfirmed), and whether `S_qual`'s **253 distinct values** —
clustered at 256–271, which looks like bit 8 of a bitmask — indicate a bitmask or
a corrupted column (`hydro_master_02b`). Every other quality column has 3–10
distinct values.
`hydro_master_05` — whether a NULL flag means "good" or "never assessed" — is
**answered for CTD**: the source states that no code displayed means the sensors
were operating normally. It remains open for the bottle master.
**Nothing in this protocol acts on a flag.** Acting on one whose semantics are
unconfirmed would be guessing with released data.
```{r}
#| label: rules_index
d_rules <- rules_all |>
transmute(rule_key, active, severity, type = rule_type, target,
scope = coalesce(scope, "all"), description) |>
arrange(desc(active), type, rule_key)
w <- tbl(d_rules, "The registry: metadata/qc_rules/rules.csv", page_length = 20)
if (knitr::is_html_output())
w <- formatStyle(w, "active",
color = styleEqual(c(TRUE, FALSE), c("inherit", "#6e7781")))
w
```
# The rules {#the-rules}
Grouped by what kind of question each asks. Rules that share a query — the three
bottle-vs-sensor calibration checks, the two spike checks — show the SQL once, under
the first of them.
```{r}
#| label: rules_active
#| output: asis
md_active <- qc_protocol_markdown(
arrange(rules_act, rule_type, rule_key), group_by = "rule_type")
cat(md_active)
```
# Parked rules
Registered, documented, and deliberately **not running**. They are kept in the
registry rather than deleted so that the reason survives — a rule that quietly
disappears looks like a rule that was never considered.
```{r}
#| label: rules_parked
#| output: asis
rules_off <- filter(rules_all, !active)
md_parked <- if (nrow(rules_off))
qc_protocol_markdown(rules_off, group_by = NULL) else
"*No parked rules.*\n"
cat(md_parked)
```
# Standards and prior art
This protocol is deliberately small and CalCOFI-specific. It is not a
reimplementation of the community standards, and where it diverges from them that
is worth stating.
### QARTOD
The IOOS *Manual for Real-Time Quality Control of In-situ Temperature and Salinity
Observations* (and its dissolved-oxygen and ocean-optics companions) is the
community reference for profile QC, and its `1` / `2` / `3` / `4` / `9` flags
(pass / not evaluated / suspect / fail / missing) are what an ERDDAP consumer
expects to find. Several tests here have direct QARTOD counterparts — gross range
(QARTOD test 6), spike (9), rate of change (10), flat line (11), density inversion
(15) — though the thresholds are ours and derived from this dataset rather than
adopted from the manual.
CalCOFI CTD is published to `erddap.calcofi.io`, where the native codes currently
pass through uninterpreted, and one of them (`6` = "data OK but taken from CTD
sensor") is meaningless for a CTD dataset. **Emitting QARTOD flags alongside the
native codes is question `calcofi_ctd-cast_11`**, and is the natural home for the
results of these rules once the flag semantics above are settled.
### `oce` — Dan Kelley
The reference R package for oceanographic data, and the closest prior art for the
profile rules here: `read.ctd()`, `ctdTrim()` (removes the soak and the upcast),
`despike()`, and `ctdDecimate()` (bins a profile to standard depths). Anyone
implementing these checks should read the *oce* CTD vignette and
*Analysis of Oceanographic Data* alongside this document.
Two deliberate differences. `ctdTrim()` discards the upcast; we keep it, in
`obs_ctd_full`, precisely so the down/up comparison is available as a check — the
disagreement is signal about sensor lag and fouling, not noise to be trimmed.
And `ctdDecimate()` resamples onto a regular grid by interpolation, whereas
`ctd_thin` *selects real scans* and never interpolates, so a thinned value is
always a measurement that was actually taken.
### `castr` — Jean-Olivier Irisson
Cast-oriented profile utilities: smoothing, despiking, and detection of the mixed
layer depth, deep chlorophyll maximum and thermocline. Relevant twice over — as a
comparison for the despiking approach, and because the Access master's `MLD_Sigma`
and `NutClineDepth` tables are exactly the derived products `castr` computes. Those
two tables are committed under
`metadata/calcofi/hydro-master/reference/` unused, so that any reimplementation can
be validated against the Access-era numbers rather than trusted.
::: {.callout-warning title="The CRAN package named `CTD` is not oceanographic"}
`CTD` on CRAN is "Connecting The Dots" — a weighted-graph pattern-discovery method
for metabolomics. It has nothing to do with conductivity-temperature-depth
profiles. It is an easy citation to make by name-matching and a hard one to notice
afterwards, so: do not cite it here.
:::
# What is not checked
Stated so that a clean run is not over-read.
- **The headline variables carry no quality flags at all.** Source flags attach to
the component sensors (`Temp1Q`, `Salt1Q`, `Ox1Q`, `Ox2Q`); the canonical types
are the *averages* of them, so the flag is lost in the mean. Only 8 of 27
published types carry any flag, over 0.05% of rows. How a sensor flag should
qualify an average of two is question `calcofi_ctd-cast_09` — a blocker.
- **Range coverage is partial.** 18 of the 27 published types have a declared
`valid_min`/`valid_max`; the rest — mostly raw voltages and derived quantities
with no agreed bound — are never range-checked.
- **The profile rules are sampled, not swept.** Spike, loop-edit and up/down
disagreement run one cruise at a time against `obs_ctd_full`; a whole-archive
sweep is the `ctd-qaqc` app's job, not this document's.
- **Dual-sensor disagreement is parked.** The thresholds per property are not
agreed, and a rule with a guessed threshold is worse than a rule that says it is
not running.
- **No rule drops, rewrites or interpolates a value.** Every result is a report.
Two blocker questions remain open with the data providers, and acting on them
before they are answered would be guessing with released data.
# Coverage {#coverage}
The build fails if a rule that runs is not described here, or carries no rationale
in its SQL header. A protocol that silently omits a check is worse than no
protocol, because it reads as completeness.
```{r}
#| label: coverage_check
qc_protocol_check(rules_all, paste(md_active, md_parked))
say(glue("active rules documented : {nrow(rules_act)}/{nrow(rules_act)}"))
say(glue("parked rules documented : {nrow(rules_off)}/{nrow(rules_off)}"))
say(glue("every active rule carries a rationale in its SQL header: yes"))
```
## Where results appear
- **[`ctd-qaqc`](https://github.com/CalCOFI/apps/tree/main/ctd-qaqc)** — runs this
registry interactively against a release, one rule or one cruise at a time, and
records review verdicts. It also runs the registry against an ad-hoc `.btl`
upload, so a cast can be checked before it is anywhere near a release.
- **[`ingest_calcofi_ctd-cast.qmd`](https://calcofi.io/workflows/ingest_calcofi_ctd-cast.html#findings)**
— runs the same registry over the parquet each ingest writes, so the notebook
reports the condition of the data it just published.
Both call `calcofi4db::qc_run_all()`. There is one engine and one registry: a
number in the app and a number in the notebook cannot disagree.
## How to comment on this document
It renders to two formats from one source:
[HTML](https://calcofi.io/workflows/ctd-cast_qa-qc-protocol.html) and
[`.docx`](https://calcofi.io/workflows/ctd-cast_qa-qc-protocol.docx). The Word
version exists so the CalCOFI CTD team can suggest edits in track-changes; it is
a **rendering, not a source**, so edits come back as comments to be applied to
`metadata/qc_rules/` rather than merged into the document.
Where the change belongs depends on what it is:
| what you want to change | where it actually lives |
|---|---|
| a threshold, or the reasoning behind one | the header comment of that rule's `metadata/qc_rules/sql/*.sql` — it is harvested into this document verbatim |
| whether a rule runs at all | `active` in `metadata/qc_rules/rules.csv` |
| the value or depth range a type is valid over | `metadata/measurement_type.csv` |
| what a quality code means | `metadata/measurement_qual.csv` |
| something only the data providers can settle | `metadata/calcofi/ctd-cast/questions.csv` — cite the `Q` label |
Anything else — the prose, the diagram, what is deliberately not checked — is in
`ctd-cast_qa-qc-protocol.qmd` itself.
## Session Info
```{r}
#| label: session_info
#| code-fold: true
sessioninfo::session_info()
```