Skip to contents

Sync local files to GCS, skipping unchanged files

Source: R/cloud.R
sync_to_gcs.Rd

Compares local files against GCS using checksums (CRC32C > MD5 > size). Only uploads files that are new or changed.

Usage

sync_to_gcs(
  local_dir,
  gcs_prefix,
  bucket,
  pattern = NULL,
  exclude = NULL,
  delete_stale = FALSE,
  log_to_gcs = FALSE,
  archive = FALSE,
  provider = NULL,
  dataset = NULL,
  parallel = TRUE,
  sidecar_dir = NULL,
  gcs_retries = 3L,
  verbose = TRUE
)

Arguments

local_dir

Directory containing files to upload

gcs_prefix

GCS destination prefix (e.g. "ingest/swfsc_ichthyo" or "archive" for archive mode)

bucket

GCS bucket name

pattern

Regex to filter local files (default: NULL = all files)

exclude

Character vector of glob patterns to skip (e.g. c(".DS_Store", "*.tmp")). Applied to relative file paths.

delete_stale

If TRUE, delete GCS files that no longer exist locally. Default FALSE. Ignored in archive mode.

log_to_gcs

If TRUE, write a timestamped JSON action log to gs://{bucket}/{gcs_prefix}/_logs/sync_YYYY-MM-DD_HHMMSS.json.

archive

If TRUE, use archive mode: creates a timestamped immutable snapshot. Requires provider and dataset.

provider

Data provider (required when archive = TRUE)

dataset

Dataset name (required when archive = TRUE)

parallel

If TRUE (default), mirror with a single gcloud storage rsync -r, which transfers concurrently. The per-file path issues one gcloud storage cp process per file and one rm per stale object, which serialises the whole upload — an ingest with a Hive-partitioned table (obs_ctd_full is 96 partitions / 4.9 GB) spends most of its wall clock in process startup. Set FALSE for the per-file path when you need the detailed per-file action tibble or crc32c-level skip reporting.

sidecar_dir

Optional second local directory whose files mirror to the same gcs_prefix. An ingest's output is split across two roots — bulk parquet under cc_stage_dir(), the JSON sidecars in the repo — but it is one directory as far as GCS and every consumer are concerned. Sidecars are copied after the main sync, and are protected from delete_stale (see Details).

gcs_retries

Attempts for the parallel rsync before giving up (default 3, backing off 15s/30s). rsync skips what already matches, so a retry re-sends only what is missing — a transient network failure should cost the remaining bytes, not the hours of compute that produced them.

verbose

Print per-file status messages (default: TRUE)

Value

In mirror mode: tibble with columns file, action, size. In archive mode: list with archive_timestamp, archive_path, created_new, files_uploaded.

Details

Two modes:

  • Mirror mode (default): syncs local_dir/ to gs://{bucket}/{gcs_prefix}/. Optionally deletes stale GCS files.

  • Archive mode (archive = TRUE): creates a timestamped immutable snapshot at gs://{bucket}/{gcs_prefix}/{timestamp}/{provider}/{dataset}/. Timestamp derived from max file mtime for reproducibility. Skips upload if files match the latest existing archive.

Examples

if (FALSE) { # \dontrun{
# mirror mode: sync parquet outputs
sync_to_gcs(
  local_dir  = "data/parquet/swfsc_ichthyo",
  gcs_prefix = "ingest/swfsc_ichthyo",
  bucket     = "calcofi-db")

# mirror mode: full GD backup with stale cleanup
sync_to_gcs(
  local_dir    = "~/My Drive/projects/calcofi/data-public",
  gcs_prefix   = "_sync",
  bucket       = "calcofi-files-public",
  delete_stale = TRUE,
  log_to_gcs   = TRUE,
  exclude      = c(".DS_Store", "*.tmp"))

# archive mode: timestamped snapshot of source CSVs
sync_to_gcs(
  local_dir  = "path/to/csv",
  gcs_prefix = "archive",
  bucket     = "calcofi-files-public",
  archive    = TRUE,
  provider   = "swfsc",
  dataset    = "ichthyo")
} # }