Sync Google Drive to GCS

1 Overview

Mirrors the entire Google Drive data-public/ folder to gs://calcofi-files-public/_sync/ using rclone. This transfers directly between cloud providers (GD API → GCS API) without needing files cached on the local machine.

This is a standalone utility — not a pipeline target. Run manually or schedule as a cron job for daily backup.

Two-tier sync:

• calcofi-files-public/_sync/ — living mirror of GD (this notebook). Deletes stale GCS files so it always matches GD.
• calcofi-files-public/archive/ — immutable timestamped snapshots (created by ingest QMDs via sync_to_gcs(archive = TRUE)). Nothing is lost permanently.

Prerequisites: rclone configured with remotes:

• gdrive-ecoquants — Google Drive (or gdrive-server on server)
• gcs-calcofi — Google Cloud Storage

Check with rclone config or rclone listremotes.

1.1 rclone exclude patterns

All sync commands below exclude these files:

Pattern	Reason
.DS_Store	macOS metadata
*.tmp	temporary files
~$*	Office lock files
*.gdoc, *.gsheet, *.gslides	Google Docs shortcuts (not real files)

2 Setup

librarian::shelf(
  DT,
  fs,
  glue,
  tibble,
  dplyr,
  quiet = T
)
options(DT.options = list(scrollX = TRUE))

# rclone remote paths — override via env vars for server deployment
# e.g. CALCOFI_GD_REMOTE="gdrive-server:projects/calcofi/data-public"
gd_remote <- Sys.getenv(
  "CALCOFI_GD_REMOTE",
  "gdrive-ecoquants:projects/calcofi/data-public"
)
gcs_remote <- Sys.getenv(
  "CALCOFI_GCS_REMOTE",
  "gcs-calcofi:calcofi-files-public/_sync"
)

# shared exclude flags for all rclone commands
rclone_excludes <- c(
  "--exclude", ".DS_Store",
  "--exclude", "*.tmp",
  "--exclude", "~$*",
  "--exclude", "*.gdoc",
  "--exclude", "*.gsheet",
  "--exclude", "*.gslides"
)

# local log directory
log_dir <- here::here("data/logs")
dir_create(log_dir)

3 Dry Run

Preview what rclone would do without making changes. This is fast and useful for verifying before a large sync.

cat(glue("dry run: {gd_remote} → {gcs_remote}"), "\n\n")

dry run: gdrive-ecoquants:projects/calcofi/data-public → gcs-calcofi:calcofi-files-public/_sync 

dry_out <- system2(
  "rclone",
  c("sync", gd_remote, gcs_remote,
    "--dry-run", rclone_excludes, "-v"),
  stdout = TRUE, stderr = TRUE
)

# parse actions from rclone verbose output
actions <- dry_out[grepl("NOTICE|INFO|Copied|Deleted", dry_out)]
cat(glue("{length(actions)} actions would be taken"), "\n\n")

322976 actions would be taken 

if (length(actions) > 0) {
  tibble(message = actions) |>
    datatable(
      caption = "rclone dry-run actions",
      rownames = FALSE
    )
}