R-clone-setup/CLAUDE.md

R-clone — Rclone→Synology Sync Monitoring Project

Mission

The office deploys client machines (Linux, GPU) that process road-survey videos/images and sync results to a Synology NAS via rclone. Nothing tracks whether syncs succeed. Goal: build a monitoring UI + alert system for the fleet. My stack: MERN (Node/Express/React/Mongo), FastAPI, intermediate Python, basic Linux.

Production architecture (from office files — do NOT commit real creds)

Each client site machine runs (via a generated docker-compose):

• algorithm3 — GPU/YOLO container, processes video → anomaly images + result CSVs
• mysql, socket-server, local-gui-backend — local site stack
• watchtower — auto-pulls new images every 300s from the private registry
• rclone-synology-sync — Alpine rclone image + sync.sh loop (this repo has exact copies)

The sync loop (sync-container/sync.sh — byte-identical to production)

• Writes rclone.conf for an SFTP remote (synodrive) → Synology at a public hostname, non-standard port, single shared user (creds in office compose only).
• Starts rclone rcd --rc-web-gui on :5572 (RC API + built-in web GUI).
• Infinite loop: for each SYNC_n=direction:local:remote env var, run rclone copy --size-only (up = local→NAS, down = NAS→local), then sleep SYNC_INTERVAL seconds (seconds — the "# minutes" comment in prod compose is wrong; effective prod interval is 120s).
• Logs to /logs/: sync.log (everything), stats.log (pretty per-round), errors.log (grep " ERROR "), completed.log (grep "Copied|Moved"), rc.log.

Production sync pairs (per site; ORG/SITENAME templated)

#	Dir	What
1	up	anomaly images → Saudi_Video_Sync/SeekRight/Anomaly/<ORG>/<SITE>/TEST
2	down	master sheets ← ClientSync/ALGORITHM_DEPLOYMENT/Master_Sheets/<ORG>/<SITE>
3	down	yolov8 weights ← ClientSync/ALGORITHM_DEPLOYMENT/Models/yolov8
4	down	NIGHT weights ← ClientSync/ALGORITHM_DEPLOYMENT/Models/NIGHT
5	up	result CSVs → ClientSync/ALGORITHM_DEPLOYMENT/csv_files/<ORG>/<SITE>

Monitoring focus = pairs 1 & 5 (client-produced files that must reach the NAS).

Known production issues (verified, mention when relevant)

1. RC_PASSWORD never set in prod compose → unauthenticated RC API on host network :5572 (full read/write to NAS + local disk from the LAN).
2. Plaintext creds in compose + git token in generate_compose.sh → flagged for rotation.
3. --size-only + copy (not sync): same-size edits never re-transfer; deletes never propagate. "Synced" is weaker than it sounds.
4. The built-in Web GUI shows nothing useful — see below.

KEY FINDING: rclone's built-in UI exists but can't see the syncs

rclone rcd --rc-web-gui serves a React GUI on :5572 (login admin/$RC_PASSWORD). Verified locally: while 5 pairs were actively copying, POST /core/stats returned all zeros. The rclone copy commands are separate OS processes; the rcd daemon only reports jobs started through its own API. So the prod GUI is decorative. Options for the real project:

• (A) Ship our own agent that tails logs + POSTs to a central collector (recommended; zero change to sync behavior), or
• (B) Rewrite sync.sh to submit jobs via rclone rc sync/copy _async=true so rcd/GUI sees them — bigger change, still no fleet view, history, or alerts.
• Adding --use-json-log to the rclone copy invocation would make log parsing trivial (structured JSON per line) — one-flag change worth proposing.

This repo = full local replica (WSL2 Ubuntu + docker-ce, NOT Docker Desktop)

docker-compose.yml        3 services (see header comment for deviations from prod)
├─ synology         atmoz/sftp:alpine — fake NAS; ./server-data ↔ /home/sr-upload/upload
│                     reachable from Windows: sftp -P 2222 sr-upload@localhost
├─ rclone-synology-sync   built from sync-container/ (EXACT prod Dockerfile+sync.sh),
│                     same 5 pairs, remote paths prefixed `upload/` (sftp chroot),
│                     SYNC_INTERVAL=60, GUI/RC on http://localhost:5572
└─ file-generator   generator/generate.sh — fake algorithm3, drops a random-bytes
                      .jpg + a results .csv into the up-sync dirs every 45s
.env                  local-only fake creds (SYNOLOGY_PASS, RC_PASSWORD, ORG, SITENAME)
client/               the "client machine's disk": image_root_dir, csv_files (sources),
                      GCBOT, yolov8, NIGHT (down targets), Sync_logs (all 5 log files)
server-data/          the "NAS content" — inspect from Windows Explorer

Verified working 2026-08-05: all 5 pairs green, files flowing both directions, errors.log has real historical failures (good test data for alert logic).

Run / operate (always via WSL)

wsl -e bash -c "cd /mnt/c/Users/seekr/OneDrive/Desktop/work/R-clone && docker compose up -d --build"
docker compose logs -f file-generator      # watch fake files being produced
tail -f client/Sync_logs/stats.log         # watch rounds
docker compose down                        # stop

WSL/Windows gotchas (cost us an hour — don't rediscover)

• WSL VM idles out when no wsl.exe session is open → dockerd dies → all restart-policy containers restart. Keep a wsl terminal open (or a background wsl -e sleep infinity) while testing. Symptom: every container "Up N seconds", sync.log shows repeated "Container started".
• /mnt/c bind mounts reject utimes() → rclone up-syncs fail with SetModTime failed: permission denied. Fixed via RCLONE_SFTP_SET_MODTIME=false env var in compose (rclone reads RCLONE_* env as config → sync.sh stays untouched).
• Shell scripts must be LF, not CRLF (sed -i 's/\r$//' after editing on Windows).
• Old production leftovers exist on this machine (user ran the office compose ~3 weeks ago): containers algorithm (dead, exit 137), mysql, socket-server (crash-loops), local-gui-backend, watchtower (polls office registry!). Parked with docker update --restart=no + stopped. Don't remove without asking; revive with docker start <name>.

Verified: RC API gives every UI metric (2026-08-05 experiment)

monitor/watch_batch.py submitted 30×5MB files via POST /sync/copy _async=true (src ./client/batch_test, throttled with core/bwlimit) and polled live:

• scheduled = core/stats.totalTransfers · done = .transfers
• moving now = .transferring[] (name, %, speed, per-file eta; max 4 shown = --transfers)
• queued = totalTransfers − transfers − len(transferring)
• overall ETA/speed/bytes = .eta, .speed, .bytes/.totalBytes
• per-file duration = core/transferred[] completed_at − started_at (+ error field) All 30 arrived; per-file report printed. Gotchas: per-file eta can be None early; totalTransfers grows during discovery (not instantly 30); always pass {"group": "job/<jobid>"} to scope stats to one job. Constraint: RC only sees jobs submitted via RC. The prod loop's rclone copy processes are invisible to it → for live per-file UI data in prod, sync.sh v2 should submit pairs via rclone rc sync/copy _async=true + poll, instead of spawning rclone copy. Log parsing alone gives only aggregate 5s stats lines

• completion events.

Monitoring stack v1 — BUILT and running (2026-08-06)

agent (per machine) → collector (FastAPI+Mongo) → React dashboard, all in compose:

• monitor/agent/agent.py — stdlib-only sidecar; tails /logs/sync.log (ro mount), regex-parses rounds/pairs/files/errors/progress, POSTs /api/ingest every 5s (empty POST = heartbeat). Works against byte-identical prod sync.sh.
• monitor/collector/main.py — FastAPI + pymongo (tz_aware=True — naive-vs-aware datetime bug otherwise). Collections: machines, pairs, events. Endpoints: POST /api/ingest, GET /api/overview (fleet + computed alerts), GET /api/machines/{id}/events?type=&limit=. Serves React build from /app/static. Alert rules: offline (no heartbeat >30s, critical), pair last_status=fail (serious), errors_1h>0 (warning).
• frontend/ — React 18 + Vite. Fleet cards (status badge, stat tiles, pair rows, live progress bar), alerts bar, click card → events panel with tabs (Files/Errors/Rounds/Pair results). Polls every 4s. Dark theme, dataviz status tokens (#0ca30c/#fab219/#ec835a/#d03b3b), icon+label never color alone. Build: cd frontend && npm run build (Windows npm; dist/ is volume-mounted into collector — rebuild frontend = just npm run build, no docker rebuild).
• Fleet demo: machine 2 = RIYADH-01 (rclone-sync-riyadh, 2 pairs, 90s interval, own client2/ dirs + generator + agent).
• UI: http://localhost:8000 · verified: both machines online, 7 pairs ok, files_1h counting. Alert-firing path not yet demo'd (user declined stopping a local container — do not stop/restart containers without asking first).

v1.1 — remote machines + RMM surface (2026-08-06)

• Agent auth: AGENT_TOKEN shared secret (in .env); collector rejects ingest without Authorization: Bearer <token> (verified 401). Empty token = auth off (never for remote use).
• RMM endpoints: GET /api/health (liveness), GET /api/alerts (lightweight polling), CORS enabled (CORS_ORIGINS env). Integration points for the user's future RMM: poll alerts, or embed /api/overview data.
• deploy/remote-machine/: self-contained bundle (compose + .env.example) for any external machine (e.g. a friend's system): rclone-sync (1 up pair from the algorithm's OUTPUT_DIR) + agent → user's collector. Remote machine needs a network path to the collector — recommend Tailscale; COLLECTOR_URL then is the tailnet IP. Bundle builds from ../../sync-container and ../../monitor/agent.
• Transfer visibility today: completed files ✓ (file_synced events), aggregate live progress per pair ✓ (5s stats → progress bar), per-file in-flight + pending queue ✗ — needs RC-submitted jobs (sync.sh v2, see RC section).

v1.2 — server-side verification / audit feature (2026-08-06) — WORKING

The audit-team answer: "did the file actually reach the Synology?" proven from the collector, no one logs into anything.

• Agent WATCH_DIRS env: mounts up-pair source dirs ro at the same container paths as sync (/sources/...), ships file inventory (name/size/mtime) with every ingest.
• Collector: own rclone remote (start.sh writes config; rclone installed via apt in Dockerfile), background thread rclone lsjson --recursive on every up-pair remote path every VERIFY_INTERVAL (45s local; use 300s+ in prod). pending = local file not in server listing; missing = pending with mtime older than MISSING_GRACE_S (600) → serious alert with file names.
• UI: 4th tile "pending → server" + amber N pending / red N missing! chips per pair. Verified live: pending counts rise as generator drops files, fall to 0 after each sync round.
• Friend's machine (compose seen 2026-08-06): logs at /home/testing/JAGAN/Prerequisites/Sync_logs, SITENAME=TEST, standard 5 pairs. deploy/remote-machine bundle fits as-is; needs Tailscale (or LAN) to reach the collector + AGENT_TOKEN. Nothing needed from his algorithm code — agent reads only Sync_logs + source dirs.

Roadmap for the actual deliverable

1. Agent (per client machine, sidecar container): tail /logs/*.log (or JSON log), parse rounds/pairs/files/errors, POST events + 60s heartbeat to collector.
2. Collector: FastAPI + MongoDB. Models: Machine, SyncPair, Round, FileEvent, Alert.
3. Server-side verification (the feature logs can't provide): collector has its own rclone SFTP remote to the NAS; rclone lsjson per site path to confirm reported uploads actually arrived; alert if a file is still missing after a grace period. Missed-heartbeat = dead machine alert (most important signal).
4. UI: React dashboard — fleet grid (last round, last heartbeat, error counts), per-machine pair detail, file history, live progress later via RC API.
5. Alerts: dashboard + email first; Slack/Teams webhook later. Rules before channels.

Conventions for this repo

• Never write real office credentials/hostnames into files here; use .env pattern and placeholders like <ORG>/<SITE>.
• sync-container/sync.sh stays byte-identical to production — replica fidelity is the point. Fixes go in compose env vars or new sidecar services, and proposed prod changes get documented in this file instead.