R-clone-setup/CLAUDE.md

R-clone — Rclone→Synology Sync Monitoring Project

Mission

The office deploys client machines (Linux, GPU) that process road-survey videos/images and sync results to a Synology NAS via rclone. Nothing tracks whether syncs succeed. Goal: build a monitoring UI + alert system for the fleet. My stack: MERN (Node/Express/React/Mongo), FastAPI, intermediate Python, basic Linux.

Production architecture (from office files — do NOT commit real creds)

Each client site machine runs (via a generated docker-compose):

• algorithm3 — GPU/YOLO container, processes video → anomaly images + result CSVs
• mysql, socket-server, local-gui-backend — local site stack
• watchtower — auto-pulls new images every 300s from the private registry
• rclone-synology-sync — Alpine rclone image + sync.sh loop (this repo has exact copies)

The sync loop (sync-container/sync.sh — byte-identical to production)

• Writes rclone.conf for an SFTP remote (synodrive) → Synology at a public hostname, non-standard port, single shared user (creds in office compose only).
• Starts rclone rcd --rc-web-gui on :5572 (RC API + built-in web GUI).
• Infinite loop: for each SYNC_n=direction:local:remote env var, run rclone copy --size-only (up = local→NAS, down = NAS→local), then sleep SYNC_INTERVAL seconds (seconds — the "# minutes" comment in prod compose is wrong; effective prod interval is 120s).
• Logs to /logs/: sync.log (everything), stats.log (pretty per-round), errors.log (grep " ERROR "), completed.log (grep "Copied|Moved"), rc.log.

Production sync pairs (per site; ORG/SITENAME templated)

#	Dir	What
1	up	anomaly images → Saudi_Video_Sync/SeekRight/Anomaly/<ORG>/<SITE>/TEST
2	down	master sheets ← ClientSync/ALGORITHM_DEPLOYMENT/Master_Sheets/<ORG>/<SITE>
3	down	yolov8 weights ← ClientSync/ALGORITHM_DEPLOYMENT/Models/yolov8
4	down	NIGHT weights ← ClientSync/ALGORITHM_DEPLOYMENT/Models/NIGHT
5	up	result CSVs → ClientSync/ALGORITHM_DEPLOYMENT/csv_files/<ORG>/<SITE>

Monitoring focus = pairs 1 & 5 (client-produced files that must reach the NAS).

Known production issues (verified, mention when relevant)

1. RC_PASSWORD never set in prod compose → unauthenticated RC API on host network :5572 (full read/write to NAS + local disk from the LAN).
2. Plaintext creds in compose + git token in generate_compose.sh → flagged for rotation.
3. --size-only + copy (not sync): same-size edits never re-transfer; deletes never propagate. "Synced" is weaker than it sounds.
4. The built-in Web GUI shows nothing useful — see below.

KEY FINDING: rclone's built-in UI exists but can't see the syncs

rclone rcd --rc-web-gui serves a React GUI on :5572 (login admin/$RC_PASSWORD). Verified locally: while 5 pairs were actively copying, POST /core/stats returned all zeros. The rclone copy commands are separate OS processes; the rcd daemon only reports jobs started through its own API. So the prod GUI is decorative. Options for the real project:

• (A) Ship our own agent that tails logs + POSTs to a central collector (recommended; zero change to sync behavior), or
• (B) Rewrite sync.sh to submit jobs via rclone rc sync/copy _async=true so rcd/GUI sees them — bigger change, still no fleet view, history, or alerts.
• Adding --use-json-log to the rclone copy invocation would make log parsing trivial (structured JSON per line) — one-flag change worth proposing.

This repo = the production monitoring stack

docker-compose.yml   mongo + collector — the permanent stack (port 4400)
monitor/collector/   FastAPI + pymongo + rclone; serves API and the built UI
monitor/agent/       stdlib-only sidecar deployed on each client machine
frontend/            React 18 + Vite dashboard → dist/ bind-mounted into collector
sync-container/      byte-identical prod sync.sh + Dockerfile (reference + used
                     by deploy/remote-machine builds) — never modify
deploy/              install bundles: agent-only/ (machine already runs prod
                     sync) and remote-machine/ (needs sync + agent)
send-to-friend/      agent-deploy.zip — the bundle actually on JAGAN-TEST-01
.backups/            pre-v2 versions of main.py/App.jsx/styles.css/.env

The simulation lab was REMOVED 2026-08-10 (user decision: not needed for production). docker-compose.sim.yml, generator/, lan-forward.ps1 are in git history; client/+client2/+server-data/ were gitignored fake data and are gone for good. Consequence: no local test fleet — alert paths can now only be exercised against a real machine. The 2026-08-05/06 sections below describe that lab; keep them as history, don't try to run them.

Run / operate

cd /opt/R-clone/R-clone-setup
sudo docker-compose up -d --build          # mongo + collector (permanent stack)
sudo docker-compose logs -f collector
sudo docker-compose down

This host has docker-compose v1 (the hyphenated binary) only — the v2 docker compose plugin is NOT installed and fails with unknown shorthand flag: 'd'. Always write docker-compose.

Environment (Linux Mint host — MOVED off Windows/WSL 2026-08-07)

The project now lives at /opt/R-clone/R-clone-setup on a Linux Mint box (vm3@vm3-mint). Every /mnt/c/... path and WSL workaround in older notes is dead — including RCLONE_SFTP_SET_MODTIME=false, which only existed because /mnt/c rejects utimes(). Current realities:

• Repo was root-owned; chown -R vm3:vm3 /opt/R-clone ran 2026-08-07. Without it, no edits, no npm run build, and git throws "dubious ownership".
• docker needs sudo — vm3 is not in the docker group.
• Ports 4000 and 8000 on this host are already taken by unrelated apps (a Node app in ~/seekright-video-calendar, and another FastAPI service). That is why the collector moved to 4400. Don't move it back.
• Backend vs frontend rebuild rules differ: main.py is COPYd into the image → a change needs sudo docker-compose up -d --build collector. frontend/dist is bind-mounted → npm run build alone is enough, no restart.
• Collector host LAN IP 192.168.1.201/22 (so the /22 spans .0–.3.255). It is DHCP — it already changed once and broke every remote agent. Reserve it.

Verified: RC API gives every UI metric (2026-08-05 experiment)

monitor/watch_batch.py submitted 30×5MB files via POST /sync/copy _async=true (src ./client/batch_test, throttled with core/bwlimit) and polled live:

• scheduled = core/stats.totalTransfers · done = .transfers
• moving now = .transferring[] (name, %, speed, per-file eta; max 4 shown = --transfers)
• queued = totalTransfers − transfers − len(transferring)
• overall ETA/speed/bytes = .eta, .speed, .bytes/.totalBytes
• per-file duration = core/transferred[] completed_at − started_at (+ error field) All 30 arrived; per-file report printed. Gotchas: per-file eta can be None early; totalTransfers grows during discovery (not instantly 30); always pass {"group": "job/<jobid>"} to scope stats to one job. Constraint: RC only sees jobs submitted via RC. The prod loop's rclone copy processes are invisible to it → for live per-file UI data in prod, sync.sh v2 should submit pairs via rclone rc sync/copy _async=true + poll, instead of spawning rclone copy. Log parsing alone gives only aggregate 5s stats lines

• completion events.

Monitoring stack v1 — BUILT and running (2026-08-06)

agent (per machine) → collector (FastAPI+Mongo) → React dashboard, all in compose:

• monitor/agent/agent.py — stdlib-only sidecar; tails /logs/sync.log (ro mount), regex-parses rounds/pairs/files/errors/progress, POSTs /api/ingest every 5s (empty POST = heartbeat). Works against byte-identical prod sync.sh.
• monitor/collector/main.py — FastAPI + pymongo (tz_aware=True — naive-vs-aware datetime bug otherwise). Collections: machines, pairs, events. Endpoints: POST /api/ingest, GET /api/overview (fleet + computed alerts), GET /api/machines/{id}/events?type=&limit=. Serves React build from /app/static. Alert rules: offline (no heartbeat >30s, critical), pair last_status=fail (serious), errors_1h>0 (warning).
• frontend/ — React 18 + Vite. Fleet cards (status badge, stat tiles, pair rows, live progress bar), alerts bar, click card → events panel with tabs (Files/Errors/Rounds/Pair results). Polls every 4s. Dark theme, dataviz status tokens (#0ca30c/#fab219/#ec835a/#d03b3b), icon+label never color alone. Build: cd frontend && npm run build (dist/ is bind-mounted into the collector — rebuild frontend = just npm run build, no docker rebuild).
• UI: http://localhost:4400 · do not stop/restart containers without asking first. (The RIYADH-01 second-machine demo died with the sim lab.)

v1.1 — remote machines + RMM surface (2026-08-06)

• Agent auth: AGENT_TOKEN shared secret (in .env); collector rejects ingest without Authorization: Bearer <token> (verified 401). Empty token = auth off (never for remote use).
• RMM endpoints: GET /api/health (liveness), GET /api/alerts (lightweight polling), CORS enabled (CORS_ORIGINS env). Integration points for the user's future RMM: poll alerts, or embed /api/overview data.
• deploy/remote-machine/: self-contained bundle (compose + .env.example) for any external machine (e.g. a friend's system): rclone-sync (1 up pair from the algorithm's OUTPUT_DIR) + agent → user's collector. Remote machine needs a network path to the collector — recommend Tailscale; COLLECTOR_URL then is the tailnet IP. Bundle builds from ../../sync-container and ../../monitor/agent.
• Transfer visibility today: completed files ✓ (file_synced events), aggregate live progress per pair ✓ (5s stats → progress bar), per-file in-flight + pending queue ✗ — needs RC-submitted jobs (sync.sh v2, see RC section).

v1.2 — server-side verification / audit feature (2026-08-06) — WORKING

The audit-team answer: "did the file actually reach the Synology?" proven from the collector, no one logs into anything.

• Agent WATCH_DIRS env: mounts up-pair source dirs ro at the same container paths as sync (/sources/...), ships file inventory (name/size/mtime) with every ingest.
• Collector: own rclone remote (start.sh writes config; rclone installed via apt in Dockerfile), background thread rclone lsjson --recursive on every up-pair remote path every VERIFY_INTERVAL (45s local; use 300s+ in prod). pending = local file not in server listing; missing = pending with mtime older than MISSING_GRACE_S (600) → serious alert with file names.
• UI: 4th tile "pending → server" + amber N pending / red N missing! chips per pair. Verified live: pending counts rise as generator drops files, fall to 0 after each sync round.
• Friend's machine (compose seen 2026-08-06): logs at /home/testing/JAGAN/Prerequisites/Sync_logs, SITENAME=TEST, standard 5 pairs. deploy/remote-machine bundle fits as-is; needs Tailscale (or LAN) to reach the collector + AGENT_TOKEN. Nothing needed from his algorithm code — agent reads only Sync_logs + source dirs.

v2 — upload audit + windowed history (2026-08-07 → 10) — CURRENT

Prompted by: "admins audit UPLOADS, not downloads; need pending/in-progress/ completed and processed-counts per hour/day/3d/week/month, ~1 month of data." Researched: rclone's native Prometheus metrics can't help (same single-rcd blindness as the GUI + unscrapeable short-lived syncs); MFT dashboards use the Completed/In-Progress/Queued/Failed taxonomy → adopted.

Collector additions (monitor/collector/main.py)

• Retention: Mongo TTL index on events.received_at, RETENTION_DAYS env (30). _ensure_retention falls back to collMod when the window changes.
• GET /api/stats — one aggregation pass over 30d computes per-machine AND fleet rollups for windows 1h/24h/3d/7d/30d: uploaded, errors, failures, rounds, bytes, avg_bps. Machines with zero activity still appear (that's the signal admins need). ~15s poll from the UI.
• GET /api/timeseries?range=&machine_id= — epoch-aligned buckets (5min/1h/3h/6h/1d per range), zero-filled, uploads + errors only.
• GET /api/machines/{id}/events — added direction=, q= (regex-escaped search over file/message/pair), skip/limit paging, returns total.
• DELETE /api/machines/{id} (agent-token auth) — retire a machine; without it a decommissioned machine alerts "offline" forever. Used to remove the CONNTEST test entry 2026-08-10.
• round_summary now splits files_up/files_down/bytes_up (download bytes are unknowable — files land outside WATCH_DIRS; never fake them).
• _machine_view: up_pairs/down_pairs split, in_progress, failed_up, uploads_1h (direction-filtered). Alerts unchanged EXCEPT download-pair failures now say "download pair failing" (still alerted, not shown as cards).
• Verification defaults to the real NAS: DEFAULT_REMOTE env (compose sets synoreal), VERIFY_ENABLED true if either SYNOLOGY_HOST or REAL_SYNOLOGY_HOST is set. Old behavior (default synodrive = the deleted fake NAS) silently skipped verification for any site not in SITE_REMOTES.

UI (v2.1 after a misstep — frontend/src/App.jsx)

First v2 cut replaced the machine cards with range-dependent tiles and dropped download pair rows + the last-round tile. User: "not understandable, it was meaningful before." Lesson recorded: the last-round-ago tile and the full pair list ARE the at-a-glance value; window counters reading 0 look like a broken dashboard. Current layout:

• Cards (top, range-INDEPENDENT, restored): last round tile (amber when

  30min stale), uploaded (1h), queued→NAS (pending), errors (1h), live progress bar, ALL 5 pair rows (up first), missing/pending chips.

• "Upload history" section (below, driven by the range selector): KPI row (uploaded / failed+errors / rounds / avg throughput+bytes), uploads-per-bucket bar chart with hover tooltip + "view as table", per-machine comparison table (the per-client speed audit). Chart empty state names the likely cause (stopped sync container) instead of rendering all-zero bars.
• Palette validated (dataviz skill): uploads #3987e5 / errors #d03b3b on #1a1a19 pass all six checks; separate stacked plots, never dual-axis.

Ops facts (2026-08-09/10)

• JAGAN-TEST-01's sync container was stopped 2026-08-07 during load-test cleanup and left stopped for 65h — the stalled-round alert caught it; the "empty graph" complaint was exactly this. Agent online ≠ sync running.
• His Master_Sheets down pair fails exit 3: remote path .../Master_Sheets/TEST/TEST doesn't exist on the NAS (ORG=SITE=TEST substitution) — create the folder or fix his compose vars.
• Load-test cleanup order that WORKS: stop sync container → delete local source files → rclone purge the NAS folder → start container. NAS-first gets re-uploaded within one round (learned the hard way).
• .env trimmed to AGENT_TOKEN + REAL_SYNOLOGY_* + ORG/SITENAME. send-to-friend/agent-deploy.zip rebuilt (old zip still had :8000).

v2.1 — public exposure + one-line installer (2026-08-11)

• Public URL: https://rclone.seekright.com → Synology reverse proxy → this VM :4400 (same pattern as the RMM hostnames). PUBLIC_URL in .env.
• Viewer auth: HTTP Basic middleware (VIEWER_USER/VIEWER_PASS in .env) gates EVERYTHING except /api/ingest, /api/health, and requests carrying the agent bearer token. Unset VIEWER_PASS = gate off. Browser prompts once.
• One-line installer (RMM deploy_agent.sh pattern, leak-lesson applied — download is authenticated, live AGENT_TOKEN injected at serve time, split-sentinel guard, exactly-once placeholder check):

  curl -fsSL -u admin "https://rclone.seekright.com/deploy_agent.sh" -o /tmp/deploy_agent.sh
  sudo bash /tmp/deploy_agent.sh <MACHINE_ID> [SITE]   # then rm the file (holds live token)
  

  Auto-discovers /logs + upload-source host paths from the machine's rclone-synology-sync container (docker inspect), fetches agent.py from GET /agent.py (single source of truth — no embedded copy to drift), writes /opt/rclone-agent/, replaces any previous rclone-agent container, starts. Re-run with no args = in-place upgrade keeping identity. MACHINE_ID is REQUIRED on fresh installs (no hostname fallback — RMM's stale-entry lesson).
• Collector build context is now ./monitor (Dockerfile at collector/Dockerfile) so the image carries agent/agent.py to serve.

v2.2 — REAL fleet sync.sh has drifted from our copy (found 2026-08-11)

First real office machine (GCBOT-1 / IRBLAMBTHAM001, image from the private registry) revealed the production sync.sh is NOT our byte-copy anymore:

• Up-pairs are change-detected: skipped unless file count or du -sb size changed; logged as SKIP: [↑ UP] <label> — unchanged (idle Ns). A forced sync runs after FORCE_SYNC_AFTER=1800s idle (" reason: …" line follows pair start). Down pairs unchanged.
• Rounds every 180s (CHECK_INTERVAL), not 120.
• Consequences: up-pairs show activity at most every 30min when idle; a pair can be absent from the dashboard until its first non-skipped run unless skips are parsed. Agent v2 parses SKIP → pair_skip event; collector records last_skip_at/last_idle_s and creates the pair as status "idle" (NEVER overwrites fail — state files update before rclone runs, so a failed up-pair is followed by skips until content changes). UI shows idle as ok.
• VERIFY_TIMEOUT env (default 300s) — 8.5k-file SFTP listings exceed 120s.
• MAX_INV_FILES now env-tunable (default 2000; GCBOT image tree is 8.5k).
• Jagan's machine runs OUR old sync.sh (bundle-built) — no SKIP lines there.
• TODO: replace sync-container/sync.sh with the real fleet version (get full file from a machine: docker exec rclone-synology-sync cat /sync.sh).

Roadmap for the actual deliverable

1. Agent (per client machine, sidecar container): tail /logs/*.log (or JSON log), parse rounds/pairs/files/errors, POST events + 60s heartbeat to collector.
2. Collector: FastAPI + MongoDB. Models: Machine, SyncPair, Round, FileEvent, Alert.
3. Server-side verification (the feature logs can't provide): collector has its own rclone SFTP remote to the NAS; rclone lsjson per site path to confirm reported uploads actually arrived; alert if a file is still missing after a grace period. Missed-heartbeat = dead machine alert (most important signal).
4. UI: React dashboard — fleet grid (last round, last heartbeat, error counts), per-machine pair detail, file history, live progress later via RC API.
5. Alerts: dashboard + email first; Slack/Teams webhook later. Rules before channels.

Conventions for this repo

• No real office credentials/hostnames in tracked files, with ONE deliberate exception: .env holds the real REAL_SYNOLOGY_* values and IS committed (see .gitignore header — private repo, creds needed on the VM after a pull). Everything else uses placeholders like <ORG>/<SITE>.
• sync-container/sync.sh stays byte-identical to production — replica fidelity is the point. Fixes go in compose env vars or new sidecar services, and proposed prod changes get documented in this file instead.
• RMM integration is PARKED (user decision 2026-08-10, R-clone fixes first). When resumed: this VM is also the production RMM server (26 field machines) — read /opt/rmm-backend/CLAUDE.md before touching anything there. Plan agreed: additive read-only proxy endpoint in rmm-backend + new rmm-ui page, developed against a scratch uvicorn (MONGODB_DB=rmm_test, spare port) and a scratch UI copy — NEVER by editing /opt/rmm-ui/src in place (vite hot-reloads straight into production).