# R-clone — Rclone→Synology Sync Monitoring Project ## Mission The office deploys client machines (Linux, GPU) that process road-survey videos/images and sync results to a Synology NAS via rclone. Nothing tracks whether syncs succeed. **Goal: build a monitoring UI + alert system** for the fleet. My stack: MERN (Node/Express/React/Mongo), FastAPI, intermediate Python, basic Linux. ## Production architecture (from office files — do NOT commit real creds) Each client site machine runs (via a generated docker-compose): - `algorithm3` — GPU/YOLO container, processes video → anomaly images + result CSVs - `mysql`, `socket-server`, `local-gui-backend` — local site stack - `watchtower` — auto-pulls new images every 300s from the private registry - `rclone-synology-sync` — Alpine rclone image + `sync.sh` loop (this repo has exact copies) ### The sync loop (`sync-container/sync.sh` — byte-identical to production) - Writes rclone.conf for an **SFTP remote** (`synodrive`) → Synology at a public hostname, non-standard port, single shared user (creds in office compose only). - Starts `rclone rcd --rc-web-gui` on **:5572** (RC API + built-in web GUI). - Infinite loop: for each `SYNC_n=direction:local:remote` env var, run `rclone copy --size-only` (up = local→NAS, down = NAS→local), then sleep `SYNC_INTERVAL` seconds (**seconds — the "# minutes" comment in prod compose is wrong; effective prod interval is 120s**). - Logs to `/logs/`: `sync.log` (everything), `stats.log` (pretty per-round), `errors.log` (grep " ERROR "), `completed.log` (grep "Copied|Moved"), `rc.log`. ### Production sync pairs (per site; ORG/SITENAME templated) | # | Dir | What | |---|-----|------| | 1 | up | anomaly images → `Saudi_Video_Sync/SeekRight/Anomaly///TEST` | | 2 | down | master sheets ← `ClientSync/ALGORITHM_DEPLOYMENT/Master_Sheets//` | | 3 | down | yolov8 weights ← `ClientSync/ALGORITHM_DEPLOYMENT/Models/yolov8` | | 4 | down | NIGHT weights ← `ClientSync/ALGORITHM_DEPLOYMENT/Models/NIGHT` | | 5 | up | result CSVs → `ClientSync/ALGORITHM_DEPLOYMENT/csv_files//` | **Monitoring focus = pairs 1 & 5** (client-produced files that must reach the NAS). ### Known production issues (verified, mention when relevant) 1. `RC_PASSWORD` never set in prod compose → **unauthenticated RC API** on host network :5572 (full read/write to NAS + local disk from the LAN). 2. Plaintext creds in compose + git token in `generate_compose.sh` → flagged for rotation. 3. `--size-only` + `copy` (not `sync`): same-size edits never re-transfer; deletes never propagate. "Synced" is weaker than it sounds. 4. **The built-in Web GUI shows nothing useful** — see below. ## KEY FINDING: rclone's built-in UI exists but can't see the syncs `rclone rcd --rc-web-gui` serves a React GUI on :5572 (login admin/$RC_PASSWORD). **Verified locally: while 5 pairs were actively copying, `POST /core/stats` returned all zeros.** The `rclone copy` commands are separate OS processes; the rcd daemon only reports jobs started through its own API. So the prod GUI is decorative. Options for the real project: - (A) Ship our own agent that tails logs + POSTs to a central collector (recommended; zero change to sync behavior), or - (B) Rewrite sync.sh to submit jobs via `rclone rc sync/copy _async=true` so rcd/GUI sees them — bigger change, still no fleet view, history, or alerts. - Adding `--use-json-log` to the rclone copy invocation would make log parsing trivial (structured JSON per line) — one-flag change worth proposing. ## This repo = the production monitoring stack ``` docker-compose.yml mongo + collector — the permanent stack (port 4400) monitor/collector/ FastAPI + pymongo + rclone; serves API and the built UI monitor/agent/ stdlib-only sidecar deployed on each client machine frontend/ React 18 + Vite dashboard → dist/ bind-mounted into collector sync-container/ byte-identical prod sync.sh + Dockerfile (reference + used by deploy/remote-machine builds) — never modify deploy/ install bundles: agent-only/ (machine already runs prod sync) and remote-machine/ (needs sync + agent) send-to-friend/ agent-deploy.zip — the bundle actually on JAGAN-TEST-01 .backups/ pre-v2 versions of main.py/App.jsx/styles.css/.env ``` **The simulation lab was REMOVED 2026-08-10** (user decision: not needed for production). `docker-compose.sim.yml`, `generator/`, `lan-forward.ps1` are in git history; `client/`+`client2/`+`server-data/` were gitignored fake data and are gone for good. Consequence: no local test fleet — alert paths can now only be exercised against a real machine. The 2026-08-05/06 sections below describe that lab; keep them as history, don't try to run them. ### Run / operate ```bash cd /opt/R-clone/R-clone-setup sudo docker-compose up -d --build # mongo + collector (permanent stack) sudo docker-compose logs -f collector sudo docker-compose down ``` **This host has docker-compose v1 (the hyphenated binary) only** — the v2 `docker compose` plugin is NOT installed and fails with `unknown shorthand flag: 'd'`. Always write `docker-compose`. ### Environment (Linux Mint host — MOVED off Windows/WSL 2026-08-07) The project now lives at **`/opt/R-clone/R-clone-setup`** on a Linux Mint box (`vm3@vm3-mint`). Every `/mnt/c/...` path and WSL workaround in older notes is dead — including `RCLONE_SFTP_SET_MODTIME=false`, which only existed because `/mnt/c` rejects `utimes()`. Current realities: - **Repo was root-owned**; `chown -R vm3:vm3 /opt/R-clone` ran 2026-08-07. Without it, no edits, no `npm run build`, and git throws "dubious ownership". - **docker needs `sudo`** — `vm3` is not in the `docker` group. - **Ports 4000 and 8000 on this host are already taken** by unrelated apps (a Node app in `~/seekright-video-calendar`, and another FastAPI service). That is why the collector moved to **4400**. Don't move it back. - **Backend vs frontend rebuild rules differ**: `main.py` is `COPY`d into the image → a change needs `sudo docker-compose up -d --build collector`. `frontend/dist` is bind-mounted → `npm run build` alone is enough, no restart. - **Collector host LAN IP `192.168.1.201/22`** (so the /22 spans .0–.3.255). It is DHCP — it already changed once and broke every remote agent. Reserve it. ## Verified: RC API gives every UI metric (2026-08-05 experiment) `monitor/watch_batch.py` submitted 30×5MB files via `POST /sync/copy _async=true` (src `./client/batch_test`, throttled with `core/bwlimit`) and polled live: - **scheduled** = `core/stats.totalTransfers` · **done** = `.transfers` - **moving now** = `.transferring[]` (name, %, speed, per-file eta; max 4 shown = `--transfers`) - **queued** = totalTransfers − transfers − len(transferring) - **overall ETA/speed/bytes** = `.eta`, `.speed`, `.bytes/.totalBytes` - **per-file duration** = `core/transferred[]` `completed_at − started_at` (+ error field) All 30 arrived; per-file report printed. Gotchas: per-file `eta` can be `None` early; totalTransfers grows during discovery (not instantly 30); always pass `{"group": "job/"}` to scope stats to one job. **Constraint**: RC only sees jobs submitted via RC. The prod loop's `rclone copy` processes are invisible to it → for live per-file UI data in prod, sync.sh v2 should submit pairs via `rclone rc sync/copy _async=true` + poll, instead of spawning `rclone copy`. Log parsing alone gives only aggregate 5s stats lines + completion events. ## Monitoring stack v1 — BUILT and running (2026-08-06) `agent (per machine) → collector (FastAPI+Mongo) → React dashboard`, all in compose: - `monitor/agent/agent.py` — stdlib-only sidecar; tails `/logs/sync.log` (ro mount), regex-parses rounds/pairs/files/errors/progress, POSTs `/api/ingest` every 5s (empty POST = heartbeat). Works against byte-identical prod sync.sh. - `monitor/collector/main.py` — FastAPI + pymongo (**tz_aware=True — naive-vs-aware datetime bug otherwise**). Collections: machines, pairs, events. Endpoints: POST `/api/ingest`, GET `/api/overview` (fleet + computed alerts), GET `/api/machines/{id}/events?type=&limit=`. Serves React build from `/app/static`. Alert rules: offline (no heartbeat >30s, critical), pair last_status=fail (serious), errors_1h>0 (warning). - `frontend/` — React 18 + Vite. Fleet cards (status badge, stat tiles, pair rows, live progress bar), alerts bar, click card → events panel with tabs (Files/Errors/Rounds/Pair results). Polls every 4s. Dark theme, dataviz status tokens (#0ca30c/#fab219/#ec835a/#d03b3b), icon+label never color alone. Build: `cd frontend && npm run build` (dist/ is bind-mounted into the collector — rebuild frontend = just `npm run build`, no docker rebuild). - UI: **http://localhost:4400** · do not stop/restart containers without asking first. (The RIYADH-01 second-machine demo died with the sim lab.) ## v1.1 — remote machines + RMM surface (2026-08-06) - **Agent auth**: `AGENT_TOKEN` shared secret (in `.env`); collector rejects ingest without `Authorization: Bearer ` (verified 401). Empty token = auth off (never for remote use). - **RMM endpoints**: GET `/api/health` (liveness), GET `/api/alerts` (lightweight polling), CORS enabled (`CORS_ORIGINS` env). Integration points for the user's future RMM: poll alerts, or embed /api/overview data. - **`deploy/remote-machine/`**: self-contained bundle (compose + .env.example) for any external machine (e.g. a friend's system): rclone-sync (1 up pair from the algorithm's OUTPUT_DIR) + agent → user's collector. Remote machine needs a network path to the collector — recommend Tailscale; COLLECTOR_URL then is the tailnet IP. Bundle builds from ../../sync-container and ../../monitor/agent. - Transfer visibility today: completed files ✓ (file_synced events), aggregate live progress per pair ✓ (5s stats → progress bar), per-file in-flight + pending queue ✗ — needs RC-submitted jobs (sync.sh v2, see RC section). ## v1.2 — server-side verification / audit feature (2026-08-06) — WORKING The audit-team answer: "did the file actually reach the Synology?" proven from the collector, no one logs into anything. - Agent `WATCH_DIRS` env: mounts up-pair source dirs ro at the same container paths as sync (`/sources/...`), ships file inventory (name/size/mtime) with every ingest. - Collector: own rclone remote (start.sh writes config; rclone installed via apt in Dockerfile), background thread `rclone lsjson --recursive` on every up-pair remote path every VERIFY_INTERVAL (45s local; use 300s+ in prod). `pending` = local file not in server listing; `missing` = pending with mtime older than MISSING_GRACE_S (600) → serious alert with file names. - UI: 4th tile "pending → server" + amber `N pending` / red `N missing!` chips per pair. Verified live: pending counts rise as generator drops files, fall to 0 after each sync round. - Friend's machine (compose seen 2026-08-06): logs at `/home/testing/JAGAN/Prerequisites/Sync_logs`, SITENAME=TEST, standard 5 pairs. deploy/remote-machine bundle fits as-is; needs Tailscale (or LAN) to reach the collector + AGENT_TOKEN. Nothing needed from his algorithm code — agent reads only Sync_logs + source dirs. ## v2 — upload audit + windowed history (2026-08-07 → 10) — CURRENT Prompted by: "admins audit UPLOADS, not downloads; need pending/in-progress/ completed and processed-counts per hour/day/3d/week/month, ~1 month of data." Researched: rclone's native Prometheus metrics can't help (same single-rcd blindness as the GUI + unscrapeable short-lived syncs); MFT dashboards use the Completed/In-Progress/Queued/Failed taxonomy → adopted. ### Collector additions (`monitor/collector/main.py`) - **Retention**: Mongo TTL index on `events.received_at`, `RETENTION_DAYS` env (30). `_ensure_retention` falls back to `collMod` when the window changes. - **GET /api/stats** — one aggregation pass over 30d computes per-machine AND fleet rollups for windows 1h/24h/3d/7d/30d: uploaded, errors, failures, rounds, bytes, avg_bps. Machines with zero activity still appear (that's the signal admins need). ~15s poll from the UI. - **GET /api/timeseries?range=&machine_id=** — epoch-aligned buckets (5min/1h/3h/6h/1d per range), zero-filled, uploads + errors only. - **GET /api/machines/{id}/events** — added `direction=`, `q=` (regex-escaped search over file/message/pair), `skip/limit` paging, returns `total`. - **DELETE /api/machines/{id}** (agent-token auth) — retire a machine; without it a decommissioned machine alerts "offline" forever. Used to remove the CONNTEST test entry 2026-08-10. - `round_summary` now splits `files_up`/`files_down`/`bytes_up` (download bytes are unknowable — files land outside WATCH_DIRS; never fake them). - `_machine_view`: `up_pairs`/`down_pairs` split, `in_progress`, `failed_up`, `uploads_1h` (direction-filtered). Alerts unchanged EXCEPT download-pair failures now say "download pair failing" (still alerted, not shown as cards). - **Verification defaults to the real NAS**: `DEFAULT_REMOTE` env (compose sets `synoreal`), `VERIFY_ENABLED` true if either SYNOLOGY_HOST or REAL_SYNOLOGY_HOST is set. Old behavior (default `synodrive` = the deleted fake NAS) silently skipped verification for any site not in SITE_REMOTES. ### UI (v2.1 after a misstep — `frontend/src/App.jsx`) First v2 cut replaced the machine cards with range-dependent tiles and dropped download pair rows + the last-round tile. **User: "not understandable, it was meaningful before."** Lesson recorded: the last-round-ago tile and the full pair list ARE the at-a-glance value; window counters reading 0 look like a broken dashboard. Current layout: - **Cards (top, range-INDEPENDENT, restored)**: last round tile (amber when >30min stale), uploaded (1h), queued→NAS (pending), errors (1h), live progress bar, ALL 5 pair rows (up first), missing/pending chips. - **"Upload history" section (below, driven by the range selector)**: KPI row (uploaded / failed+errors / rounds / avg throughput+bytes), uploads-per-bucket bar chart with hover tooltip + "view as table", per-machine comparison table (the per-client speed audit). Chart empty state names the likely cause (stopped sync container) instead of rendering all-zero bars. - Palette validated (dataviz skill): uploads #3987e5 / errors #d03b3b on #1a1a19 pass all six checks; separate stacked plots, never dual-axis. ### Ops facts (2026-08-09/10) - JAGAN-TEST-01's sync container was stopped 2026-08-07 during load-test cleanup and left stopped for 65h — the stalled-round alert caught it; the "empty graph" complaint was exactly this. Agent online ≠ sync running. - His Master_Sheets down pair fails exit 3: remote path `.../Master_Sheets/TEST/TEST` doesn't exist on the NAS (ORG=SITE=TEST substitution) — create the folder or fix his compose vars. - Load-test cleanup order that WORKS: stop sync container → delete local source files → `rclone purge` the NAS folder → start container. NAS-first gets re-uploaded within one round (learned the hard way). - `.env` trimmed to AGENT_TOKEN + REAL_SYNOLOGY_* + ORG/SITENAME. send-to-friend/agent-deploy.zip rebuilt (old zip still had :8000). ## v2.1 — public exposure + one-line installer (2026-08-11) - **Public URL**: https://rclone.seekright.com → Synology reverse proxy → this VM :4400 (same pattern as the RMM hostnames). `PUBLIC_URL` in .env. - **Viewer auth**: HTTP Basic middleware (VIEWER_USER/VIEWER_PASS in .env) gates EVERYTHING except /api/ingest, /api/health, and requests carrying the agent bearer token. Unset VIEWER_PASS = gate off. Browser prompts once. - **One-line installer** (RMM deploy_agent.sh pattern, leak-lesson applied — download is authenticated, live AGENT_TOKEN injected at serve time, split-sentinel guard, exactly-once placeholder check): ``` curl -fsSL -u admin "https://rclone.seekright.com/deploy_agent.sh" -o /tmp/deploy_agent.sh sudo bash /tmp/deploy_agent.sh [SITE] # then rm the file (holds live token) ``` Auto-discovers /logs + upload-source host paths from the machine's rclone-synology-sync container (docker inspect), fetches agent.py from GET /agent.py (single source of truth — no embedded copy to drift), writes /opt/rclone-agent/, replaces any previous rclone-agent container, starts. Re-run with no args = in-place upgrade keeping identity. MACHINE_ID is REQUIRED on fresh installs (no hostname fallback — RMM's stale-entry lesson). - Collector build context is now ./monitor (Dockerfile at collector/Dockerfile) so the image carries agent/agent.py to serve. ## v2.2 — REAL fleet sync.sh has drifted from our copy (found 2026-08-11) First real office machine (GCBOT-1 / IRBLAMBTHAM001, image from the private registry) revealed the production sync.sh is NOT our byte-copy anymore: - **Up-pairs are change-detected**: skipped unless file count or `du -sb` size changed; logged as `SKIP: [↑ UP]