rmm-backend/CLAUDE.md

SeekRight Pulse RMM — Production Deployment Map

This VM (vm3-mint, LAN 192.168.1.201, Tailscale 100.79.183.41) is the production server for the SeekRight Pulse RMM system. Field agents and the public dashboard both talk to services running here.

FIELD-SYSTEM SAFETY IS THE TOP PRIORITY

The agents run on ~26 production toll-plaza machines across India with no easy physical access. A broken agent rollout can strand the whole fleet. Every change MUST follow these rules:

1. Backward compatible always. Old agents must keep working against a new backend, and new agents against an old backend. Additive changes only: new heartbeat-response keys (old agents ignore them) and new Optional telemetry fields (old backends… see rule 2). Never rename/remove existing API fields, endpoints, or whitelist command keys agents depend on.
2. A telemetry field takes three places: agent payload (deploy_agent.sh), TelemetryPayload in central_api_prototype.py, and the UI. Pydantic silently DROPS unknown fields — a missing model field fails invisibly.
3. Agent code is fail-soft. Anything added to the agent must swallow its own errors and never break the heartbeat loop. Follow the existing try/except-print pattern. No new inbound ports, no shells for data the Python stdlib/psutil can read, outbound connections to our server (plus the existing ipify/Rocket.Chat calls) only.
4. Canary before fleet. Update NH-8 (or one expendable node) first, verify version + token_ok + features on its card, then batch the rest. Each update = ~5 min agent downtime + Rocket.Chat down/up alert pair.
5. Whitelist discipline. Agents execute ONLY commands in commands.json. Keep new commands read-only unless explicitly required; remember the file is served to ALL agents (no per-client commands) and read per request.
6. Token injection is sacred. The server replaces exactly the assignment AGENT_TOKEN="__AGENT_TOKEN__" in the served installer (count=1) and the installer's sentinel check uses a split placeholder ("__AGENT_""TOKEN__") so injection can't rewrite it. Broke once (2026-07-16, agents got empty tokens); don't reintroduce.
7. Never flip strict auth (set_strict.py) until every system that matters shows agent_token_ok: true — strict mode locks tokenless agents out AND blocks them from downloading the installer to fix themselves.
8. Verify before rollout: bash -n deploy_agent.sh, extract the embedded agent (between the heredoc markers) and ast.parse it, and exercise new functions against a mock where feasible.

Agent version history

• legacy / pre-3.2 — no token, no version reporting, old two-call poll (/api/telemetry + /api/get-command). Still on: HYDTOT pair + offline nodes.
• 3.2-auth — X-Agent-Token auth, single /api/heartbeat call, sends cpu_temp + agent_version. Deployed fleet-wide 2026-07-16.
• 3.4-net (current template, NOT yet fleet-deployed) — dashboard-configured SHIFT path for video fetch, SHIFT folder search, 6-hourly link-speed probe vs our own server, local interface IPs in every heartbeat (import socket is stdlib IP discovery — NOT a websocket; no packets sent, nothing listens).

Architecture

Internet
  │
  ▼
Synology reverse proxy (takeleapindia.synology.me / 106.51.70.111)
  ├── https://rmm.seekright.com          → this VM : vite port (currently 4173)
  └── https://rmm-backend.seekright.com  → this VM : 8000
                                                │
Field agents (X-Agent-Token) ───────────────────┤
                                                ▼
                        FastAPI backend (uvicorn :8000, 4 workers)
                                                │
                                MongoDB localhost:27017, db `rmm_db`

Components

Backend — /opt/rmm-backend

• FastAPI app central_api_prototype.py, run by systemd rmm-backend.service (User=root): venv/bin/uvicorn central_api_prototype:app --host 0.0.0.0 --port 8000 --workers 4
• Env: systemd EnvironmentFile=/opt/rmm-backend/.env injects vars at service start. The APP_ENV / .env.production / .env.development logic in the code is mostly vestigial here: values from .env are already in the process environment and load_dotenv never overrides them. Editing .env requires a service restart to take effect.
• Data: MongoDB rmm_db (clients, logs, config collections). Alerts go to Rocket.Chat via ROCKETCHAT_WEBHOOK_URL.
• commands.json = whitelist of commands agents may run. Read from disk per request (no restart needed). Also editable live from the dashboard config editor (/api load/save endpoints) — meaning production edits land in the working tree and will conflict with git pull (see gotchas).
• Agent auth: agents send X-Agent-Token (value AGENT_TOKEN in .env). Mode lives in Mongo config._id=agent_auth: grace (default — tokenless legacy agents still accepted) or strict (reject without token). Flip with set_strict.py only after every field agent has been updated (deploy_agent.sh / the update_agent command).
• Dashboard login: POST /api/login, credentials DASHBOARD_USERNAME / DASHBOARD_PASSWORD from .env, JWT signed with JWT_SECRET_KEY.

Frontend — /opt/rmm-ui

• React + Vite, run by systemd rmm-frontend.service (User=root): npm run dev -- --mode production (a vite dev server, not a static build).
• Port is set in vite.config.js (server.port, currently 4173 — must match the Synology reverse-proxy upstream); allowedHosts: ["rmm.seekright.com"] must include the public hostname or vite rejects proxied requests.
• Vite watches files: pulled code changes hot-reload automatically, and a vite.config.js change makes vite restart itself — BUT an in-process restart keeps whatever port vite already bound (including an auto-bumped one like 4174 after a port collision). To change ports for real: sudo systemctl restart rmm-frontend.service.
• API base URL comes from /opt/rmm-ui/.env.production (VITE_API_BASE_URL=https://rmm-backend.seekright.com).

Reverse proxy — Synology (NOT on this VM)

• No nginx/apache runs on this VM. TLS + routing for both public hostnames is done on the Synology at takeleapindia.synology.me (106.51.70.111).
• If the vite port changes in vite.config.js, the Synology reverse-proxy upstream port must be updated to match — otherwise the site 502s. This exact thing happened 2026-07-16 (commit dc604a4 moved 4173 → 6173).

Field agents (remote machines)

• Run /opt/seekright-agent/client_agent_prototype.py on each monitored node; poll GET /api/get-command and push POST /api/telemetry.
• Installed/updated via deploy_agent.sh (served by the backend; in strict mode downloading it requires a valid agent token).

Agent token rotation (and why the download is authenticated)

The served installer has the LIVE agent token injected into it. Until 2026-08-03 that download was only gated in strict mode, so in grace mode https://rmm-backend.seekright.com/deploy_agent.sh handed the fleet token to anyone on the internet. It is now authenticated in EVERY mode (agent token via X-Agent-Token header or ?token=, or a logged-in dashboard session).

To rotate the token:

1. .env: move the current value to AGENT_TOKEN_PREVIOUS=, set a new AGENT_TOKEN= (python3 -c "import secrets; print(secrets.token_urlsafe(32))"). Both are accepted while both are set — agents authenticate the self-update with the old token and receive the new one.
2. Restart the backend, then run update_agent across the fleet.
3. When every active node shows agent_token_ok: true AND is on the new token, delete AGENT_TOKEN_PREVIOUS from .env and restart. Only then flip strict (set_strict.py) — that is what actually closes the agent endpoints.

Note: grace mode means the agent endpoints accept UNAUTHENTICATED requests today. Rotating the token stops the leak, but strict mode is what enforces it.

Installing the agent on a NEW system

curl -fsSL "https://rmm-backend.seekright.com/deploy_agent.sh" -o /tmp/deploy_agent.sh
sudo bash /tmp/deploy_agent.sh <CLIENT_ID>        # e.g. KRBOT-Narwana

• ALWAYS pass a CLIENT_ID on a fresh install — the fallback is the machine's hostname (that's how the stale hamsadmin-MS-7E07 client entry happened).
• ALWAYS download from the server, never copy deploy_agent.sh out of the repo: the server injects the live agent token at download time.
• Re-running with no args is a safe in-place upgrade (keeps the node's identity) — the update_agent fleet command relies on this.
• In strict auth mode the download itself needs the token: append ?token=<AGENT_TOKEN from .env> to the URL.
• After install: the node appears in the dashboard in ~30s; set its SHIFT folder path in the UI card so video fetch/search works.

How to deploy

Backend:

cd /opt/rmm-backend
git status                      # working tree should be clean — see gotchas
git pull
sudo systemctl restart rmm-backend.service
systemctl status rmm-backend.service --no-pager -n 20   # check for crash loop
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8000/docs  # expect 200

Frontend:

cd /opt/rmm-ui
git pull                        # vite hot-reloads; restarts itself on config change
# only if deps changed: npm install && sudo systemctl restart rmm-frontend.service
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: rmm.seekright.com' http://localhost:4173/
curl -sk -o /dev/null -w '%{http_code}\n' https://rmm.seekright.com/   # via Synology

Gotchas / history

• Don't hand-edit tracked files on this server (commands.json, .env are tracked in git). Local edits block git pull. Commit changes to the repo instead. Dashboard edits to commands.json also dirty the tree — push them upstream after changing them.

• 2026-07-16: pull of dc604a4 changed the vite port 4173 → 6173 and broke https://rmm.seekright.com (Synology still forwarded to 4173). Resolved by reverting vite.config.js to port 4173 in the working tree. That revert is a LOCAL change to a tracked file — commit and push it (or the next pull will conflict and re-break the site by pulling 6173 back in).

• The backend service must be restarted after every backend pull — uvicorn has no auto-reload in the unit. A long-running process can silently serve weeks-old code (happened Jul 6 → Jul 16).

• Adding a telemetry field takes THREE places: agent payload (in deploy_agent.sh), TelemetryPayload in central_api_prototype.py, and the UI. Pydantic silently DROPS any field missing from TelemetryPayload — agents sent cpu_temp for a while with the backend discarding it and no error anywhere (found 2026-07-16).

• Ports on this VM: 8000 backend, 6173 rmm-ui vite, 5173 auditor-portal vite (localhost only), 7514 + others auditor portal, 4430–4433 MeshCentral, Grafana also runs here. Check ss -tlnp before assigning a new port.

• 2026-07-23 INCIDENT: the server's own disk hit 100% → mongod fatally aborted ("Writing to log file failed") and stayed down ~22h (no Restart= in its unit) → dashboard showed zero systems while agents kept heartbeating into a dead DB. Mitigations now in code: per-heartbeat telemetry stdout dumps are opt-in (VERBOSE_TELEMETRY=true in .env to re-enable) and the backend Rocket.Chat alerts when its own / passes 85%. Recommended systemd hardening: a mongod override with Restart=on-failure, and SystemMaxUse=2G in journald.conf.

Other services on this VM (not part of RMM deploys)

• meshcentral.service — MeshCentral remote management (node, ports 4430–4433)
• grafana — monitoring dashboards
• /opt/auditor-portal — separate project (its own vite/node dev servers)