# SeekRight Pulse RMM — Production Deployment Map

This VM (`vm3-mint`, LAN `192.168.1.201`, Tailscale `100.79.183.41`) is the
**production server** for the SeekRight Pulse RMM system. Field agents and the
public dashboard both talk to services running here.

## FIELD-SYSTEM SAFETY IS THE TOP PRIORITY

The agents run on ~26 production toll-plaza machines across India with no easy
physical access. A broken agent rollout can strand the whole fleet. Every
change MUST follow these rules:

1. **Backward compatible always.** Old agents must keep working against a new
   backend, and new agents against an old backend. Additive changes only:
   new heartbeat-response keys (old agents ignore them) and new Optional
   telemetry fields (old backends… see rule 2). Never rename/remove existing
   API fields, endpoints, or whitelist command keys agents depend on.
2. **A telemetry field takes three places**: agent payload (deploy_agent.sh),
   `TelemetryPayload` in central_api_prototype.py, and the UI. Pydantic
   silently DROPS unknown fields — a missing model field fails invisibly.
3. **Agent code is fail-soft.** Anything added to the agent must swallow its
   own errors and never break the heartbeat loop. Follow the existing
   try/except-print pattern. No new inbound ports, no shells for data the
   Python stdlib/psutil can read, outbound connections to our server (plus
   the existing ipify/Rocket.Chat calls) only.
4. **Canary before fleet.** Update NH-8 (or one expendable node) first,
   verify version + token_ok + features on its card, then batch the rest.
   Each update = ~5 min agent downtime + Rocket.Chat down/up alert pair.
5. **Whitelist discipline.** Agents execute ONLY commands in commands.json.
   Keep new commands read-only unless explicitly required; remember the file
   is served to ALL agents (no per-client commands) and read per request.
6. **Token injection is sacred.** The server replaces exactly the assignment
   `AGENT_TOKEN="__AGENT_TOKEN__"` in the served installer (count=1) and the
   installer's sentinel check uses a split placeholder (`"__AGENT_""TOKEN__"`)
   so injection can't rewrite it. Broke once (2026-07-16, agents got empty
   tokens); don't reintroduce.
7. **Never flip strict auth** (`set_strict.py`) until every system that
   matters shows `agent_token_ok: true` — strict mode locks tokenless agents
   out AND blocks them from downloading the installer to fix themselves.
8. **Verify before rollout**: `bash -n deploy_agent.sh`, extract the embedded
   agent (between the heredoc markers) and `ast.parse` it, and exercise new
   functions against a mock where feasible.

## Agent version history

- **legacy / pre-3.2** — no token, no version reporting, old two-call poll
  (`/api/telemetry` + `/api/get-command`). Still on: HYDTOT pair + offline nodes.
- **3.2-auth** — X-Agent-Token auth, single `/api/heartbeat` call, sends
  cpu_temp + agent_version. Deployed fleet-wide 2026-07-16.
- **3.4-net** (current template, NOT yet fleet-deployed) — dashboard-configured
  SHIFT path for video fetch, SHIFT folder search, 6-hourly link-speed probe
  vs our own server, local interface IPs in every heartbeat (`import socket`
  is stdlib IP discovery — NOT a websocket; no packets sent, nothing listens).

## Architecture

```
Internet
  │
  ▼
Synology reverse proxy (takeleapindia.synology.me / 106.51.70.111)
  ├── https://rmm.seekright.com          → this VM : vite port (currently 4173)
  └── https://rmm-backend.seekright.com  → this VM : 8000
                                                │
Field agents (X-Agent-Token) ───────────────────┤
                                                ▼
                        FastAPI backend (uvicorn :8000, 4 workers)
                                                │
                                MongoDB localhost:27017, db `rmm_db`
```

## Components

### Backend — `/opt/rmm-backend`
- FastAPI app `central_api_prototype.py`, run by **systemd `rmm-backend.service`**
  (User=root): `venv/bin/uvicorn central_api_prototype:app --host 0.0.0.0 --port 8000 --workers 4`
- Env: systemd `EnvironmentFile=/opt/rmm-backend/.env` injects vars **at service
  start**. The `APP_ENV` / `.env.production` / `.env.development` logic in the
  code is mostly vestigial here: values from `.env` are already in the process
  environment and `load_dotenv` never overrides them. **Editing `.env` requires
  a service restart to take effect.**
- Data: MongoDB `rmm_db` (`clients`, `logs`, `config` collections). Alerts go to
  Rocket.Chat via `ROCKETCHAT_WEBHOOK_URL`.
- `commands.json` = whitelist of commands agents may run. Read from disk **per
  request** (no restart needed). Also editable live from the dashboard config
  editor (`/api` load/save endpoints) — meaning production edits land in the
  working tree and will conflict with `git pull` (see gotchas).
- Agent auth: agents send `X-Agent-Token` (value `AGENT_TOKEN` in `.env`). Mode
  lives in Mongo `config._id=agent_auth`: `grace` (default — tokenless legacy
  agents still accepted) or `strict` (reject without token). Flip with
  `set_strict.py` only after every field agent has been updated
  (`deploy_agent.sh` / the `update_agent` command).
- Dashboard login: `POST /api/login`, credentials `DASHBOARD_USERNAME` /
  `DASHBOARD_PASSWORD` from `.env`, JWT signed with `JWT_SECRET_KEY`.

### Frontend — `/opt/rmm-ui`
- React + Vite, run by **systemd `rmm-frontend.service`** (User=root):
  `npm run dev -- --mode production` (a vite **dev server**, not a static build).
- Port is set in `vite.config.js` (`server.port`, currently **4173** — must
  match the Synology reverse-proxy upstream);
  `allowedHosts: ["rmm.seekright.com"]` must include the public hostname or
  vite rejects proxied requests.
- Vite watches files: pulled code changes hot-reload automatically, and a
  `vite.config.js` change makes vite restart itself — BUT an in-process
  restart keeps whatever port vite already bound (including an auto-bumped
  one like 4174 after a port collision). To change ports for real:
  `sudo systemctl restart rmm-frontend.service`.
- API base URL comes from `/opt/rmm-ui/.env.production`
  (`VITE_API_BASE_URL=https://rmm-backend.seekright.com`).

### Reverse proxy — Synology (NOT on this VM)
- No nginx/apache runs on this VM. TLS + routing for both public hostnames is
  done on the Synology at `takeleapindia.synology.me` (106.51.70.111).
- **If the vite port changes in `vite.config.js`, the Synology reverse-proxy
  upstream port must be updated to match** — otherwise the site 502s.
  This exact thing happened 2026-07-16 (commit `dc604a4` moved 4173 → 6173).

### Field agents (remote machines)
- Run `/opt/seekright-agent/client_agent_prototype.py` on each monitored node;
  poll `GET /api/get-command` and push `POST /api/telemetry`.
- Installed/updated via `deploy_agent.sh` (served by the backend; in strict
  mode downloading it requires a valid agent token).

## Agent token rotation (and why the download is authenticated)

The served installer has the LIVE agent token injected into it. Until
2026-08-03 that download was only gated in strict mode, so in grace mode
`https://rmm-backend.seekright.com/deploy_agent.sh` handed the fleet token to
anyone on the internet. It is now authenticated in EVERY mode (agent token via
`X-Agent-Token` header or `?token=`, or a logged-in dashboard session).

To rotate the token:
1. `.env`: move the current value to `AGENT_TOKEN_PREVIOUS=`, set a new
   `AGENT_TOKEN=` (`python3 -c "import secrets; print(secrets.token_urlsafe(32))"`).
   Both are accepted while both are set — agents authenticate the self-update
   with the old token and receive the new one.
2. Restart the backend, then run `update_agent` across the fleet.
3. When every active node shows `agent_token_ok: true` AND is on the new token,
   delete `AGENT_TOKEN_PREVIOUS` from `.env` and restart. Only then flip strict
   (`set_strict.py`) — that is what actually closes the agent endpoints.

Note: grace mode means the agent endpoints accept UNAUTHENTICATED requests
today. Rotating the token stops the leak, but strict mode is what enforces it.

## Installing the agent on a NEW system

```bash
curl -fsSL "https://rmm-backend.seekright.com/deploy_agent.sh" -o /tmp/deploy_agent.sh
sudo bash /tmp/deploy_agent.sh <CLIENT_ID>        # e.g. KRBOT-Narwana
```

- ALWAYS pass a CLIENT_ID on a fresh install — the fallback is the machine's
  hostname (that's how the stale `hamsadmin-MS-7E07` client entry happened).
- ALWAYS download from the server, never copy deploy_agent.sh out of the repo:
  the server injects the live agent token at download time.
- Re-running with no args is a safe in-place upgrade (keeps the node's
  identity) — the `update_agent` fleet command relies on this.
- In strict auth mode the download itself needs the token:
  append `?token=<AGENT_TOKEN from .env>` to the URL.
- After install: the node appears in the dashboard in ~30s; set its SHIFT
  folder path in the UI card so video fetch/search works.

## How to deploy

Backend:
```bash
cd /opt/rmm-backend
git status                      # working tree should be clean — see gotchas
git pull
sudo systemctl restart rmm-backend.service
systemctl status rmm-backend.service --no-pager -n 20   # check for crash loop
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8000/docs  # expect 200
```

Frontend:
```bash
cd /opt/rmm-ui
git pull                        # vite hot-reloads; restarts itself on config change
# only if deps changed: npm install && sudo systemctl restart rmm-frontend.service
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: rmm.seekright.com' http://localhost:4173/
curl -sk -o /dev/null -w '%{http_code}\n' https://rmm.seekright.com/   # via Synology
```

## Gotchas / history

- **Don't hand-edit tracked files on this server** (`commands.json`, `.env`
  are tracked in git). Local edits block `git pull`. Commit changes to the repo
  instead. Dashboard edits to `commands.json` also dirty the tree — push them
  upstream after changing them.
- 2026-07-16: pull of `dc604a4` changed the vite port 4173 → 6173 and broke
  https://rmm.seekright.com (Synology still forwarded to 4173). Resolved by
  reverting `vite.config.js` to port 4173 in the working tree. That revert is
  a LOCAL change to a tracked file — commit and push it (or the next pull
  will conflict and re-break the site by pulling 6173 back in).
- The backend service must be restarted after every backend pull — uvicorn has
  no auto-reload in the unit. A long-running process can silently serve
  weeks-old code (happened Jul 6 → Jul 16).
- Adding a telemetry field takes THREE places: agent payload (in
  `deploy_agent.sh`), `TelemetryPayload` in `central_api_prototype.py`, and the
  UI. Pydantic silently DROPS any field missing from `TelemetryPayload` —
  agents sent `cpu_temp` for a while with the backend discarding it and no
  error anywhere (found 2026-07-16).
- Ports on this VM: 8000 backend, 6173 rmm-ui vite, 5173 auditor-portal vite
  (localhost only), 7514 + others auditor portal, 4430–4433 MeshCentral,
  Grafana also runs here. Check `ss -tlnp` before assigning a new port.

- 2026-07-23 INCIDENT: the server's own disk hit 100% → mongod fatally aborted
  ("Writing to log file failed") and stayed down ~22h (no Restart= in its unit)
  → dashboard showed zero systems while agents kept heartbeating into a dead DB.
  Mitigations now in code: per-heartbeat telemetry stdout dumps are opt-in
  (`VERBOSE_TELEMETRY=true` in .env to re-enable) and the backend Rocket.Chat
  alerts when its own `/` passes 85%. Recommended systemd hardening: a mongod
  override with `Restart=on-failure`, and `SystemMaxUse=2G` in journald.conf.

## Other services on this VM (not part of RMM deploys)

- `meshcentral.service` — MeshCentral remote management (node, ports 4430–4433)
- `grafana` — monitoring dashboards
- `/opt/auditor-portal` — separate project (its own vite/node dev servers)