fix: authenticate installer download, rotate-capable agent token, thermal alerts, vitals history

- deploy_agent.sh download now requires auth in EVERY mode (agent token via
  header/?token=, or a dashboard session). It was gated on strict mode only,
  so in grace mode the served installer published the live fleet token to the
  internet
- accept AGENT_TOKEN_PREVIOUS alongside AGENT_TOKEN so a rotation can roll
  through the fleet; record agent_token_current per node to know when the
  previous token can be dropped
- /api/agent-token (dashboard-auth) so the UI can build the install command
- installer: fall back to the default server URL when an inherited one is
  unreachable (stale Tailscale address stranded a fresh install), and derive
  the version banner from the embedded agent instead of a hardcoded string
- CPU thermal alerts to Rocket.Chat at 85C with hysteresis clearing at 75C
- telemetry_history: 1-minute vitals samples, 7-day TTL (~37 MB fleet-wide),
  plus /api/history and /api/history-bulk for dashboard sparklines
- server watches its own disk (85%) after the 2026-07-23 full-disk outage that
  killed mongod; per-heartbeat telemetry logging now opt-in via VERBOSE_TELEMETRY
- /api/logs returns the newest 25 slim entries per client instead of the full
  history (2.2 MB every 3s was most of the server's egress); gzip middleware
- agent speed probe right-sized to 4/2 MB once a day
- server file listing/delete endpoints, search clearing, WAN IP capture

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CLAUDE.md

@@ -4,6 +4,54 @@ This VM (`vm3-mint`, LAN `192.168.1.201`, Tailscale `100.79.183.41`) is the
 **production server** for the SeekRight Pulse RMM system. Field agents and the
 public dashboard both talk to services running here.
 
+## FIELD-SYSTEM SAFETY IS THE TOP PRIORITY
+
+The agents run on ~26 production toll-plaza machines across India with no easy
+physical access. A broken agent rollout can strand the whole fleet. Every
+change MUST follow these rules:
+
+1. **Backward compatible always.** Old agents must keep working against a new
+   backend, and new agents against an old backend. Additive changes only:
+   new heartbeat-response keys (old agents ignore them) and new Optional
+   telemetry fields (old backends… see rule 2). Never rename/remove existing
+   API fields, endpoints, or whitelist command keys agents depend on.
+2. **A telemetry field takes three places**: agent payload (deploy_agent.sh),
+   `TelemetryPayload` in central_api_prototype.py, and the UI. Pydantic
+   silently DROPS unknown fields — a missing model field fails invisibly.
+3. **Agent code is fail-soft.** Anything added to the agent must swallow its
+   own errors and never break the heartbeat loop. Follow the existing
+   try/except-print pattern. No new inbound ports, no shells for data the
+   Python stdlib/psutil can read, outbound connections to our server (plus
+   the existing ipify/Rocket.Chat calls) only.
+4. **Canary before fleet.** Update NH-8 (or one expendable node) first,
+   verify version + token_ok + features on its card, then batch the rest.
+   Each update = ~5 min agent downtime + Rocket.Chat down/up alert pair.
+5. **Whitelist discipline.** Agents execute ONLY commands in commands.json.
+   Keep new commands read-only unless explicitly required; remember the file
+   is served to ALL agents (no per-client commands) and read per request.
+6. **Token injection is sacred.** The server replaces exactly the assignment
+   `AGENT_TOKEN="__AGENT_TOKEN__"` in the served installer (count=1) and the
+   installer's sentinel check uses a split placeholder (`"__AGENT_""TOKEN__"`)
+   so injection can't rewrite it. Broke once (2026-07-16, agents got empty
+   tokens); don't reintroduce.
+7. **Never flip strict auth** (`set_strict.py`) until every system that
+   matters shows `agent_token_ok: true` — strict mode locks tokenless agents
+   out AND blocks them from downloading the installer to fix themselves.
+8. **Verify before rollout**: `bash -n deploy_agent.sh`, extract the embedded
+   agent (between the heredoc markers) and `ast.parse` it, and exercise new
+   functions against a mock where feasible.
+
+## Agent version history
+
+- **legacy / pre-3.2** — no token, no version reporting, old two-call poll
+  (`/api/telemetry` + `/api/get-command`). Still on: HYDTOT pair + offline nodes.
+- **3.2-auth** — X-Agent-Token auth, single `/api/heartbeat` call, sends
+  cpu_temp + agent_version. Deployed fleet-wide 2026-07-16.
+- **3.4-net** (current template, NOT yet fleet-deployed) — dashboard-configured
+  SHIFT path for video fetch, SHIFT folder search, 6-hourly link-speed probe
+  vs our own server, local interface IPs in every heartbeat (`import socket`
+  is stdlib IP discovery — NOT a websocket; no packets sent, nothing listens).
+
 ## Architecture
 
 ```
@@ -73,6 +121,45 @@ Field agents (X-Agent-Token) ─────────────────
 - Installed/updated via `deploy_agent.sh` (served by the backend; in strict
   mode downloading it requires a valid agent token).
 
+## Agent token rotation (and why the download is authenticated)
+
+The served installer has the LIVE agent token injected into it. Until
+2026-08-03 that download was only gated in strict mode, so in grace mode
+`https://rmm-backend.seekright.com/deploy_agent.sh` handed the fleet token to
+anyone on the internet. It is now authenticated in EVERY mode (agent token via
+`X-Agent-Token` header or `?token=`, or a logged-in dashboard session).
+
+To rotate the token:
+1. `.env`: move the current value to `AGENT_TOKEN_PREVIOUS=`, set a new
+   `AGENT_TOKEN=` (`python3 -c "import secrets; print(secrets.token_urlsafe(32))"`).
+   Both are accepted while both are set — agents authenticate the self-update
+   with the old token and receive the new one.
+2. Restart the backend, then run `update_agent` across the fleet.
+3. When every active node shows `agent_token_ok: true` AND is on the new token,
+   delete `AGENT_TOKEN_PREVIOUS` from `.env` and restart. Only then flip strict
+   (`set_strict.py`) — that is what actually closes the agent endpoints.
+
+Note: grace mode means the agent endpoints accept UNAUTHENTICATED requests
+today. Rotating the token stops the leak, but strict mode is what enforces it.
+
+## Installing the agent on a NEW system
+
+```bash
+curl -fsSL "https://rmm-backend.seekright.com/deploy_agent.sh" -o /tmp/deploy_agent.sh
+sudo bash /tmp/deploy_agent.sh <CLIENT_ID>        # e.g. KRBOT-Narwana
+```
+
+- ALWAYS pass a CLIENT_ID on a fresh install — the fallback is the machine's
+  hostname (that's how the stale `hamsadmin-MS-7E07` client entry happened).
+- ALWAYS download from the server, never copy deploy_agent.sh out of the repo:
+  the server injects the live agent token at download time.
+- Re-running with no args is a safe in-place upgrade (keeps the node's
+  identity) — the `update_agent` fleet command relies on this.
+- In strict auth mode the download itself needs the token:
+  append `?token=<AGENT_TOKEN from .env>` to the URL.
+- After install: the node appears in the dashboard in ~30s; set its SHIFT
+  folder path in the UI card so video fetch/search works.
+
 ## How to deploy
 
 Backend:
@@ -117,6 +204,14 @@ curl -sk -o /dev/null -w '%{http_code}\n' https://rmm.seekright.com/   # via Syn
   (localhost only), 7514 + others auditor portal, 4430–4433 MeshCentral,
   Grafana also runs here. Check `ss -tlnp` before assigning a new port.
 
+- 2026-07-23 INCIDENT: the server's own disk hit 100% → mongod fatally aborted
+  ("Writing to log file failed") and stayed down ~22h (no Restart= in its unit)
+  → dashboard showed zero systems while agents kept heartbeating into a dead DB.
+  Mitigations now in code: per-heartbeat telemetry stdout dumps are opt-in
+  (`VERBOSE_TELEMETRY=true` in .env to re-enable) and the backend Rocket.Chat
+  alerts when its own `/` passes 85%. Recommended systemd hardening: a mongod
+  override with `Restart=on-failure`, and `SystemMaxUse=2G` in journald.conf.
+
 ## Other services on this VM (not part of RMM deploys)
 
 - `meshcentral.service` — MeshCentral remote management (node, ports 4430–4433)