Compare commits
8 Commits
25e9b5e120
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| c929176cfa | |||
| b9aeedc5ef | |||
| 80e1bd8e3a | |||
| 9987b3a45a | |||
| 9baf93bfef | |||
| 6e05180e7c | |||
| 647c6ae7f0 | |||
| 06c8f62ee6 |
9
.env
9
.env
@@ -1,3 +1,12 @@
|
|||||||
ROCKETCHAT_WEBHOOK_URL="https://text.seekright.com/hooks/6a0eb41ea88367bc6e101f0c/gvXfD8zSNRti43kEabqdE9tMCvNi9zAAoGfbao7cxcKbiW6R"
|
ROCKETCHAT_WEBHOOK_URL="https://text.seekright.com/hooks/6a0eb41ea88367bc6e101f0c/gvXfD8zSNRti43kEabqdE9tMCvNi9zAAoGfbao7cxcKbiW6R"
|
||||||
MONGODB_URI="mongodb://localhost:27017"
|
MONGODB_URI="mongodb://localhost:27017"
|
||||||
MONGODB_DB="rmm_db"
|
MONGODB_DB="rmm_db"
|
||||||
|
|
||||||
|
# --- Authentication (added 2026-07-16) ---
|
||||||
|
# Shared secret every agent must present in the X-Agent-Token header.
|
||||||
|
AGENT_TOKEN="170LRiZafLZThrIGQ6VlP8K9FPw7i8InwzM3dVoYnK4"
|
||||||
|
# Signing key for dashboard login tokens.
|
||||||
|
JWT_SECRET_KEY="Gme4fl7mCxTQv9b7H7iahacoEG6j5R4Av7kLAPQpzgVzHjINNScKIDWZ_tJsf93a"
|
||||||
|
# Dashboard login. CHANGE THESE, then restart the backend.
|
||||||
|
DASHBOARD_USERNAME="root"
|
||||||
|
DASHBOARD_PASSWORD="seekright159@"
|
||||||
|
|||||||
@@ -1,3 +1,3 @@
|
|||||||
ROCKETCHAT_WEBHOOK_URL=""
|
ROCKETCHAT_WEBHOOK_URL="https://text.seekright.com/hooks/6a22ad73a88367bc6e1095a2/xcHkbDSce5rCwofdBZF3ri2rb6ntxT2mahqT6Gg7iKdectYk"
|
||||||
MONGODB_URI="mongodb://localhost:27017"
|
MONGODB_URI="mongodb://localhost:27017"
|
||||||
MONGODB_DB="rmm_db_dev"
|
MONGODB_DB="rmm_db_dev"
|
||||||
|
|||||||
1
.gitignore
vendored
1
.gitignore
vendored
@@ -7,3 +7,4 @@ logs.json
|
|||||||
.DS_Store
|
.DS_Store
|
||||||
.vscode/
|
.vscode/
|
||||||
.idea/
|
.idea/
|
||||||
|
file_transfers/
|
||||||
|
|||||||
143
AGENT_DEPLOYMENT_GUIDE.md
Normal file
143
AGENT_DEPLOYMENT_GUIDE.md
Normal file
@@ -0,0 +1,143 @@
|
|||||||
|
# 🛡️ SeekRight RMM Agent — Client Deployment Guide
|
||||||
|
### *AnyDesk Remote Access Workflow*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📋 Prerequisites (before you connect)
|
||||||
|
|
||||||
|
| What you need | Where to get it |
|
||||||
|
|---|---|
|
||||||
|
| AnyDesk installed on **your** machine | anydesk.com |
|
||||||
|
| AnyDesk installed on the **client** machine | Ask client to install from anydesk.com |
|
||||||
|
| Client's AnyDesk ID / address | Ask client to share it with you |
|
||||||
|
| `deploy_agent.sh` file | In your workspace: `RMM-BE/rmm-backend/deploy_agent.sh` |
|
||||||
|
| Client ID to register with | Agreed upon name (e.g. `NH-8`, `BS-13`) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔗 Step 1 — Connect to the Client via AnyDesk
|
||||||
|
|
||||||
|
1. Open **AnyDesk** on your machine.
|
||||||
|
2. Enter the **client's AnyDesk ID** in the address bar.
|
||||||
|
3. Click **Connect** → client approves the session.
|
||||||
|
4. Once inside: right-click the desktop or press `Ctrl+Alt+T` to open a **Terminal**.
|
||||||
|
|
||||||
|
**Tip:** Ask the client to set up **unattended access** in AnyDesk settings so you don't need them to approve each connection.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📤 Step 2 — Transfer the Deploy Script
|
||||||
|
|
||||||
|
### Option A — AnyDesk File Transfer (easiest, recommended)
|
||||||
|
1. In AnyDesk, click the **File Manager** icon in the top toolbar.
|
||||||
|
2. Drag `deploy_agent.sh` from your machine to the **client's Desktop** or `~/Downloads`.
|
||||||
|
|
||||||
|
### Option B — Download directly on the client machine
|
||||||
|
```bash
|
||||||
|
curl -O http://rmm-backend.seekright.com/deploy_agent.sh
|
||||||
|
# OR if curl is not available:
|
||||||
|
wget http://rmm-backend.seekright.com/deploy_agent.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚙️ Step 3 — Run the Deployment Script
|
||||||
|
|
||||||
|
In the client's terminal, navigate to where you placed the script and run it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Make the script executable (only needed the first time)
|
||||||
|
chmod +x deploy_agent.sh
|
||||||
|
|
||||||
|
# Run with the agreed Client ID (replace NH-8 with the actual ID)
|
||||||
|
sudo ./deploy_agent.sh NH-8
|
||||||
|
```
|
||||||
|
|
||||||
|
> **The script MUST be run with `sudo`** — it installs a systemd service and writes to `/opt/seekright-agent/`.
|
||||||
|
|
||||||
|
### What you'll see during deployment:
|
||||||
|
```
|
||||||
|
🚀 Preparing SeekRight RMM Agent installation...
|
||||||
|
📍 Target Client ID: NH-8
|
||||||
|
🌐 Central Server URL: http://rmm-backend.seekright.com
|
||||||
|
🔄 Updating system package indexes...
|
||||||
|
📁 Creating installation workspace...
|
||||||
|
📝 Generating client agent file...
|
||||||
|
⚙️ Configuring background systemd daemon service...
|
||||||
|
🔄 Reloading system daemons and starting agent service...
|
||||||
|
|
||||||
|
✨ ============================================= ✨
|
||||||
|
✅ SeekRight RMM Agent successfully deployed!
|
||||||
|
📍 Location: /opt/seekright-agent/client_agent_prototype.py
|
||||||
|
📋 System Service: seekright-agent.service
|
||||||
|
✨ ============================================= ✨
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ Step 4 — Verify the Agent is Running
|
||||||
|
|
||||||
|
After deployment, the script shows live logs automatically. Confirm you see:
|
||||||
|
|
||||||
|
```
|
||||||
|
seekright-agent[XXXXX]: Starting Agent for NH-8...
|
||||||
|
seekright-agent[XXXXX]: [*] Telemetry pushed successfully.
|
||||||
|
seekright-agent[XXXXX]: [*] Checking for commands...
|
||||||
|
seekright-agent[XXXXX]: -> No pending commands.
|
||||||
|
```
|
||||||
|
|
||||||
|
- ✅ **`Telemetry pushed successfully`** → agent is live and sending data.
|
||||||
|
- ❌ **`SyntaxError` or `exit-code`** → redeploy with the latest version of the script.
|
||||||
|
|
||||||
|
Press `Ctrl+C` to exit the log view. The agent continues running in the background as a system service.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔍 Step 5 — Confirm in the RMM Dashboard
|
||||||
|
|
||||||
|
1. Open the **SeekRight Pulse RMM** dashboard in your browser.
|
||||||
|
2. Within ~30 seconds the new client node appears with a 🟢 **Online** status.
|
||||||
|
3. Verify that disk drives, CPU, and memory telemetry are loading correctly.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🛠️ Useful Troubleshooting Commands (run on the client machine)
|
||||||
|
|
||||||
|
| Task | Command |
|
||||||
|
|---|---|
|
||||||
|
| Check agent status | `sudo systemctl status seekright-agent` |
|
||||||
|
| View live logs | `sudo journalctl -u seekright-agent -f` |
|
||||||
|
| Restart the agent | `sudo systemctl restart seekright-agent` |
|
||||||
|
| Stop the agent | `sudo systemctl stop seekright-agent` |
|
||||||
|
| View installed agent file | `cat /opt/seekright-agent/client_agent_prototype.py` |
|
||||||
|
| Re-run full deployment | `sudo ./deploy_agent.sh <CLIENT_ID>` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🆔 Client ID Naming Convention
|
||||||
|
|
||||||
|
Use a consistent, short identifier. No spaces.
|
||||||
|
|
||||||
|
| ✅ Good | ❌ Avoid |
|
||||||
|
|---|---|
|
||||||
|
| `NH-8` | `New HP Workstation 8` |
|
||||||
|
| `BS-13` | `bs 13 machine` |
|
||||||
|
| `SHOWROOM-PC-01` | `showroom pc` |
|
||||||
|
|
||||||
|
> If no Client ID is provided, the script defaults to the machine's **hostname** (e.g. `nh8gglamsrv01`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔄 Re-deploying / Updating the Agent
|
||||||
|
|
||||||
|
Whenever you push a bug fix or new version of `deploy_agent.sh`, simply re-run on the client:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo ./deploy_agent.sh NH-8
|
||||||
|
```
|
||||||
|
|
||||||
|
The script is **idempotent** — safely overwrites the previous installation and restarts the service.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*SeekRight Pulse RMM · Internal Documentation · v1.0*
|
||||||
124
CLAUDE.md
Normal file
124
CLAUDE.md
Normal file
@@ -0,0 +1,124 @@
|
|||||||
|
# SeekRight Pulse RMM — Production Deployment Map
|
||||||
|
|
||||||
|
This VM (`vm3-mint`, LAN `192.168.1.201`, Tailscale `100.79.183.41`) is the
|
||||||
|
**production server** for the SeekRight Pulse RMM system. Field agents and the
|
||||||
|
public dashboard both talk to services running here.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
Internet
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Synology reverse proxy (takeleapindia.synology.me / 106.51.70.111)
|
||||||
|
├── https://rmm.seekright.com → this VM : vite port (currently 4173)
|
||||||
|
└── https://rmm-backend.seekright.com → this VM : 8000
|
||||||
|
│
|
||||||
|
Field agents (X-Agent-Token) ───────────────────┤
|
||||||
|
▼
|
||||||
|
FastAPI backend (uvicorn :8000, 4 workers)
|
||||||
|
│
|
||||||
|
MongoDB localhost:27017, db `rmm_db`
|
||||||
|
```
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
### Backend — `/opt/rmm-backend`
|
||||||
|
- FastAPI app `central_api_prototype.py`, run by **systemd `rmm-backend.service`**
|
||||||
|
(User=root): `venv/bin/uvicorn central_api_prototype:app --host 0.0.0.0 --port 8000 --workers 4`
|
||||||
|
- Env: systemd `EnvironmentFile=/opt/rmm-backend/.env` injects vars **at service
|
||||||
|
start**. The `APP_ENV` / `.env.production` / `.env.development` logic in the
|
||||||
|
code is mostly vestigial here: values from `.env` are already in the process
|
||||||
|
environment and `load_dotenv` never overrides them. **Editing `.env` requires
|
||||||
|
a service restart to take effect.**
|
||||||
|
- Data: MongoDB `rmm_db` (`clients`, `logs`, `config` collections). Alerts go to
|
||||||
|
Rocket.Chat via `ROCKETCHAT_WEBHOOK_URL`.
|
||||||
|
- `commands.json` = whitelist of commands agents may run. Read from disk **per
|
||||||
|
request** (no restart needed). Also editable live from the dashboard config
|
||||||
|
editor (`/api` load/save endpoints) — meaning production edits land in the
|
||||||
|
working tree and will conflict with `git pull` (see gotchas).
|
||||||
|
- Agent auth: agents send `X-Agent-Token` (value `AGENT_TOKEN` in `.env`). Mode
|
||||||
|
lives in Mongo `config._id=agent_auth`: `grace` (default — tokenless legacy
|
||||||
|
agents still accepted) or `strict` (reject without token). Flip with
|
||||||
|
`set_strict.py` only after every field agent has been updated
|
||||||
|
(`deploy_agent.sh` / the `update_agent` command).
|
||||||
|
- Dashboard login: `POST /api/login`, credentials `DASHBOARD_USERNAME` /
|
||||||
|
`DASHBOARD_PASSWORD` from `.env`, JWT signed with `JWT_SECRET_KEY`.
|
||||||
|
|
||||||
|
### Frontend — `/opt/rmm-ui`
|
||||||
|
- React + Vite, run by **systemd `rmm-frontend.service`** (User=root):
|
||||||
|
`npm run dev -- --mode production` (a vite **dev server**, not a static build).
|
||||||
|
- Port is set in `vite.config.js` (`server.port`, currently **4173** — must
|
||||||
|
match the Synology reverse-proxy upstream);
|
||||||
|
`allowedHosts: ["rmm.seekright.com"]` must include the public hostname or
|
||||||
|
vite rejects proxied requests.
|
||||||
|
- Vite watches files: pulled code changes hot-reload automatically, and a
|
||||||
|
`vite.config.js` change makes vite restart itself — BUT an in-process
|
||||||
|
restart keeps whatever port vite already bound (including an auto-bumped
|
||||||
|
one like 4174 after a port collision). To change ports for real:
|
||||||
|
`sudo systemctl restart rmm-frontend.service`.
|
||||||
|
- API base URL comes from `/opt/rmm-ui/.env.production`
|
||||||
|
(`VITE_API_BASE_URL=https://rmm-backend.seekright.com`).
|
||||||
|
|
||||||
|
### Reverse proxy — Synology (NOT on this VM)
|
||||||
|
- No nginx/apache runs on this VM. TLS + routing for both public hostnames is
|
||||||
|
done on the Synology at `takeleapindia.synology.me` (106.51.70.111).
|
||||||
|
- **If the vite port changes in `vite.config.js`, the Synology reverse-proxy
|
||||||
|
upstream port must be updated to match** — otherwise the site 502s.
|
||||||
|
This exact thing happened 2026-07-16 (commit `dc604a4` moved 4173 → 6173).
|
||||||
|
|
||||||
|
### Field agents (remote machines)
|
||||||
|
- Run `/opt/seekright-agent/client_agent_prototype.py` on each monitored node;
|
||||||
|
poll `GET /api/get-command` and push `POST /api/telemetry`.
|
||||||
|
- Installed/updated via `deploy_agent.sh` (served by the backend; in strict
|
||||||
|
mode downloading it requires a valid agent token).
|
||||||
|
|
||||||
|
## How to deploy
|
||||||
|
|
||||||
|
Backend:
|
||||||
|
```bash
|
||||||
|
cd /opt/rmm-backend
|
||||||
|
git status # working tree should be clean — see gotchas
|
||||||
|
git pull
|
||||||
|
sudo systemctl restart rmm-backend.service
|
||||||
|
systemctl status rmm-backend.service --no-pager -n 20 # check for crash loop
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8000/docs # expect 200
|
||||||
|
```
|
||||||
|
|
||||||
|
Frontend:
|
||||||
|
```bash
|
||||||
|
cd /opt/rmm-ui
|
||||||
|
git pull # vite hot-reloads; restarts itself on config change
|
||||||
|
# only if deps changed: npm install && sudo systemctl restart rmm-frontend.service
|
||||||
|
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: rmm.seekright.com' http://localhost:4173/
|
||||||
|
curl -sk -o /dev/null -w '%{http_code}\n' https://rmm.seekright.com/ # via Synology
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gotchas / history
|
||||||
|
|
||||||
|
- **Don't hand-edit tracked files on this server** (`commands.json`, `.env`
|
||||||
|
are tracked in git). Local edits block `git pull`. Commit changes to the repo
|
||||||
|
instead. Dashboard edits to `commands.json` also dirty the tree — push them
|
||||||
|
upstream after changing them.
|
||||||
|
- 2026-07-16: pull of `dc604a4` changed the vite port 4173 → 6173 and broke
|
||||||
|
https://rmm.seekright.com (Synology still forwarded to 4173). Resolved by
|
||||||
|
reverting `vite.config.js` to port 4173 in the working tree. That revert is
|
||||||
|
a LOCAL change to a tracked file — commit and push it (or the next pull
|
||||||
|
will conflict and re-break the site by pulling 6173 back in).
|
||||||
|
- The backend service must be restarted after every backend pull — uvicorn has
|
||||||
|
no auto-reload in the unit. A long-running process can silently serve
|
||||||
|
weeks-old code (happened Jul 6 → Jul 16).
|
||||||
|
- Adding a telemetry field takes THREE places: agent payload (in
|
||||||
|
`deploy_agent.sh`), `TelemetryPayload` in `central_api_prototype.py`, and the
|
||||||
|
UI. Pydantic silently DROPS any field missing from `TelemetryPayload` —
|
||||||
|
agents sent `cpu_temp` for a while with the backend discarding it and no
|
||||||
|
error anywhere (found 2026-07-16).
|
||||||
|
- Ports on this VM: 8000 backend, 6173 rmm-ui vite, 5173 auditor-portal vite
|
||||||
|
(localhost only), 7514 + others auditor portal, 4430–4433 MeshCentral,
|
||||||
|
Grafana also runs here. Check `ss -tlnp` before assigning a new port.
|
||||||
|
|
||||||
|
## Other services on this VM (not part of RMM deploys)
|
||||||
|
|
||||||
|
- `meshcentral.service` — MeshCentral remote management (node, ports 4430–4433)
|
||||||
|
- `grafana` — monitoring dashboards
|
||||||
|
- `/opt/auditor-portal` — separate project (its own vite/node dev servers)
|
||||||
@@ -11,12 +11,17 @@ else:
|
|||||||
# Fallback to load default .env if any variables are not yet defined
|
# Fallback to load default .env if any variables are not yet defined
|
||||||
load_dotenv()
|
load_dotenv()
|
||||||
|
|
||||||
from fastapi import FastAPI, HTTPException
|
from fastapi import FastAPI, HTTPException, Depends, Header, Request, BackgroundTasks
|
||||||
from fastapi.responses import PlainTextResponse
|
from fastapi.responses import PlainTextResponse
|
||||||
from fastapi.middleware.cors import CORSMiddleware
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
from pydantic import BaseModel
|
from pydantic import BaseModel
|
||||||
from typing import Dict, Any, List
|
from typing import Dict, Any, List, Optional
|
||||||
import uvicorn
|
import uvicorn
|
||||||
|
import base64
|
||||||
|
import hmac
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
app = FastAPI(title="RMM Central API")
|
app = FastAPI(title="RMM Central API")
|
||||||
|
|
||||||
@@ -29,6 +34,58 @@ app.add_middleware(
|
|||||||
allow_headers=["*"],
|
allow_headers=["*"],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# JWT-like HMAC Secure Token Utilities
|
||||||
|
JWT_SECRET_KEY = os.getenv("JWT_SECRET_KEY", "seekright_rmm_secret_key_2026_default")
|
||||||
|
|
||||||
|
def generate_token(username: str) -> str:
|
||||||
|
# Expire in 1 day (24 hours)
|
||||||
|
expiry = (datetime.utcnow() + timedelta(days=1)).isoformat()
|
||||||
|
payload = {
|
||||||
|
"username": username,
|
||||||
|
"expires": expiry
|
||||||
|
}
|
||||||
|
payload_json = json.dumps(payload)
|
||||||
|
payload_b64 = base64.urlsafe_b64encode(payload_json.encode('utf-8')).decode('utf-8').rstrip('=')
|
||||||
|
|
||||||
|
sig = hmac.new(JWT_SECRET_KEY.encode('utf-8'), payload_b64.encode('utf-8'), hashlib.sha256).hexdigest()
|
||||||
|
return f"{payload_b64}.{sig}"
|
||||||
|
|
||||||
|
def verify_token(token: str) -> bool:
|
||||||
|
try:
|
||||||
|
parts = token.split(".")
|
||||||
|
if len(parts) != 2:
|
||||||
|
return False
|
||||||
|
payload_b64, sig = parts
|
||||||
|
|
||||||
|
# Verify signature
|
||||||
|
expected_sig = hmac.new(JWT_SECRET_KEY.encode('utf-8'), payload_b64.encode('utf-8'), hashlib.sha256).hexdigest()
|
||||||
|
if not hmac.compare_digest(sig, expected_sig):
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Decode and check expiry
|
||||||
|
padding = 4 - (len(payload_b64) % 4)
|
||||||
|
if padding < 4:
|
||||||
|
payload_b64 += "=" * padding
|
||||||
|
payload_json = base64.urlsafe_b64decode(payload_b64.encode('utf-8')).decode('utf-8')
|
||||||
|
payload = json.loads(payload_json)
|
||||||
|
|
||||||
|
expires_dt = datetime.fromisoformat(payload["expires"])
|
||||||
|
if datetime.utcnow() > expires_dt:
|
||||||
|
return False # Expired
|
||||||
|
|
||||||
|
return payload["username"] == "root"
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
# FastAPI dependency to secure UI endpoints
|
||||||
|
async def get_current_user(authorization: str = Header(None)):
|
||||||
|
if not authorization or not authorization.startswith("Bearer "):
|
||||||
|
raise HTTPException(status_code=401, detail="Missing or invalid authentication credentials")
|
||||||
|
token = authorization.split(" ")[1]
|
||||||
|
if not verify_token(token):
|
||||||
|
raise HTTPException(status_code=401, detail="Authentication token is invalid or has expired")
|
||||||
|
return "root"
|
||||||
|
|
||||||
import os
|
import os
|
||||||
import json
|
import json
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
@@ -47,17 +104,35 @@ db = mongo_client[DB_NAME]
|
|||||||
|
|
||||||
clients_collection = db["clients"]
|
clients_collection = db["clients"]
|
||||||
logs_collection = db["logs"]
|
logs_collection = db["logs"]
|
||||||
|
config_collection = db["config"]
|
||||||
|
|
||||||
|
# --- Agent authentication ---
|
||||||
|
# Shared secret agents present in X-Agent-Token. Auth mode ("grace"|"strict") is
|
||||||
|
# stored in the config collection so it can be flipped live from the dashboard:
|
||||||
|
# grace = accept authenticated and legacy (tokenless) agents; used during rollout
|
||||||
|
# strict = reject any request without a valid token
|
||||||
|
AGENT_TOKEN = os.getenv("AGENT_TOKEN", "")
|
||||||
|
|
||||||
|
def get_agent_auth_mode() -> str:
|
||||||
|
doc = config_collection.find_one({"_id": "agent_auth"})
|
||||||
|
return (doc or {}).get("mode", "grace")
|
||||||
|
|
||||||
|
def is_valid_agent_token(token: Optional[str]) -> bool:
|
||||||
|
return bool(AGENT_TOKEN) and token is not None and hmac.compare_digest(token, AGENT_TOKEN)
|
||||||
|
|
||||||
|
async def require_agent_token(x_agent_token: str = Header(None)):
|
||||||
|
"""Dependency guarding every agent-facing endpoint. Enforces only in strict mode;
|
||||||
|
in grace mode it lets legacy tokenless agents through so a fleet can migrate."""
|
||||||
|
if get_agent_auth_mode() == "strict" and not is_valid_agent_token(x_agent_token):
|
||||||
|
raise HTTPException(status_code=401, detail="Invalid or missing agent token")
|
||||||
|
return is_valid_agent_token(x_agent_token)
|
||||||
|
|
||||||
MAX_ENTRIES_PER_CLIENT = 100 # Keep the latest 100 historical logs per client node
|
MAX_ENTRIES_PER_CLIENT = 100 # Keep the latest 100 historical logs per client node
|
||||||
|
|
||||||
ROCKETCHAT_WEBHOOK_URL = os.getenv("ROCKETCHAT_WEBHOOK_URL", "")
|
ROCKETCHAT_WEBHOOK_URL = os.getenv("ROCKETCHAT_WEBHOOK_URL", "")
|
||||||
|
|
||||||
# In-memory storage to prevent duplicate/spam storage notifications
|
# Alert states are now stored directly in the MongoDB client documents
|
||||||
active_storage_alerts = set()
|
# to support multi-worker environments safely.
|
||||||
|
|
||||||
# Tracks the timestamp when a System DOWN notification was last sent for a client ID
|
|
||||||
# Key: client_id (str), Value: datetime object of the last notification time
|
|
||||||
last_down_notification_times = {}
|
|
||||||
|
|
||||||
def send_rocketchat_notification(text: str, color: str = "#808080"):
|
def send_rocketchat_notification(text: str, color: str = "#808080"):
|
||||||
"""
|
"""
|
||||||
@@ -123,18 +198,31 @@ def append_to_logs(log_type: str, client_id: str, detail: Any):
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"[!] Error writing log to MongoDB: {e}")
|
print(f"[!] Error writing log to MongoDB: {e}")
|
||||||
|
|
||||||
def mark_client_active(client_id: str, clients: Dict[str, Any]):
|
def mark_client_active(client_id: str):
|
||||||
if client_id in clients:
|
try:
|
||||||
was_active = clients[client_id].get("active", False)
|
now_str = datetime.now().isoformat()
|
||||||
if not was_active:
|
# Atomically check if transitioned from offline to online (active is not True)
|
||||||
if client_id in last_down_notification_times:
|
old_doc = clients_collection.find_one_and_update(
|
||||||
|
{"_id": client_id, "active": {"$ne": True}},
|
||||||
|
{"$set": {"active": True, "last_seen": now_str, "last_down_alert_time": None}},
|
||||||
|
return_document=False
|
||||||
|
)
|
||||||
|
if old_doc:
|
||||||
|
# Client transitioned from inactive/None to active!
|
||||||
|
# Send UP recovery alert only if they had a down alert time logged
|
||||||
|
if old_doc.get("last_down_alert_time"):
|
||||||
send_rocketchat_notification(
|
send_rocketchat_notification(
|
||||||
text=f"✅ **System UP:** Client `{client_id}` has recovered and is back online.",
|
text=f"✅ **System UP:** Client `{client_id}` has recovered and is back online.",
|
||||||
color="#2ecc71"
|
color="#2ecc71"
|
||||||
)
|
)
|
||||||
last_down_notification_times.pop(client_id, None)
|
else:
|
||||||
clients[client_id]["active"] = True
|
# Client is already active, just update their last_seen timestamp
|
||||||
clients[client_id]["last_seen"] = datetime.now().isoformat()
|
clients_collection.update_one(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$set": {"last_seen": now_str}}
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"[!] Error marking client active: {e}")
|
||||||
|
|
||||||
def load_clients() -> Dict[str, Any]:
|
def load_clients() -> Dict[str, Any]:
|
||||||
try:
|
try:
|
||||||
@@ -142,94 +230,93 @@ def load_clients() -> Dict[str, Any]:
|
|||||||
clients_collection.delete_many({"_id": {"$in": ["", None]}})
|
clients_collection.delete_many({"_id": {"$in": ["", None]}})
|
||||||
clients_collection.delete_many({"_id": {"$regex": "^\\s*$"}})
|
clients_collection.delete_many({"_id": {"$regex": "^\\s*$"}})
|
||||||
|
|
||||||
# If collection is empty, seed it with default registry
|
|
||||||
if clients_collection.count_documents({}) == 0:
|
|
||||||
default_data = {
|
|
||||||
"client_1": {"pending_command": "none", "last_seen": None, "active": False},
|
|
||||||
"client_2": {"pending_command": "none", "last_seen": None, "active": False},
|
|
||||||
"client_3": {"pending_command": "none", "last_seen": None, "active": False}
|
|
||||||
}
|
|
||||||
save_clients(default_data)
|
|
||||||
return default_data
|
|
||||||
|
|
||||||
# Load all documents
|
# Load all documents
|
||||||
cursor = clients_collection.find()
|
cursor = clients_collection.find()
|
||||||
data = {}
|
data = {}
|
||||||
for doc in cursor:
|
for doc in cursor:
|
||||||
cid = doc["_id"]
|
cid = doc["_id"]
|
||||||
data[cid] = {k: v for k, v in doc.items() if k != "_id"}
|
|
||||||
|
|
||||||
# Compute active status dynamically based on 30-second heartbeat check-ins
|
|
||||||
modified = False
|
|
||||||
for cid, info in data.items():
|
|
||||||
last_seen_str = info.get("last_seen")
|
|
||||||
was_active = info.get("active", False)
|
|
||||||
is_active = False
|
|
||||||
|
|
||||||
|
# Compute active status dynamically for reads, but do NOT write back to database or alert
|
||||||
|
last_seen_str = doc.get("last_seen")
|
||||||
|
is_active = doc.get("active", False)
|
||||||
if last_seen_str:
|
if last_seen_str:
|
||||||
try:
|
try:
|
||||||
last_seen_dt = datetime.fromisoformat(last_seen_str)
|
last_seen_dt = datetime.fromisoformat(last_seen_str)
|
||||||
if (datetime.now() - last_seen_dt).total_seconds() < 30:
|
if (datetime.now() - last_seen_dt).total_seconds() >= 30:
|
||||||
is_active = True
|
is_active = False
|
||||||
except Exception:
|
except Exception:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
if was_active != is_active:
|
info = {k: v for k, v in doc.items() if k != "_id"}
|
||||||
info["active"] = is_active
|
info["active"] = is_active
|
||||||
modified = True
|
data[cid] = info
|
||||||
|
|
||||||
# Active status transitions alerts
|
|
||||||
if is_active:
|
|
||||||
# Transitions from offline (DOWN) to online (UP)
|
|
||||||
if cid in last_down_notification_times:
|
|
||||||
send_rocketchat_notification(
|
|
||||||
text=f"✅ **System UP:** Client `{cid}` has recovered and is back online.",
|
|
||||||
color="#2ecc71"
|
|
||||||
)
|
|
||||||
last_down_notification_times.pop(cid, None)
|
|
||||||
else:
|
|
||||||
# Transitions from online (UP) to offline (DOWN) for the first time
|
|
||||||
now = datetime.now()
|
|
||||||
last_down_notification_times[cid] = now
|
|
||||||
send_rocketchat_notification(
|
|
||||||
text=f"🚨 **System DOWN:** Client `{cid}` has missed heartbeats for over 30 seconds.",
|
|
||||||
color="#e74c3c"
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
# No transition (state remains the same)
|
|
||||||
if not is_active:
|
|
||||||
# Client remains offline. Check if we should send a repeating reminder.
|
|
||||||
# We ONLY send a reminder if it is already tracked in last_down_notification_times.
|
|
||||||
# This prevents sending alerts for historically offline systems at server boot!
|
|
||||||
if cid in last_down_notification_times:
|
|
||||||
now = datetime.now()
|
|
||||||
elapsed_seconds = (now - last_down_notification_times[cid]).total_seconds()
|
|
||||||
if elapsed_seconds >= 3600: # 1 hour (3600 seconds)
|
|
||||||
last_down_notification_times[cid] = now
|
|
||||||
send_rocketchat_notification(
|
|
||||||
text=f"🚨 **System STILL DOWN:** Client `{cid}` remains offline (reminder sent every hour).",
|
|
||||||
color="#e74c3c"
|
|
||||||
)
|
|
||||||
|
|
||||||
if modified:
|
|
||||||
save_clients(data)
|
|
||||||
|
|
||||||
return data
|
return data
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"[!] Error loading clients from MongoDB: {e}")
|
print(f"[!] Error loading clients from MongoDB: {e}")
|
||||||
return {}
|
return {}
|
||||||
|
|
||||||
def save_clients(data: Dict[str, Any]):
|
import asyncio
|
||||||
try:
|
|
||||||
for cid, info in data.items():
|
async def monitor_heartbeats_loop():
|
||||||
# Use upsert to update existing client or insert new client automatically
|
while True:
|
||||||
clients_collection.update_one(
|
try:
|
||||||
{"_id": cid},
|
now = datetime.now()
|
||||||
{"$set": info},
|
timeout_time = (now - timedelta(seconds=30)).isoformat()
|
||||||
upsert=True
|
|
||||||
)
|
# Find clients that are active in DB but haven't checked in for 30s
|
||||||
except Exception as e:
|
cursor = clients_collection.find({
|
||||||
print(f"[!] Error saving clients to MongoDB: {e}")
|
"active": True,
|
||||||
|
"last_seen": {"$lt": timeout_time}
|
||||||
|
})
|
||||||
|
for doc in cursor:
|
||||||
|
cid = doc["_id"]
|
||||||
|
now_str = now.isoformat()
|
||||||
|
|
||||||
|
# Atomically update to active = False
|
||||||
|
# Ensures only one request/worker triggers the state transition and sends the alert
|
||||||
|
old_doc = clients_collection.find_one_and_update(
|
||||||
|
{"_id": cid, "active": True, "last_seen": doc["last_seen"]},
|
||||||
|
{"$set": {"active": False, "last_down_alert_time": now_str}},
|
||||||
|
return_document=False
|
||||||
|
)
|
||||||
|
if old_doc:
|
||||||
|
send_rocketchat_notification(
|
||||||
|
text=f"🚨 **System DOWN:** Client `{cid}` has missed heartbeats for over 30 seconds.",
|
||||||
|
color="#e74c3c"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Hourly reminders for STILL DOWN clients
|
||||||
|
reminder_time = (now - timedelta(hours=1)).isoformat()
|
||||||
|
cursor_still_down = clients_collection.find({
|
||||||
|
"active": False,
|
||||||
|
"last_down_alert_time": {"$lt": reminder_time}
|
||||||
|
})
|
||||||
|
for doc in cursor_still_down:
|
||||||
|
cid = doc["_id"]
|
||||||
|
old_last_down = doc["last_down_alert_time"]
|
||||||
|
now_str = now.isoformat()
|
||||||
|
|
||||||
|
# Atomically update reminder timestamp
|
||||||
|
old_doc = clients_collection.find_one_and_update(
|
||||||
|
{"_id": cid, "active": False, "last_down_alert_time": old_last_down},
|
||||||
|
{"$set": {"last_down_alert_time": now_str}},
|
||||||
|
return_document=False
|
||||||
|
)
|
||||||
|
if old_doc:
|
||||||
|
send_rocketchat_notification(
|
||||||
|
text=f"🚨 **System STILL DOWN:** Client `{cid}` remains offline (reminder sent every hour).",
|
||||||
|
color="#e74c3c"
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"[!] Error in heartbeat monitoring loop: {e}")
|
||||||
|
|
||||||
|
await asyncio.sleep(10)
|
||||||
|
|
||||||
|
@app.on_event("startup")
|
||||||
|
async def startup_event():
|
||||||
|
asyncio.create_task(monitor_heartbeats_loop())
|
||||||
|
print("[*] Heartbeat monitoring background task initialized.")
|
||||||
|
|
||||||
class TelemetryPayload(BaseModel):
|
class TelemetryPayload(BaseModel):
|
||||||
cpu_percent: float
|
cpu_percent: float
|
||||||
@@ -238,6 +325,10 @@ class TelemetryPayload(BaseModel):
|
|||||||
memory_free_gb: float
|
memory_free_gb: float
|
||||||
disks: List[Dict[str, Any]]
|
disks: List[Dict[str, Any]]
|
||||||
gpus: List[Dict[str, Any]]
|
gpus: List[Dict[str, Any]]
|
||||||
|
cpu_temp: Optional[float] = None
|
||||||
|
agent_version: Optional[str] = None
|
||||||
|
speed_upload_mbps: Optional[float] = None
|
||||||
|
speed_download_mbps: Optional[float] = None
|
||||||
|
|
||||||
class CommandResultPayload(BaseModel):
|
class CommandResultPayload(BaseModel):
|
||||||
command: str
|
command: str
|
||||||
@@ -245,8 +336,55 @@ class CommandResultPayload(BaseModel):
|
|||||||
stdout: str
|
stdout: str
|
||||||
stderr: str
|
stderr: str
|
||||||
|
|
||||||
|
class SearchResultsPayload(BaseModel):
|
||||||
|
query: str
|
||||||
|
results: List[Dict[str, Any]]
|
||||||
|
searched_path: Optional[str] = None
|
||||||
|
error: Optional[str] = None
|
||||||
|
|
||||||
|
class LoginPayload(BaseModel):
|
||||||
|
username: str
|
||||||
|
password: str
|
||||||
|
|
||||||
|
DASHBOARD_USERNAME = os.getenv("DASHBOARD_USERNAME", "root")
|
||||||
|
DASHBOARD_PASSWORD = os.getenv("DASHBOARD_PASSWORD", "seekright159@")
|
||||||
|
|
||||||
|
@app.post("/api/login")
|
||||||
|
async def login(payload: LoginPayload):
|
||||||
|
user_ok = hmac.compare_digest(payload.username, DASHBOARD_USERNAME)
|
||||||
|
pass_ok = hmac.compare_digest(payload.password, DASHBOARD_PASSWORD)
|
||||||
|
if user_ok and pass_ok:
|
||||||
|
token = generate_token("root")
|
||||||
|
return {"token": token}
|
||||||
|
raise HTTPException(status_code=401, detail="Invalid username or password")
|
||||||
|
|
||||||
|
class AuthModePayload(BaseModel):
|
||||||
|
mode: str
|
||||||
|
|
||||||
|
@app.get("/api/agent-auth-mode")
|
||||||
|
async def read_agent_auth_mode(current_user: str = Depends(get_current_user)):
|
||||||
|
return {"mode": get_agent_auth_mode()}
|
||||||
|
|
||||||
|
@app.post("/api/set-agent-auth-mode")
|
||||||
|
async def set_agent_auth_mode(payload: AuthModePayload, current_user: str = Depends(get_current_user)):
|
||||||
|
if payload.mode not in ("grace", "strict"):
|
||||||
|
raise HTTPException(status_code=400, detail="mode must be 'grace' or 'strict'")
|
||||||
|
config_collection.update_one({"_id": "agent_auth"}, {"$set": {"mode": payload.mode}}, upsert=True)
|
||||||
|
return {"status": "success", "mode": payload.mode}
|
||||||
|
|
||||||
|
@app.get("/api/agent-versions")
|
||||||
|
async def agent_versions(current_user: str = Depends(get_current_user)):
|
||||||
|
"""Migration dashboard: which agent version each node reports and whether its token is valid."""
|
||||||
|
out = {}
|
||||||
|
for doc in clients_collection.find({}, {"agent_version": 1, "agent_token_ok": 1}):
|
||||||
|
out[doc["_id"]] = {
|
||||||
|
"agent_version": doc.get("agent_version", "unknown"),
|
||||||
|
"agent_token_ok": doc.get("agent_token_ok", False),
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
|
||||||
@app.get("/api/get-commands")
|
@app.get("/api/get-commands")
|
||||||
async def get_whitelisted_commands(platform: str):
|
async def get_whitelisted_commands(platform: str, token_valid: bool = Depends(require_agent_token)):
|
||||||
"""
|
"""
|
||||||
Endpoint for remote agents to fetch their OS-specific whitelisted commands.
|
Endpoint for remote agents to fetch their OS-specific whitelisted commands.
|
||||||
"""
|
"""
|
||||||
@@ -258,9 +396,8 @@ async def get_whitelisted_commands(platform: str):
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
raise HTTPException(status_code=500, detail=f"Failed to load commands: {e}")
|
raise HTTPException(status_code=500, detail=f"Failed to load commands: {e}")
|
||||||
|
|
||||||
|
|
||||||
@app.get("/api/get-command")
|
@app.get("/api/get-command")
|
||||||
async def get_command(client_id: str):
|
async def get_command(client_id: str, token_valid: bool = Depends(require_agent_token)):
|
||||||
"""
|
"""
|
||||||
Endpoint for clients to poll for pending commands.
|
Endpoint for clients to poll for pending commands.
|
||||||
Client Agent hits this endpoint to ask: "Do I have any work to do?"
|
Client Agent hits this endpoint to ask: "Do I have any work to do?"
|
||||||
@@ -268,93 +405,151 @@ async def get_command(client_id: str):
|
|||||||
if not client_id or not client_id.strip():
|
if not client_id or not client_id.strip():
|
||||||
raise HTTPException(status_code=400, detail="client_id cannot be empty")
|
raise HTTPException(status_code=400, detail="client_id cannot be empty")
|
||||||
|
|
||||||
clients = load_clients()
|
now_str = datetime.now().isoformat()
|
||||||
|
# Dynamic fleet auto-registration
|
||||||
|
clients_collection.update_one(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$setOnInsert": {"pending_command": "none", "active": True, "last_seen": now_str}},
|
||||||
|
upsert=True
|
||||||
|
)
|
||||||
|
|
||||||
# 1. Auto-register new client if we haven't seen it yet
|
# Get and reset command atomically
|
||||||
if client_id not in clients:
|
updated_doc = clients_collection.find_one_and_update(
|
||||||
print(f"[+] Dynamic fleet auto-registration: Registered new agent '{client_id}'")
|
{"_id": client_id},
|
||||||
clients[client_id] = {"pending_command": "none", "last_seen": None, "active": True}
|
{"$set": {"pending_command": "none"}},
|
||||||
|
return_document=False # Returns the state before update
|
||||||
|
)
|
||||||
|
|
||||||
# 2. Fetch the command
|
cmd = "none"
|
||||||
cmd = clients[client_id].get("pending_command", "none")
|
if updated_doc:
|
||||||
|
cmd = updated_doc.get("pending_command", "none")
|
||||||
|
|
||||||
# Log if a valid pending command was polled
|
|
||||||
if cmd != "none":
|
if cmd != "none":
|
||||||
append_to_logs("command_polled", client_id, {"command": cmd})
|
append_to_logs("command_polled", client_id, {"command": cmd})
|
||||||
|
|
||||||
# 3. Reset the command to "none", mark active, and update last_seen timestamp
|
mark_client_active(client_id)
|
||||||
clients[client_id]["pending_command"] = "none"
|
|
||||||
mark_client_active(client_id, clients)
|
|
||||||
|
|
||||||
save_clients(clients)
|
|
||||||
return {"command": cmd}
|
return {"command": cmd}
|
||||||
|
|
||||||
@app.post("/api/schedule-command")
|
@app.post("/api/schedule-command")
|
||||||
async def schedule_command(client_id: str, command: str):
|
async def schedule_command(client_id: str, command: str, current_user: str = Depends(get_current_user)):
|
||||||
"""
|
"""
|
||||||
Endpoint for your Dashboard/UI to schedule a new command.
|
Endpoint for your Dashboard/UI to schedule a new command.
|
||||||
Supports a comma-separated list of client IDs for batch fleet updates.
|
Supports a comma-separated list of client IDs for batch fleet updates.
|
||||||
"""
|
"""
|
||||||
clients = load_clients()
|
|
||||||
target_ids = [cid.strip() for cid in client_id.split(",") if cid.strip()]
|
target_ids = [cid.strip() for cid in client_id.split(",") if cid.strip()]
|
||||||
|
|
||||||
if not target_ids:
|
if not target_ids:
|
||||||
raise HTTPException(status_code=400, detail="No target client IDs specified")
|
raise HTTPException(status_code=400, detail="No target client IDs specified")
|
||||||
|
|
||||||
for cid in target_ids:
|
for cid in target_ids:
|
||||||
if cid not in clients:
|
clients_collection.update_one(
|
||||||
clients[cid] = {"pending_command": "none", "last_seen": None, "active": False}
|
{"_id": cid},
|
||||||
clients[cid]["pending_command"] = command
|
{"$set": {"pending_command": command}},
|
||||||
# Log command scheduling per system
|
upsert=True
|
||||||
|
)
|
||||||
append_to_logs("command_scheduled", cid, {"command": command})
|
append_to_logs("command_scheduled", cid, {"command": command})
|
||||||
|
|
||||||
save_clients(clients)
|
|
||||||
return {"status": "success", "message": f"Command '{command}' scheduled for {', '.join(target_ids)}"}
|
return {"status": "success", "message": f"Command '{command}' scheduled for {', '.join(target_ids)}"}
|
||||||
|
|
||||||
@app.post("/api/telemetry")
|
@app.post("/api/telemetry")
|
||||||
async def receive_telemetry(client_id: str, payload: TelemetryPayload):
|
async def receive_telemetry(client_id: str, payload: TelemetryPayload, token_valid: bool = Depends(require_agent_token)):
|
||||||
"""
|
"""
|
||||||
Endpoint for agents to push their live hardware telemetry.
|
Endpoint for agents to push their live hardware telemetry.
|
||||||
"""
|
"""
|
||||||
if not client_id or not client_id.strip():
|
if not client_id or not client_id.strip():
|
||||||
raise HTTPException(status_code=400, detail="client_id cannot be empty")
|
raise HTTPException(status_code=400, detail="client_id cannot be empty")
|
||||||
|
|
||||||
clients = load_clients()
|
now_str = datetime.now().isoformat()
|
||||||
|
# Auto-register client if we see it through telemetry first
|
||||||
|
clients_collection.update_one(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$setOnInsert": {"pending_command": "none", "active": True, "last_seen": now_str}},
|
||||||
|
upsert=True
|
||||||
|
)
|
||||||
|
|
||||||
# Auto-register new client if we see it through telemetry first
|
mark_client_active(client_id)
|
||||||
if client_id not in clients:
|
|
||||||
print(f"[+] Dynamic fleet auto-registration: Registered new agent '{client_id}' through telemetry")
|
|
||||||
clients[client_id] = {"pending_command": "none", "last_seen": None, "active": True}
|
|
||||||
|
|
||||||
# Update last_seen timestamp, active status, and store the current telemetry details
|
# Get current alert states from database to prevent duplicate alerts
|
||||||
mark_client_active(client_id, clients)
|
client_doc = clients_collection.find_one({"_id": client_id}) or {}
|
||||||
clients[client_id]["telemetry"] = payload.dict()
|
storage_alerts = client_doc.get("active_storage_alerts", [])
|
||||||
save_clients(clients)
|
gpu_alert_sent = client_doc.get("gpu_alert_sent", False)
|
||||||
|
|
||||||
# Log telemetry history event
|
storage_modified = False
|
||||||
append_to_logs("telemetry", client_id, payload.dict())
|
gpu_modified = False
|
||||||
|
|
||||||
# Storage limits warning checker
|
# Storage warning checker
|
||||||
for disk in payload.disks:
|
for disk in payload.disks:
|
||||||
mount = disk.get("mount", "/")
|
mount = disk.get("mount", "/")
|
||||||
percent = disk.get("percent", 0.0)
|
percent = disk.get("percent", 0.0)
|
||||||
alert_key = f"{client_id}:{mount}"
|
|
||||||
|
|
||||||
if percent >= 80.0:
|
if percent >= 80.0:
|
||||||
if alert_key not in active_storage_alerts:
|
if mount not in storage_alerts:
|
||||||
active_storage_alerts.add(alert_key)
|
storage_alerts.append(mount)
|
||||||
|
storage_modified = True
|
||||||
send_rocketchat_notification(
|
send_rocketchat_notification(
|
||||||
text=f"⚠️ **Storage Warning:** Client `{client_id}` disk `{mount}` is at **{percent}%** capacity.",
|
text=f"⚠️ **Storage Warning:** Client `{client_id}` disk `{mount}` is at **{percent}%** capacity.",
|
||||||
color="#f39c12"
|
color="#f39c12"
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
if alert_key in active_storage_alerts:
|
if mount in storage_alerts:
|
||||||
active_storage_alerts.remove(alert_key)
|
storage_alerts.remove(mount)
|
||||||
|
storage_modified = True
|
||||||
send_rocketchat_notification(
|
send_rocketchat_notification(
|
||||||
text=f"✅ **Storage Recovered:** Client `{client_id}` disk `{mount}` has cleared warning state and is at **{percent}%**.",
|
text=f"✅ **Storage Recovered:** Client `{client_id}` disk `{mount}` has cleared warning state and is at **{percent}%**.",
|
||||||
color="#2ecc71"
|
color="#2ecc71"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# GPU Failure Monitor
|
||||||
|
if len(payload.gpus) == 0:
|
||||||
|
if not gpu_alert_sent:
|
||||||
|
gpu_alert_sent = True
|
||||||
|
gpu_modified = True
|
||||||
|
send_rocketchat_notification(
|
||||||
|
text=f"🚨 **GPU Failure:** Client `{client_id}` is not reporting any GPU data! (nvidia-smi has failed or is missing)",
|
||||||
|
color="#e74c3c"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
if gpu_alert_sent:
|
||||||
|
gpu_alert_sent = False
|
||||||
|
gpu_modified = True
|
||||||
|
send_rocketchat_notification(
|
||||||
|
text=f"✅ **GPU Recovered:** Client `{client_id}` is reporting GPU data successfully again.",
|
||||||
|
color="#2ecc71"
|
||||||
|
)
|
||||||
|
|
||||||
|
update_fields = {
|
||||||
|
"telemetry": payload.dict()
|
||||||
|
}
|
||||||
|
if storage_modified:
|
||||||
|
update_fields["active_storage_alerts"] = storage_alerts
|
||||||
|
if gpu_modified:
|
||||||
|
update_fields["gpu_alert_sent"] = gpu_alert_sent
|
||||||
|
|
||||||
|
clients_collection.update_one(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$set": update_fields}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Log telemetry history event
|
||||||
|
append_to_logs("telemetry", client_id, payload.dict())
|
||||||
|
|
||||||
|
data = payload.dict()
|
||||||
|
client_telemetry[client_id] = data
|
||||||
|
|
||||||
|
# Log received stats
|
||||||
|
print(f"\n[TELEMETRY RECEIVED] from {client_id}:")
|
||||||
|
print(f" CPU: {data['cpu_percent']}%")
|
||||||
|
print(f" RAM: {data['memory_percent']}% ({data['memory_free_gb']} GB Free / {data['memory_total_gb']} GB Total)")
|
||||||
|
|
||||||
|
for disk in data['disks']:
|
||||||
|
print(f" Drive {disk['mount']}: {disk['percent']}% Used ({disk['free_gb']} GB Free / {disk['total_gb']} GB Total)")
|
||||||
|
|
||||||
|
for gpu in data['gpus']:
|
||||||
|
power_str = f"Power: {gpu.get('power_draw', 'N/A')} / {gpu.get('power_limit', 'N/A')}"
|
||||||
|
fan_str = f"Fan: {gpu.get('fan_speed', 'N/A')}"
|
||||||
|
print(f" GPU [{gpu['name']}]: Core: {gpu['utilization']}, Temp: {gpu['temp']}, VRAM: {gpu['memory_used']}/{gpu['memory_total']}, {power_str}, {fan_str}")
|
||||||
|
|
||||||
|
return {"status": "success"}
|
||||||
|
|
||||||
data = payload.dict()
|
data = payload.dict()
|
||||||
client_telemetry[client_id] = data
|
client_telemetry[client_id] = data
|
||||||
|
|
||||||
@@ -374,21 +569,22 @@ async def receive_telemetry(client_id: str, payload: TelemetryPayload):
|
|||||||
return {"status": "success"}
|
return {"status": "success"}
|
||||||
|
|
||||||
@app.get("/api/telemetry")
|
@app.get("/api/telemetry")
|
||||||
async def get_all_telemetry():
|
async def get_all_telemetry(current_user: str = Depends(get_current_user)):
|
||||||
"""
|
"""
|
||||||
Endpoint to view the live dashboard data of all clients.
|
Endpoint to view the live dashboard data of all clients.
|
||||||
"""
|
"""
|
||||||
return client_telemetry
|
clients = load_clients()
|
||||||
|
return {cid: info.get("telemetry", {}) for cid, info in clients.items() if info.get("telemetry")}
|
||||||
|
|
||||||
@app.get("/api/clients")
|
@app.get("/api/clients")
|
||||||
async def get_clients_api():
|
async def get_clients_api(current_user: str = Depends(get_current_user)):
|
||||||
"""
|
"""
|
||||||
Endpoint for the React UI to fetch the live active client registry with computed statuses.
|
Endpoint for the React UI to fetch the live active client registry with computed statuses.
|
||||||
"""
|
"""
|
||||||
return load_clients()
|
return load_clients()
|
||||||
|
|
||||||
@app.get("/api/logs")
|
@app.get("/api/logs")
|
||||||
async def get_logs_history():
|
async def get_logs_history(current_user: str = Depends(get_current_user)):
|
||||||
"""
|
"""
|
||||||
Endpoint to retrieve logs history grouped by client_id.
|
Endpoint to retrieve logs history grouped by client_id.
|
||||||
"""
|
"""
|
||||||
@@ -414,7 +610,7 @@ async def get_logs_history():
|
|||||||
return {}
|
return {}
|
||||||
|
|
||||||
@app.post("/api/command-result")
|
@app.post("/api/command-result")
|
||||||
async def receive_command_result(client_id: str, payload: CommandResultPayload):
|
async def receive_command_result(client_id: str, payload: CommandResultPayload, token_valid: bool = Depends(require_agent_token)):
|
||||||
"""
|
"""
|
||||||
Endpoint for remote agents to push execution results back to the central server.
|
Endpoint for remote agents to push execution results back to the central server.
|
||||||
"""
|
"""
|
||||||
@@ -425,7 +621,7 @@ async def receive_command_result(client_id: str, payload: CommandResultPayload):
|
|||||||
return {"status": "success"}
|
return {"status": "success"}
|
||||||
|
|
||||||
@app.get("/api/get-raw-commands")
|
@app.get("/api/get-raw-commands")
|
||||||
async def get_raw_commands():
|
async def get_raw_commands(current_user: str = Depends(get_current_user)):
|
||||||
"""
|
"""
|
||||||
Endpoint for the UI to load the complete commands.json whitelist file.
|
Endpoint for the UI to load the complete commands.json whitelist file.
|
||||||
"""
|
"""
|
||||||
@@ -437,7 +633,7 @@ async def get_raw_commands():
|
|||||||
raise HTTPException(status_code=500, detail=f"Failed to read commands: {e}")
|
raise HTTPException(status_code=500, detail=f"Failed to read commands: {e}")
|
||||||
|
|
||||||
@app.post("/api/save-commands")
|
@app.post("/api/save-commands")
|
||||||
async def save_commands(commands: Dict[str, Any]):
|
async def save_commands(commands: Dict[str, Any], current_user: str = Depends(get_current_user)):
|
||||||
"""
|
"""
|
||||||
Endpoint for the UI to save changes back to commands.json.
|
Endpoint for the UI to save changes back to commands.json.
|
||||||
"""
|
"""
|
||||||
@@ -502,3 +698,342 @@ if __name__ == "__main__":
|
|||||||
print("[*] Starting FastAPI Central Server...")
|
print("[*] Starting FastAPI Central Server...")
|
||||||
# uvicorn runs the FastAPI app on port 8000
|
# uvicorn runs the FastAPI app on port 8000
|
||||||
uvicorn.run("central_api_prototype:app", host="0.0.0.0", port=8000, reload=True)
|
uvicorn.run("central_api_prototype:app", host="0.0.0.0", port=8000, reload=True)
|
||||||
|
|
||||||
|
# ============================================================
|
||||||
|
# Remote File Transfer (video fetch from agent TAKELEAP folders)
|
||||||
|
# ============================================================
|
||||||
|
import os
|
||||||
|
FILE_TRANSFER_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "file_transfers")
|
||||||
|
os.makedirs(FILE_TRANSFER_DIR, exist_ok=True)
|
||||||
|
|
||||||
|
class FileStatusPayload(BaseModel):
|
||||||
|
filename: str
|
||||||
|
status: str
|
||||||
|
message: Optional[str] = ""
|
||||||
|
progress_percent: Optional[float] = None
|
||||||
|
|
||||||
|
def set_file_transfer_state(client_id: str, filename: str, status: str, message: str = "", extra: Optional[Dict[str, Any]] = None):
|
||||||
|
state = {
|
||||||
|
"filename": filename,
|
||||||
|
"status": status,
|
||||||
|
"message": message,
|
||||||
|
"updated": datetime.now().isoformat()
|
||||||
|
}
|
||||||
|
if extra:
|
||||||
|
state.update(extra)
|
||||||
|
clients_collection.update_one(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$set": {"file_transfer": state}},
|
||||||
|
upsert=True
|
||||||
|
)
|
||||||
|
|
||||||
|
@app.post("/api/request-file")
|
||||||
|
async def request_file(client_id: str, filename: str, current_user: str = Depends(get_current_user)):
|
||||||
|
safe_name = os.path.basename(filename.strip())
|
||||||
|
if not client_id.strip() or not safe_name:
|
||||||
|
raise HTTPException(status_code=400, detail="client_id and filename are required")
|
||||||
|
|
||||||
|
clients_collection.update_one(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$set": {"pending_file_request": safe_name}},
|
||||||
|
upsert=True
|
||||||
|
)
|
||||||
|
set_file_transfer_state(client_id, safe_name, "requested", "Waiting for agent to poll...")
|
||||||
|
append_to_logs("file_requested", client_id, {"filename": safe_name})
|
||||||
|
return {"status": "success", "message": f"File '{safe_name}' requested from {client_id}"}
|
||||||
|
|
||||||
|
@app.post("/api/set-shift-path")
|
||||||
|
async def set_shift_path(client_id: str, shift_path: str = "", current_user: str = Depends(get_current_user)):
|
||||||
|
"""
|
||||||
|
Store where this node keeps its SHIFT folder (varies per site:
|
||||||
|
/mnt/<disk-uuid>/SR/SHIFT or /mnt/<disk-uuid>/TAKELEAP/SHIFT).
|
||||||
|
Delivered to the agent in every heartbeat response.
|
||||||
|
"""
|
||||||
|
path = shift_path.strip()
|
||||||
|
result = clients_collection.update_one({"_id": client_id}, {"$set": {"shift_path": path}})
|
||||||
|
if result.matched_count == 0:
|
||||||
|
raise HTTPException(status_code=404, detail="Unknown client_id")
|
||||||
|
append_to_logs("shift_path_set", client_id, {"shift_path": path})
|
||||||
|
return {"status": "success", "shift_path": path}
|
||||||
|
|
||||||
|
@app.post("/api/request-search")
|
||||||
|
async def request_search(client_id: str, query: str, current_user: str = Depends(get_current_user)):
|
||||||
|
"""
|
||||||
|
Queue a filename search in the node's SHIFT folder. The agent picks the
|
||||||
|
query up on its next heartbeat and posts matches to /api/search-results.
|
||||||
|
"""
|
||||||
|
q = query.strip()
|
||||||
|
if not q:
|
||||||
|
raise HTTPException(status_code=400, detail="Empty search query")
|
||||||
|
now_str = datetime.now().isoformat()
|
||||||
|
result = clients_collection.update_one(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$set": {"pending_search": q,
|
||||||
|
"file_search": {"query": q, "status": "searching", "results": [], "requested_at": now_str}}}
|
||||||
|
)
|
||||||
|
if result.matched_count == 0:
|
||||||
|
raise HTTPException(status_code=404, detail="Unknown client_id")
|
||||||
|
append_to_logs("search_requested", client_id, {"query": q})
|
||||||
|
return {"status": "success"}
|
||||||
|
|
||||||
|
@app.post("/api/search-results")
|
||||||
|
async def receive_search_results(client_id: str, payload: SearchResultsPayload, token_valid: bool = Depends(require_agent_token)):
|
||||||
|
now_str = datetime.now().isoformat()
|
||||||
|
clients_collection.update_one(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$set": {"file_search": {
|
||||||
|
"query": payload.query,
|
||||||
|
"status": "error" if payload.error else "done",
|
||||||
|
"results": payload.results[:200],
|
||||||
|
"searched_path": payload.searched_path,
|
||||||
|
"error": payload.error,
|
||||||
|
"completed_at": now_str,
|
||||||
|
}}}
|
||||||
|
)
|
||||||
|
append_to_logs("search_results", client_id, {"query": payload.query, "count": len(payload.results), "error": payload.error})
|
||||||
|
return {"status": "success"}
|
||||||
|
|
||||||
|
@app.get("/api/get-file-request")
|
||||||
|
async def get_file_request(client_id: str, token_valid: bool = Depends(require_agent_token)):
|
||||||
|
if not client_id or not client_id.strip():
|
||||||
|
raise HTTPException(status_code=400, detail="client_id cannot be empty")
|
||||||
|
|
||||||
|
doc = clients_collection.find_one_and_update(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$set": {"pending_file_request": "none"}},
|
||||||
|
return_document=False
|
||||||
|
)
|
||||||
|
fname = doc.get("pending_file_request", "none") if doc else "none"
|
||||||
|
return {"filename": fname}
|
||||||
|
|
||||||
|
@app.post("/api/file-transfer-status")
|
||||||
|
async def update_file_transfer_status(client_id: str, payload: FileStatusPayload, token_valid: bool = Depends(require_agent_token)):
|
||||||
|
if not client_id or not client_id.strip():
|
||||||
|
raise HTTPException(status_code=400, detail="client_id cannot be empty")
|
||||||
|
set_file_transfer_state(client_id, payload.filename, payload.status, payload.message or "", {"progress_percent": payload.progress_percent})
|
||||||
|
append_to_logs("file_transfer_status", client_id, payload.dict())
|
||||||
|
return {"status": "success"}
|
||||||
|
|
||||||
|
@app.post("/api/cancel-upload")
|
||||||
|
async def cancel_upload(client_id: str, filename: str):
|
||||||
|
safe_name = os.path.basename(filename.strip())
|
||||||
|
set_file_transfer_state(client_id, safe_name, "cancelled", "Upload cancelled by user")
|
||||||
|
|
||||||
|
client_dir = os.path.join(FILE_TRANSFER_DIR, os.path.basename(client_id.strip()))
|
||||||
|
if os.path.exists(client_dir):
|
||||||
|
for f in os.listdir(client_dir):
|
||||||
|
if f.startswith(f"{safe_name}.part"):
|
||||||
|
try:
|
||||||
|
os.remove(os.path.join(client_dir, f))
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
return {"status": "success"}
|
||||||
|
|
||||||
|
@app.get("/api/received-chunks")
|
||||||
|
async def received_chunks(client_id: str, filename: str, chunk_size: int, file_size: int, token_valid: bool = Depends(require_agent_token)):
|
||||||
|
"""Resume support: report which chunk indexes are already stored for this file.
|
||||||
|
Parts whose size doesn't match the current chunking scheme (stale from an
|
||||||
|
earlier run with a different chunk size) are deleted so they can't corrupt
|
||||||
|
the final stitch."""
|
||||||
|
safe_name = os.path.basename(filename.strip())
|
||||||
|
if not client_id.strip() or not safe_name or chunk_size <= 0 or file_size < 0:
|
||||||
|
raise HTTPException(status_code=400, detail="invalid parameters")
|
||||||
|
|
||||||
|
total_chunks = 1 if file_size == 0 else (file_size + chunk_size - 1) // chunk_size
|
||||||
|
last_expected = file_size - (total_chunks - 1) * chunk_size
|
||||||
|
|
||||||
|
client_dir = os.path.join(FILE_TRANSFER_DIR, os.path.basename(client_id.strip()))
|
||||||
|
received = []
|
||||||
|
if os.path.isdir(client_dir):
|
||||||
|
prefix = f"{safe_name}.part"
|
||||||
|
for entry in os.listdir(client_dir):
|
||||||
|
if not entry.startswith(prefix) or not entry[len(prefix):].isdigit():
|
||||||
|
continue
|
||||||
|
idx = int(entry[len(prefix):])
|
||||||
|
path = os.path.join(client_dir, entry)
|
||||||
|
expected = chunk_size if idx < total_chunks - 1 else last_expected
|
||||||
|
try:
|
||||||
|
if idx < total_chunks and os.path.getsize(path) == expected:
|
||||||
|
received.append(idx)
|
||||||
|
else:
|
||||||
|
os.remove(path)
|
||||||
|
except OSError:
|
||||||
|
pass # part is mid-write or locked; report as not received
|
||||||
|
return {"received": sorted(received)}
|
||||||
|
|
||||||
|
@app.post("/api/upload-chunk")
|
||||||
|
async def upload_chunk(client_id: str, filename: str, chunk_index: int, request: Request, token_valid: bool = Depends(require_agent_token)):
|
||||||
|
safe_name = os.path.basename(filename.strip())
|
||||||
|
if not client_id.strip() or not safe_name:
|
||||||
|
raise HTTPException(status_code=400, detail="client_id and filename are required")
|
||||||
|
|
||||||
|
doc = clients_collection.find_one({"_id": client_id})
|
||||||
|
if doc and doc.get("file_transfer", {}).get("filename") == safe_name and doc.get("file_transfer", {}).get("status") == "cancelled":
|
||||||
|
return {"status": "cancelled"}
|
||||||
|
|
||||||
|
client_dir = os.path.join(FILE_TRANSFER_DIR, os.path.basename(client_id.strip()))
|
||||||
|
os.makedirs(client_dir, exist_ok=True)
|
||||||
|
dest_path = os.path.join(client_dir, f"{safe_name}.part{chunk_index}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(dest_path, "wb") as f:
|
||||||
|
async for chunk in request.stream():
|
||||||
|
f.write(chunk)
|
||||||
|
except Exception as e:
|
||||||
|
raise HTTPException(status_code=500, detail=f"Failed to store chunk: {e}")
|
||||||
|
|
||||||
|
return {"status": "success", "chunk_index": chunk_index}
|
||||||
|
|
||||||
|
def enforce_storage_limit(max_bytes: int = 20 * 1024**3):
|
||||||
|
"""Scan FILE_TRANSFER_DIR recursively and delete oldest completed files if total size > max_bytes."""
|
||||||
|
all_files = []
|
||||||
|
total_size = 0
|
||||||
|
for root, dirs, files in os.walk(FILE_TRANSFER_DIR):
|
||||||
|
for name in files:
|
||||||
|
if ".part" in name:
|
||||||
|
continue
|
||||||
|
path = os.path.join(root, name)
|
||||||
|
try:
|
||||||
|
stat = os.stat(path)
|
||||||
|
all_files.append((stat.st_mtime, path, stat.st_size))
|
||||||
|
total_size += stat.st_size
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if total_size > max_bytes:
|
||||||
|
all_files.sort(key=lambda x: x[0])
|
||||||
|
for mtime, path, size in all_files:
|
||||||
|
if total_size <= max_bytes:
|
||||||
|
break
|
||||||
|
try:
|
||||||
|
os.remove(path)
|
||||||
|
total_size -= size
|
||||||
|
print(f"Deleted old file {path} to free space.")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error deleting file {path}: {e}")
|
||||||
|
|
||||||
|
@app.post("/api/upload-complete")
|
||||||
|
async def upload_complete(client_id: str, filename: str, total_chunks: int, background_tasks: BackgroundTasks, token_valid: bool = Depends(require_agent_token)):
|
||||||
|
safe_name = os.path.basename(filename.strip())
|
||||||
|
client_dir = os.path.join(FILE_TRANSFER_DIR, os.path.basename(client_id.strip()))
|
||||||
|
final_path = os.path.join(client_dir, safe_name)
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(final_path, "wb") as outfile:
|
||||||
|
for i in range(total_chunks):
|
||||||
|
part_path = os.path.join(client_dir, f"{safe_name}.part{i}")
|
||||||
|
if not os.path.exists(part_path):
|
||||||
|
raise HTTPException(status_code=400, detail=f"Missing chunk {i}")
|
||||||
|
with open(part_path, "rb") as infile:
|
||||||
|
outfile.write(infile.read())
|
||||||
|
os.remove(part_path)
|
||||||
|
except Exception as e:
|
||||||
|
set_file_transfer_state(client_id, safe_name, "error", f"Upload finalize failed: {e}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Failed to finalize file: {e}")
|
||||||
|
|
||||||
|
size_mb = round(os.path.getsize(final_path) / (1024 * 1024), 2)
|
||||||
|
set_file_transfer_state(client_id, safe_name, "ready", f"File received ({size_mb} MB)", {"size_mb": size_mb, "progress_percent": 100.0})
|
||||||
|
append_to_logs("file_received", client_id, {"filename": safe_name, "size_mb": size_mb})
|
||||||
|
send_rocketchat_notification(
|
||||||
|
text=f"📥 **File Received:** `{safe_name}` ({size_mb} MB) uploaded from client `{client_id}`.",
|
||||||
|
color="#2ecc71"
|
||||||
|
)
|
||||||
|
background_tasks.add_task(enforce_storage_limit)
|
||||||
|
return {"status": "success", "size_mb": size_mb}
|
||||||
|
|
||||||
|
@app.get("/api/file-transfers")
|
||||||
|
async def get_file_transfers(current_user: str = Depends(get_current_user)):
|
||||||
|
try:
|
||||||
|
cursor = clients_collection.find({}, {"file_transfer": 1})
|
||||||
|
return {doc["_id"]: doc.get("file_transfer") for doc in cursor if doc.get("file_transfer")}
|
||||||
|
except Exception as e:
|
||||||
|
print(f"[!] Error loading file transfers: {e}")
|
||||||
|
return {}
|
||||||
|
|
||||||
|
from fastapi.responses import FileResponse, Response
|
||||||
|
|
||||||
|
@app.get("/deploy_agent.sh")
|
||||||
|
async def get_agent_installer(x_agent_token: str = Header(None), token: Optional[str] = None):
|
||||||
|
# In strict mode the installer carries the live token, so downloading it must
|
||||||
|
# itself be authenticated (header for agent self-update, ?token= for humans).
|
||||||
|
if get_agent_auth_mode() == "strict" and not (is_valid_agent_token(x_agent_token) or is_valid_agent_token(token)):
|
||||||
|
raise HTTPException(status_code=401, detail="Invalid or missing agent token")
|
||||||
|
path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "deploy_agent.sh")
|
||||||
|
if not os.path.isfile(path):
|
||||||
|
raise HTTPException(status_code=404, detail="Installer script not found on server")
|
||||||
|
# Normalize CRLF -> LF (edited on Windows, executed by bash on Linux) and inject
|
||||||
|
# the live agent token so a freshly downloaded installer bakes it into the agent.
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
content = f.read().replace(b"\r\n", b"\n")
|
||||||
|
# Replace only the assignment, not the "was I injected?" sentinel comparison
|
||||||
|
# a few lines below it — a blanket replace turns that check into an
|
||||||
|
# always-true self-comparison and the installer discards the injected token.
|
||||||
|
content = content.replace(
|
||||||
|
b'AGENT_TOKEN="__AGENT_TOKEN__"',
|
||||||
|
b'AGENT_TOKEN="' + AGENT_TOKEN.encode("utf-8") + b'"',
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
return Response(
|
||||||
|
content=content,
|
||||||
|
media_type="text/x-shellscript",
|
||||||
|
headers={"Content-Disposition": 'attachment; filename="deploy_agent.sh"'},
|
||||||
|
)
|
||||||
|
|
||||||
|
@app.get("/api/download-file")
|
||||||
|
async def download_file(client_id: str, filename: str, token: Optional[str] = None, inline: bool = False, authorization: str = Header(None)):
|
||||||
|
authed = False
|
||||||
|
if authorization and authorization.startswith("Bearer ") and verify_token(authorization.split(" ")[1]):
|
||||||
|
authed = True
|
||||||
|
if token and verify_token(token):
|
||||||
|
authed = True
|
||||||
|
if not authed:
|
||||||
|
raise HTTPException(status_code=401, detail="Authentication token is invalid or has expired")
|
||||||
|
|
||||||
|
safe_name = os.path.basename(filename.strip())
|
||||||
|
path = os.path.join(FILE_TRANSFER_DIR, os.path.basename(client_id.strip()), safe_name)
|
||||||
|
if not os.path.isfile(path):
|
||||||
|
raise HTTPException(status_code=404, detail="File not found on server")
|
||||||
|
|
||||||
|
media_type = "video/mp4" if safe_name.lower().endswith(".mp4") else "application/octet-stream"
|
||||||
|
if inline:
|
||||||
|
return FileResponse(path, media_type=media_type)
|
||||||
|
return FileResponse(path, media_type=media_type, filename=safe_name)
|
||||||
|
|
||||||
|
@app.post("/api/heartbeat")
|
||||||
|
async def receive_heartbeat(client_id: str, payload: TelemetryPayload,
|
||||||
|
token_valid: bool = Depends(require_agent_token)):
|
||||||
|
await receive_telemetry(client_id, payload)
|
||||||
|
# Record migration status so the dashboard can confirm the fleet is upgraded
|
||||||
|
# and every node is authenticated before auth is flipped to strict.
|
||||||
|
clients_collection.update_one(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$set": {"agent_version": payload.agent_version or "unknown",
|
||||||
|
"speed_upload_mbps": payload.speed_upload_mbps,
|
||||||
|
"speed_download_mbps": payload.speed_download_mbps,
|
||||||
|
"agent_token_ok": token_valid}},
|
||||||
|
)
|
||||||
|
doc = clients_collection.find_one_and_update(
|
||||||
|
{"_id": client_id},
|
||||||
|
{"$set": {"pending_command": "none", "pending_file_request": "none", "pending_search": "none"}},
|
||||||
|
return_document=False
|
||||||
|
)
|
||||||
|
cmd = "none"
|
||||||
|
fname = "none"
|
||||||
|
search = "none"
|
||||||
|
shift_path = None
|
||||||
|
if doc:
|
||||||
|
cmd = doc.get("pending_command", "none")
|
||||||
|
fname = doc.get("pending_file_request", "none")
|
||||||
|
search = doc.get("pending_search", "none") or "none"
|
||||||
|
shift_path = doc.get("shift_path") or None
|
||||||
|
|
||||||
|
if cmd != "none":
|
||||||
|
append_to_logs("command_polled", client_id, {"command": cmd})
|
||||||
|
|
||||||
|
return {
|
||||||
|
"status": "success",
|
||||||
|
"command": cmd,
|
||||||
|
"filename": fname,
|
||||||
|
"search": search,
|
||||||
|
"shift_path": shift_path
|
||||||
|
}
|
||||||
|
|||||||
366
commands.json
366
commands.json
@@ -1,42 +1,169 @@
|
|||||||
{
|
{
|
||||||
"nt": {
|
"nt": {
|
||||||
"whoami": ["whoami"],
|
"whoami": [
|
||||||
"ls": ["cmd.exe", "/c", "dir"],
|
"whoami"
|
||||||
"echo": ["cmd.exe", "/c", "echo", "The prototype executed the echo command successfully on Windows!"],
|
],
|
||||||
"reboot": ["cmd.exe", "/c", "echo", "SIMULATED WINDOWS REBOOT"],
|
"ls": [
|
||||||
"systeminfo": ["systeminfo"],
|
"cmd.exe",
|
||||||
"ipconfig": ["ipconfig"],
|
"/c",
|
||||||
"date": ["cmd.exe", "/c", "date", "/t"],
|
"dir"
|
||||||
"check_space": ["powershell.exe", "-Command", "Get-PSDrive -PSProvider FileSystem"],
|
],
|
||||||
"check_takeleap": ["python", "client_agent_prototype.py", "--check-takeleap"],
|
"echo": [
|
||||||
"list_takeleap": ["python", "client_agent_prototype.py", "--list-takeleap"],
|
"cmd.exe",
|
||||||
"list_takeleap_size": ["python", "client_agent_prototype.py", "--list-takeleap-size"],
|
"/c",
|
||||||
"list_takeleap_shift_size": ["python", "client_agent_prototype.py", "--list-takeleap-sub-size", "SHIFT"],
|
"echo",
|
||||||
"list_takeleap_shift_files": ["python", "client_agent_prototype.py", "--list-takeleap-sub-files", "SHIFT"],
|
"The prototype executed the echo command successfully on Windows!"
|
||||||
"list_takeleap_errors_size": ["python", "client_agent_prototype.py", "--list-takeleap-sub-size", "Error_Videos"],
|
],
|
||||||
"list_takeleap_errors_files": ["python", "client_agent_prototype.py", "--list-takeleap-sub-files", "Error_Videos"],
|
"reboot": [
|
||||||
"list_takeleap_upload_size": ["python", "client_agent_prototype.py", "--list-takeleap-sub-size", "UPLOAD_FOLDER"],
|
"cmd.exe",
|
||||||
"list_takeleap_upload_files": ["python", "client_agent_prototype.py", "--list-takeleap-sub-files", "UPLOAD_FOLDER"]
|
"/c",
|
||||||
|
"echo",
|
||||||
|
"SIMULATED WINDOWS REBOOT"
|
||||||
|
],
|
||||||
|
"systeminfo": [
|
||||||
|
"systeminfo"
|
||||||
|
],
|
||||||
|
"ipconfig": [
|
||||||
|
"ipconfig"
|
||||||
|
],
|
||||||
|
"date": [
|
||||||
|
"cmd.exe",
|
||||||
|
"/c",
|
||||||
|
"date",
|
||||||
|
"/t"
|
||||||
|
],
|
||||||
|
"check_space": [
|
||||||
|
"powershell.exe",
|
||||||
|
"-Command",
|
||||||
|
"Get-PSDrive -PSProvider FileSystem"
|
||||||
|
],
|
||||||
|
"check_takeleap": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--check-takeleap"
|
||||||
|
],
|
||||||
|
"list_takeleap": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap"
|
||||||
|
],
|
||||||
|
"list_takeleap_size": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-size"
|
||||||
|
],
|
||||||
|
"list_takeleap_shift_size": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"SHIFT"
|
||||||
|
],
|
||||||
|
"list_takeleap_shift_files": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"SHIFT"
|
||||||
|
],
|
||||||
|
"list_takeleap_errors_size": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"Error_Videos"
|
||||||
|
],
|
||||||
|
"list_takeleap_errors_files": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"Error_Videos"
|
||||||
|
],
|
||||||
|
"list_takeleap_upload_size": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"UPLOAD_FOLDER"
|
||||||
|
],
|
||||||
|
"list_takeleap_upload_files": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"UPLOAD_FOLDER"
|
||||||
|
]
|
||||||
},
|
},
|
||||||
"posix": {
|
"posix": {
|
||||||
"whoami": ["whoami"],
|
"whoami": [
|
||||||
"ls": ["ls", "-la"],
|
"whoami"
|
||||||
"echo": ["echo", "The prototype executed the echo command successfully on POSIX/WSL!"],
|
],
|
||||||
"pwd": ["pwd"],
|
"ls": [
|
||||||
"date": ["date"],
|
"ls",
|
||||||
"time": ["date", "+%T"],
|
"-la"
|
||||||
"systeminfo": ["uname", "-a"],
|
],
|
||||||
"ipconfig": ["ip", "addr"],
|
"echo": [
|
||||||
"check_space": ["df", "-h"],
|
"echo",
|
||||||
"memory": ["free", "-h"],
|
"The prototype executed the echo command successfully on POSIX/WSL!"
|
||||||
"ps": ["ps", "aux"],
|
],
|
||||||
"kill_python": ["pkill", "-9", "python"],
|
"pwd": [
|
||||||
"ping_google": ["ping", "-c", "4", "google.com"],
|
"pwd"
|
||||||
"env": ["env"],
|
],
|
||||||
"hostname": ["hostname"],
|
"date": [
|
||||||
"reboot": ["sudo", "reboot"],
|
"date"
|
||||||
"restart_mysql": ["sudo", "systemctl", "restart", "mysql"],
|
],
|
||||||
"mysql_status": ["sudo", "systemctl", "status", "mysql"],
|
"time": [
|
||||||
|
"date",
|
||||||
|
"+%T"
|
||||||
|
],
|
||||||
|
"systeminfo": [
|
||||||
|
"uname",
|
||||||
|
"-a"
|
||||||
|
],
|
||||||
|
"ipconfig": [
|
||||||
|
"ip",
|
||||||
|
"addr"
|
||||||
|
],
|
||||||
|
"check_space": [
|
||||||
|
"df",
|
||||||
|
"-h"
|
||||||
|
],
|
||||||
|
"memory": [
|
||||||
|
"free",
|
||||||
|
"-h"
|
||||||
|
],
|
||||||
|
"ps": [
|
||||||
|
"ps",
|
||||||
|
"aux"
|
||||||
|
],
|
||||||
|
"kill_python": [
|
||||||
|
"pkill",
|
||||||
|
"-9",
|
||||||
|
"python"
|
||||||
|
],
|
||||||
|
"ping_google": [
|
||||||
|
"ping",
|
||||||
|
"-c",
|
||||||
|
"4",
|
||||||
|
"google.com"
|
||||||
|
],
|
||||||
|
"env": [
|
||||||
|
"env"
|
||||||
|
],
|
||||||
|
"hostname": [
|
||||||
|
"hostname"
|
||||||
|
],
|
||||||
|
"reboot": [
|
||||||
|
"sudo",
|
||||||
|
"reboot"
|
||||||
|
],
|
||||||
|
"restart_mysql": [
|
||||||
|
"sudo",
|
||||||
|
"systemctl",
|
||||||
|
"restart",
|
||||||
|
"mysql"
|
||||||
|
],
|
||||||
|
"mysql_status": [
|
||||||
|
"sudo",
|
||||||
|
"systemctl",
|
||||||
|
"status",
|
||||||
|
"mysql"
|
||||||
|
],
|
||||||
"repair_mysql": [
|
"repair_mysql": [
|
||||||
"sudo",
|
"sudo",
|
||||||
"mysqlcheck",
|
"mysqlcheck",
|
||||||
@@ -54,17 +181,162 @@
|
|||||||
"--optimize",
|
"--optimize",
|
||||||
"Test_seekright"
|
"Test_seekright"
|
||||||
],
|
],
|
||||||
"storage_devices": ["lsblk"],
|
"restart_rustdesk": [
|
||||||
"mounted_disks": ["df", "-hT"],
|
"sudo",
|
||||||
"netstat": ["ss", "-tulnp"],
|
"systemctl",
|
||||||
"check_takeleap": ["python3", "/opt/seekright-agent/client_agent_prototype.py", "--check-takeleap"],
|
"restart",
|
||||||
"list_takeleap": ["python3", "/opt/seekright-agent/client_agent_prototype.py", "--list-takeleap"],
|
"rustdesk"
|
||||||
"list_takeleap_size": ["python3", "/opt/seekright-agent/client_agent_prototype.py", "--list-takeleap-size"],
|
],
|
||||||
"list_takeleap_shift_size": ["python3", "/opt/seekright-agent/client_agent_prototype.py", "--list-takeleap-sub-size", "SHIFT"],
|
"check_display": [
|
||||||
"list_takeleap_shift_files": ["python3", "/opt/seekright-agent/client_agent_prototype.py", "--list-takeleap-sub-files", "SHIFT"],
|
"bash",
|
||||||
"list_takeleap_errors_size": ["python3", "/opt/seekright-agent/client_agent_prototype.py", "--list-takeleap-sub-size", "Error_Videos"],
|
"-c",
|
||||||
"list_takeleap_errors_files": ["python3", "/opt/seekright-agent/client_agent_prototype.py", "--list-takeleap-sub-files", "Error_Videos"],
|
"for i in /sys/class/drm/*/status; do echo \"$i: $(cat $i)\"; done"
|
||||||
"list_takeleap_upload_size": ["python3", "/opt/seekright-agent/client_agent_prototype.py", "--list-takeleap-sub-size", "UPLOAD_FOLDER"],
|
],
|
||||||
"list_takeleap_upload_files": ["python3", "/opt/seekright-agent/client_agent_prototype.py", "--list-takeleap-sub-files", "UPLOAD_FOLDER"]
|
"drm_details": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"ls -l /sys/class/drm && echo '---' && cat /sys/class/drm/*/status"
|
||||||
|
],
|
||||||
|
"headless_check": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"connected=0; for f in /sys/class/drm/*/status; do if grep -q connected $f; then connected=1; fi; done; if [ $connected -eq 1 ]; then echo DISPLAY_CONNECTED; else echo HEADLESS; fi"
|
||||||
|
],
|
||||||
|
"framebuffer_check": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"ls -l /dev/fb* && cat /proc/fb"
|
||||||
|
],
|
||||||
|
"install_temp_tools": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"sudo apt update && sudo apt install -y lm-sensors"
|
||||||
|
],
|
||||||
|
"detect_sensors": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"sudo sensors-detect --auto"
|
||||||
|
],
|
||||||
|
"gpu_temp": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader"
|
||||||
|
],
|
||||||
|
"temperature": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"sensors"
|
||||||
|
],
|
||||||
|
"check_edid": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"for f in /sys/class/drm/*/edid; do echo $f; [ -s $f ] && echo 'EDID PRESENT' || echo 'NO EDID'; done"
|
||||||
|
],
|
||||||
|
"xrandr_display": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"xrandr --query 2>&1"
|
||||||
|
],
|
||||||
|
"nvidia_status": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"nvidia-smi"
|
||||||
|
],
|
||||||
|
"check_display_session": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"echo DISPLAY=$DISPLAY && loginctl list-sessions"
|
||||||
|
],
|
||||||
|
"check_gpu_display": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"lspci | grep -i -E 'vga|display|3d'"
|
||||||
|
],
|
||||||
|
"rustdesk_status": [
|
||||||
|
"sudo",
|
||||||
|
"systemctl",
|
||||||
|
"status",
|
||||||
|
"rustdesk"
|
||||||
|
],
|
||||||
|
"storage_devices": [
|
||||||
|
"lsblk"
|
||||||
|
],
|
||||||
|
"mounted_disks": [
|
||||||
|
"df",
|
||||||
|
"-hT"
|
||||||
|
],
|
||||||
|
"netstat": [
|
||||||
|
"ss",
|
||||||
|
"-tulnp"
|
||||||
|
],
|
||||||
|
"update_agent": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"set -e; PY=/opt/seekright-agent/client_agent_prototype.py; URL=$(sed -n 's/^CENTRAL_BASE = \"\\(.*\\)\"/\\1/p' \"$PY\" | head -1); TOK=$(sed -n 's/^AGENT_TOKEN = \"\\(.*\\)\"/\\1/p' \"$PY\" | head -1); curl -fsSL -H \"X-Agent-Token: $TOK\" \"$URL/deploy_agent.sh\" -o /tmp/sr_update.sh; test -s /tmp/sr_update.sh; systemd-run --collect --description='SeekRight agent self-update' bash /tmp/sr_update.sh >/dev/null 2>&1; echo 'Update launched detached via systemd-run; agent will restart on the new version shortly.'"
|
||||||
|
],
|
||||||
|
"check_takeleap": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--check-takeleap"
|
||||||
|
],
|
||||||
|
"list_takeleap": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap"
|
||||||
|
],
|
||||||
|
"list_takeleap_size": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-size"
|
||||||
|
],
|
||||||
|
"list_takeleap_shift_size": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"SHIFT"
|
||||||
|
],
|
||||||
|
"list_takeleap_shift_files": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"SHIFT"
|
||||||
|
],
|
||||||
|
"list_takeleap_errors_size": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"Error_Videos"
|
||||||
|
],
|
||||||
|
"list_takeleap_errors_files": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"Error_Videos"
|
||||||
|
],
|
||||||
|
"list_takeleap_upload_size": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"UPLOAD_FOLDER"
|
||||||
|
],
|
||||||
|
"list_takeleap_upload_files": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"UPLOAD_FOLDER"
|
||||||
|
],
|
||||||
|
"last_reboot": [
|
||||||
|
"last",
|
||||||
|
"reboot"
|
||||||
|
],
|
||||||
|
"dummy": [
|
||||||
|
"last",
|
||||||
|
"reboot"
|
||||||
|
],
|
||||||
|
"update_diag": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"echo HOST:$(hostname); echo USER:$(id -un); echo MACHINE_ID:$(cat /etc/machine-id 2>/dev/null); echo AGENT_PROCS:; pgrep -af client_agent_prototype 2>/dev/null; echo APT_PROCS:; ps aux | grep -E \"apt-get|apt |dpkg|unattended\" | grep -v grep; echo UPDATE_UNITS:; systemctl list-units \"run-r*\" --no-legend --all 2>&1 | head -5"
|
||||||
|
]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
332
commands_new.json
Normal file
332
commands_new.json
Normal file
@@ -0,0 +1,332 @@
|
|||||||
|
{
|
||||||
|
"nt": {
|
||||||
|
"whoami": [
|
||||||
|
"whoami"
|
||||||
|
],
|
||||||
|
"ls": [
|
||||||
|
"cmd.exe",
|
||||||
|
"/c",
|
||||||
|
"dir"
|
||||||
|
],
|
||||||
|
"echo": [
|
||||||
|
"cmd.exe",
|
||||||
|
"/c",
|
||||||
|
"echo",
|
||||||
|
"The prototype executed the echo command successfully on Windows!"
|
||||||
|
],
|
||||||
|
"reboot": [
|
||||||
|
"cmd.exe",
|
||||||
|
"/c",
|
||||||
|
"echo",
|
||||||
|
"SIMULATED WINDOWS REBOOT"
|
||||||
|
],
|
||||||
|
"systeminfo": [
|
||||||
|
"systeminfo"
|
||||||
|
],
|
||||||
|
"ipconfig": [
|
||||||
|
"ipconfig"
|
||||||
|
],
|
||||||
|
"date": [
|
||||||
|
"cmd.exe",
|
||||||
|
"/c",
|
||||||
|
"date",
|
||||||
|
"/t"
|
||||||
|
],
|
||||||
|
"check_space": [
|
||||||
|
"powershell.exe",
|
||||||
|
"-Command",
|
||||||
|
"Get-PSDrive -PSProvider FileSystem"
|
||||||
|
],
|
||||||
|
"check_takeleap": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--check-takeleap"
|
||||||
|
],
|
||||||
|
"list_takeleap": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap"
|
||||||
|
],
|
||||||
|
"list_takeleap_size": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-size"
|
||||||
|
],
|
||||||
|
"list_takeleap_shift_size": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"SHIFT"
|
||||||
|
],
|
||||||
|
"list_takeleap_shift_files": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"SHIFT"
|
||||||
|
],
|
||||||
|
"list_takeleap_errors_size": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"Error_Videos"
|
||||||
|
],
|
||||||
|
"list_takeleap_errors_files": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"Error_Videos"
|
||||||
|
],
|
||||||
|
"list_takeleap_upload_size": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"UPLOAD_FOLDER"
|
||||||
|
],
|
||||||
|
"list_takeleap_upload_files": [
|
||||||
|
"python",
|
||||||
|
"client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"UPLOAD_FOLDER"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"posix": {
|
||||||
|
"whoami": [
|
||||||
|
"whoami"
|
||||||
|
],
|
||||||
|
"ls": [
|
||||||
|
"ls",
|
||||||
|
"-la"
|
||||||
|
],
|
||||||
|
"echo": [
|
||||||
|
"echo",
|
||||||
|
"The prototype executed the echo command successfully on POSIX/WSL!"
|
||||||
|
],
|
||||||
|
"pwd": [
|
||||||
|
"pwd"
|
||||||
|
],
|
||||||
|
"date": [
|
||||||
|
"date"
|
||||||
|
],
|
||||||
|
"time": [
|
||||||
|
"date",
|
||||||
|
"+%T"
|
||||||
|
],
|
||||||
|
"systeminfo": [
|
||||||
|
"uname",
|
||||||
|
"-a"
|
||||||
|
],
|
||||||
|
"ipconfig": [
|
||||||
|
"ip",
|
||||||
|
"addr"
|
||||||
|
],
|
||||||
|
"check_space": [
|
||||||
|
"df",
|
||||||
|
"-h"
|
||||||
|
],
|
||||||
|
"memory": [
|
||||||
|
"free",
|
||||||
|
"-h"
|
||||||
|
],
|
||||||
|
"ps": [
|
||||||
|
"ps",
|
||||||
|
"aux"
|
||||||
|
],
|
||||||
|
"kill_python": [
|
||||||
|
"pkill",
|
||||||
|
"-9",
|
||||||
|
"python"
|
||||||
|
],
|
||||||
|
"ping_google": [
|
||||||
|
"ping",
|
||||||
|
"-c",
|
||||||
|
"4",
|
||||||
|
"google.com"
|
||||||
|
],
|
||||||
|
"env": [
|
||||||
|
"env"
|
||||||
|
],
|
||||||
|
"hostname": [
|
||||||
|
"hostname"
|
||||||
|
],
|
||||||
|
"reboot": [
|
||||||
|
"sudo",
|
||||||
|
"reboot"
|
||||||
|
],
|
||||||
|
"restart_mysql": [
|
||||||
|
"sudo",
|
||||||
|
"systemctl",
|
||||||
|
"restart",
|
||||||
|
"mysql"
|
||||||
|
],
|
||||||
|
"mysql_status": [
|
||||||
|
"sudo",
|
||||||
|
"systemctl",
|
||||||
|
"status",
|
||||||
|
"mysql"
|
||||||
|
],
|
||||||
|
"repair_mysql": [
|
||||||
|
"sudo",
|
||||||
|
"mysqlcheck",
|
||||||
|
"--defaults-file=/root/.my.cnf",
|
||||||
|
"--auto-repair",
|
||||||
|
"--optimize",
|
||||||
|
"Test_seekright",
|
||||||
|
"tbl_site_anomaly"
|
||||||
|
],
|
||||||
|
"repair_mysql_db": [
|
||||||
|
"sudo",
|
||||||
|
"mysqlcheck",
|
||||||
|
"--defaults-file=/root/.my.cnf",
|
||||||
|
"--auto-repair",
|
||||||
|
"--optimize",
|
||||||
|
"Test_seekright"
|
||||||
|
],
|
||||||
|
"restart_rustdesk": [
|
||||||
|
"sudo",
|
||||||
|
"systemctl",
|
||||||
|
"restart",
|
||||||
|
"rustdesk"
|
||||||
|
],
|
||||||
|
"check_display": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"for i in /sys/class/drm/*/status; do echo \"$i: $(cat $i)\"; done"
|
||||||
|
],
|
||||||
|
"drm_details": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"ls -l /sys/class/drm && echo '---' && cat /sys/class/drm/*/status"
|
||||||
|
],
|
||||||
|
"headless_check": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"connected=0; for f in /sys/class/drm/*/status; do if grep -q connected $f; then connected=1; fi; done; if [ $connected -eq 1 ]; then echo DISPLAY_CONNECTED; else echo HEADLESS; fi"
|
||||||
|
],
|
||||||
|
"framebuffer_check": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"ls -l /dev/fb* && cat /proc/fb"
|
||||||
|
],
|
||||||
|
"install_temp_tools": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"sudo apt update && sudo apt install -y lm-sensors"
|
||||||
|
],
|
||||||
|
"detect_sensors": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"sudo sensors-detect --auto"
|
||||||
|
],
|
||||||
|
"gpu_temp": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader"
|
||||||
|
],
|
||||||
|
"temperature": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"sensors"
|
||||||
|
],
|
||||||
|
"check_edid": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"for f in /sys/class/drm/*/edid; do echo $f; [ -s $f ] && echo 'EDID PRESENT' || echo 'NO EDID'; done"
|
||||||
|
],
|
||||||
|
"xrandr_display": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"xrandr --query 2>&1"
|
||||||
|
],
|
||||||
|
"nvidia_status": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"nvidia-smi"
|
||||||
|
],
|
||||||
|
"check_display_session": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"echo DISPLAY=$DISPLAY && loginctl list-sessions"
|
||||||
|
],
|
||||||
|
"check_gpu_display": [
|
||||||
|
"bash",
|
||||||
|
"-c",
|
||||||
|
"lspci | grep -i -E 'vga|display|3d'"
|
||||||
|
],
|
||||||
|
"rustdesk_status": [
|
||||||
|
"sudo",
|
||||||
|
"systemctl",
|
||||||
|
"status",
|
||||||
|
"rustdesk"
|
||||||
|
],
|
||||||
|
"storage_devices": [
|
||||||
|
"lsblk"
|
||||||
|
],
|
||||||
|
"mounted_disks": [
|
||||||
|
"df",
|
||||||
|
"-hT"
|
||||||
|
],
|
||||||
|
"netstat": [
|
||||||
|
"ss",
|
||||||
|
"-tulnp"
|
||||||
|
],
|
||||||
|
"check_takeleap": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--check-takeleap"
|
||||||
|
],
|
||||||
|
"list_takeleap": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap"
|
||||||
|
],
|
||||||
|
"list_takeleap_size": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-size"
|
||||||
|
],
|
||||||
|
"list_takeleap_shift_size": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"SHIFT"
|
||||||
|
],
|
||||||
|
"list_takeleap_shift_files": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"SHIFT"
|
||||||
|
],
|
||||||
|
"list_takeleap_errors_size": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"Error_Videos"
|
||||||
|
],
|
||||||
|
"list_takeleap_errors_files": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"Error_Videos"
|
||||||
|
],
|
||||||
|
"list_takeleap_upload_size": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-size",
|
||||||
|
"UPLOAD_FOLDER"
|
||||||
|
],
|
||||||
|
"list_takeleap_upload_files": [
|
||||||
|
"python3",
|
||||||
|
"/opt/seekright-agent/client_agent_prototype.py",
|
||||||
|
"--list-takeleap-sub-files",
|
||||||
|
"UPLOAD_FOLDER"
|
||||||
|
],
|
||||||
|
"last_reboot": [
|
||||||
|
"last",
|
||||||
|
"reboot"
|
||||||
|
],
|
||||||
|
"dummy": [
|
||||||
|
"last",
|
||||||
|
"reboot"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
551
deploy_agent.sh
551
deploy_agent.sh
@@ -10,26 +10,59 @@ if [ "$EUID" -ne 0 ]; then
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
# ====================================================
|
# ====================================================
|
||||||
# ⚙️ CONFIGURATION: Set your Client ID & Central URL
|
# ⚙️ CONFIGURATION: Client ID, Central URL, Agent Token
|
||||||
# ====================================================
|
# ====================================================
|
||||||
CLIENT_ID=""
|
# Resolution priority for each value: explicit CLI arg > existing install > default.
|
||||||
|
# Reading from the existing install makes re-running with no args a safe in-place
|
||||||
|
# upgrade — the node keeps its identity. This is what fleet self-update relies on.
|
||||||
|
# Usage: deploy_agent.sh [CLIENT_ID] [CENTRAL_URL] [AGENT_TOKEN]
|
||||||
|
EXISTING_PY="/opt/seekright-agent/client_agent_prototype.py"
|
||||||
|
|
||||||
# For Production deployment, use: "http://rmm-backend.seekright.com"
|
CLIENT_ID="${1:-}"
|
||||||
# For Local WSL testing on your laptop, use: "http://172.18.16.1:8000"
|
CENTRAL_URL="${2:-}"
|
||||||
CENTRAL_URL="http://rmm-backend.seekright.com"
|
ARG_TOKEN="${3:-}"
|
||||||
|
|
||||||
# Check if Client ID is set; if not, check command line arguments, otherwise default to hostname
|
if [ -f "$EXISTING_PY" ]; then
|
||||||
if [ -z "$CLIENT_ID" ]; then
|
if [ -z "$CLIENT_ID" ]; then
|
||||||
CLIENT_ID="${1:-$(hostname)}"
|
CLIENT_ID="$(sed -n 's/^CLIENT_ID = "\(.*\)"/\1/p' "$EXISTING_PY" | head -1)"
|
||||||
|
fi
|
||||||
|
if [ -z "$CENTRAL_URL" ]; then
|
||||||
|
CENTRAL_URL="$(sed -n 's/^CENTRAL_BASE = "\(.*\)"/\1/p' "$EXISTING_PY" | head -1)"
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo "🚀 Preparing SeekRight RMM Agent installation..."
|
# Fallbacks when nothing was provided and there is no prior install
|
||||||
|
if [ -z "$CLIENT_ID" ]; then CLIENT_ID="$(hostname)"; fi
|
||||||
|
if [ -z "$CENTRAL_URL" ]; then CENTRAL_URL="http://rmm-backend.seekright.com"; fi
|
||||||
|
|
||||||
|
# Agent token. The server injects the live token into the line below at download
|
||||||
|
# time (replacing __AGENT_TOKEN__). Priority: CLI arg > server-injected > existing install.
|
||||||
|
AGENT_TOKEN="__AGENT_TOKEN__"
|
||||||
|
if [ -n "$ARG_TOKEN" ]; then
|
||||||
|
AGENT_TOKEN="$ARG_TOKEN"
|
||||||
|
elif [ "$AGENT_TOKEN" = "__AGENT_""TOKEN__" ]; then
|
||||||
|
# (placeholder split in two so the server's token injection can't rewrite this
|
||||||
|
# comparison — it must only replace the assignment above)
|
||||||
|
if [ -f "$EXISTING_PY" ]; then
|
||||||
|
AGENT_TOKEN="$(sed -n 's/^AGENT_TOKEN = "\(.*\)"/\1/p' "$EXISTING_PY" | head -1)"
|
||||||
|
else
|
||||||
|
AGENT_TOKEN=""
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "🚀 Preparing SeekRight RMM Agent installation (agent v3.2-auth)..."
|
||||||
echo "📍 Target Client ID: $CLIENT_ID"
|
echo "📍 Target Client ID: $CLIENT_ID"
|
||||||
echo "🌐 Central Server URL: $CENTRAL_URL"
|
echo "🌐 Central Server URL: $CENTRAL_URL"
|
||||||
|
|
||||||
# 1. Install System Dependencies
|
# 1. Install System Dependencies
|
||||||
echo "🔄 Updating system package indexes and installing python dependencies..."
|
echo "🔄 Updating system package indexes and installing python dependencies..."
|
||||||
apt-get update -y
|
|
||||||
|
# Fix potential MySQL GPG key expiration issues before updating (common issue on Ubuntu 22.04)
|
||||||
|
apt-key del B7B3B788A8D3785C || true
|
||||||
|
wget -q -O - https://repo.mysql.com/RPM-GPG-KEY-mysql-2025 | apt-key add - || true
|
||||||
|
wget -q -O - https://repo.mysql.com/RPM-GPG-KEY-mysql-2025 | gpg --dearmor --yes -o /usr/share/keyrings/mysql-apt-config.gpg || true
|
||||||
|
|
||||||
|
apt-get update -y || true
|
||||||
apt-get install -y python3 python3-pip python3-psutil
|
apt-get install -y python3 python3-pip python3-psutil
|
||||||
|
|
||||||
# 2. Create Destination Directories
|
# 2. Create Destination Directories
|
||||||
@@ -41,6 +74,7 @@ chmod 755 /opt/seekright-agent
|
|||||||
echo "📝 Generating client agent file..."
|
echo "📝 Generating client agent file..."
|
||||||
cat << 'EOF' > /opt/seekright-agent/client_agent_prototype.py
|
cat << 'EOF' > /opt/seekright-agent/client_agent_prototype.py
|
||||||
import urllib.request
|
import urllib.request
|
||||||
|
import urllib.parse
|
||||||
import json
|
import json
|
||||||
import time
|
import time
|
||||||
import subprocess
|
import subprocess
|
||||||
@@ -50,17 +84,49 @@ import sys
|
|||||||
import string
|
import string
|
||||||
import threading
|
import threading
|
||||||
|
|
||||||
|
def get_cpu_temp():
|
||||||
|
if not hasattr(psutil, "sensors_temperatures"):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
temps = psutil.sensors_temperatures()
|
||||||
|
if not temps:
|
||||||
|
return None
|
||||||
|
for name in ['coretemp', 'k10temp', 'acpitz', 'cpu_thermal']:
|
||||||
|
if name in temps and temps[name]:
|
||||||
|
return temps[name][0].current
|
||||||
|
for name, entries in temps.items():
|
||||||
|
if entries:
|
||||||
|
return entries[0].current
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return None
|
||||||
|
|
||||||
|
AGENT_VERSION = "3.3-shift"
|
||||||
|
|
||||||
CLIENT_ID = "TEMPLATE_CLIENT_ID"
|
CLIENT_ID = "TEMPLATE_CLIENT_ID"
|
||||||
CENTRAL_BASE = "TEMPLATE_CENTRAL_URL"
|
CENTRAL_BASE = "TEMPLATE_CENTRAL_URL"
|
||||||
|
AGENT_TOKEN = "TEMPLATE_AGENT_TOKEN"
|
||||||
|
|
||||||
|
# Attach the shared headers (agent token + tunnel bypass) to every outbound request
|
||||||
|
# by installing a default opener, so we don't have to touch each call site.
|
||||||
|
_opener = urllib.request.build_opener()
|
||||||
|
_opener.addheaders = [('ngrok-skip-browser-warning', 'true'), ('X-Agent-Token', AGENT_TOKEN)]
|
||||||
|
urllib.request.install_opener(_opener)
|
||||||
|
|
||||||
CENTRAL_API_URL = f"{CENTRAL_BASE}/api/get-command?client_id={CLIENT_ID}"
|
CENTRAL_API_URL = f"{CENTRAL_BASE}/api/get-command?client_id={CLIENT_ID}"
|
||||||
CENTRAL_TELEMETRY_URL = f"{CENTRAL_BASE}/api/telemetry?client_id={CLIENT_ID}"
|
CENTRAL_HEARTBEAT_URL = f"{CENTRAL_BASE}/api/heartbeat?client_id={CLIENT_ID}"
|
||||||
|
|
||||||
platform_name = "nt" if os.name == "nt" else "posix"
|
platform_name = "nt" if os.name == "nt" else "posix"
|
||||||
CENTRAL_COMMANDS_URL = f"{CENTRAL_BASE}/api/get-commands?platform={platform_name}"
|
CENTRAL_COMMANDS_URL = f"{CENTRAL_BASE}/api/get-commands?platform={platform_name}"
|
||||||
|
|
||||||
|
CENTRAL_FILE_REQUEST_URL = f"{CENTRAL_BASE}/api/get-file-request?client_id={CLIENT_ID}"
|
||||||
|
CENTRAL_FILE_STATUS_URL = f"{CENTRAL_BASE}/api/file-transfer-status?client_id={CLIENT_ID}"
|
||||||
|
|
||||||
CACHED_COMMANDS = {}
|
CACHED_COMMANDS = {}
|
||||||
command_in_progress = False
|
command_in_progress = False
|
||||||
|
file_transfer_in_progress = False
|
||||||
|
SHIFT_PATH = None # per-site SHIFT folder, configured on the dashboard, delivered via heartbeat
|
||||||
|
search_in_progress = False
|
||||||
|
|
||||||
def run_command_async(command_key, safe_cmd_list, is_shutdown_or_restart):
|
def run_command_async(command_key, safe_cmd_list, is_shutdown_or_restart):
|
||||||
global command_in_progress
|
global command_in_progress
|
||||||
@@ -87,12 +153,43 @@ def run_command_async(command_key, safe_cmd_list, is_shutdown_or_restart):
|
|||||||
finally:
|
finally:
|
||||||
command_in_progress = False
|
command_in_progress = False
|
||||||
|
|
||||||
|
def execute_command_async(command_key):
|
||||||
|
# Resolve a command key received via heartbeat against the whitelist and run it.
|
||||||
|
# (The heartbeat loop sets command_in_progress=True before dispatching here, so
|
||||||
|
# every exit path must clear it unless run_command_async takes over that duty.)
|
||||||
|
global command_in_progress
|
||||||
|
try:
|
||||||
|
allowed_commands = load_allowed_commands()
|
||||||
|
if command_key not in allowed_commands:
|
||||||
|
warning_msg = f"Server requested unknown/unwhitelisted command: '{command_key}'. Ignored."
|
||||||
|
print(f" -> [!] SECURITY WARNING: {warning_msg}")
|
||||||
|
send_rocket_chat_notification("⚠️ SECURITY WARNING", warning_msg, "#FFA500")
|
||||||
|
command_in_progress = False
|
||||||
|
return
|
||||||
|
|
||||||
|
print(f" -> Executing approved command: '{command_key}'")
|
||||||
|
safe_cmd_list = allowed_commands[command_key]
|
||||||
|
is_shutdown_or_restart = any(x in command_key.lower() for x in ["reboot", "restart", "kill_python"])
|
||||||
|
if is_shutdown_or_restart:
|
||||||
|
print(" -> Shutdown/restart command detected. Sending pre-execution notification...")
|
||||||
|
send_rocket_chat_notification(
|
||||||
|
f"🔄 Executing: {command_key}",
|
||||||
|
f"System/agent is performing an action: {' '.join(safe_cmd_list)}\nConnection may drop temporarily.",
|
||||||
|
"#FFA500"
|
||||||
|
)
|
||||||
|
time.sleep(3)
|
||||||
|
# run_command_async clears command_in_progress in its finally block
|
||||||
|
run_command_async(command_key, safe_cmd_list, is_shutdown_or_restart)
|
||||||
|
except Exception as e:
|
||||||
|
print(f" -> [!] Command dispatch error: {e}")
|
||||||
|
command_in_progress = False
|
||||||
|
|
||||||
def load_allowed_commands():
|
def load_allowed_commands():
|
||||||
global CACHED_COMMANDS
|
global CACHED_COMMANDS
|
||||||
try:
|
try:
|
||||||
req = urllib.request.Request(CENTRAL_COMMANDS_URL)
|
req = urllib.request.Request(CENTRAL_COMMANDS_URL)
|
||||||
req.add_header('ngrok-skip-browser-warning', 'true')
|
req.add_header('ngrok-skip-browser-warning', 'true')
|
||||||
with urllib.request.urlopen(req, timeout=3) as response:
|
with urllib.request.urlopen(req, timeout=15) as response:
|
||||||
commands = json.loads(response.read().decode())
|
commands = json.loads(response.read().decode())
|
||||||
if commands:
|
if commands:
|
||||||
CACHED_COMMANDS = commands
|
CACHED_COMMANDS = commands
|
||||||
@@ -121,11 +218,42 @@ def send_rocket_chat_notification(title, message, color="#764FA5"):
|
|||||||
}
|
}
|
||||||
data = json.dumps(payload).encode('utf-8')
|
data = json.dumps(payload).encode('utf-8')
|
||||||
req = urllib.request.Request(ROCKET_CHAT_WEBHOOK, data=data, headers={'Content-Type': 'application/json'}, method='POST')
|
req = urllib.request.Request(ROCKET_CHAT_WEBHOOK, data=data, headers={'Content-Type': 'application/json'}, method='POST')
|
||||||
urllib.request.urlopen(req, timeout=3)
|
urllib.request.urlopen(req, timeout=15)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f" -> [!] Failed to send webhook: {e}")
|
print(f" -> [!] Failed to send webhook: {e}")
|
||||||
|
|
||||||
def collect_and_send_telemetry():
|
def get_linux_labels():
|
||||||
|
labels = {}
|
||||||
|
try:
|
||||||
|
if os.path.exists("/dev/disk/by-label"):
|
||||||
|
for label in os.listdir("/dev/disk/by-label"):
|
||||||
|
try:
|
||||||
|
full_path = os.path.join("/dev/disk/by-label", label)
|
||||||
|
real_dev = os.path.realpath(full_path)
|
||||||
|
labels[real_dev] = label
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return labels
|
||||||
|
|
||||||
|
def get_windows_label(drive):
|
||||||
|
try:
|
||||||
|
import ctypes
|
||||||
|
volumeNameBuffer = ctypes.create_unicode_buffer(260)
|
||||||
|
rc = ctypes.windll.kernel32.GetVolumeInformationW(
|
||||||
|
ctypes.c_wchar_p(drive),
|
||||||
|
volumeNameBuffer,
|
||||||
|
ctypes.sizeof(volumeNameBuffer),
|
||||||
|
None, None, None, None, 0
|
||||||
|
)
|
||||||
|
if rc:
|
||||||
|
return volumeNameBuffer.value
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return ""
|
||||||
|
|
||||||
|
def send_heartbeat():
|
||||||
try:
|
try:
|
||||||
gpus = []
|
gpus = []
|
||||||
try:
|
try:
|
||||||
@@ -151,18 +279,55 @@ def collect_and_send_telemetry():
|
|||||||
except FileNotFoundError:
|
except FileNotFoundError:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
|
linux_labels = get_linux_labels() if os.name != "nt" else {}
|
||||||
disks = []
|
disks = []
|
||||||
|
seen_devices = set() # Deduplicate: skip bind-mounts that reuse the same device
|
||||||
for part in psutil.disk_partitions(all=False):
|
for part in psutil.disk_partitions(all=False):
|
||||||
|
# Skip loop devices and unknown/empty fstypes
|
||||||
if 'loop' in part.device or 'loop' in part.fstype or part.fstype == '':
|
if 'loop' in part.device or 'loop' in part.fstype or part.fstype == '':
|
||||||
continue
|
continue
|
||||||
|
# Skip snap & squashfs mounts
|
||||||
if part.mountpoint.startswith('/snap') or part.fstype == 'squashfs':
|
if part.mountpoint.startswith('/snap') or part.fstype == 'squashfs':
|
||||||
continue
|
continue
|
||||||
|
# Skip boot, WSL internal, and other virtual/system paths
|
||||||
if part.mountpoint.startswith('/boot') or part.mountpoint.startswith('/mnt/wsl'):
|
if part.mountpoint.startswith('/boot') or part.mountpoint.startswith('/mnt/wsl'):
|
||||||
continue
|
continue
|
||||||
|
if part.mountpoint.startswith('/var/snap') or part.mountpoint.startswith('/run'):
|
||||||
|
continue
|
||||||
|
# Skip virtual/overlay filesystems
|
||||||
|
if part.fstype in ('tmpfs', 'devtmpfs', 'overlay', 'aufs', 'fusectl',
|
||||||
|
'proc', 'sysfs', 'cgroup', 'cgroup2', 'pstore',
|
||||||
|
'securityfs', 'debugfs', 'tracefs', 'bpf'):
|
||||||
|
continue
|
||||||
|
# Deduplicate: each physical device reported only once (eliminates bind-mounts)
|
||||||
|
if part.device in seen_devices:
|
||||||
|
continue
|
||||||
|
seen_devices.add(part.device)
|
||||||
try:
|
try:
|
||||||
usage = psutil.disk_usage(part.mountpoint)
|
usage = psutil.disk_usage(part.mountpoint)
|
||||||
|
|
||||||
|
# Resolve label/friendly name
|
||||||
|
label = ""
|
||||||
|
if os.name == "nt":
|
||||||
|
label = get_windows_label(part.mountpoint)
|
||||||
|
if not label:
|
||||||
|
if part.mountpoint == "C:\\":
|
||||||
|
label = "OS"
|
||||||
|
else:
|
||||||
|
_mp = part.mountpoint.strip('\\')
|
||||||
|
label = f"Local Disk ({_mp})"
|
||||||
|
else:
|
||||||
|
label = linux_labels.get(part.device, "")
|
||||||
|
if not label:
|
||||||
|
if part.mountpoint == "/":
|
||||||
|
label = "Computer"
|
||||||
|
else:
|
||||||
|
label = os.path.basename(part.mountpoint) or "Volume"
|
||||||
|
|
||||||
disks.append({
|
disks.append({
|
||||||
"mount": part.mountpoint,
|
"mount": part.mountpoint,
|
||||||
|
"device": part.device,
|
||||||
|
"label": label,
|
||||||
"total_gb": round(usage.total / (1024**3), 2),
|
"total_gb": round(usage.total / (1024**3), 2),
|
||||||
"free_gb": round(usage.free / (1024**3), 2),
|
"free_gb": round(usage.free / (1024**3), 2),
|
||||||
"percent": usage.percent
|
"percent": usage.percent
|
||||||
@@ -172,20 +337,60 @@ def collect_and_send_telemetry():
|
|||||||
|
|
||||||
payload = {
|
payload = {
|
||||||
"cpu_percent": psutil.cpu_percent(interval=0.5),
|
"cpu_percent": psutil.cpu_percent(interval=0.5),
|
||||||
|
"cpu_temp": get_cpu_temp(),
|
||||||
"memory_percent": psutil.virtual_memory().percent,
|
"memory_percent": psutil.virtual_memory().percent,
|
||||||
"memory_total_gb": round(psutil.virtual_memory().total / (1024**3), 2),
|
"memory_total_gb": round(psutil.virtual_memory().total / (1024**3), 2),
|
||||||
"memory_free_gb": round(psutil.virtual_memory().available / (1024**3), 2),
|
"memory_free_gb": round(psutil.virtual_memory().available / (1024**3), 2),
|
||||||
"disks": disks,
|
"disks": disks,
|
||||||
"gpus": gpus
|
"gpus": gpus,
|
||||||
|
"agent_version": AGENT_VERSION
|
||||||
}
|
}
|
||||||
|
|
||||||
data = json.dumps(payload).encode('utf-8')
|
data = json.dumps(payload).encode('utf-8')
|
||||||
req = urllib.request.Request(CENTRAL_TELEMETRY_URL, data=data, headers={'Content-Type': 'application/json'}, method='POST')
|
req = urllib.request.Request(CENTRAL_HEARTBEAT_URL, data=data, headers={'Content-Type': 'application/json'}, method='POST')
|
||||||
req.add_header('ngrok-skip-browser-warning', 'true')
|
req.add_header('ngrok-skip-browser-warning', 'true')
|
||||||
urllib.request.urlopen(req, timeout=3)
|
with urllib.request.urlopen(req, timeout=30) as response:
|
||||||
print("[*] Telemetry pushed successfully.")
|
res = json.loads(response.read().decode())
|
||||||
|
|
||||||
|
# Handle commands
|
||||||
|
cmd = res.get("command", "none")
|
||||||
|
if cmd != "none":
|
||||||
|
global command_in_progress
|
||||||
|
if not command_in_progress:
|
||||||
|
command_in_progress = True
|
||||||
|
t = threading.Thread(target=execute_command_async, args=(cmd,))
|
||||||
|
t.daemon = True
|
||||||
|
t.start()
|
||||||
|
|
||||||
|
# Handle file requests
|
||||||
|
fname = res.get("filename", "none")
|
||||||
|
if fname != "none":
|
||||||
|
global file_transfer_in_progress
|
||||||
|
if not file_transfer_in_progress:
|
||||||
|
file_transfer_in_progress = True
|
||||||
|
t = threading.Thread(target=handle_file_request_async, args=(fname,))
|
||||||
|
t.daemon = True
|
||||||
|
t.start()
|
||||||
|
|
||||||
|
# Server-configured SHIFT folder for this node
|
||||||
|
global SHIFT_PATH
|
||||||
|
sp = res.get("shift_path")
|
||||||
|
if sp:
|
||||||
|
SHIFT_PATH = sp
|
||||||
|
|
||||||
|
# Handle video search requests
|
||||||
|
squery = res.get("search", "none")
|
||||||
|
if squery and squery != "none":
|
||||||
|
global search_in_progress
|
||||||
|
if not search_in_progress:
|
||||||
|
search_in_progress = True
|
||||||
|
t = threading.Thread(target=run_file_search, args=(squery,))
|
||||||
|
t.daemon = True
|
||||||
|
t.start()
|
||||||
|
|
||||||
|
print("[*] Heartbeat successful.")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"[!] Failed to push telemetry: {e}")
|
print(f"[!] Failed heartbeat: {e}")
|
||||||
|
|
||||||
def send_command_result_to_server(command: str, returncode: int, stdout: str, stderr: str):
|
def send_command_result_to_server(command: str, returncode: int, stdout: str, stderr: str):
|
||||||
try:
|
try:
|
||||||
@@ -205,7 +410,7 @@ def send_command_result_to_server(command: str, returncode: int, stdout: str, st
|
|||||||
method='POST'
|
method='POST'
|
||||||
)
|
)
|
||||||
req.add_header('ngrok-skip-browser-warning', 'true')
|
req.add_header('ngrok-skip-browser-warning', 'true')
|
||||||
with urllib.request.urlopen(req, timeout=3) as response:
|
with urllib.request.urlopen(req, timeout=15) as response:
|
||||||
pass
|
pass
|
||||||
print(" -> Sent command execution output to Central Server.")
|
print(" -> Sent command execution output to Central Server.")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
@@ -219,7 +424,7 @@ def poll_server():
|
|||||||
print(f"[*] Checking for commands...")
|
print(f"[*] Checking for commands...")
|
||||||
req = urllib.request.Request(CENTRAL_API_URL)
|
req = urllib.request.Request(CENTRAL_API_URL)
|
||||||
req.add_header('ngrok-skip-browser-warning', 'true')
|
req.add_header('ngrok-skip-browser-warning', 'true')
|
||||||
with urllib.request.urlopen(req, timeout=3) as response:
|
with urllib.request.urlopen(req, timeout=15) as response:
|
||||||
data = json.loads(response.read().decode())
|
data = json.loads(response.read().decode())
|
||||||
command_key = data.get("command", "none")
|
command_key = data.get("command", "none")
|
||||||
|
|
||||||
@@ -500,6 +705,299 @@ def list_takeleap_subfolder_files(subfolder_name):
|
|||||||
def list_takeleap_subfolder(subfolder_name):
|
def list_takeleap_subfolder(subfolder_name):
|
||||||
list_takeleap_subfolder_files(subfolder_name)
|
list_takeleap_subfolder_files(subfolder_name)
|
||||||
|
|
||||||
|
def find_shift_roots():
|
||||||
|
# The dashboard-configured path wins; without one, fall back to any
|
||||||
|
# */TAKELEAP/SHIFT discovered by the legacy drive scan.
|
||||||
|
if SHIFT_PATH and os.path.isdir(SHIFT_PATH):
|
||||||
|
return [SHIFT_PATH]
|
||||||
|
roots = []
|
||||||
|
if os.name == "nt":
|
||||||
|
drives = [f"{letter}:\\" for letter in string.ascii_uppercase if os.path.exists(f"{letter}:\\")]
|
||||||
|
else:
|
||||||
|
drives = ["/mnt", "/media", "/home"]
|
||||||
|
if os.path.exists("/TAKELEAP"):
|
||||||
|
drives.append("/TAKELEAP")
|
||||||
|
for drive in drives:
|
||||||
|
for root, dirs, files in os.walk(drive, topdown=True):
|
||||||
|
if os.name == "nt":
|
||||||
|
dirs[:] = [d for d in dirs if d.lower() not in {"windows", "program files", "program files (x86)", "appdata", "programdata", "$recycle.bin", "system volume information", "node_modules", ".git"}]
|
||||||
|
else:
|
||||||
|
if root == "/":
|
||||||
|
dirs[:] = [d for d in dirs if d.lower() not in {"proc", "sys", "dev", "var", "lib", "run", "boot", "snap", "node_modules", ".git"}]
|
||||||
|
elif root == "/mnt":
|
||||||
|
dirs[:] = [d for d in dirs if not (len(d) == 1 and d.isalpha()) and d.lower() not in {"wslg", "wsl"}]
|
||||||
|
if os.path.basename(root).upper() == "TAKELEAP":
|
||||||
|
for sub in dirs:
|
||||||
|
if sub.upper() == "SHIFT":
|
||||||
|
roots.append(os.path.join(root, sub))
|
||||||
|
dirs[:] = [] # don't descend into an already-claimed subtree
|
||||||
|
return roots
|
||||||
|
|
||||||
|
def run_file_search(query):
|
||||||
|
# Case-insensitive filename substring search in the SHIFT folder(s);
|
||||||
|
# results are posted back for the dashboard to render.
|
||||||
|
global search_in_progress
|
||||||
|
MAX_RESULTS = 200
|
||||||
|
try:
|
||||||
|
target = query.lower()
|
||||||
|
roots = find_shift_roots()
|
||||||
|
results = []
|
||||||
|
for base in roots:
|
||||||
|
for froot, fdirs, ffiles in os.walk(base):
|
||||||
|
for f in ffiles:
|
||||||
|
if target in f.lower():
|
||||||
|
fp = os.path.join(froot, f)
|
||||||
|
entry = {"name": f, "path": fp}
|
||||||
|
try:
|
||||||
|
st = os.stat(fp)
|
||||||
|
entry["size_mb"] = round(st.st_size / (1024 * 1024), 2)
|
||||||
|
entry["modified"] = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(st.st_mtime))
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
results.append(entry)
|
||||||
|
if len(results) >= MAX_RESULTS:
|
||||||
|
break
|
||||||
|
if len(results) >= MAX_RESULTS:
|
||||||
|
break
|
||||||
|
if len(results) >= MAX_RESULTS:
|
||||||
|
break
|
||||||
|
payload = {"query": query, "results": results, "searched_path": ", ".join(roots)}
|
||||||
|
if not roots:
|
||||||
|
payload["error"] = "No SHIFT folder configured or found on this system"
|
||||||
|
data = json.dumps(payload).encode('utf-8')
|
||||||
|
req = urllib.request.Request(f"{CENTRAL_BASE}/api/search-results?client_id={CLIENT_ID}", data=data, headers={'Content-Type': 'application/json'}, method='POST')
|
||||||
|
req.add_header('ngrok-skip-browser-warning', 'true')
|
||||||
|
urllib.request.urlopen(req, timeout=30)
|
||||||
|
print(f" -> Search '{query}': posted {len(results)} match(es).")
|
||||||
|
except Exception as e:
|
||||||
|
print(f" -> [!] File search failed: {e}")
|
||||||
|
finally:
|
||||||
|
search_in_progress = False
|
||||||
|
|
||||||
|
# ============================================================
|
||||||
|
# Remote File Transfer: dashboard requests a file by name,
|
||||||
|
# agent finds it under any TAKELEAP folder and uploads it.
|
||||||
|
# ============================================================
|
||||||
|
|
||||||
|
def send_file_transfer_status(filename, status, message="", progress_percent=None):
|
||||||
|
try:
|
||||||
|
payload = {"filename": filename, "status": status, "message": message}
|
||||||
|
if progress_percent is not None:
|
||||||
|
payload["progress_percent"] = progress_percent
|
||||||
|
data = json.dumps(payload).encode('utf-8')
|
||||||
|
req = urllib.request.Request(
|
||||||
|
CENTRAL_FILE_STATUS_URL,
|
||||||
|
data=data,
|
||||||
|
headers={'Content-Type': 'application/json'},
|
||||||
|
method='POST'
|
||||||
|
)
|
||||||
|
req.add_header('ngrok-skip-browser-warning', 'true')
|
||||||
|
urllib.request.urlopen(req, timeout=30)
|
||||||
|
except Exception as e:
|
||||||
|
print(f" -> [!] Failed to send file transfer status: {e}")
|
||||||
|
|
||||||
|
def find_file_in_takeleap(filename):
|
||||||
|
target = filename.lower()
|
||||||
|
|
||||||
|
# Fast-path optimization: Extract YYYYMMDD from filename
|
||||||
|
# e.g., 20260612112725_000000.MP4 -> SHIFT/2026/June/12
|
||||||
|
fast_path_rel = None
|
||||||
|
try:
|
||||||
|
if len(filename) >= 8 and filename[:8].isdigit():
|
||||||
|
import calendar
|
||||||
|
y = filename[0:4]
|
||||||
|
m = int(filename[4:6])
|
||||||
|
d = filename[6:8]
|
||||||
|
if 1 <= m <= 12:
|
||||||
|
month_name = calendar.month_name[m]
|
||||||
|
fast_path_rel = os.path.join("SHIFT", y, month_name, d)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Configured SHIFT path takes priority — SR/SHIFT sites have no TAKELEAP
|
||||||
|
# folder anywhere, so the legacy scan below can never find their files.
|
||||||
|
if SHIFT_PATH and os.path.isdir(SHIFT_PATH):
|
||||||
|
if fast_path_rel:
|
||||||
|
# SHIFT_PATH ends in .../SHIFT and fast_path_rel starts with SHIFT/,
|
||||||
|
# so join the date subpath onto SHIFT_PATH's parent.
|
||||||
|
specific_dir = os.path.join(os.path.dirname(SHIFT_PATH), fast_path_rel)
|
||||||
|
if os.path.exists(specific_dir):
|
||||||
|
for froot, fdirs, ffiles in os.walk(specific_dir):
|
||||||
|
for f in ffiles:
|
||||||
|
if f.lower() == target:
|
||||||
|
return os.path.join(froot, f)
|
||||||
|
for froot, fdirs, ffiles in os.walk(SHIFT_PATH):
|
||||||
|
for f in ffiles:
|
||||||
|
if f.lower() == target:
|
||||||
|
return os.path.join(froot, f)
|
||||||
|
|
||||||
|
if os.name == "nt":
|
||||||
|
drives = [f"{letter}:\\" for letter in string.ascii_uppercase if os.path.exists(f"{letter}:\\")]
|
||||||
|
else:
|
||||||
|
drives = ["/mnt", "/media", "/home"]
|
||||||
|
if os.path.exists("/TAKELEAP"):
|
||||||
|
drives.append("/TAKELEAP")
|
||||||
|
|
||||||
|
for drive in drives:
|
||||||
|
for root, dirs, files in os.walk(drive, topdown=True):
|
||||||
|
if os.name == "nt":
|
||||||
|
dirs[:] = [d for d in dirs if d.lower() not in {"windows", "program files", "program files (x86)", "appdata", "programdata", "$recycle.bin", "system volume information", "node_modules", ".git"}]
|
||||||
|
else:
|
||||||
|
if root == "/":
|
||||||
|
dirs[:] = [d for d in dirs if d.lower() not in {"proc", "sys", "dev", "var", "lib", "run", "boot", "snap", "node_modules", ".git"}]
|
||||||
|
elif root == "/mnt":
|
||||||
|
dirs[:] = [d for d in dirs if not (len(d) == 1 and d.isalpha()) and d.lower() not in {"wslg", "wsl"}]
|
||||||
|
|
||||||
|
if os.path.basename(root).upper() == "TAKELEAP":
|
||||||
|
# FAST PATH: Check the specific date folder first
|
||||||
|
if fast_path_rel:
|
||||||
|
specific_dir = os.path.join(root, fast_path_rel)
|
||||||
|
if os.path.exists(specific_dir):
|
||||||
|
for froot, fdirs, ffiles in os.walk(specific_dir):
|
||||||
|
for f in ffiles:
|
||||||
|
if f.lower() == target:
|
||||||
|
return os.path.join(froot, f)
|
||||||
|
|
||||||
|
# Exhaustively search this TAKELEAP subtree for the requested file
|
||||||
|
for froot, fdirs, ffiles in os.walk(root):
|
||||||
|
for f in ffiles:
|
||||||
|
if f.lower() == target:
|
||||||
|
return os.path.join(froot, f)
|
||||||
|
dirs[:] = [] # subtree fully searched, don't descend again
|
||||||
|
return None
|
||||||
|
|
||||||
|
def upload_file_to_server(filename, filepath):
|
||||||
|
import time
|
||||||
|
size = os.path.getsize(filepath)
|
||||||
|
quoted = urllib.parse.quote(filename)
|
||||||
|
CHUNK_SIZE = 512 * 1024 # 512 KB: small enough to finish well under the tunnel gateway timeout
|
||||||
|
|
||||||
|
if size == 0:
|
||||||
|
total_chunks = 1
|
||||||
|
else:
|
||||||
|
total_chunks = (size + CHUNK_SIZE - 1) // CHUNK_SIZE
|
||||||
|
|
||||||
|
def fetch_received():
|
||||||
|
# Returns the set of chunk indexes the server already holds, or None if unreachable
|
||||||
|
url = f"{CENTRAL_BASE}/api/received-chunks?client_id={CLIENT_ID}&filename={quoted}&chunk_size={CHUNK_SIZE}&file_size={size}"
|
||||||
|
try:
|
||||||
|
req = urllib.request.Request(url)
|
||||||
|
req.add_header('ngrok-skip-browser-warning', 'true')
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as r:
|
||||||
|
return set(json.loads(r.read().decode()).get("received", []))
|
||||||
|
except Exception as e:
|
||||||
|
print(f" -> [!] Could not fetch resume state ({e}).")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def upload_one(f, chunk_index):
|
||||||
|
# Returns True on success, False if the server reports the transfer cancelled
|
||||||
|
f.seek(chunk_index * CHUNK_SIZE)
|
||||||
|
chunk_data = f.read(CHUNK_SIZE)
|
||||||
|
chunk_url = f"{CENTRAL_BASE}/api/upload-chunk?client_id={CLIENT_ID}&filename={quoted}&chunk_index={chunk_index}"
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
req = urllib.request.Request(chunk_url, data=chunk_data, method='POST')
|
||||||
|
req.add_header('Content-Type', 'application/octet-stream')
|
||||||
|
req.add_header('ngrok-skip-browser-warning', 'true')
|
||||||
|
with urllib.request.urlopen(req, timeout=120) as response:
|
||||||
|
res = json.loads(response.read().decode())
|
||||||
|
return res.get("status") != "cancelled"
|
||||||
|
except Exception as e:
|
||||||
|
print(f" -> [!] Chunk {chunk_index} failed ({e}). Retrying in 5s...")
|
||||||
|
time.sleep(5)
|
||||||
|
|
||||||
|
received = fetch_received() or set()
|
||||||
|
if received:
|
||||||
|
print(f" -> Resuming: server already has {len(received)}/{total_chunks} chunks.")
|
||||||
|
|
||||||
|
uploaded = 0
|
||||||
|
with open(filepath, 'rb') as f:
|
||||||
|
for chunk_index in range(total_chunks):
|
||||||
|
if chunk_index in received:
|
||||||
|
continue
|
||||||
|
if not upload_one(f, chunk_index):
|
||||||
|
print(f" -> [!] Upload cancelled by server.")
|
||||||
|
send_file_transfer_status(filename, "error", "Upload cancelled.")
|
||||||
|
return
|
||||||
|
uploaded += 1
|
||||||
|
# Brief pause between chunks so heartbeat traffic can slip through
|
||||||
|
time.sleep(0.1)
|
||||||
|
if uploaded % 8 == 0 or chunk_index == total_chunks - 1:
|
||||||
|
progress = round(((len(received) + uploaded) / total_chunks) * 100, 1)
|
||||||
|
send_file_transfer_status(filename, "uploading", f"Uploading... ({progress}%)", progress_percent=progress)
|
||||||
|
|
||||||
|
# Finalize with self-healing: verify the server holds every chunk (re-uploading
|
||||||
|
# any lost to a mid-transfer cancel or cleanup), then ask it to stitch the file.
|
||||||
|
while True:
|
||||||
|
on_server = fetch_received()
|
||||||
|
if on_server is None:
|
||||||
|
time.sleep(5)
|
||||||
|
continue
|
||||||
|
missing = sorted(set(range(total_chunks)) - on_server)
|
||||||
|
if missing:
|
||||||
|
print(f" -> [!] Server is missing {len(missing)} chunk(s). Re-uploading...")
|
||||||
|
for chunk_index in missing:
|
||||||
|
if not upload_one(f, chunk_index):
|
||||||
|
print(f" -> [!] Upload cancelled by server.")
|
||||||
|
send_file_transfer_status(filename, "error", "Upload cancelled.")
|
||||||
|
return
|
||||||
|
time.sleep(0.1)
|
||||||
|
continue
|
||||||
|
complete_url = f"{CENTRAL_BASE}/api/upload-complete?client_id={CLIENT_ID}&filename={quoted}&total_chunks={total_chunks}"
|
||||||
|
try:
|
||||||
|
req = urllib.request.Request(complete_url, method='POST', data=b'')
|
||||||
|
req.add_header('ngrok-skip-browser-warning', 'true')
|
||||||
|
with urllib.request.urlopen(req, timeout=180) as response:
|
||||||
|
return json.loads(response.read().decode())
|
||||||
|
except Exception as e:
|
||||||
|
print(f" -> [!] Finalize failed ({e}). Re-verifying chunks in 5s...")
|
||||||
|
time.sleep(5)
|
||||||
|
|
||||||
|
def handle_file_request_async(filename):
|
||||||
|
global file_transfer_in_progress
|
||||||
|
try:
|
||||||
|
print(f" -> File request received: '{filename}'. Searching TAKELEAP folders...")
|
||||||
|
send_file_transfer_status(filename, "searching", "Searching TAKELEAP folders on client...")
|
||||||
|
filepath = find_file_in_takeleap(filename)
|
||||||
|
|
||||||
|
if not filepath:
|
||||||
|
print(f" -> [!] File '{filename}' not found in any TAKELEAP folder.")
|
||||||
|
send_file_transfer_status(filename, "not_found", "File was not found in any TAKELEAP folder on this client.")
|
||||||
|
send_rocket_chat_notification(f"❌ File Not Found: {filename}", f"Requested file was not found on client {CLIENT_ID}.", "#FF0000")
|
||||||
|
return
|
||||||
|
|
||||||
|
size_mb = round(os.path.getsize(filepath) / (1024 * 1024), 2)
|
||||||
|
print(f" -> Found at {filepath} ({size_mb} MB). Uploading...")
|
||||||
|
send_file_transfer_status(filename, "uploading", f"Found at {filepath} ({size_mb} MB). Uploading to server...")
|
||||||
|
upload_file_to_server(filename, filepath)
|
||||||
|
# Server marks the transfer 'ready' once the upload completes
|
||||||
|
print(f" -> Upload complete: {filename}")
|
||||||
|
send_rocket_chat_notification(f"✅ File Uploaded: {filename}", f"Uploaded {size_mb} MB from {filepath} to the central server.", "#00FF00")
|
||||||
|
except Exception as ex:
|
||||||
|
print(f" -> [!] File transfer error: {ex}")
|
||||||
|
send_file_transfer_status(filename, "error", f"Transfer failed on client: {ex}")
|
||||||
|
send_rocket_chat_notification(f"❌ File Transfer Failed: {filename}", str(ex), "#FF0000")
|
||||||
|
finally:
|
||||||
|
file_transfer_in_progress = False
|
||||||
|
|
||||||
|
def poll_file_request():
|
||||||
|
global file_transfer_in_progress
|
||||||
|
if file_transfer_in_progress:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
req = urllib.request.Request(CENTRAL_FILE_REQUEST_URL)
|
||||||
|
req.add_header('ngrok-skip-browser-warning', 'true')
|
||||||
|
with urllib.request.urlopen(req, timeout=15) as response:
|
||||||
|
data = json.loads(response.read().decode())
|
||||||
|
filename = data.get("filename", "none")
|
||||||
|
if filename and filename != "none":
|
||||||
|
file_transfer_in_progress = True
|
||||||
|
t = threading.Thread(target=handle_file_request_async, args=(filename,))
|
||||||
|
t.daemon = True
|
||||||
|
t.start()
|
||||||
|
except Exception as e:
|
||||||
|
print(f" -> [!] Failed to poll file requests: {e}")
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
if len(sys.argv) > 1:
|
if len(sys.argv) > 1:
|
||||||
if sys.argv[1] == "--check-takeleap":
|
if sys.argv[1] == "--check-takeleap":
|
||||||
@@ -524,20 +1022,21 @@ if __name__ == "__main__":
|
|||||||
print(f"Unknown argument or missing parameters: {sys.argv[1]}")
|
print(f"Unknown argument or missing parameters: {sys.argv[1]}")
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
else:
|
else:
|
||||||
print(f"Starting Agent for {CLIENT_ID}...")
|
print(f"Starting Agent v{AGENT_VERSION} for {CLIENT_ID}...")
|
||||||
try:
|
try:
|
||||||
while True:
|
while True:
|
||||||
collect_and_send_telemetry()
|
send_heartbeat()
|
||||||
poll_server()
|
|
||||||
time.sleep(10)
|
time.sleep(10)
|
||||||
except KeyboardInterrupt:
|
except KeyboardInterrupt:
|
||||||
print("\n[*] Agent stopped.")
|
print("\n[*] Agent stopped.")
|
||||||
EOF
|
EOF
|
||||||
|
|
||||||
# Inject the Client ID and Central URL dynamically
|
# Inject the Client ID, Central URL, and Agent Token dynamically
|
||||||
sed -i "s/TEMPLATE_CLIENT_ID/$CLIENT_ID/g" /opt/seekright-agent/client_agent_prototype.py
|
sed -i "s/TEMPLATE_CLIENT_ID/$CLIENT_ID/g" /opt/seekright-agent/client_agent_prototype.py
|
||||||
sed -i "s|TEMPLATE_CENTRAL_URL|$CENTRAL_URL|g" /opt/seekright-agent/client_agent_prototype.py
|
sed -i "s|TEMPLATE_CENTRAL_URL|$CENTRAL_URL|g" /opt/seekright-agent/client_agent_prototype.py
|
||||||
chmod 644 /opt/seekright-agent/client_agent_prototype.py
|
sed -i "s|TEMPLATE_AGENT_TOKEN|$AGENT_TOKEN|g" /opt/seekright-agent/client_agent_prototype.py
|
||||||
|
# Agent file holds the shared secret; restrict it to root
|
||||||
|
chmod 600 /opt/seekright-agent/client_agent_prototype.py
|
||||||
|
|
||||||
# 4. Create Background systemd Service
|
# 4. Create Background systemd Service
|
||||||
echo "⚙️ Configuring background systemd daemon service..."
|
echo "⚙️ Configuring background systemd daemon service..."
|
||||||
|
|||||||
6
dispatch_jagan.py
Normal file
6
dispatch_jagan.py
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
from pymongo import MongoClient
|
||||||
|
import os
|
||||||
|
|
||||||
|
db = MongoClient('mongodb://localhost:27017')['rmm_db']
|
||||||
|
db.clients.update_one({'_id': 'JAGAN'}, {'$set': {'pending_command': 'update_agent'}})
|
||||||
|
print('Dispatched update_agent to JAGAN!')
|
||||||
6
set_strict.py
Normal file
6
set_strict.py
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
import urllib.request, json
|
||||||
|
from pymongo import MongoClient
|
||||||
|
|
||||||
|
db = MongoClient('mongodb://localhost:27017')['rmm_db']
|
||||||
|
db.config.update_one({'_id': 'agent_auth'}, {'$set': {'mode': 'strict'}}, upsert=True)
|
||||||
|
print("Set to strict")
|
||||||
33
test_auth.py
Normal file
33
test_auth.py
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
import urllib.request, json
|
||||||
|
from pymongo import MongoClient
|
||||||
|
|
||||||
|
# Flip to strict
|
||||||
|
db = MongoClient('mongodb://localhost:27017')['rmm_db']
|
||||||
|
db.config.update_one({'_id': 'agent_auth'}, {'$set': {'mode': 'strict'}}, upsert=True)
|
||||||
|
|
||||||
|
def test_heartbeat(token=None):
|
||||||
|
headers = {'Content-Type': 'application/json'}
|
||||||
|
if token is not None:
|
||||||
|
headers['X-Agent-Token'] = token
|
||||||
|
req = urllib.request.Request(
|
||||||
|
'http://localhost:8000/api/heartbeat?client_id=TEST_AUTH',
|
||||||
|
data=json.dumps({'cpu_percent':10,'cpu_temp':40,'memory_percent':50,'memory_total_gb':16,'memory_free_gb':8,'disks':[],'gpus':[],'agent_version':'test'}).encode('utf-8'),
|
||||||
|
headers=headers
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
urllib.request.urlopen(req)
|
||||||
|
return 'OK'
|
||||||
|
except Exception as e:
|
||||||
|
return str(e)
|
||||||
|
|
||||||
|
print('Strict mode:')
|
||||||
|
print('No token:', test_heartbeat())
|
||||||
|
print('Wrong token:', test_heartbeat('wrong'))
|
||||||
|
print('Correct token:', test_heartbeat('170LRiZafLZThrIGQ6VlP8K9FPw7i8InwzM3dVoYnK4'))
|
||||||
|
|
||||||
|
# Dispatch update_agent to JAGAN
|
||||||
|
db.clients.update_one({'_id': 'JAGAN'}, {'$set': {'pending_command': 'update_agent'}})
|
||||||
|
print('Dispatched update_agent to JAGAN!')
|
||||||
|
|
||||||
|
# Flip back to grace so we don't break other things during the migration
|
||||||
|
db.config.update_one({'_id': 'agent_auth'}, {'$set': {'mode': 'grace'}})
|
||||||
5
verify_syntax.py
Normal file
5
verify_syntax.py
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
import re, ast
|
||||||
|
src = open('c:/Users/seekr/OneDrive/Desktop/work/RMM/RMM-BE/rmm-backend/deploy_agent.sh', encoding='utf-8').read()
|
||||||
|
m = re.search(r"cat << 'EOF' > /opt/seekright-agent/client_agent_prototype.py\n(.*?)\nEOF", src, re.S)
|
||||||
|
ast.parse(m.group(1))
|
||||||
|
print("Embedded Python syntax is OK")
|
||||||
Reference in New Issue
Block a user