kaushik/RMM-UI-BE
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
kawsik
2026-06-02 10:37:57 +05:30
commit b4acb8d1df
49 changed files with 15350 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
.gitignore vendored Normal file
View File

@@ -0,0 +1,59 @@
# Node.js
node_modules/
dist/
dist-ssr/
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
# Python
venv/
__pycache__/
*.pyc
*.pyo
*.pyd
.Python
env/
pip-log.txt
pip-delete-this-directory.txt
develop-eggs/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# Environment / Credentials
.env
.env.local
.env.development.local
.env.test.local
.env.production.local
*.local
# Logs and databases
logs/
*.log
logs.json
# IDEs/Editors
.vscode/
!.vscode/extensions.json
.idea/
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

62
01_architecture_overview.md Normal file
View File

@@ -0,0 +1,62 @@
# 01 - Architecture Overview
This documentation suite serves as a complete playbook to build a production-grade monitoring pipeline from scratch.
## The Architecture Stack
Our monitoring infrastructure is built on a **PULL-based architecture** combined with **PUSH-based alerting**.
1. **Node Exporters (Remote Nodes):** Tiny agents installed on every server/desktop. They sit on a port (usually `9100` for CPU/RAM, `9835` for GPUs) and simply expose raw hardware metrics.
2. **Prometheus (Central Server):** The brain. It actively **PULLS** (scrapes) data from all the remote Node Exporters every 15 seconds. It also evaluates Alert Rules (e.g., "Is a server offline?").
3. **Grafana (Central Server):** The visualizer. It connects to the Prometheus database to draw beautiful, real-time dashboards.
4. **Alertmanager (Central Server):** The router. When Prometheus detects an anomaly, it **PUSHES** an alert to Alertmanager. Alertmanager groups, delays, and routes the alert to external systems like Email and Rocket.Chat.
```mermaid
graph TD
subgraph Remote Nodes
N1[Node Exporter 1]
N2[Node Exporter 2]
G1[GPU Exporter]
end
subgraph Central Server
P[Prometheus]
A[Alertmanager]
G[Grafana]
end
subgraph Notifications
E[Email]
R[Rocket.Chat]
end
P -- "Pulls metrics (15s)" --> N1
P -- "Pulls metrics (15s)" --> N2
P -- "Pulls metrics (15s)" --> G1
G -- "Queries DB" --> P
P -- "Pushes alerts" --> A
A -- "Routes alert" --> E
A -- "Routes alert" --> R
```
## Setup Sequence
To deploy this infrastructure from scratch, you must follow the steps in this order:
1. Setup the Central Server (`02_central_server_setup.md`)
2. Setup the Remote Nodes (`03_remote_node_setup.md`)
3. Review the Day-to-Day Operations (`04_operations_and_debugging.md`)
---
## Understanding Alert Timing
Timing configurations exist in both Prometheus and Alertmanager to ensure you receive timely alerts without being spammed by false positives.
### The Chronological Flow
- **T=0s:** A remote server suddenly loses power.
- **T=15s (Prometheus `scrape_interval`):** The security guard checks the shop. Prometheus attempts its regular 15s scrape, gets a connection error, and marks the server state as `PENDING`.
- **T=25s (Prometheus `for: 10s`):** The guard waits to be sure. After being down for a full 10 seconds, Prometheus promotes the alert from `PENDING` to `FIRING`.
- **T=35s (Alertmanager `group_wait: 10s`):** The boss waits for more news. Alertmanager receives the firing alert, but waits 10 seconds to see if any *other* servers crash at the same time so it can bundle them into a single message. It then sends the first notification to Rocket.Chat/Email.
### The Follow-up Rules (Alertmanager)
- **`group_interval: 1m`:** If a brand *new* server crashes immediately after that first notification was sent, Alertmanager waits 1 minute before sending an update to avoid spamming the channel.
- **`repeat_interval: 5m`:** If nobody fixes the server and it remains offline, Alertmanager will re-send the exact same notification every 5 minutes until it is resolved. *(Note: `repeat_interval` must always be larger than `group_interval`!)*
- **`resolve_timeout: 5m`:** When the server finally comes back online, Alertmanager waits 5 minutes before sending the green `RESOLVED` notification. This ensures the server is completely stable and not just flickering on and off.

323
02_central_server_setup.md Normal file
View File

@@ -0,0 +1,323 @@
# 02 - Central Server Setup
_Follow these instructions to build the "Brain" of your monitoring architecture. All of these components (Prometheus, Grafana, and Alertmanager) will be installed on your central server._
---
## 1. INSTALLING PROMETHEUS (The Database & Scraper)
**1. Download Prometheus**
```bash
wget https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz
```
**2. Extract Prometheus**
```bash
tar -xvf prometheus-2.51.2.linux-amd64.tar.gz
```
**3. Create a restricted system user for security**
```bash
sudo useradd --no-create-home --shell /bin/false prometheus
```
**4. Create the Configuration and Data folders**
```bash
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
```
**5. Hand ownership of those folders to the new user**
```bash
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
```
**6. Copy the main engine and spell-checker to the secure Linux bin folder**
```bash
sudo cp prometheus-2.51.2.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.51.2.linux-amd64/promtool /usr/local/bin/
```
**7. Hand ownership of the executables to the new user**
```bash
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
```
**8. Create the main configuration file**
```bash
sudo nano /etc/prometheus/prometheus.yml
```
_Paste this exact code into the file and save:_
```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus_self_monitor"
static_configs:
- targets: ["localhost:9090"]
- job_name: "laptop_server_1"
static_configs:
- targets: ["localhost:9100"]
```
_Verify the YAML syntax is perfect so you don't crash the server:_
```bash
promtool check config /etc/prometheus/prometheus.yml
```
**9. Create the background service file**
```bash
sudo nano /etc/systemd/system/prometheus.service
```
_Paste this exact code into the file and save:_
```ini
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/
[Install]
WantedBy=multi-user.target
```
**10. Turn Prometheus on forever**
```bash
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
```
---
## 2. INSTALLING ALERTMANAGER (For Notifications)
**1. Download and Extract Alertmanager**
```bash
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar -xvf alertmanager-0.27.0.linux-amd64.tar.gz
```
**2. Create a restricted user and folders**
```bash
sudo useradd --no-create-home --shell /bin/false alertmanager
sudo mkdir /etc/alertmanager
sudo mkdir /var/lib/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager
```
**3. Move the binary and hand over ownership**
```bash
sudo cp alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
```
**4. Create the Configuration File**
```bash
sudo nano /etc/alertmanager/alertmanager.yml
```
_Paste this code to route alerts to both Rocket.Chat and Gmail simultaneously:_
```yaml
global:
resolve_timeout: 5m
smtp_smarthost: "smtp.gmail.com:587"
smtp_from: "knmkaushik@gmail.com"
smtp_auth_username: "knmkaushik@gmail.com"
smtp_auth_password: "YOUR_APP_PASSWORD"
route:
receiver: "fallback-do-nothing"
group_wait: 10s
group_interval: 1m
repeat_interval: 5m
routes:
- receiver: "rocket-chat-alerts"
continue: true
- receiver: "email-alerts"
continue: true
receivers:
- name: "fallback-do-nothing"
- name: "rocket-chat-alerts"
webhook_configs:
- url: "YOUR_ROCKETCHAT_WEBHOOK_URL"
send_resolved: true
- name: "email-alerts"
email_configs:
- to: "kawsik97@gmail.com"
send_resolved: true
```
**5. Check syntax, then hand over ownership of the config file**
```bash
amtool check-config /etc/alertmanager/alertmanager.yml
sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml
```
**6. Create the background service file**
```bash
sudo nano /etc/systemd/system/alertmanager.service
```
_Paste this exact code into the file and save:_
```ini
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager
[Install]
WantedBy=multi-user.target
```
**7. Turn Alertmanager on forever**
```bash
sudo systemctl daemon-reload
sudo systemctl start alertmanager
sudo systemctl enable alertmanager
```
**8. Configure Rocket.Chat Webhook Script**
Rocket.Chat natively only understands `{"text": "..."}` payloads, but Alertmanager sends structured JSON. To fix this, you must add a transformation script in the Rocket.Chat admin panel:
1. Go to your Rocket.Chat Admin Panel -> **Integrations** -> **Incoming**.
2. Create or Edit the incoming webhook for your channel (e.g., `#devops-alerts`).
3. Toggle **Script Enabled** to **ON**.
4. Paste the following into the Script box:
```javascript
class Script {
process_incoming_request({ request }) {
var content = request.content;
var status = content.status ? content.status.toUpperCase() : 'UNKNOWN';
var alertName = 'Unknown';
var instance = '';
var summary = '';
if (content.alerts && content.alerts[0]) {
var a = content.alerts[0];
if (a.labels) {
alertName = a.labels.alertname || 'Unknown';
instance = a.labels.instance || '';
}
if (a.annotations) {
summary = a.annotations.summary || '';
}
}
var icon = status === 'FIRING' ? ':red_circle:' : ':white_check_mark:';
var msg = icon + ' *' + status + '* — ' + alertName;
if (instance) { msg = msg + ' (' + instance + ')'; }
if (summary) { msg = msg + '\n' + summary; }
return {
content: {
text: msg
}
};
}
}
```
5. Make sure the "Script Sandbox" is set to "Secure Sandbox".
6. Click **Save** and use the generated Webhook URL in your `alertmanager.yml`.
---
## 3. LINKING PROMETHEUS & ALERTMANAGER
Prometheus must be told that Alertmanager exists, and where the "Rules" are stored.
**1. Open the main Prometheus config**
```bash
sudo nano /etc/prometheus/prometheus.yml
```
**2. Update the `alerting` and `rule_files` blocks at the top:**
```yaml
alerting:
alertmanagers:
- static_configs:
- targets: ["localhost:9093"]
rule_files:
- "rules.yml"
```
_(Save and exit)._
**3. Create the Rules file (Where the Alert Logic lives):**
```bash
sudo nano /etc/prometheus/rules.yml
```
**4. Paste this exact rule and save:**
_(This logic says: If any server goes offline for more than 10 seconds, fire a CRITICAL alert)._
```yaml
groups:
- name: hardware_alerts
rules:
- alert: ServerDown
expr: up == 0
for: 10s
labels:
severity: critical
annotations:
summary: "Server {{ $labels.instance }} is {{ if eq $value 0.0 }}DOWN{{ else }}UP{{ end }}!"
description: "Server {{ $labels.instance }} has been unreachable for more than 10 seconds."
```
**5. Hand over ownership and restart Prometheus:**
```bash
sudo chown prometheus:prometheus /etc/prometheus/rules.yml
sudo systemctl restart prometheus
```
---
## 4. INSTALLING GRAFANA (THE DASHBOARD UI)
_Grafana runs on port 3000 by default. Use this to visualize Prometheus data._
**1. Download the Grafana security key:**
```bash
sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
```
**2. Add the official Grafana repository to Linux:**
```bash
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
```
**3. Install Grafana and start the service:**
```bash
sudo apt-get update
sudo apt-get install grafana
sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
```
> **You have now successfully built the Central Server! Proceed to `03_remote_node_setup.md` to add your remote computers to the dashboard.**

165
03_remote_node_setup.md Normal file
View File

@@ -0,0 +1,165 @@
# 03 - Remote Node Setup
_Follow these instructions on **every remote computer/server** you want to monitor. These machines will run lightweight exporters that gather hardware metrics, waiting for the Central Server to pull the data._
---
## 1. INSTALLING NODE EXPORTER (For standard hardware data)
_Node Exporter tracks CPU, RAM, Disk Space, and Network speeds._
**1. Download Node Exporter**
```bash
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
```
**2. Extract Node Exporter**
```bash
tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz
```
**3. Create a restricted system user for security**
```bash
sudo useradd --no-create-home --shell /bin/false node_exporter
```
**4. Copy the main engine to the secure Linux bin folder**
```bash
sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
```
**5. Hand ownership of the executable to the new user**
```bash
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
```
**6. Create the background service file**
```bash
sudo nano /etc/systemd/system/node_exporter.service
```
_Paste this exact code into the file and save:_
```ini
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
```
**7. Turn Node Exporter on forever**
```bash
sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
```
_(The Node Exporter is now running on port `9100`)._
---
## 2. INSTALLING NVIDIA GPU EXPORTER (Only for servers with GPUs)
_Note: The server must already have Nvidia drivers installed so the `nvidia-smi` command works in the terminal. Node Exporter does NOT monitor GPUs._
**1. Download the Exporter**
```bash
wget https://github.com/utkuozdemir/nvidia_gpu_exporter/releases/download/v1.4.1/nvidia_gpu_exporter_1.4.1_linux_x86_64.tar.gz
```
> **[WARNING] Panic `[ms]` Error:** Always use version `v1.4.1` or newer. Older versions crash on modern drivers because they output a unit called `[ms]`, which breaks Prometheus's strict naming rules.
**2. Extract it**
```bash
tar -xvf nvidia_gpu_exporter_1.4.1_linux_x86_64.tar.gz
```
**3. Create a restricted system user**
```bash
sudo useradd --no-create-home --shell /bin/false nvidia_exporter
```
**4. Copy the engine to the secure bin folder**
```bash
sudo cp nvidia_gpu_exporter /usr/local/bin/
```
**5. Hand ownership to the new user**
```bash
sudo chown nvidia_exporter:nvidia_exporter /usr/local/bin/nvidia_gpu_exporter
```
**6. Create the background service file**
```bash
sudo nano /etc/systemd/system/nvidia_gpu_exporter.service
```
_Paste this exact code into the file and save:_
```ini
[Unit]
Description=Nvidia GPU Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=nvidia_exporter
Group=nvidia_exporter
Type=simple
ExecStart=/usr/local/bin/nvidia_gpu_exporter
[Install]
WantedBy=multi-user.target
```
**7. Turn it on forever**
```bash
sudo systemctl daemon-reload
sudo systemctl start nvidia_gpu_exporter
sudo systemctl enable nvidia_gpu_exporter
```
_(The GPU Exporter is now running on port `9835`)._
---
## 3. ADDING THE REMOTE TARGET TO PROMETHEUS
Once your remote servers have their exporters running, you must configure the **Central Server** to scrape them.
### If the remote server is on the SAME network:
Simply use its Local IP Address. Go to your **Central Server** and edit `prometheus.yml`:
```yaml
- job_name: "office_desktop_1"
static_configs:
- targets: ["192.168.1.150:9100"] # For Node Exporter
- job_name: "office_desktop_1_gpu"
static_configs:
- targets: ["192.168.1.150:9835"] # For GPU Exporter
```
### If the remote server is across the internet (Ngrok/Cloudflare Tunnels):
If the server is remote and behind a firewall, you must expose the ports securely using a tunnel like Ngrok or Cloudflare.
For example, if you run Ngrok on the remote server pointing to port 9100:
```bash
ngrok http 9100
```
It gives you a URL: `https://my-custom-tunnel.ngrok-free.dev`.
Go to your **Central Server** and add it to `prometheus.yml` using `scheme: https`:
```yaml
- job_name: "remote_gpu_server"
scheme: https
static_configs:
- targets: ["my-custom-tunnel.ngrok-free.dev"]
```
> **IMPORTANT:** After editing `prometheus.yml`, always restart the Prometheus service on the Central Server: `sudo systemctl restart prometheus`.

126
04_operations_and_debugging.md Normal file
View File

@@ -0,0 +1,126 @@
# 04 - Operations and Debugging
_This document serves as your day-to-day playbook for maintaining the monitoring infrastructure, writing queries, building dashboards, and debugging system crashes._
---
## 1. ESSENTIAL PROMQL QUERIES
_Use these queries in the Prometheus search bar (port 9090) or inside Grafana panels to monitor your hardware._
**1. RAM: Percentage Available**
```promql
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
```
**2. CPU: Percentage Actively Used**
_(Node Exporter tracks how long the CPU is "idle". We subtract the idle speed from 100% to get the active usage)._
```promql
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
```
**3. DISK (SSD/HDD): Percentage Free Space (Root Drive)**
```promql
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100
```
**4. NETWORK: Total Upload Speed in Megabytes/second (MB/s)**
```promql
rate(node_network_transmit_bytes_total[5m]) / 1024 / 1024
```
**5. HARDWARE: Server Uptime in Days**
```promql
(time() - node_boot_time_seconds) / 86400
```
**6. SERVERS: Is a server down? (1 = UP, 0 = DOWN)**
```promql
up
```
---
## 2. GRAFANA DASHBOARD TEMPLATES
When importing templates from `grafana.com/dashboards`, use these exact IDs for pre-built, beautiful hardware graphs:
- **Node Exporter Full (CPU/RAM/Network/Disk):** `1860`
- **Nvidia GPU Exporter (Temperature/VRAM/Power):** `14574`
### How to Fix "Blank" or "No Data" Dashboards
1. **Variables Mismatch:** Look at the dropdown menus at the very top of the dashboard. If they say "N/A", click them and select your specific `job_name` (e.g. `remote_gpu_server`).
2. **PromQL Mismatch:** If a specific graph is still blank, click the `...` next to the graph title and hit **Edit**. Look at the raw PromQL code. Sometimes community dashboards hardcode filters like `{job="gpu"}`. Simply delete that strict filter so your specific data can flow through!
---
## 3. MANAGING LINUX SERVICES (Crash, Reboot, & Maintenance)
Because we configured all components (Prometheus, Node Exporter, Grafana, Alertmanager) as Linux `systemd` services and used the `enable` command, **they will start automatically every time you boot or reboot your server.**
### 1. Check the Status of All Services at Once
If you just rebooted your server and want to ensure everything turned on properly, run this:
```bash
sudo systemctl status prometheus node_exporter grafana-server alertmanager
```
_Tip: If the output locks your terminal and shows `(END)` at the bottom, just press the `q` key on your keyboard to quit the reader mode and return to your prompt._
### 2. Restarting a Service After a Configuration Change
If you edit a configuration file (like `prometheus.yml` or `alertmanager.yml`), the changes won't apply until you restart the service. If a service ever crashes and you want to force it to start fresh, use `restart`:
```bash
# Restart Prometheus
sudo systemctl restart prometheus
# Restart Alertmanager
sudo systemctl restart alertmanager
# Restart Grafana
sudo systemctl restart grafana-server
```
### 3. Turning Services On / Off Manually
If you need to intentionally stop a service to perform maintenance (or trigger a fake alert):
```bash
sudo systemctl stop node_exporter
```
To start it back up:
```bash
sudo systemctl start node_exporter
```
---
## 4. DEBUGGING CRASHES WITH LOGS
If a service says it is "failed" or "inactive" when you check its status, you can view its exact crash logs by checking the Linux `journalctl` (the master log book).
**To see the last 50 logs for Alertmanager (useful for debugging 500 errors or webhook failures):**
```bash
sudo journalctl -u alertmanager -n 50 --no-pager
```
**To stream the logs live in real-time (press `Ctrl+C` to stop):**
```bash
sudo journalctl -u alertmanager -f
```
*(You can replace `alertmanager` with `prometheus`, `node_exporter`, or `grafana-server` to view logs for any other component).*
---
## 5. COMMON TROUBLESHOOTING
### 1. Browser Blocks HTTP Exporters
If your exporter is running (e.g., `curl http://localhost:9100/metrics` works in the terminal) but the browser says "Connection Refused", make sure your browser didn't automatically force `https://`. Exporters only run on plain `http://` by default!
### 2. "field repeat_interval already set" Error in Alertmanager
Alertmanager will crash on startup if you have duplicate keys in the YAML file (like declaring `repeat_interval` twice in the `route` block). Always run the syntax checker before restarting:
```bash
amtool check-config /etc/alertmanager/alertmanager.yml
```
### 3. Rocket.Chat 500 Internal Server Error
If `journalctl` shows Alertmanager successfully trying to send an alert but getting a `500 Internal Server Error`, the issue is on the Rocket.Chat server. Make sure:
1. The **"Allow to overwrite destination channel"** toggle is ON in the webhook settings.
2. The transformation script (from `02_central_server_setup.md`) is successfully pasted into the Rocket.Chat integration and **Script Enabled** is ON.
3. "Script Sandbox" is set to "Secure Sandbox".

145
05_architecture_roadmap.md Normal file
View File

@@ -0,0 +1,145 @@
# Global Monitoring Architecture — Full Roadmap
## What We Are Building (The Big Picture)
A fully automated, zero-touch monitoring system that:
- Tracks CPU, RAM, Disk, GPU health of 100+ client computers across the world.
- Sends instant alerts to Rocket.Chat and Email when a machine goes offline or overheats.
- Lets you remotely execute scripts on any or all machines without touching them physically.
- Scales from 10 to 1000 machines without changing the core architecture.
---
## The Complete Architecture Diagram
```
┌─────────────────────────────────────────────────────────────────────┐
│ CLIENT MACHINES (100+) │
│ Each machine runs 2 silent background agents: │
│ ┌──────────────────────┐ ┌──────────────────────────┐ │
│ │ Node Exporter :9100 │ │ MeshCentral Agent │ │
│ │ (Hardware Metrics) │ │ (RMM Command Channel) │ │
│ └──────────────────────┘ └──────────────────────────┘ │
└──────────────────────────────────────┬──────────────────────────────┘
│ Both agents "call home"
│ through client's firewall
▼
┌─────────────────────────────────────────────────────────────────────┐
│ CLOUDFLARE (Free Global Relay) │
│ Acts as a secure public-facing entry point for your private server │
│ Gives your desktop a real public URL (e.g. rmm.yourcompany.com) │
└──────────────────────────────────────┬──────────────────────────────┘
│ Cloudflare Tunnel
▼
┌─────────────────────────────────────────────────────────────────────┐
│ YOUR CENTRAL SERVER (Always-On Linux) │
│ │
│ ┌─────────────────┐ ┌────────────────┐ ┌──────────────────────┐ │
│ │ MeshCentral │ │ Prometheus │ │ Grafana Dashboard │ │
│ │ RMM Dashboard │ │ :9090 │ │ :3000 │ │
│ │ │ │ │ │ │ │
│ │ - See all 100 │ │ - Scrapes │ │ - Visual graphs of │ │
│ │ machines │ │ metrics │ │ CPU/RAM/GPU │ │
│ │ - Open remote │ │ from all │ │ - Real-time and │ │
│ │ terminals │ │ clients │ │ historical data │ │
│ │ - Push scripts │ │ - Evaluates │ └──────────────────────┘ │
│ │ to all │ │ alert rules │ │
│ └─────────────────┘ └───────┬────────┘ ┌──────────────────────┐ │
│ │ │ Alertmanager │ │
│ └──────────►│ :9093 │ │
│ │ │ │
│ │ Routes alerts to: │ │
│ │ - Rocket.Chat │ │
│ │ - Gmail │ │
│ └──────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
```
---
## What Software is Required (and Why)
### On Every Client Machine (100+ remote desktops)
| Software | Purpose | Cost |
| ----------------------- | ------------------------------------------------------------------------------------- | ---- |
| **Node Exporter** | Exposes CPU, RAM, Disk, Network metrics on port 9100 | Free |
| **Nvidia GPU Exporter** | Exposes GPU temp and VRAM metrics on port 9835 (only if GPU exists) | Free |
| **MeshCentral Agent** | Maintains a silent reverse tunnel back to your RMM server for remote script execution | Free |
### On Your Central Server (Always-On Linux Machine)
| Software | Purpose | Cost |
| ------------------------------------- | ------------------------------------------------------------------------------ | ---- |
| **MeshCentral** | RMM dashboard — deploy scripts to all clients in one click | Free |
| **Cloudflare Tunnel (`cloudflared`)** | Creates a secure public URL for your local server without touching your router | Free |
| **Prometheus** | Scrapes and stores time-series hardware metrics from all clients | Free |
| **Alertmanager** | Receives firing alerts from Prometheus and routes to Rocket.Chat/Gmail | Free |
| **Grafana** | Connects to Prometheus and draws real-time dashboards | Free |
---
## The Build Phases (In Order)
### Phase 1: Central Server Foundation ✅ DONE
- Install Prometheus, Alertmanager, Grafana on central server.
- Configure `alertmanager.yml` with Rocket.Chat webhook and Gmail SMTP.
- Write `rules.yml` with the `ServerDown` alert.
### Phase 2: Remote Agent Automation ✅ DONE
- Write `deploy_exporters.sh` to silently install Node Exporter and GPU Exporter.
- Test the script on a remote machine via RustDesk.
### Phase 3: MeshCentral RMM Setup (CURRENT PHASE)
- Install NodeJS on your central Linux server.
- Install and configure MeshCentral.
- Expose it to the internet via a Cloudflare Tunnel.
- Install the MeshCentral Agent on one test machine.
- Verify you can push a script remotely from the web dashboard.
### Phase 4: Networking — Connect Prometheus to Remote Clients
- Install Tailscale (mesh VPN) on the Central Server and all 100 client machines.
- Each client gets a stable private IP (e.g., `100.85.x.x`).
- Add all 100 Tailscale IPs to `prometheus.yml` as scrape targets.
### Phase 5: Scale and Polish
- Add all 100 machines to MeshCentral.
- Use MeshCentral's "Run Script on Group" feature to deploy `deploy_exporters.sh` to all machines simultaneously.
- Import Grafana dashboard templates (Node Exporter ID: `1860`, GPU ID: `14574`).
- Set up Grafana Alerting for CPU/RAM thresholds.
---
## WSL vs Production Linux — What Works Where
| Feature | WSL (Testing) | Native Linux (Production) |
| ----------------------------------------- | ---------------------------- | ------------------------- |
| Install MeshCentral | ✅ Yes | ✅ Yes |
| Open MeshCentral in browser | ✅ Yes | ✅ Yes |
| Install MeshCentral Agent on same machine | ✅ Yes | ✅ Yes |
| Expose to internet via Cloudflare Tunnel | ⚠️ Tricky (WSL networking) | ✅ Easy |
| Run as background service on reboot | ❌ WSL doesn't auto-start | ✅ Yes (systemd) |
| Accept real remote client connections | ⚠️ Possible with workarounds | ✅ Yes |
**Conclusion:** WSL is perfect for Phase 3 learning and testing. You can install MeshCentral, learn the dashboard, test pushing scripts locally, and understand how everything works. When you are ready to go live with real clients, you simply run the same commands on a native Linux machine and it works perfectly.
---
## Immediate Next Step
Install NodeJS on your WSL terminal and deploy MeshCentral locally.
```bash
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g meshcentral
node /usr/lib/node_modules/meshcentral
```
Then open `http://localhost:4430` in your browser to see the MeshCentral dashboard!

440
06_meshcentral_setup_guide.md Normal file
View File

@@ -0,0 +1,440 @@
# MeshCentral RMM — Complete Setup Guide
## From Zero to Connected (With All Challenges Documented)
---
## What is MeshCentral and Why We Need It
MeshCentral is a free, open-source Remote Monitoring and Management (RMM) tool. It lets you see all your remote machines in one browser dashboard, open root terminals, transfer files, and push scripts to 100 machines simultaneously — without using RustDesk or asking for passwords every time.
**The Goal:** Install MeshCentral on the central server once. Install the lightweight agent on each client machine once. After that, manage everything from a web browser forever.
---
## Architecture Overview
```
Client Machine (anywhere in the world)
└── MeshCentral Agent (background service, runs as root)
└── Tailscale (mesh VPN, gives stable 100.x.x.x IP)
│
│ Direct encrypted tunnel (no middleman)
▼
Central Server (your always-on Linux machine / WSL)
└── MeshCentral Server (NodeJS app, port 4430)
└── Tailscale (same mesh VPN, stable 100.x.x.x IP)
```
---
## Part 1: Setting Up the Central Server (Your Laptop/Desktop)
### Prerequisites
- Linux machine or WSL2 on Windows
- NodeJS v20+
- Tailscale account (free at tailscale.com)
### Step 1: Install NodeJS
```bash
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs
```
Verify: `node --version` should show v20.x.x
### Step 2: Install MeshCentral Locally
```bash
mkdir ~/meshcentral
cd ~/meshcentral
npm install meshcentral
```
> **CRITICAL:** Do NOT use `sudo npm install -g meshcentral`.
> Installing globally puts MeshCentral in `/usr/lib/` which is root-owned.
> MeshCentral will fail with `EACCES: permission denied, mkdir '/usr/lib/meshcentral-data'`.
> Always install locally in your home directory.
### Step 3: Install Tailscale on the Central Server
```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
```
> **CRITICAL — DO NOT PRESS Ctrl+C:**
> When `sudo tailscale up` runs, it prints an authentication URL and then WAITS.
> You MUST open that URL in a browser and log in FIRST.
> Only after successful login will the terminal print `Success.` and return to the prompt.
> If you press Ctrl+C before authenticating, Tailscale will be installed but NOT logged in.
> You will have to run `sudo tailscale up` again and get a NEW URL to authenticate.
Open the authentication URL in a browser. Log in with Google/GitHub. After the terminal prints `Success.`, get your stable IP:
```bash
tailscale ip -4
# Example output: 100.66.205.2
```
Save this IP — it is your permanent MeshCentral server address.
### Step 4: Configure MeshCentral
```bash
cat > ~/meshcentral/meshcentral-data/config.json << 'EOF'
{
"settings": {
"cert": "YOUR_TAILSCALE_IP",
"port": 4430,
"redirport": 4431
},
"domains": {
"": {
"title": "My RMM"
}
}
}
EOF
```
Replace `YOUR_TAILSCALE_IP` with your actual Tailscale IP (e.g., `100.66.205.2`).
### Step 5: Start MeshCentral
```bash
cd ~/meshcentral && node node_modules/meshcentral
```
Expected healthy startup output:
```
MeshCentral HTTP redirection server running on port 4431.
MeshCentral v1.1.59, Hybrid (LAN + WAN) mode.
MeshCentral HTTPS server running on 100.66.205.2:4430.
```
### Step 6: Access the Dashboard
Open your browser and go to:
```
https://localhost:4430
```
Accept the certificate warning (self-signed cert is normal).
**The first account you create automatically becomes the Site Administrator.**
### Step 7: Create a Device Group
In the dashboard:
- Click **My Devices**
- Click **Add Device Group**
- Name it (e.g., `Client Machines`)
- Click OK
---
## Part 2: Connecting a New Client Machine
Repeat these steps for every new machine you want to manage.
### Step 1: Install Tailscale on the Client Machine
```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
```
Open the authentication URL. Log in with the **SAME Tailscale account** as the server. This is critical — both must be on the same account to communicate.
Verify connection:
```bash
tailscale ip -4
# Should give a 100.x.x.x IP
ping 100.66.205.2 # Should reach the server
```
### Step 2: Get the Agent Install Command
In the MeshCentral browser:
- Go to **My Devices**
- Click on your Device Group
- Click **Add Agent**
- Select **Linux / BSD**
- Copy the generated command
The command will look like:
```bash
(wget "https://100.66.205.2:4430/meshagents?script=1" --no-check-certificate -O ./meshinstall.sh) && chmod 755 ./meshinstall.sh && sudo ./meshinstall.sh https://100.66.205.2:4430 'YOUR_GROUP_KEY'
```
### Step 3: Run the Agent Install Command on the Client
On the client machine terminal:
```bash
cd /tmp && (wget "https://100.66.205.2:4430/meshagents?script=1" --no-check-certificate -O ./meshinstall.sh) && chmod 755 ./meshinstall.sh && sudo ./meshinstall.sh https://100.66.205.2:4430 'YOUR_GROUP_KEY'
```
Expected output:
```
Downloading agent #6...
Agent downloaded.
...Installing service [DONE]
-> Starting service... [OK]
```
### Step 4: Verify the Agent is Running
```bash
sudo systemctl status meshagent
```
Should show `active (running)`.
### Step 5: Confirm in the Dashboard
The machine should appear in MeshCentral within 10 seconds showing **"Agent, Powered"** with a blue refresh icon.
---
## Part 3: Using MeshCentral to Deploy Scripts
### Running a Script on One Machine
1. Go to **My Devices**
2. Right-click the machine → **Terminal** (for interactive use)
OR
3. Check the checkbox next to the machine
4. Click **Group Action** → **Run commands**
5. Select **Linux/BSD/macOS Command Shell**
6. Select **Run as agent** (runs as root on Linux)
7. Select **Commands from file**
8. Click **Choose file** → select your `.sh` script
9. Click **OK**
### Running a Script on ALL 100 Machines Simultaneously
1. Click **Select All** at the top
2. Click **Group Action** → **Run commands**
3. Upload your script and click OK
4. Watch the output for each machine stream in simultaneously
---
## Part 4: Challenges and Issues Faced (With Solutions)
### Issue 1: Permission Denied on First Run
**Error:**
```
Error: EACCES: permission denied, mkdir '/usr/lib/meshcentral-data'
```
**Cause:** MeshCentral was installed globally with `sudo npm install -g`, placing it in the root-owned `/usr/lib/` directory.
**Fix:** Install locally without sudo:
```bash
mkdir ~/meshcentral && cd ~/meshcentral
npm install meshcentral
node node_modules/meshcentral
```
---
### Issue 2: Permission Denied When Saving Script in nano
**Error:** `[ Error writing test_install.sh: Permission denied ]`
**Cause:** Terminal was sitting in a restricted system folder (like `/` or `/root`).
**Fix:** Always create files in `/tmp` which allows all users to write:
```bash
cd /tmp
nano test_install.sh
```
---
### Issue 3: Script Fails with "command not found" After Clipboard Paste
**Error:** `test_install.sh: line 27: Update: command not found`
**Cause:** RustDesk/AnyDesk clipboard dropped the `#` symbol from comments. Linux interpreted the English word "Update" as a command.
**Fix:** Remove ALL comments and blank lines from scripts before pasting over remote desktop:
```bash
# Comments like this get corrupted when pasted via RustDesk
# Solution: Strip the script to pure commands only
```
---
### Issue 4: MeshCentral Running in LAN Mode Only
**Symptom:** MeshCentral startup shows `LAN mode` — agents from other networks cannot connect.
**Cause:** No `cert` was set in `config.json`. MeshCentral defaults to LAN-only mode.
**Fix:** Set the `cert` field in `config.json` to the server's public hostname or Tailscale IP:
```json
{
"settings": {
"cert": "100.66.205.2",
"port": 4430
}
}
```
Server will restart in `Hybrid (LAN + WAN) mode`.
---
### Issue 5: Agent Stuck in SYN-SENT State (Port Mismatch)
**Symptom:** Agent connects to `104.16.x.x:1025` and hangs forever.
**Cause:** MeshCentral was running on port 1025. The `.msh` file embedded port 1025 in the server URL. But Cloudflare Quick Tunnels only listen on port 443. The connection attempt to `:1025` on Cloudflare was silently blocked.
**Diagnosis:**
```bash
sudo ss -tp | grep meshagent
# Shows: SYN-SENT ... 104.16.x.x:1025
```
**Fix (with Cloudflare):** Use `setcap` to allow NodeJS to bind to port 443, and enable `tlsoffload` in config so the agent connects on the standard HTTPS port.
**Better Fix:** Switch to Tailscale (see Issue 6).
---
### Issue 6: Certificate Hash Mismatch with Cloudflare Tunnel
**Error:**
```
Agent bad web cert hash (Agent:9969c2afd9 != Server:a364860c2f)
holding connection (106.51.126.27:43912)
```
**Cause:** Cloudflare "terminates TLS" — it acts as a middleman and presents its own TLS certificate to the agent instead of MeshCentral's certificate. The agent's `.msh` file contains MeshCentral's certificate fingerprint. When the agent sees Cloudflare's certificate instead, the fingerprints don't match and the agent refuses to connect.
**Why this cannot be easily fixed with Cloudflare Quick Tunnels:** The only way to fix it properly is to use a real Cloudflare account with a named tunnel and an Origin Certificate — so MeshCentral uses the same certificate that Cloudflare presents externally.
**The Clean Fix: Use Tailscale Instead**
Tailscale does not terminate TLS. It creates a direct encrypted tunnel between machines without intercepting TLS certificates. The agent connects directly to MeshCentral and sees its real certificate — the fingerprint always matches.
**Setup:**
```bash
# On server
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up # Login with Google/GitHub
tailscale ip -4 # Note this IP (e.g., 100.66.205.2)
# On client
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up # DO NOT PRESS Ctrl+C — wait for browser login then Success.
```
> **CRITICAL — DO NOT PRESS Ctrl+C during `sudo tailscale up`:**
> The command prints a URL and waits. Open the URL in ANY browser (even on your laptop).
> Log in with the SAME account. The terminal will then print `Success.` on its own.
> If you press Ctrl+C too early, the machine will show as logged out.
> Run `tailscale status` to check — if it says `Logged out`, run `sudo tailscale up` again.
Both machines get stable `100.x.x.x` IPs. Update MeshCentral config to use the Tailscale IP. Agent connects cleanly.
---
### Issue 7: Windows Browser Cannot Reach WSL Tailscale IP
**Symptom:** Browser shows `ERR_CONNECTION_TIMED_OUT` when visiting `https://100.66.205.2:4430`
**Cause:** Windows cannot directly route to the Tailscale IP assigned to WSL. The Tailscale IP is only reachable from other Tailscale machines on the network, not from the Windows host itself.
**Fix:** From your own laptop's Windows browser, always use `localhost` to access WSL services:
```
https://localhost:4430 ← Use this from your laptop browser
https://100.x.x.x:4430 ← Use this from OTHER machines on Tailscale
```
---
### Issue 8: MeshCentral Terminal vs Regular Terminal Permissions
**Symptom:** Running `apt-get install` in the regular terminal gives `Permission denied`.
**Cause:** The regular terminal runs as the normal user (`kawsik`). The MeshCentral Terminal runs as `root` because the agent service itself runs as root.
**Rule:**
- Regular terminal: needs `sudo` for system commands
- MeshCentral Terminal: already root, never needs `sudo`
---
### Issue 9: Tailscale Shows "Logged Out" After Ctrl+C
**Symptom:** `tailscale status` shows `Logged out` even after running `sudo tailscale up`.
**Cause:** The user pressed Ctrl+C while `sudo tailscale up` was waiting for browser authentication. This cancelled the authentication process before it completed. Tailscale is installed but not authenticated.
**Fix:** Simply run `sudo tailscale up` again. A new authentication URL will be printed. Open it in any browser (including your laptop's browser), log in, and wait for `Success.` to appear in the terminal. Never press Ctrl+C while the URL is shown.
```bash
sudo tailscale up
# Opens URL → open it in browser → log in → wait for Success. → done
tailscale status # Should now show Connected
```
---
### Issue 10: Native Linux Client Cannot Ping WSL Tailscale IP (100% Packet Loss)
**Symptom:** `ping 100.66.205.2` from a native Linux client (like PopOS) gives 100% packet loss. The Tailscale dashboard shows both machines as Connected but they cannot communicate.
**Cause:** The MeshCentral server's Tailscale is installed inside **WSL2**, which is behind double NAT — the home router AND Windows' internal NAT. Other WSL machines (like `bs13`) can reach it through Tailscale relay servers. But native Linux machines sometimes cannot punch through both NAT layers to reach a WSL Tailscale node.
**Diagnosis:**
```bash
tailscale status # Check if machine is actually online
tailscale ping 100.66.205.2 # Tailscale's own connectivity check
```
**Fix Option 1 (Simple):** Make sure Tailscale on the client is actually authenticated (run `tailscale status` — if it says `Logged out`, re-authenticate with `sudo tailscale up`).
**Fix Option 2 (Production):** Install Tailscale on the **Windows host** instead of WSL, giving it a proper host-level IP without double NAT. All clients can then reach the Windows Tailscale IP directly.
**Fix Option 3 (Production):** Move MeshCentral to a cloud VPS (Oracle Free Tier, etc.) so it has a real public IP with no NAT issues at all.
---
### Issue 11: RustDesk "Failed to get capturer display info" on COSMIC Desktop (PopOS 24.04)
**Symptom:** RustDesk connects, password accepted, but immediately shows `Failed to get capturer display info`.
**Cause:** PopOS 24.04 uses the **COSMIC desktop environment** which is Wayland-only. There is no X11 option. RustDesk has limited support for Wayland/COSMIC screen capture.
**Root Cause Detail:** `echo $XDG_SESSION_TYPE` returns empty when run over SSH because no display session is attached to the SSH terminal. Running `loginctl list-sessions` showed session on `tty1` — a text console, not a graphical session. The machine was sitting at the COSMIC login screen with no user logged into the GUI.
**Identification Commands:**
```bash
cat /etc/X11/default-display-manager # Shows: /usr/bin/cosmic-greeter
echo $XDG_SESSION_TYPE # Empty = no display in this terminal
loginctl list-sessions # Shows TTY type (tty1 = text, not GUI)
```
**Fix:** Enable auto-login for COSMIC so the machine automatically logs into the desktop on boot:
```bash
sudo mkdir -p /etc/sddm.conf.d
sudo nano /etc/sddm.conf.d/autologin.conf
```
Add:
```ini
[Autologin]
User=byfar
Session=cosmic
```
Then reboot. After reboot, COSMIC logs in automatically and RustDesk can capture the screen.
**Alternative:** Skip RustDesk entirely. Use MeshCentral's built-in **Terminal** (root shell) and **Web-VNC** features for remote access instead. These work regardless of display server type.
---
## Part 5: Quick Reference — Adding Machine #N to MeshCentral
This is the exact 3-command sequence to run on any new Linux client machine:
```bash
# Step 1: Install Tailscale
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
# ⚠️ A URL will appear — open it in your laptop browser and log in with the SAME account
# ⚠️ DO NOT press Ctrl+C — wait until the terminal prints Success.
tailscale status # Must show Connected before proceeding
# Step 2: Install MeshCentral Agent
cd /tmp && (wget "https://100.66.205.2:4430/meshagents?script=1" --no-check-certificate -O ./meshinstall.sh) && chmod 755 ./meshinstall.sh && sudo ./meshinstall.sh https://100.66.205.2:4430 'YOUR_GROUP_KEY'
# Step 3: Verify
sudo systemctl status meshagent
```
The machine will appear in the MeshCentral dashboard within 10 seconds.
> **If the agent install hangs connecting:** Run `tailscale status` — if it shows `Logged out`, the Tailscale session expired. Run `sudo tailscale up` again, authenticate in the browser, then retry the agent install.
---
## Part 6: Production Checklist
- [ ] Central server is always-on (disable sleep/hibernate)
- [ ] MeshCentral starts automatically on server reboot (configure as systemd service)
- [ ] Tailscale starts automatically on all machines (enabled by default after install)
- [ ] All client machines enrolled in Tailscale with the same account
- [ ] MeshCentral agent enabled as systemd service on all clients (`systemctl enable meshagent`)
- [ ] Node Exporter deployed and running on all clients (port 9100)
- [ ] All Tailscale IPs added to `prometheus.yml` as scrape targets

77
07_connect_remote_machine.md Normal file
View File

@@ -0,0 +1,77 @@
# Connect a New Remote Machine to MeshCentral
## Commands Only — Run These on the Remote Machine
---
## Step 1 — Install Tailscale
```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
```
> ⚠️ A URL will appear. Open it in **any browser** (your laptop is fine).
> Log in with **knmkaushik@gmail.com** (same account as all other machines).
> **DO NOT press Ctrl+C** — wait until the terminal prints `Success.`
Verify:
```bash
tailscale status
# Must show: pop-os ... Connected
```
---
## Step 2 — Install MeshCentral Agent
```bash
cd /tmp && \
wget "https://100.66.205.2:4430/meshagents?script=1" --no-check-certificate -O ./meshinstall.sh && \
chmod 755 ./meshinstall.sh && \
sudo ./meshinstall.sh https://100.66.205.2:4430 '7TS2JNCPPk4H6zUjPNYIy5SOy6kyUQRsmkIoyd6jHlzweRwcEPRxZ8oZ2vLV6nZi'
```
Expected output:
```
...Installing service [DONE]
-> Starting service... [OK]
```
---
## Step 3 — Verify Agent is Running
```bash
sudo systemctl status meshagent
# Must show: active (running)
```
---
## Done
The machine will appear in the MeshCentral dashboard at `https://localhost:4430` on your laptop within 10 seconds.
---
## If Something Goes Wrong
| Problem | Fix |
|---|---|
| `tailscale status` shows `Logged out` | Run `sudo tailscale up` again, open the new URL in browser |
| Agent install hangs at `Connecting to 100.66.205.2` | Tailscale not connected — fix Step 1 first |
| Machine not appearing in dashboard | Run `sudo systemctl restart meshagent` |
| Agent install succeeds but no green dot | Wait 30 seconds then hard refresh browser |
---
## Quick Reference Info
| Item | Value |
|---|---|
| MeshCentral Server IP | `100.66.205.2` |
| MeshCentral Port | `4430` |
| Dashboard URL (from laptop) | `https://localhost:4430` |
| Dashboard URL (from Tailscale machine) | `https://100.66.205.2:4430` |
| Tailscale Account | `knmkaushik@gmail.com` |
| Device Group Key | `7TS2JNCPPk4H6zUjPNYIy5SOy6kyUQRsmkIoyd6jHlzweRwcEPRxZ8oZ2vLV6nZi` |

469
08_central_server_production_setup.md Normal file
View File

@@ -0,0 +1,469 @@
# Central Server Production Setup
## Fresh Linux Machine → Full Monitoring + RMM Stack
---
## What This Document Covers
Setting up your central server from a fresh Ubuntu/PopOS Linux install with:
- **Tailscale** — private network backbone connecting all machines
- **MeshCentral** — RMM dashboard for remote management of all client machines
- **Prometheus** — metrics collection engine
- **Alertmanager** — alert routing to Rocket.Chat and Gmail
- **Grafana** — visualization dashboards
---
## Requirements
| Requirement | Minimum | Recommended |
|---|---|---|
| OS | Ubuntu 22.04 / 24.04 LTS | Ubuntu 24.04 LTS (native, NOT WSL) |
| RAM | 2 GB | 4 GB |
| Disk | 20 GB | 50 GB |
| CPU | 2 cores | 4 cores |
| Network | Always-on internet | Always-on internet |
| Power | No sleep/hibernate | UPS recommended |
> ⚠️ **WSL is for testing only.** For production, use a native Linux machine (bare metal or VM).
> WSL does not auto-start services on Windows boot and has double-NAT networking issues.
---
## Accounts You Need Before Starting
### 1. Tailscale Account (Free)
- Go to: **https://tailscale.com**
- Click **Get Started**
- Sign in with Google or GitHub — whichever you use on all machines
- You do not need to configure anything on the website — just create the account
- Free plan supports up to 100 machines, which is exactly what we need
### 2. Tailscale is the ONLY external account required
MeshCentral, Prometheus, Alertmanager, and Grafana are all self-hosted on your machine.
No cloud accounts needed for any of them.
---
## Part 1: System Preparation
### Update the system
```bash
sudo apt-get update && sudo apt-get upgrade -y
```
### Disable sleep and hibernate (critical for a server)
```bash
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
```
### Create a dedicated user for running services (optional but recommended)
```bash
sudo useradd -r -s /bin/false prometheus
sudo useradd -r -s /bin/false alertmanager
```
---
## Part 2: Install Tailscale
```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
```
> ⚠️ A URL will appear. Open it in a browser and log in with your Tailscale account.
> **DO NOT press Ctrl+C** — wait for `Success.`
Get your server's stable Tailscale IP:
```bash
tailscale ip -4
# Example: 100.66.205.2 ← save this, you will use it everywhere
```
Verify Tailscale starts on boot (it does by default):
```bash
sudo systemctl is-enabled tailscaled
# Should print: enabled
```
---
## Part 3: Install NodeJS and MeshCentral
### Install NodeJS v20
```bash
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs
node --version # Should show v20.x.x
```
### Install MeshCentral locally
```bash
mkdir ~/meshcentral
cd ~/meshcentral
npm install meshcentral
```
> ⚠️ Do NOT use `sudo npm install -g meshcentral` — permission errors will occur.
### Configure MeshCentral
Replace `YOUR_TAILSCALE_IP` with the IP from Part 2:
```bash
mkdir -p ~/meshcentral/meshcentral-data
cat > ~/meshcentral/meshcentral-data/config.json << 'EOF'
{
"settings": {
"cert": "YOUR_TAILSCALE_IP",
"port": 4430,
"redirport": 4431
},
"domains": {
"": {
"title": "My RMM"
}
}
}
EOF
```
### Test MeshCentral starts correctly
```bash
cd ~/meshcentral && node node_modules/meshcentral
```
Expected output:
```
MeshCentral HTTP redirection server running on port 4431.
MeshCentral v1.1.x, Hybrid (LAN + WAN) mode.
MeshCentral HTTPS server running on YOUR_TAILSCALE_IP:4430.
```
Open browser at `https://localhost:4430` — accept the certificate warning.
**Create your admin account** — the first account created is automatically the Site Administrator.
Then press **Ctrl+C** to stop it. We will run it as a service next.
### Run MeshCentral as a systemd service (auto-starts on reboot)
```bash
cat > /tmp/meshcentral.service << EOF
[Unit]
Description=MeshCentral RMM Server
After=network.target
[Service]
Type=simple
User=$USER
WorkingDirectory=$HOME/meshcentral
ExecStart=/usr/bin/node $HOME/meshcentral/node_modules/meshcentral
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
sudo mv /tmp/meshcentral.service /etc/systemd/system/meshcentral.service
sudo systemctl daemon-reload
sudo systemctl enable meshcentral
sudo systemctl start meshcentral
sudo systemctl status meshcentral
```
Verify it is running:
```bash
sudo systemctl status meshcentral
# Must show: active (running)
```
---
## Part 4: Install Prometheus
### Download and install
```bash
cd /tmp
PROM_VERSION="2.51.2"
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
tar xvf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
sudo mv prometheus-${PROM_VERSION}.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-${PROM_VERSION}.linux-amd64/promtool /usr/local/bin/
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo mv prometheus-${PROM_VERSION}.linux-amd64/prometheus.yml /etc/prometheus/
sudo mv prometheus-${PROM_VERSION}.linux-amd64/consoles /etc/prometheus/
sudo mv prometheus-${PROM_VERSION}.linux-amd64/console_libraries /etc/prometheus/
```
### Configure prometheus.yml
```bash
sudo nano /etc/prometheus/prometheus.yml
```
Replace the entire file contents with:
```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
rule_files:
- "/etc/prometheus/rules.yml"
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# Add client machines here as you connect them via Tailscale
# - job_name: 'node_exporter'
# static_configs:
# - targets:
# - '100.119.113.98:9100' # bs13
# - '100.76.222.82:9100' # pop-os / byfar
```
### Create alert rules file
```bash
sudo tee /etc/prometheus/rules.yml << 'EOF'
groups:
- name: server_alerts
rules:
- alert: ServerDown
expr: up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Server {{ $labels.instance }} is down"
description: "{{ $labels.instance }} has been unreachable for more than 1 minute."
- alert: HighCPU
expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU on {{ $labels.instance }}"
- alert: HighRAM
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
for: 5m
labels:
severity: warning
annotations:
summary: "High RAM on {{ $labels.instance }}"
- alert: LowDisk
expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
for: 5m
labels:
severity: warning
annotations:
summary: "Low disk on {{ $labels.instance }}"
EOF
```
### Run Prometheus as a systemd service
```bash
sudo tee /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Monitoring
After=network.target
[Service]
User=root
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.retention.time=30d \
--storage.tsdb.retention.size=10GB \
--web.listen-address=0.0.0.0:9090
Restart=always
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus
```
Verify: Open `http://localhost:9090` in your browser — you should see the Prometheus UI.
---
## Part 5: Install Alertmanager
### Download and install
```bash
cd /tmp
AM_VERSION="0.27.0"
wget https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz
tar xvf alertmanager-${AM_VERSION}.linux-amd64.tar.gz
sudo mv alertmanager-${AM_VERSION}.linux-amd64/alertmanager /usr/local/bin/
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
```
### Configure Alertmanager
```bash
sudo tee /etc/alertmanager/alertmanager.yml << 'EOF'
global:
resolve_timeout: 5m
route:
group_by: ['alertname']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'default'
receivers:
- name: 'default'
webhook_configs:
- url: 'YOUR_ROCKETCHAT_WEBHOOK_URL'
send_resolved: true
EOF
```
Replace `YOUR_ROCKETCHAT_WEBHOOK_URL` with your actual Rocket.Chat webhook URL.
### Run as systemd service
```bash
sudo tee /etc/systemd/system/alertmanager.service << 'EOF'
[Unit]
Description=Alertmanager
After=network.target
[Service]
User=root
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yml \
--storage.path=/var/lib/alertmanager
Restart=always
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager
```
---
## Part 6: Install Grafana
```bash
sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana
sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server
```
Verify: Open `http://localhost:3000` — default login is `admin` / `admin`. Change the password on first login.
### Connect Grafana to Prometheus
1. Go to **Connections → Data Sources → Add data source**
2. Select **Prometheus**
3. Set URL to `http://localhost:9090`
4. Click **Save & Test** — should show "Data source is working"
### Import recommended dashboards
- Go to **Dashboards → Import**
- Enter ID `1860` → Load → Import (Node Exporter Full)
- Enter ID `14574` → Load → Import (NVIDIA GPU Metrics)
---
## Part 7: Verify All Services Are Running
```bash
sudo systemctl status meshcentral prometheus alertmanager grafana-server tailscaled
```
All five should show `active (running)`.
Quick port check:
```bash
ss -tlnp | grep -E "4430|9090|9093|3000"
```
Expected:
```
LISTEN *:4430 → MeshCentral
LISTEN *:9090 → Prometheus
LISTEN *:9093 → Alertmanager
LISTEN *:3000 → Grafana
```
---
## Part 8: Adding Client Machines
For each new client machine, see `07_connect_remote_machine.md`.
After the client is connected to MeshCentral and Tailscale, add its Tailscale IP to Prometheus:
```bash
sudo nano /etc/prometheus/prometheus.yml
```
Add under `scrape_configs`:
```yaml
- job_name: 'node_exporter'
static_configs:
- targets:
- '100.x.x.x:9100' # machine name
```
Reload Prometheus without restarting:
```bash
curl -X POST http://localhost:9090/-/reload
```
---
## Part 9: Production Checklist
- [ ] Native Linux (not WSL) — services survive reboots
- [ ] All 5 services enabled with `systemctl enable`
- [ ] Sleep/hibernate disabled
- [ ] MeshCentral admin account created and saved
- [ ] Tailscale logged in and showing `Connected`
- [ ] Grafana default password changed
- [ ] At least one client machine connected and showing in Prometheus targets
- [ ] At least one Grafana dashboard imported and showing data
- [ ] Test alert fired and received in Rocket.Chat
---
## Quick Reference
| Service | Port | URL |
|---|---|---|
| MeshCentral | 4430 | `https://localhost:4430` |
| Prometheus | 9090 | `http://localhost:9090` |
| Alertmanager | 9093 | `http://localhost:9093` |
| Grafana | 3000 | `http://localhost:3000` |
| Tailscale | — | `tailscale status` |
| Command | Purpose |
|---|---|
| `sudo systemctl restart meshcentral` | Restart RMM server |
| `sudo systemctl restart prometheus` | Restart metrics collector |
| `curl -X POST http://localhost:9090/-/reload` | Reload Prometheus config without restart |
| `tailscale ip -4` | Get this machine's Tailscale IP |
| `tailscale status` | See all connected machines |

405
09_central_system_setup.md Normal file
View File

@@ -0,0 +1,405 @@
# Central Monitoring System — Complete Setup
## Prometheus + Grafana + Alertmanager + systemd Process Management
Run every command below on your **central Linux server** (native Linux, not WSL).
---
## Prerequisites
- Ubuntu 22.04 or 24.04 LTS (native, always-on machine)
- You have already installed Tailscale and MeshCentral (see `08_central_server_production_setup.md`)
- Your Tailscale IP is noted (run `tailscale ip -4` to get it)
- You are running all commands as your normal user (use `sudo` where shown)
---
## Part 1: Prometheus
### Step 1 — Download and Install
```bash
cd /tmp
PROM_VERSION="2.51.2"
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
tar xvf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
cd prometheus-${PROM_VERSION}.linux-amd64
sudo mv prometheus promtool /usr/local/bin/
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo mv prometheus.yml consoles console_libraries /etc/prometheus/
```
Verify:
```bash
prometheus --version
# Should print: prometheus, version 2.51.2
```
### Step 2 — Write the Config File
```bash
sudo tee /etc/prometheus/prometheus.yml << 'EOF'
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
rule_files:
- "/etc/prometheus/rules.yml"
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# Add each client machine's Tailscale IP here after connecting them
# - job_name: 'node_exporter'
# static_configs:
# - targets:
# - '100.x.x.x:9100' # machine name
# - '100.x.x.x:9100' # machine name
EOF
```
### Step 3 — Write Alert Rules
```bash
sudo tee /etc/prometheus/rules.yml << 'EOF'
groups:
- name: server_alerts
rules:
- alert: ServerDown
expr: up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Server {{ $labels.instance }} is DOWN"
description: "{{ $labels.instance }} has been unreachable for over 1 minute."
- alert: HighCPU
expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU on {{ $labels.instance }}"
description: "CPU usage is above 85% for 5 minutes."
- alert: HighRAM
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
for: 5m
labels:
severity: warning
annotations:
summary: "High RAM on {{ $labels.instance }}"
description: "RAM usage is above 90% for 5 minutes."
- alert: LowDisk
expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
for: 5m
labels:
severity: warning
annotations:
summary: "Low disk on {{ $labels.instance }}"
description: "Root disk has less than 15% free space."
- alert: HighGPUTemp
expr: nvidia_smi_temperature_gpu > 85
for: 5m
labels:
severity: critical
annotations:
summary: "GPU overheating on {{ $labels.instance }}"
description: "GPU temperature is above 85°C."
EOF
```
### Step 4 — Create systemd Service
```bash
sudo tee /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Monitoring
Documentation=https://prometheus.io/docs/
After=network.target
[Service]
User=root
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.retention.time=30d \
--storage.tsdb.retention.size=10GB \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus
```
Expected: `active (running)`
Verify in browser: `http://localhost:9090`
### Step 5 — Adding a New Client to Prometheus
After connecting a client machine via Tailscale and deploying Node Exporter:
```bash
sudo nano /etc/prometheus/prometheus.yml
```
Add the machine under `scrape_configs`:
```yaml
- job_name: "node_exporter"
static_configs:
- targets:
- "100.x.x.x:9100" # machine hostname
```
Reload without restarting (since `--web.enable-lifecycle` is set):
```bash
curl -X POST http://localhost:9090/-/reload
```
Check `http://localhost:9090/targets` — new machine should appear green.
---
## Part 2: Alertmanager
### Step 1 — Download and Install
```bash
cd /tmp
AM_VERSION="0.27.0"
wget https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz
tar xvf alertmanager-${AM_VERSION}.linux-amd64.tar.gz
cd alertmanager-${AM_VERSION}.linux-amd64
sudo mv alertmanager amtool /usr/local/bin/
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
```
### Step 2 — Configure Alertmanager
```bash
sudo tee /etc/alertmanager/alertmanager.yml << 'EOF'
global:
resolve_timeout: 5m
smtp_smarthost: "smtp.gmail.com:587"
smtp_from: "no-reply@seekright.com"
smtp_auth_username: "no-reply@seekright.com"
smtp_auth_password: "clzfywewriqxlvqv"
route:
receiver: "fallback-do-nothing"
group_wait: 10s # ← ADD THIS
group_interval: 1m # ← ADD THIS (must be ≤ repeat_interval)
repeat_interval: 1m # now this actually works
routes:
- receiver: "rocket-chat-alerts"
continue: true
- receiver: "email-alerts"
continue: true
receivers:
- name: "fallback-do-nothing"
- name: "rocket-chat-alerts"
webhook_configs: # Changed from slack_configs
- url: "https://text.seekright.com/hooks/6a05a268a88367bc6e0fe318/aNjZzmLy2wB3R4BR7ZtoLtAZYyfNyfLAoQMszmhfBDveKrB>
send_resolved: true
- name: "email-alerts"
EOF
```
Replace:
- `YOUR_GMAIL@gmail.com` — your Gmail address
- `YOUR_GMAIL_APP_PASSWORD` — Gmail App Password (not your login password — create one at myaccount.google.com → Security → App Passwords)
- `YOUR_ROCKETCHAT_WEBHOOK_URL` — your Rocket.Chat incoming webhook URL
### Step 3 — Create systemd Service
```bash
sudo tee /etc/systemd/system/alertmanager.service << 'EOF'
[Unit]
Description=Alertmanager
Documentation=https://prometheus.io/docs/alerting/alertmanager/
After=network.target
[Service]
User=root
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yml \
--storage.path=/var/lib/alertmanager \
--web.listen-address=0.0.0.0:9093
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager
```
Verify in browser: `http://localhost:9093`
---
## Part 3: Grafana
### Step 1 — Install via apt (Official Repo)
```bash
sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana
sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server
```
Verify in browser: `http://localhost:3000`
Default login: `admin` / `admin` — change password immediately on first login.
### Step 2 — Connect Grafana to Prometheus
1. Go to **Connections → Data Sources → Add data source**
2. Choose **Prometheus**
3. Set URL: `http://localhost:9090`
4. Click **Save & Test** — must show "Data source is working"
### Step 3 — Import Dashboards
Go to **Dashboards → Import** and import these IDs one by one:
| Dashboard ID | What It Shows |
| ------------ | ------------------------------------------------------------- |
| `1860` | Node Exporter Full — CPU, RAM, Disk, Network for all machines |
| `14574` | NVIDIA GPU Metrics — GPU temp, VRAM, utilization |
---
## Part 4: Process Management Reference
### Check all service statuses at once
```bash
sudo systemctl status prometheus alertmanager grafana-server
```
### Start / Stop / Restart any service
```bash
sudo systemctl start prometheus
sudo systemctl stop prometheus
sudo systemctl restart prometheus
sudo systemctl start alertmanager
sudo systemctl stop alertmanager
sudo systemctl restart alertmanager
sudo systemctl start grafana-server
sudo systemctl stop grafana-server
sudo systemctl restart grafana-server
```
### Check live logs for any service
```bash
sudo journalctl -u prometheus -f
sudo journalctl -u alertmanager -f
sudo journalctl -u grafana-server -f
```
### Reload Prometheus config without restarting
```bash
curl -X POST http://localhost:9090/-/reload
# Only works because we set --web.enable-lifecycle
```
### Check all listening ports
```bash
ss -tlnp | grep -E "9090|9093|3000|4430"
```
### Check all enabled services (confirms they survive reboot)
```bash
sudo systemctl is-enabled prometheus alertmanager grafana-server
# All three must print: enabled
```
---
## Part 5: Final Verification Checklist
Run this after complete setup to confirm everything is working:
```bash
echo "=== Service Status ===" && \
sudo systemctl is-active prometheus alertmanager grafana-server && \
echo "=== Prometheus Targets ===" && \
curl -s http://localhost:9090/api/v1/targets | python3 -c "import sys,json; [print(t['labels']['job'], t['health']) for t in json.load(sys.stdin)['data']['activeTargets']]" && \
echo "=== Ports Listening ===" && \
ss -tlnp | grep -E "9090|9093|3000"
```
Expected output:
```
=== Service Status ===
active
active
active
=== Prometheus Targets ===
prometheus up
node_exporter up
=== Ports Listening ===
LISTEN *:9090 prometheus
LISTEN *:9093 alertmanager
LISTEN *:3000 grafana
```
---
## Quick Reference
| Service | Port | Config File | Log Command |
| ------------ | ---- | -------------------------------------------- | --------------------------------- |
| Prometheus | 9090 | `/etc/prometheus/prometheus.yml` | `journalctl -u prometheus -f` |
| Alertmanager | 9093 | `/etc/alertmanager/alertmanager.yml` | `journalctl -u alertmanager -f` |
| Grafana | 3000 | `/etc/grafana/grafana.ini` | `journalctl -u grafana-server -f` |
| MeshCentral | 4430 | `~/meshcentral/meshcentral-data/config.json` | `journalctl -u meshcentral -f` |
| Tailscale | — | — | `tailscale status` |

211
10_script_explanations.md Normal file
View File

@@ -0,0 +1,211 @@
# Script Files — Line by Line Explanation
---
## 1. `deploy_exporters.sh`
**Purpose:** Installs Node Exporter and (optionally) Nvidia GPU Exporter on a remote client machine as permanent background services.
**How it is used:** Pushed to remote machines via MeshCentral's "Run Commands" feature. Runs as root. Never needs to be touched manually on each machine.
---
### Full Script with Explanation
```bash
#!/bin/bash
```
**Shebang line.** Tells the operating system: "Use the Bash shell to run this file." Without this, the OS doesn't know which interpreter to use.
---
```bash
set -e
```
**Exit on any error.** If ANY command in the script fails, the entire script stops immediately instead of blindly continuing. Without this, a failed download might be followed by an attempt to install a corrupt file — causing silent broken installs.
---
```bash
echo "Starting Automated Exporter Installation..."
if [ "$EUID" -ne 0 ]; then echo "❌ Error: Please run this script with sudo."; exit 1; fi
```
**Root check.** `$EUID` is the current user's ID. Root's ID is always `0`. If the script is not running as root, it prints an error and exits immediately. All the commands below (installing files to `/usr/local/bin`, creating system users, writing systemd services) require root — if we don't check here, they will fail silently.
---
```bash
NODE_VERSION="1.7.0"
GPU_VERSION="1.4.1"
```
**Version variables.** Defined once at the top so you only need to change the version number in one place when upgrading. Without this, you would have to find and replace the version string in 4 different places.
---
```bash
cd /tmp
wget -q https://github.com/prometheus/node_exporter/releases/download/v${NODE_VERSION}/node_exporter-${NODE_VERSION}.linux-amd64.tar.gz
tar -xf node_exporter-${NODE_VERSION}.linux-amd64.tar.gz
```
**Download and extract.** `cd /tmp` — work in a temporary folder that all users can write to. `wget -q` — download silently (no progress bar, cleaner output for remote execution). `tar -xf` — extract the downloaded archive.
---
```bash
if ! id "node_exporter" &>/dev/null; then useradd --no-create-home --shell /bin/false node_exporter; fi
```
**Create a dedicated system user (if it doesn't already exist).** This is a security best practice — Node Exporter runs as its own unprivileged user `node_exporter` instead of root.
- `id "node_exporter"` — checks if the user already exists
- `! id` — if it does NOT exist, run the useradd command
- `--no-create-home` — no home directory needed (it's a service account, not a human)
- `--shell /bin/false` — this user cannot log in interactively (prevents abuse if account is compromised)
---
```bash
cp node_exporter-${NODE_VERSION}.linux-amd64/node_exporter /usr/local/bin/
chown node_exporter:node_exporter /usr/local/bin/node_exporter
```
**Install the binary.** `cp` — copy the extracted binary to `/usr/local/bin/` which is the standard location for manually installed programs on Linux. `chown` — change the file owner to the `node_exporter` user so only that user can execute it.
---
```bash
cat << 'EOF' > /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
EOF
```
**Write the systemd service file.** This is what makes Node Exporter a background service that:
- Starts automatically on every reboot (`WantedBy=multi-user.target`)
- Waits for the network to be ready before starting (`After=network-online.target`)
- Runs as the `node_exporter` user (not root)
- Automatically restarts if it crashes
`cat << 'EOF' > file` is a "here document" — it writes everything between `EOF` markers directly into the file without needing a separate text editor.
---
```bash
systemctl daemon-reload
systemctl enable --now node_exporter
```
**Register and start the service.**
- `daemon-reload` — tells systemd to re-read all service files (required after creating a new .service file)
- `enable` — registers the service to start on boot
- `--now` — also starts it immediately right now (instead of waiting for next reboot)
---
```bash
if command -v nvidia-smi &> /dev/null; then
# ... install GPU exporter ...
else
echo "No Nvidia GPU detected. Skipping GPU Exporter installation."
fi
```
**Conditional GPU detection.** `command -v nvidia-smi` checks if the `nvidia-smi` tool exists (it only exists on machines with Nvidia drivers installed). If found, the GPU exporter is installed. If not found, the script silently skips — so the same script works on both GPU and non-GPU machines without any modification.
---
## 2. `test_install.sh`
**Purpose:** A safe, harmless script to verify that MeshCentral's remote execution works before deploying the real `deploy_exporters.sh`. It installs `cowsay` and `htop` — two completely harmless utilities — just to prove the remote execution pipeline works end to end.
**How it is used:** Run first on a test machine via MeshCentral before ever running `deploy_exporters.sh`. If this works, you know `deploy_exporters.sh` will work too.
---
### Full Script with Explanation
```bash
#!/bin/bash
set -e
```
Same as above — Bash interpreter + exit on error.
---
```bash
echo "Starting Dummy Installation Test..."
if [ "$EUID" -ne 0 ]; then
echo "❌ Error: Please run this script with sudo."
exit 1
fi
```
Same root check as in `deploy_exporters.sh`. Even a test script needs root because `apt-get install` requires root.
---
```bash
apt-get update -yq
apt-get install -yq cowsay htop
```
**Install the test tools.**
- `apt-get update` — refresh the package list from Ubuntu's package servers
- `-y` — automatically answer "yes" to all prompts (required for non-interactive remote execution — there is no human at the keyboard to press Y)
- `-q` — quiet mode, minimal output
`cowsay` — prints ASCII art of a cow saying a message. Completely harmless.
`htop` — an interactive process viewer, like a better Task Manager for Linux.
---
```bash
if command -v htop &> /dev/null; then
echo "✅ 'htop' installed successfully!"
fi
if command -v /usr/games/cowsay &> /dev/null; then
echo "✅ 'cowsay' installed successfully!"
/usr/games/cowsay "Hello from the automated script! ..."
fi
```
**Verify the installation actually worked.** Instead of trusting that `apt-get` succeeded, the script independently checks whether the commands now exist using `command -v`. If `htop` is not found after the install, something silently failed and this catches it.
`/usr/games/cowsay` is the full path because `cowsay` installs into `/usr/games/` which is sometimes not in the system PATH.
---
```bash
echo "To clean up and remove these test tools later, just run:"
echo "sudo apt-get remove --purge -y cowsay htop"
```
**Cleanup instructions.** Tells you exactly how to undo what the script did. Good practice for any test script.
---
## Summary: How the Scripts Relate to the Whole System
```
Central Server (you)
└── MeshCentral Dashboard (browser)
└── Group Action → Run Commands
├── test_install.sh ← Run FIRST to verify pipeline works
└── deploy_exporters.sh ← Run AFTER test passes, on all real machines
├── Installs Node Exporter (always)
└── Installs GPU Exporter (only if Nvidia GPU detected)
│
▼
Client Machine now exposes:
├── port 9100 → Node Exporter (CPU, RAM, Disk, Network)
└── port 9835 → GPU Exporter (temp, VRAM, utilization)
│
▼
Prometheus scrapes these ports via Tailscale IP
│
▼
Grafana displays the data as dashboards
│
▼
Alertmanager sends alerts to Rocket.Chat / Gmail
```

83
11_synology_vpn_setup.md Normal file
View File

@@ -0,0 +1,83 @@
# Synology VPN Server Setup Guide
This document outlines the architecture, security implications, and exact steps required to set up a traditional OpenVPN server on a Synology NAS. This method replaces third-party SaaS VPNs (like Tailscale) with a 100% self-hosted and self-controlled infrastructure.
---
## 1. The Architecture (How the Flow Works)
When an employee works from home and needs to access the office network, here is the exact path their data takes:
1. **The Employee's Laptop** runs the OpenVPN Client software.
2. The laptop sends heavily encrypted traffic across the public internet to your office's **Public Internet IP Address** (or your DDNS name, like `office.synology.me`).
3. The traffic hits your office's physical **Internet Router** (your ISP modem/router).
4. Because you set up a **Port Forwarding Rule**, the router catches that traffic and says, *"Ah, this is VPN traffic! I will pass this directly to the Synology NAS."*
5. The **Synology NAS** receives the traffic, verifies the cryptographic certificate, decrypts the request, and grants the employee access to the office files and servers.
---
## 2. What Exactly Are We Exposing? (Security Implications)
**You are exposing exactly ONE port on your office router: `UDP 1194`.**
Is this safe? Yes, if configured correctly.
* When a hacker scans the internet and hits port `1194` on your router, the Synology VPN server will not even respond unless the hacker presents a highly complex, pre-generated cryptographic Certificate.
* To hackers, the port appears "dead" or invisible unless they have the exact `.ovpn` configuration file that you exported.
* **Warning:** The only risk is if the Synology NAS software itself has an unpatched security flaw. Because of this, it is critical that you keep your Synology DSM software updated at all times.
---
## 3. Step-by-Step Setup Guide
### Phase 1: Establish a Permanent Address (DDNS)
Most office internet connections change their Public IP address periodically. To ensure employees can always find the office, you need a name that auto-updates.
1. Log into your Synology NAS.
2. Go to **Control Panel** -> **External Access** -> **DDNS**.
3. Click **Add**.
4. Service Provider: Select **Synology**.
5. Hostname: Pick a name (e.g., `seekright.synology.me`).
6. Click **Test Connection** and then **OK**.
*Now, employees will connect to `seekright.synology.me` instead of a random string of numbers.*
### Phase 2: Install and Configure the VPN Server
1. Open the Synology **Package Center**.
2. Search for and install **VPN Server**.
3. Open the VPN Server app. On the left menu, click **OpenVPN**.
4. Check the box to **Enable OpenVPN server**.
5. Set the following:
* **Dynamic IP address:** `10.8.0.0` (This is the virtual IP range for connected laptops).
* **Max concurrent connections:** Pick a number (e.g., 20).
* **Port:** `1194`
* **Protocol:** `UDP`
* **Encryption:** `AES-256-CBC` (High security).
* Check the box: **Allow clients to access server's LAN** (Crucial: This lets them see printers and other office servers, not just the NAS).
6. Click **Apply**.
### Phase 3: Export the Client Configuration
1. Still on the OpenVPN screen in Synology, click the button that says **Export Configuration**.
2. This downloads a `.zip` file to your computer. Extract it.
3. Open the file named `VPNConfig.ovpn` using a text editor (like Notepad).
4. Find the line that says `remote YOUR_SERVER_IP 1194`.
5. Change `YOUR_SERVER_IP` to the DDNS name you created in Phase 1 (e.g., `remote seekright.synology.me 1194`).
6. Save the file. *This file is the "key" that you will securely email to your employees.*
### Phase 4: Office Router Port Forwarding (The Critical Step)
You must physically (or remotely) log into the main internet router for your office. Every router is different (Cisco, Unifi, Netgear, ISP Router), but the concept is identical:
1. Find the local IP address of your Synology NAS (e.g., `192.168.1.50`).
2. Log into the Office Router's admin page.
3. Find the **Port Forwarding** or **NAT** settings.
4. Create a new rule:
* **External Port:** `1194`
* **Internal Port:** `1194`
* **Protocol:** `UDP`
* **Internal IP / Target:** `192.168.1.50` (Your Synology's local IP).
5. Save and apply.
### Phase 5: Connect the Client
1. On the employee's laptop, download and install the official **OpenVPN Connect** client (available for Windows, Mac, iOS, Android).
2. Open the app and click the **+** or **Import Profile** button.
3. Upload the `VPNConfig.ovpn` file you created in Phase 3.
4. The employee enters their Synology Username and Password.
5. Click **Connect**.
They are now securely inside the office network.

48
12_cto_architecture_pitch.md Normal file
View File

@@ -0,0 +1,48 @@
# Executive Brief: Remote Infrastructure Monitoring Architecture
**To:** Chief Technology Officer (CTO)
**Subject:** Architecture & Security Proposal for 100-Node Remote Monitoring Infrastructure
## 1. Executive Summary
We are deploying a fleet of 100+ remote Linux servers globally. To ensure maximum uptime, we must monitor their hardware health (CPU/GPU) and have emergency remote terminal access.
The primary engineering challenge is **Secure Network Connectivity**. We cannot expose 100 remote Linux servers to the public internet. We must establish a secure, encrypted tunnel between our Central Monitoring Server and the 100 remote nodes.
We evaluated two architectural approaches: the traditional **Hub-and-Spoke VPN (Synology OpenVPN)** and the modern **Zero-Trust Mesh VPN (Tailscale/WireGuard)**.
---
## 2. Option A: Traditional Hub-and-Spoke (Synology OpenVPN)
In this architecture, our office Synology NAS acts as the central gateway. All 100 remote servers funnel their data through our office internet router to reach the central monitoring server.
### The Implementation Flow:
1. **DDNS (Dynamic DNS):** We configure the Synology NAS to broadcast its changing IP address to a static URL (e.g., `office.synology.me`).
2. **Port Forwarding:** We modify the main office internet router (firewall) to punch a hole for `UDP Port 1194`, forwarding it to the Synology.
3. **Certificate Generation:** We export a static `.ovpn` cryptographic certificate from the Synology.
4. **Client Deployment:** We manually SSH into all 100 remote machines, install the OpenVPN client, and inject the `.ovpn` certificate.
### Security & Scaling Risks (Why we advise against this):
* **Single Point of Failure:** If the office internet goes down, or the Synology restarts, all 100 machines disconnect instantly.
* **The Revocation Problem:** All 100 machines share the exact same `.ovpn` certificate. If one remote machine is physically stolen, we must revoke the certificate on the Synology to lock the hacker out. However, doing so instantly disconnects the other 99 healthy machines, requiring us to spend ~15 hours manually deploying a new certificate to all of them.
* **Firewall Risk:** We are forced to open an inbound port on the corporate office firewall.
---
## 3. Option B: Zero-Trust Mesh VPN (Tailscale) - *Recommended*
Tailscale is an enterprise overlay network built on the WireGuard protocol. Instead of funneling traffic through a central office bottleneck, it creates a peer-to-peer mesh.
### The Implementation Flow:
1. **Control Plane Integration:** We create a corporate Tailscale account (or self-host Headscale) to act as the central identity provider.
2. **Key Generation:** We generate a "Reusable Auth Key" from the admin console.
3. **Client Deployment:** We run a single, automated script on the 100 remote machines: `tailscale up --authkey=YOUR_KEY`. The machines authenticate and join the network silently in 30 seconds.
### Security & Scaling Benefits:
* **Zero Open Ports:** Neither the remote machines nor the central office router need any open firewall ports. All connections are outbound. The corporate firewall remains 100% closed.
* **Isolated Node Identities:** Every single machine mathematically generates its own unique encryption key. If Machine #42 is stolen, we click "Revoke" in the dashboard. Machine #42 goes permanently dark, and the other 99 machines are completely unaffected.
* **Peer-to-Peer Speed:** If the central monitoring server is in AWS, the remote nodes talk directly to AWS. Traffic never bounces through the office internet, saving bandwidth and reducing latency.
* **Automated Key Rotation:** OpenVPN requires manual certificate replacement every 1-3 years. Tailscale rotates node keys automatically in the background every few hours.
## 4. Conclusion & Next Steps
To ensure scalability and adhere to modern Zero-Trust security principles, we recommend deploying the **Tailscale Mesh Architecture**. It reduces manual engineering overhead from ~15 hours to ~45 minutes, eliminates single points of failure, and keeps the corporate office firewall completely closed.

87
13_master_deployment_manual.md Normal file
View File

@@ -0,0 +1,87 @@
# Master Deployment Manual: Scalable RMM & Monitoring Infrastructure
This document serves as the master reference guide for deploying, connecting, and managing the end-to-end Remote Monitoring and Management (RMM) infrastructure.
---
## 1. System Architecture Overview
The infrastructure is split into two primary environments:
### A. The Central System (`vm3`)
The central command center. It is responsible for gathering data, alerting admins, and providing remote terminal access.
* **Grafana (Port 3000):** The visualization UI where humans view metrics.
* **Prometheus (Port 9090):** The time-series database that reaches out to scrape metrics from clients.
* **Alertmanager (Port 9093):** Evaluates Prometheus rules and fires emails or Rocket.Chat messages.
* **MeshCentral (Port 4430):** The RMM dashboard that allows admins to open secure remote terminals to any client.
### B. The Remote Systems (Client Nodes)
The 100+ endpoints distributed globally that are being monitored.
* **Node Exporter (Port 9100):** Exposes base hardware metrics (CPU, RAM, Disk).
* **Nvidia GPU Exporter (Port 9835):** Exposes GPU metrics (Temperature, VRAM, Utilization).
* **MeshAgent:** A lightweight background process that connects back to MeshCentral.
---
## 2. The Network Connectivity Layer
Because Prometheus relies on a "Pull" architecture (scraping), the Central System must have a secure, encrypted tunnel to reach the Remote Systems without exposing them to the public internet. We support two models:
### Model A: Zero-Trust Mesh VPN (Tailscale) - *Primary Recommendation*
* **Why:** Peer-to-peer, automated key rotation, zero IP sub-net collisions, and no router configuration required.
* **How it's done:**
1. Admin generates a "Reusable, Pre-Approved Auth Key" in the Tailscale web console.
2. On the remote client, run: `curl -fsSL https://tailscale.com/install.sh | sh && sudo tailscale up --authkey=YOUR_KEY`
3. The machine instantly joins the `100.x.x.x` network.
### Model B: Traditional Hub-and-Spoke (Synology OpenVPN) - *Alternative*
* **Why:** 100% self-hosted; prevents reliance on third-party SaaS servers.
* **How it's done:**
1. **DDNS:** Configure Synology to broadcast to `yourcompany.synology.me`.
2. **Port Forwarding:** The office IT Admin configures the Cisco Router to forward `UDP Port 1194` to the Synology's Local IP.
3. **Certificates:** Admin exports the `VPNConfig.ovpn` file and edits the IP to match the DDNS name.
4. **Client Setup:** Move the `.ovpn` file to `/etc/openvpn/client.conf`, create an `auth.txt` file for automated login, and run `sudo systemctl enable --now openvpn@client`.
---
## 3. Central System Setup Guide
To stand up the central server (`vm3`), execute the automated `setup_central_server.sh` script.
**What the script does autonomously:**
1. Installs Tailscale to join the secure network.
2. Creates the `meshcentral` user and securely installs Node.js and the MeshCentral package.
3. Downloads the latest Prometheus and Alertmanager binaries, extracts them, and places them in `/usr/local/bin`.
4. Creates Prometheus configuration files (`prometheus.yml`), mapping it to the Alertmanager target.
5. Installs Grafana via the official APT repository.
6. Creates robust `systemd` service files for every component to ensure they auto-restart if the server reboots.
---
## 4. Remote Client Setup Guide
To onboard a new remote server, execute the `deploy_exporters.sh` script on the client machine.
**What the script does autonomously:**
1. Installs Node Exporter and sets it to run as a systemd background service.
2. Detects if an Nvidia GPU is present using `nvidia-smi`.
3. If a GPU is found, downloads and installs the Nvidia GPU Exporter.
4. Uses a `curl` webhook to fire a notification to a Rocket.Chat channel, alerting the admin that the deployment succeeded and providing the exporter statuses.
**MeshAgent Deployment:**
1. Log into the MeshCentral web UI (`https://100.x.x.x:4430`).
2. Create a Device Group.
3. Click "Add Agent" and copy the provided bash script.
4. Paste it into the client terminal to install the MeshAgent.
---
## 5. Security Protocols & Client Objections
When dealing with B2B deployments, clients may object to installing VPNs or Remote Control agents on their internal servers.
**Objection 1: "We do not want you to have unattended remote terminal access to our servers."**
* **Solution:** Configure MeshCentral to use **User-Consent Mode**. The MeshAgent will still run, but if an admin tries to open a terminal, a prompt will appear on the client's screen forcing them to click "Accept" before the connection is allowed.
**Objection 2: "We do not allow external VPNs (Tailscale/OpenVPN) inside our network."**
* **Solution:** Abandon the "Pull" architecture. Do not install Tailscale or OpenVPN. Instead, install a **Prometheus Agent** (like Grafana Alloy) on the client machine. The agent will "Push" metrics outbound over standard HTTPS (Port 443) to your Central Server. This requires zero inbound open ports and ensures you have no network access to their internal systems, providing maximum security for paranoid clients.

209
14_rmm_central_production_deployment.md Normal file
View File

@@ -0,0 +1,209 @@
# Seekright Pulse RMM - Central System Production Deployment Manual
## Fresh Linux Production VM Setup & Operations Guide (Transitional ngrok Testing Setup)
This document serves as the master production setup and deployment manual for hosting the **Seekright Pulse RMM Central System** on a dedicated Virtual Machine (VM).
This guide focuses on a **Transitional ngrok Testing Setup** designed to get your central backend and database running on a remote VM immediately and securely, matching your local network behavior, before transitioning to a full Nginx configuration later.
---
## 1. Why Use ngrok for VM Testing?
Running **ngrok** as a service on the remote VM offers several major benefits for transitional testing:
1. **Zero Open Ports:** You **do not** need to open port `8000` to the public internet on the VM firewall. ngrok creates a secure outbound TCP connection, protecting your VM backend from external scanners.
2. **URL Reusability:** Remote client agents can continue connecting to your established public ngrok HTTPS URL without requiring any code edits or DNS configurations.
3. **No SSL Setup Required:** ngrok automatically provides secure, pre-configured HTTPS certificates out-of-the-box.
---
## 2. Production Port Map & Access Control (ngrok Setup)
When using ngrok, the internal ports are locked down strictly to the loopback interface (`127.0.0.1`), ensuring maximum security during testing.
| Service | Local Port | Exposed Globally? | Access Method / Firewall |
|---|---|---|---|
| **FastAPI Backend** | `8000` | **No** (Directly) | Bound strictly to `127.0.0.1`. Exposed externally ONLY via the secure ngrok tunnel |
| **MongoDB Community** | `27017` | **No** | Bound strictly to `127.0.0.1`. No external access permitted |
| **Prometheus Server** | `9090` | **No** | Bound strictly to `127.0.0.1`. Accessible locally on VM or via SSH port forwarding |
---
## 3. Step-by-Step Installation Guide
Execute these steps chronologically on your remote VM.
### Step 1: OS Preparation
Update system index dependencies, disable sleep loops, and install required tools:
```bash
sudo apt-get update && sudo apt-get upgrade -y
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
sudo apt-get install -y git curl wget build-essential python3 python3-pip python3-venv
```
---
### Step 2: Install and Configure MongoDB
Install MongoDB Community Edition 8.0 and lock it down to the loopback interface.
```bash
# 1. Import the public GPG key
curl -fsSL https://www.mongodb.org/static/pgp/server-8.0.asc | sudo gpg --dearmor -o /usr/share/keyrings/mongodb-server-8.0.gpg
# 2. Add the MongoDB 8.0 repository for Ubuntu 24.04 (Noble)
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-8.0.gpg ] https://repo.mongodb.org/apt/ubuntu noble/mongodb-org/8.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list
# 3. Update repositories and install packages
sudo apt-get update
sudo apt-get install -y mongodb-org
# 4. Ensure bindIp is set strictly to 127.0.0.1 in the config file
sudo sed -i 's/bindIp: 127.0.0.1/bindIp: 127.0.0.1/g' /etc/mongod.conf
# 5. Start and enable service to survive system reboots
sudo systemctl daemon-reload
sudo systemctl enable mongod
sudo systemctl start mongod
# Verify service is running
sudo systemctl status mongod
```
---
### Step 3: Set Up the FastAPI Backend
Deploy the backend application code under `/opt/rmm-backend` and install dependencies.
```bash
# 1. Create dedicated application directory
sudo mkdir -p /opt/rmm-backend
sudo chown -R $USER:$USER /opt/rmm-backend
cd /opt/rmm-backend
# 2. Copy central_api_prototype.py and commands.json into /opt/rmm-backend/
# 3. Initialize Python Virtual Environment (Venv)
python3 -m venv venv
source venv/bin/activate
# 4. Install production dependencies
pip install --upgrade pip
pip install fastapi uvicorn pymongo dnspython python-dotenv httpx psutil
# 5. Configure production .env file
cat > /opt/rmm-backend/.env << 'EOF'
ROCKETCHAT_WEBHOOK_URL="https://text.seekright.com/hooks/6a0eb41ea88367bc6e101f0c/gvXfD8zSNRti43kEabqdE9tMCvNi9zAAoGfbao7cxcKbiW6R"
MONGODB_URI="mongodb://localhost:27017"
MONGODB_DB="rmm_db"
EOF
# Deactivate the environment
deactivate
```
#### Run FastAPI as a systemd Background Service
Configure Uvicorn to run as a persistent background daemon:
```bash
sudo tee /etc/systemd/system/rmm-backend.service << 'EOF'
[Unit]
Description=Seekright Pulse RMM Backend Service
After=network.target mongod.service
[Service]
Type=simple
User=root
WorkingDirectory=/opt/rmm-backend
ExecStart=/opt/rmm-backend/venv/bin/uvicorn central_api_prototype:app --host 127.0.0.1 --port 8000 --workers 4
Restart=always
RestartSec=5
EnvironmentFile=/opt/rmm-backend/.env
[Install]
WantedBy=multi-user.target
EOF
# Reload, enable, and launch the daemon
sudo systemctl daemon-reload
sudo systemctl enable rmm-backend
sudo systemctl start rmm-backend
```
---
### Step 4: Install and Run ngrok as a Persistent VM Service
Since a remote VM is managed via SSH, launching `ngrok http 8000` directly in the shell will terminate as soon as the SSH connection closes. To keep ngrok running persistently in the background on the VM, install and run it as a **systemd service**.
#### A. Install ngrok on the VM
```bash
# 1. Clean any previous broken list files
sudo rm -f /etc/apt/sources.list.d/ngrok.list
# 2. Add the official ngrok signing key and repository configuration
curl -sSL https://ngrok-agent.s3.amazonaws.com/ngrok.asc \
| sudo tee /etc/apt/trusted.gpg.d/ngrok.asc >/dev/null \
&& echo "deb https://ngrok-agent.s3.amazonaws.com buster main" \
| sudo tee /etc/apt/sources.list.d/ngrok.list
# 3. Update repositories and install ngrok
sudo apt-get update && sudo apt-get install -y ngrok
```
#### B. Authenticate ngrok
Run this command on the VM, substituting `<YOUR_AUTHTOKEN>` with your actual ngrok token:
```bash
sudo ngrok config add-authtoken <YOUR_AUTHTOKEN>
```
#### C. Create the ngrok systemd Service File
```bash
sudo tee /etc/systemd/system/ngrok.service << 'EOF'
[Unit]
Description=ngrok secure tunnel
After=network.target rmm-backend.service
[Service]
ExecStart=/usr/bin/ngrok http 8000 --log=stdout
Restart=always
RestartSec=5
User=root
[Install]
WantedBy=multi-user.target
EOF
# Enable and start the service
sudo systemctl daemon-reload
sudo systemctl enable ngrok
sudo systemctl start ngrok
```
#### D. Retrieve Your Public ngrok URL from the VM
Because the ngrok agent runs silently in the background, query the local ngrok client API to find the active HTTPS endpoint:
```bash
curl http://localhost:4040/api/tunnels
```
Look for the `"public_url"` value (e.g. `https://xxxx.ngrok-free.app`) in the JSON output.
---
### Step 5: Configure Firewall Protection (UFW)
Since ngrok tunnels traffic privately, secure your server by locking down all ports except SSH (`22`).
```bash
# Allow SSH access
sudo ufw allow ssh
# Enable the firewall
sudo ufw enable
# Verify
sudo ufw status verbose
```
---
## 4. Testing Your Setup
Once the VM configurations are applied, you can test the entire RMM loop securely:
1. **Remote Agents:** Configure `client_agent_prototype.py` on client machines to target the public ngrok HTTPS URL retrieved from **Step 4 (Part D)** of the VM.
2. **Local Dashboard UI:** Run `rmm-ui` locally on your machine. Update `const API_BASE` in your local `App.jsx` to target the same public ngrok HTTPS URL.
3. **Verify:** Launch your local frontend dashboard. You will see metrics and heartbeat logs from the client nodes streaming securely through your VM's background ngrok tunnel into the VM's MongoDB database.

90
15_push_architecture_guide.md Normal file
View File

@@ -0,0 +1,90 @@
# The "Push" Architecture (Prometheus Remote Write)
When a client refuses to use VPNs or Outbound Tunnels (like Cloudflare), we must use the **Push Architecture**. In this model, the remote client acts like a normal web browser. It wakes up every 15 seconds, packages the CPU/GPU data, and securely "uploads" it over HTTPS to your Central Server.
## 1. The Architecture Flow Diagram
```mermaid
graph LR
subgraph "Paranoid Client Network (100% Locked Down)"
NE[Node Exporter<br>Port 9100]
GE[GPU Exporter<br>Port 9835]
PA[Prometheus Agent<br>Scrapes local ports]
NE -.->|Scraped locally| PA
GE -.->|Scraped locally| PA
end
subgraph "The Public Internet"
HTTPS[HTTPS POST Request<br>Username & Password]
end
subgraph "Your Central Office Network"
ROUTER[Office Router<br>Port Forward 443]
NGINX[Nginx Reverse Proxy<br>Checks Password & SSL]
PROM[Central Prometheus<br>Port 9090 Receiver]
GRAF[Grafana<br>Port 3000]
end
PA ===>|Pushes Data Out| HTTPS
HTTPS ===>|Arrives at Public IP| ROUTER
ROUTER ===>|Routes to vm3| NGINX
NGINX ===>|If Password OK| PROM
PROM ===>|Stores Data| GRAF
```
## 2. What Needs to be Installed?
### On the Central Server (`vm3`)
Your Central Server acts as the public "mailbox" receiving data from 100 clients.
* **Prometheus:** Standard installation, but we must turn on the "Receiver" switch.
* **Nginx:** A highly secure web server. Prometheus does not have built-in passwords for receiving data. We put Nginx in front of Prometheus to act as a bouncer. If the client doesn't provide the correct username/password, Nginx blocks them.
* **Certbot (Let's Encrypt):** A tool that generates a free SSL certificate so the data travels over secure HTTPS.
### On the Remote Client
The client's firewall remains completely closed. No inbound connections are allowed.
* **Node & GPU Exporters:** These run normally, gathering hardware data locally.
* **Prometheus Agent:** We install a tiny version of Prometheus called "Agent Mode". It does not store data. Its only job is to scrape the local Node Exporter and push the data out to the internet.
---
## 3. How the Configuration Happens (Step-by-Step)
### Step 1: Central Router Setup
You log into your physical Cisco/ISP router at your main office. You create a Port Forwarding rule: **Route all internet traffic on Port 443 to the local IP of `vm3` (e.g., `192.168.1.199`).**
### Step 2: Nginx Bouncer Setup (Central Server)
On `vm3`, you install Nginx and create a password file (`.htpasswd`). You configure Nginx with this rule:
> *"If a request comes in on HTTPS (443), check if they have the password. If yes, forward the data to Prometheus on local Port 9090."*
### Step 3: Prometheus Receiver (Central Server)
By default, Prometheus is a "Pull" system. We edit the `prometheus.service` systemd file and add the startup flag: `--web.enable-remote-write-receiver`. This tells Prometheus it is allowed to catch incoming pushed data.
### Step 4: The Agent Setup (Remote Client)
On the remote client, we install the Prometheus binary, but we run it with a special configuration file (`agent.yml`).
The configuration looks exactly like this:
```yaml
# 1. Gather data from the local machine
scrape_configs:
- job_name: 'client_hardware'
static_configs:
- targets: ['localhost:9100', 'localhost:9835']
# 2. Push the data out to the Central Server
remote_write:
- url: "https://metrics.seekright.com/api/v1/write"
basic_auth:
username: "admin"
password: "SuperSecretPassword123"
```
## 4. The Final Result
1. The Prometheus Agent on the client wakes up.
2. It looks at `localhost:9100` and sees "CPU is at 45%".
3. It packages that string into a tiny web request.
4. It attaches the username `admin` and the password `SuperSecretPassword123` to the header.
5. It fires the request out across the public internet to `https://metrics.seekright.com/api/v1/write`.
6. Your office router catches it and sends it to `vm3`.
7. Nginx on `vm3` checks the password, says "Access Granted", and hands the CPU data to Prometheus.
8. You see the CPU data appear on Grafana.

5402
Global Server Monitoring Architecture.md Normal file
View File

File diff suppressed because it is too large Load Diff

470
central_api_prototype.py Normal file
View File

@@ -0,0 +1,470 @@
from dotenv import load_dotenv
load_dotenv()
from fastapi import FastAPI, HTTPException
from fastapi.responses import PlainTextResponse
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import Dict, Any, List
import uvicorn
app = FastAPI(title="RMM Central API")
# Enable CORS for frontend dashboard queries (React dev server runs on a separate port)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
import os
import json
from datetime import datetime
# In-memory storage for client hardware telemetry
client_telemetry = {}
from pymongo import MongoClient
# Database Connection: Configured via environment variable with local fallback
MONGODB_URI = os.getenv("MONGODB_URI", "mongodb://localhost:27017")
DB_NAME = os.getenv("MONGODB_DB", "rmm_db")
mongo_client = MongoClient(MONGODB_URI)
db = mongo_client[DB_NAME]
clients_collection = db["clients"]
logs_collection = db["logs"]
MAX_ENTRIES_PER_CLIENT = 100 # Keep the latest 100 historical logs per client node
ROCKETCHAT_WEBHOOK_URL = os.getenv("ROCKETCHAT_WEBHOOK_URL", "")
# In-memory storage to prevent duplicate/spam storage notifications
active_storage_alerts = set()
# Tracks the timestamp when a System DOWN notification was last sent for a client ID
# Key: client_id (str), Value: datetime object of the last notification time
last_down_notification_times = {}
def send_rocketchat_notification(text: str, color: str = "#808080"):
"""
Dispatches a formatted notification payload to the Rocket.Chat incoming webhook.
"""
if not ROCKETCHAT_WEBHOOK_URL:
try:
print(f"[Rocket.Chat Simulation] {text}")
except UnicodeEncodeError:
# Fallback for Windows consoles that do not support printing unicode emojis
sanitized_text = text.encode('ascii', errors='backslashreplace').decode('ascii')
print(f"[Rocket.Chat Simulation] {sanitized_text}")
return
payload = {
"text": text,
"attachments": [
{
"color": color,
"ts": datetime.now().isoformat()
}
]
}
try:
import urllib.request
import json
req = urllib.request.Request(
ROCKETCHAT_WEBHOOK_URL,
data=json.dumps(payload).encode('utf-8'),
headers={'Content-Type': 'application/json'}
)
with urllib.request.urlopen(req, timeout=3.0) as response:
pass
except Exception as e:
print(f"[!] Error sending Rocket.Chat notification: {e}")
def append_to_logs(log_type: str, client_id: str, detail: Any):
"""
Appends a new event log grouped by client_id in MongoDB logs collection.
Ensures fair-share log limits per system so noisy clients never overwrite others.
"""
try:
entry = {
"client_id": client_id,
"timestamp": datetime.now().isoformat(),
"type": log_type,
"detail": detail
}
logs_collection.insert_one(entry)
# Enforce fair-share logging limits per system (FIFO circular buffer)
count = logs_collection.count_documents({"client_id": client_id})
if count > MAX_ENTRIES_PER_CLIENT:
oldest_docs = logs_collection.find(
{"client_id": client_id},
{"_id": 1}
).sort("timestamp", 1).limit(count - MAX_ENTRIES_PER_CLIENT)
ids_to_delete = [doc["_id"] for doc in oldest_docs]
if ids_to_delete:
logs_collection.delete_many({"_id": {"$in": ids_to_delete}})
except Exception as e:
print(f"[!] Error writing log to MongoDB: {e}")
def load_clients() -> Dict[str, Any]:
try:
# If collection is empty, seed it with default registry
if clients_collection.count_documents({}) == 0:
default_data = {
"client_1": {"pending_command": "none", "last_seen": None, "active": False},
"client_2": {"pending_command": "none", "last_seen": None, "active": False},
"client_3": {"pending_command": "none", "last_seen": None, "active": False}
}
save_clients(default_data)
return default_data
# Load all documents
cursor = clients_collection.find()
data = {}
for doc in cursor:
cid = doc["_id"]
data[cid] = {k: v for k, v in doc.items() if k != "_id"}
# Compute active status dynamically based on 30-second heartbeat check-ins
modified = False
for cid, info in data.items():
last_seen_str = info.get("last_seen")
was_active = info.get("active", False)
is_active = False
if last_seen_str:
try:
last_seen_dt = datetime.fromisoformat(last_seen_str)
if (datetime.now() - last_seen_dt).total_seconds() < 30:
is_active = True
except Exception:
pass
if was_active != is_active:
info["active"] = is_active
modified = True
# Active status transitions alerts
if is_active:
# Transitions from offline (DOWN) to online (UP)
if cid in last_down_notification_times:
send_rocketchat_notification(
text=f"✅ **System UP:** Client `{cid}` has recovered and is back online.",
color="#2ecc71"
)
last_down_notification_times.pop(cid, None)
else:
# Transitions from online (UP) to offline (DOWN) for the first time
now = datetime.now()
last_down_notification_times[cid] = now
send_rocketchat_notification(
text=f"🚨 **System DOWN:** Client `{cid}` has missed heartbeats for over 30 seconds.",
color="#e74c3c"
)
else:
# No transition (state remains the same)
if not is_active:
# Client remains offline. Check if we should send a repeating reminder.
# We ONLY send a reminder if it is already tracked in last_down_notification_times.
# This prevents sending alerts for historically offline systems at server boot!
if cid in last_down_notification_times:
now = datetime.now()
elapsed_seconds = (now - last_down_notification_times[cid]).total_seconds()
if elapsed_seconds >= 3600: # 1 hour (3600 seconds)
last_down_notification_times[cid] = now
send_rocketchat_notification(
text=f"🚨 **System STILL DOWN:** Client `{cid}` remains offline (reminder sent every hour).",
color="#e74c3c"
)
if modified:
save_clients(data)
return data
except Exception as e:
print(f"[!] Error loading clients from MongoDB: {e}")
return {}
def save_clients(data: Dict[str, Any]):
try:
for cid, info in data.items():
# Use upsert to update existing client or insert new client automatically
clients_collection.update_one(
{"_id": cid},
{"$set": info},
upsert=True
)
except Exception as e:
print(f"[!] Error saving clients to MongoDB: {e}")
class TelemetryPayload(BaseModel):
cpu_percent: float
memory_percent: float
memory_total_gb: float
memory_free_gb: float
disks: List[Dict[str, Any]]
gpus: List[Dict[str, Any]]
class CommandResultPayload(BaseModel):
command: str
returncode: int
stdout: str
stderr: str
@app.get("/api/get-commands")
async def get_whitelisted_commands(platform: str):
"""
Endpoint for remote agents to fetch their OS-specific whitelisted commands.
"""
commands_file = os.path.join(os.path.dirname(os.path.abspath(__file__)), "commands.json")
try:
with open(commands_file, "r") as f:
data = json.load(f)
return data.get(platform, {})
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to load commands: {e}")
@app.get("/api/get-command")
async def get_command(client_id: str):
"""
Endpoint for clients to poll for pending commands.
Client Agent hits this endpoint to ask: "Do I have any work to do?"
"""
clients = load_clients()
# 1. Auto-register new client if we haven't seen it yet
if client_id not in clients:
print(f"[+] Dynamic fleet auto-registration: Registered new agent '{client_id}'")
clients[client_id] = {"pending_command": "none", "last_seen": None, "active": True}
# 2. Fetch the command
cmd = clients[client_id].get("pending_command", "none")
# Log if a valid pending command was polled
if cmd != "none":
append_to_logs("command_polled", client_id, {"command": cmd})
# 3. Reset the command to "none", mark active, and update last_seen timestamp
clients[client_id]["pending_command"] = "none"
clients[client_id]["active"] = True
clients[client_id]["last_seen"] = datetime.now().isoformat()
save_clients(clients)
return {"command": cmd}
@app.post("/api/schedule-command")
async def schedule_command(client_id: str, command: str):
"""
Endpoint for your Dashboard/UI to schedule a new command.
Supports a comma-separated list of client IDs for batch fleet updates.
"""
clients = load_clients()
target_ids = [cid.strip() for cid in client_id.split(",") if cid.strip()]
if not target_ids:
raise HTTPException(status_code=400, detail="No target client IDs specified")
for cid in target_ids:
if cid not in clients:
clients[cid] = {"pending_command": "none", "last_seen": None, "active": False}
clients[cid]["pending_command"] = command
# Log command scheduling per system
append_to_logs("command_scheduled", cid, {"command": command})
save_clients(clients)
return {"status": "success", "message": f"Command '{command}' scheduled for {', '.join(target_ids)}"}
@app.post("/api/telemetry")
async def receive_telemetry(client_id: str, payload: TelemetryPayload):
"""
Endpoint for agents to push their live hardware telemetry.
"""
clients = load_clients()
# Auto-register new client if we see it through telemetry first
if client_id not in clients:
print(f"[+] Dynamic fleet auto-registration: Registered new agent '{client_id}' through telemetry")
clients[client_id] = {"pending_command": "none", "last_seen": None, "active": True}
# Update last_seen timestamp, active status, and store the current telemetry details
clients[client_id]["last_seen"] = datetime.now().isoformat()
clients[client_id]["active"] = True
clients[client_id]["telemetry"] = payload.dict()
save_clients(clients)
# Log telemetry history event
append_to_logs("telemetry", client_id, payload.dict())
# Storage limits warning checker
for disk in payload.disks:
mount = disk.get("mount", "/")
percent = disk.get("percent", 0.0)
alert_key = f"{client_id}:{mount}"
if percent >= 80.0:
if alert_key not in active_storage_alerts:
active_storage_alerts.add(alert_key)
send_rocketchat_notification(
text=f"⚠️ **Storage Warning:** Client `{client_id}` disk `{mount}` is at **{percent}%** capacity.",
color="#f39c12"
)
else:
if alert_key in active_storage_alerts:
active_storage_alerts.remove(alert_key)
send_rocketchat_notification(
text=f"✅ **Storage Recovered:** Client `{client_id}` disk `{mount}` has cleared warning state and is at **{percent}%**.",
color="#2ecc71"
)
data = payload.dict()
client_telemetry[client_id] = data
# Temporarily print the data to the console for the user to see!
print(f"\n[TELEMETRY RECEIVED] from {client_id}:")
print(f" CPU: {data['cpu_percent']}%")
print(f" RAM: {data['memory_percent']}% ({data['memory_free_gb']} GB Free / {data['memory_total_gb']} GB Total)")
for disk in data['disks']:
print(f" Drive {disk['mount']}: {disk['percent']}% Used ({disk['free_gb']} GB Free / {disk['total_gb']} GB Total)")
for gpu in data['gpus']:
power_str = f"Power: {gpu.get('power_draw', 'N/A')} / {gpu.get('power_limit', 'N/A')}"
fan_str = f"Fan: {gpu.get('fan_speed', 'N/A')}"
print(f" GPU [{gpu['name']}]: Core: {gpu['utilization']}, Temp: {gpu['temp']}, VRAM: {gpu['memory_used']}/{gpu['memory_total']}, {power_str}, {fan_str}")
return {"status": "success"}
@app.get("/api/telemetry")
async def get_all_telemetry():
"""
Endpoint to view the live dashboard data of all clients.
"""
return client_telemetry
@app.get("/api/clients")
async def get_clients_api():
"""
Endpoint for the React UI to fetch the live active client registry with computed statuses.
"""
return load_clients()
@app.get("/api/logs")
async def get_logs_history():
"""
Endpoint to retrieve logs history grouped by client_id.
"""
try:
cursor = logs_collection.find().sort("timestamp", 1)
grouped_logs = {}
for doc in cursor:
cid = doc.get("client_id")
if not cid:
continue
if cid not in grouped_logs:
grouped_logs[cid] = []
entry = {
"timestamp": doc.get("timestamp"),
"type": doc.get("type"),
"detail": doc.get("detail")
}
grouped_logs[cid].append(entry)
return grouped_logs
except Exception as e:
print(f"[!] Error loading logs from MongoDB: {e}")
return {}
@app.post("/api/command-result")
async def receive_command_result(client_id: str, payload: CommandResultPayload):
"""
Endpoint for remote agents to push execution results back to the central server.
"""
append_to_logs("command_result", client_id, payload.dict())
return {"status": "success"}
@app.get("/api/get-raw-commands")
async def get_raw_commands():
"""
Endpoint for the UI to load the complete commands.json whitelist file.
"""
commands_file = os.path.join(os.path.dirname(os.path.abspath(__file__)), "commands.json")
try:
with open(commands_file, "r") as f:
return json.load(f)
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to read commands: {e}")
@app.post("/api/save-commands")
async def save_commands(commands: Dict[str, Any]):
"""
Endpoint for the UI to save changes back to commands.json.
"""
commands_file = os.path.join(os.path.dirname(os.path.abspath(__file__)), "commands.json")
try:
with open(commands_file, "w") as f:
json.dump(commands, f, indent=2)
return {"status": "success", "message": "commands.json saved successfully"}
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to save commands: {e}")
@app.get("/metrics", response_class=PlainTextResponse)
async def get_prometheus_metrics():
"""
Exposes the latest client telemetry in standard Prometheus text format.
"""
clients = load_clients()
metrics_lines = []
for client_id, data in clients.items():
# Active status (1.0 for active/online, 0.0 for inactive/offline)
active_val = 1.0 if data.get("active", False) else 0.0
metrics_lines.append(f'rmm_client_active{{client_id="{client_id}"}} {active_val}')
# Telemetry stats
telemetry = data.get("telemetry")
if telemetry:
cpu = telemetry.get("cpu_percent", 0.0)
ram = telemetry.get("memory_percent", 0.0)
ram_free = telemetry.get("memory_free_gb", 0.0)
ram_total = telemetry.get("memory_total_gb", 0.0)
metrics_lines.append(f'rmm_cpu_utilization{{client_id="{client_id}"}} {cpu}')
metrics_lines.append(f'rmm_memory_utilization{{client_id="{client_id}"}} {ram}')
metrics_lines.append(f'rmm_memory_free_bytes{{client_id="{client_id}"}} {ram_free * (1024**3)}')
metrics_lines.append(f'rmm_memory_total_bytes{{client_id="{client_id}"}} {ram_total * (1024**3)}')
# Disk mounts
for disk in telemetry.get("disks", []):
mount = disk.get("mount", "/")
disk_percent = disk.get("percent", 0.0)
metrics_lines.append(f'rmm_disk_utilization{{client_id="{client_id}",mount="{mount}"}} {disk_percent}')
# GPU utilization and temps
for gpu in telemetry.get("gpus", []):
gpu_name = gpu.get("name", "GPU")
# Utilization percent parsing
gpu_util_str = str(gpu.get("utilization", "0"))
gpu_util = float(gpu_util_str.replace("%", "").strip())
# Temperature parsing
gpu_temp_str = str(gpu.get("temp", "0"))
gpu_temp = float(gpu_temp_str.replace("C", "").strip())
metrics_lines.append(f'rmm_gpu_utilization{{client_id="{client_id}",gpu_name="{gpu_name}"}} {gpu_util}')
metrics_lines.append(f'rmm_gpu_temperature{{client_id="{client_id}",gpu_name="{gpu_name}"}} {gpu_temp}')
return "\n".join(metrics_lines) + "\n"
if __name__ == "__main__":
print("[*] Starting FastAPI Central Server...")
# uvicorn runs the FastAPI app on port 8000
uvicorn.run("central_api_prototype:app", host="0.0.0.0", port=8000, reload=True)

100
cheatsheet.md Normal file
View File

@@ -0,0 +1,100 @@
# PROMETHEUS CHEAT SHEET
Last Updated: 2026-05-11 14:44:00
---
## Architecture Quick Reference
Central server runs:
- Prometheus → port 9090
- Grafana → port 3000
- Alertmanager → port 9093
Each remote server runs:
- Node Exporter → port 9100
---
## Bridge Reference
| Prometheus concept | MERN/Express equivalent |
|---|---|
| prometheus.yml | Express config / .env |
| scrape_interval | setInterval() |
| Node Exporter | /health endpoint |
| job_name | route group name |
| labels | MongoDB document fields |
| PromQL | MongoDB query language |
| rate() | calculating speed from counter |
| Grafana | React frontend |
| Alertmanager | nodemailer |
| systemd service | PM2 process |
| promtool check | ESLint |
| recording rule | database view |
| federation | API gateway pattern |
---
## Essential Commands
### Install sequence
wget <url> → Downloads file from internet
tar -xvf <filename> → Unzips a .tar.gz file
sudo useradd --no-create-home --shell /bin/false prometheus → Creates restricted user
sudo mkdir <folder_path> → Creates a new folder as admin
sudo chown user:group <folder_path> → Changes the owner of a folder
sudo cp <file> <destination> → Copies a file to a new location
### systemd commands
sudo systemctl daemon-reload → Reloads Linux after creating a new .service file
sudo systemctl start <service> → Turns a service ON right now
sudo systemctl enable <service> → Tells Linux to turn it on automatically after reboot
sudo systemctl status <service> → Checks if the service is running or crashed
### Validation
[Empty]
### Debugging
[Empty]
---
## prometheus.yml Skeleton
```yaml
global:
scrape_interval: 15s
scrape_configs:
- job_name: "example_job"
static_configs:
- targets: ["IP:PORT"]
```
## alertmanager.yml Skeleton
[Empty]
## Alert Rule Skeleton
[Empty]
## Node Exporter systemd service
[Empty]
---
## PromQL Quick Reference
### My 7 servers queries
[Empty]
### Aggregation
[Empty]
---
## Alert Rules For 7 Servers
[Empty]
---
## Troubleshooting Decision Tree
[Empty]
---
## Common Errors and Fixes
[Empty]

228
client_agent_prototype.py Normal file
View File

@@ -0,0 +1,228 @@
import urllib.request
import json
import time
import subprocess
import psutil
CLIENT_ID = "client_1"
import os
# Using the public ngrok URL so the agent can connect from anywhere in the world!
CENTRAL_API_URL = f"https://culprit-campfire-galore.ngrok-free.dev/api/get-command?client_id={CLIENT_ID}"
CENTRAL_TELEMETRY_URL = f"https://culprit-campfire-galore.ngrok-free.dev/api/telemetry?client_id={CLIENT_ID}"
platform_name = "nt" if os.name == "nt" else "posix"
CENTRAL_COMMANDS_URL = f"https://culprit-campfire-galore.ngrok-free.dev/api/get-commands?platform={platform_name}"
# SECURITY MECHANISM: The "Whitelist"
# The agent NEVER executes arbitrary strings from the internet.
# It dynamically pulls safe command arrays from the central FastAPI server configuration.
# An in-memory cache is used as a resilient fallback if the central server goes offline.
CACHED_COMMANDS = {}
def load_allowed_commands():
global CACHED_COMMANDS
try:
req = urllib.request.Request(CENTRAL_COMMANDS_URL)
with urllib.request.urlopen(req, timeout=3) as response:
commands = json.loads(response.read().decode())
if commands:
CACHED_COMMANDS = commands
return CACHED_COMMANDS
except Exception as e:
print(f" -> [!] Failed to load commands from central API: {e}")
if CACHED_COMMANDS:
print(" -> Returning last successfully cached whitelist.")
return CACHED_COMMANDS
# Return fallback safe command in case the server is offline on startup
return {"whoami": ["whoami"]}
ROCKET_CHAT_WEBHOOK = "https://text.seekright.com/hooks/6a0eb41ea88367bc6e101f0c/gvXfD8zSNRti43kEabqdE9tMCvNi9zAAoGfbao7cxcKbiW6R"
def send_rocket_chat_notification(title, message, color="#764FA5"):
try:
if not message:
message = "No output provided."
payload = {
"alias": f"Agent: {CLIENT_ID}",
"text": "Execution Report",
"attachments": [{
"title": title,
"text": message,
"color": color
}]
}
data = json.dumps(payload).encode('utf-8')
req = urllib.request.Request(ROCKET_CHAT_WEBHOOK, data=data, headers={'Content-Type': 'application/json'}, method='POST')
urllib.request.urlopen(req, timeout=3)
except Exception as e:
print(f" -> [!] Failed to send webhook: {e}")
def collect_and_send_telemetry():
try:
# 1. Gather comprehensive GPU stats
gpus = []
try:
# Query multiple specific fields (including detailed power and fan speed metrics)
result = subprocess.run(
["nvidia-smi", "--query-gpu=name,utilization.gpu,temperature.gpu,memory.used,memory.total,power.draw,power.limit,fan.speed", "--format=csv,noheader"],
capture_output=True, text=True
)
if result.returncode == 0:
for line in result.stdout.strip().split('\n'):
if line:
parts = [p.strip() for p in line.split(',')]
if len(parts) >= 8:
gpus.append({
"name": parts[0],
"utilization": parts[1],
"temp": parts[2] + "C",
"memory_used": parts[3],
"memory_total": parts[4],
"power_draw": parts[5],
"power_limit": parts[6],
"fan_speed": parts[7]
})
except FileNotFoundError:
pass # nvidia-smi not installed, ignore
# 2. Gather ALL Disk Partitions (Filtering snaps, loops, and virtual mounts)
disks = []
for part in psutil.disk_partitions(all=False):
# Ignore loop devices, snaps, system, and WSL virtual mountpoints
if 'loop' in part.device or 'loop' in part.fstype or part.fstype == '':
continue
if part.mountpoint.startswith('/snap') or part.fstype == 'squashfs':
continue
if part.mountpoint.startswith('/boot') or part.mountpoint.startswith('/mnt/wsl'):
continue
try:
usage = psutil.disk_usage(part.mountpoint)
disks.append({
"mount": part.mountpoint,
"total_gb": round(usage.total / (1024**3), 2),
"free_gb": round(usage.free / (1024**3), 2),
"percent": usage.percent
})
except PermissionError:
continue
# 3. Assemble Payload
payload = {
"cpu_percent": psutil.cpu_percent(interval=0.5),
"memory_percent": psutil.virtual_memory().percent,
"memory_total_gb": round(psutil.virtual_memory().total / (1024**3), 2),
"memory_free_gb": round(psutil.virtual_memory().available / (1024**3), 2),
"disks": disks,
"gpus": gpus
}
data = json.dumps(payload).encode('utf-8')
req = urllib.request.Request(CENTRAL_TELEMETRY_URL, data=data, headers={'Content-Type': 'application/json'}, method='POST')
urllib.request.urlopen(req, timeout=3)
print("[*] Telemetry pushed successfully.")
except Exception as e:
print(f"[!] Failed to push telemetry: {e}")
def send_command_result_to_server(command: str, returncode: int, stdout: str, stderr: str):
try:
base_url = CENTRAL_API_URL.split("/api/")[0]
url = f"{base_url}/api/command-result?client_id={CLIENT_ID}"
payload = {
"command": command,
"returncode": returncode,
"stdout": stdout,
"stderr": stderr
}
data = json.dumps(payload).encode('utf-8')
req = urllib.request.Request(
url,
data=data,
headers={'Content-Type': 'application/json'},
method='POST'
)
with urllib.request.urlopen(req, timeout=3) as response:
pass
print(" -> Sent command execution output to Central Server.")
except Exception as e:
print(f" -> [!] Failed to send command result to server: {e}")
def poll_server():
try:
print(f"[*] Checking for commands...")
# 1. Fetch data from Central API
req = urllib.request.Request(CENTRAL_API_URL)
with urllib.request.urlopen(req, timeout=3) as response:
data = json.loads(response.read().decode())
command_key = data.get("command", "none")
# 2. Process the command
if command_key == "none":
print(" -> No pending commands.")
else:
# Load whitelisted commands dynamically from commands.json
allowed_commands = load_allowed_commands()
if command_key in allowed_commands:
print(f" -> Executing approved command: '{command_key}'")
safe_cmd_list = allowed_commands[command_key]
# Special check: If rebooting or restarting agent, send pre-execution notification
# to ensure the network buffer has time to transmit the status before connection drops!
is_shutdown_or_restart = any(x in command_key.lower() for x in ["reboot", "restart", "kill_python"])
if is_shutdown_or_restart:
print(" -> Shutdown/restart command detected. Sending pre-execution success notification...")
send_rocket_chat_notification(
f"🔄 Executing: {command_key}",
f"System/agent is performing a scheduled action: {' '.join(safe_cmd_list)}\nConnection may drop temporarily.",
"#FFA500"
)
time.sleep(3) # Give 3 seconds for the HTTP payload to hit the Rocket.Chat server!
try:
# Execute securely. capture_output prevents it from hanging.
result = subprocess.run(safe_cmd_list, capture_output=True, text=True)
output = result.stdout.strip()
error_output = result.stderr.strip() if result.stderr else ""
# Post execution results to Central Server
send_command_result_to_server(command_key, result.returncode, output, error_output)
if not is_shutdown_or_restart:
if result.returncode == 0:
print(f" -> Output: {output}")
send_rocket_chat_notification(f"✅ Success: {command_key}", output, "#00FF00")
else:
print(f" -> Error: {error_output}")
send_rocket_chat_notification(f"❌ Failed: {command_key}", error_output, "#FF0000")
except Exception as ex:
print(f" -> [!] Execution Error: {ex}")
send_command_result_to_server(command_key, -1, "", str(ex))
if not is_shutdown_or_restart:
send_rocket_chat_notification(f"❌ Execution Error: {command_key}", str(ex), "#FF0000")
else:
# If a hacker injects "rm -rf /" into the database, it hits this block and fails securely.
warning_msg = f"Server requested unknown/malicious command: '{command_key}'. Ignored."
print(f" -> [!] SECURITY WARNING: {warning_msg}")
send_rocket_chat_notification("⚠️ SECURITY WARNING", warning_msg, "#FFA500")
except Exception as e:
print(f" -> [!] Failed to connect to central server: {e}")
if __name__ == "__main__":
print(f"Starting Agent for {CLIENT_ID}...")
print("[*] Running in continuous background polling mode (every 60 seconds).")
print("[*] Press Ctrl+C to stop the agent.")
try:
while True:
collect_and_send_telemetry()
poll_server()
time.sleep(10) # Poll every 10 seconds
except KeyboardInterrupt:
print("\n[*] Agent stopped by user.")

61
clients.json Normal file
View File

@@ -0,0 +1,61 @@
{
"client_1": {
"pending_command": "none",
"last_seen": null
},
"client_2": {
"pending_command": "none",
"last_seen": "2026-05-28T10:28:07.688334"
},
"client_3": {
"pending_command": "none",
"last_seen": "2026-06-01T11:59:54.713609",
"telemetry": {
"cpu_percent": 0.9,
"memory_percent": 10.9,
"memory_total_gb": 125.66,
"memory_free_gb": 112.01,
"disks": [
{
"mount": "/",
"total_gb": 453.44,
"free_gb": 22.8,
"percent": 94.7
}
],
"gpus": [
{
"name": "NVIDIA GeForce RTX 2080 Ti",
"utilization": "0 %",
"temp": "38C",
"memory_used": "295 MiB",
"memory_total": "11264 MiB",
"power_draw": "9.49 W",
"power_limit": "300.00 W",
"fan_speed": "0 %"
}
]
},
"active": true
},
"kawsik": {
"pending_command": "none",
"last_seen": "2026-06-01T11:59:52.394837",
"active": true,
"telemetry": {
"cpu_percent": 0.1,
"memory_percent": 7.1,
"memory_total_gb": 15.54,
"memory_free_gb": 14.43,
"disks": [
{
"mount": "/",
"total_gb": 1006.85,
"free_gb": 952.71,
"percent": 0.3
}
],
"gpus": []
}
}
}

52
commands.json Normal file
View File

@@ -0,0 +1,52 @@
{
"nt": {
"whoami": ["whoami"],
"ls": ["cmd.exe", "/c", "dir"],
"echo": ["cmd.exe", "/c", "echo", "The prototype executed the echo command successfully on Windows!"],
"reboot": ["cmd.exe", "/c", "echo", "SIMULATED WINDOWS REBOOT"],
"systeminfo": ["systeminfo"],
"ipconfig": ["ipconfig"],
"date": ["cmd.exe", "/c", "date", "/t"],
"check_space": ["powershell.exe", "-Command", "Get-PSDrive -PSProvider FileSystem"]
},
"posix": {
"whoami": ["whoami"],
"ls": ["ls", "-la"],
"echo": ["echo", "The prototype executed the echo command successfully on POSIX/WSL!"],
"pwd": ["pwd"],
"date": ["date"],
"time": ["date", "+%T"],
"systeminfo": ["uname", "-a"],
"ipconfig": ["ip", "addr"],
"check_space": ["df", "-h"],
"memory": ["free", "-h"],
"ps": ["ps", "aux"],
"kill_python": ["pkill", "-9", "python"],
"ping_google": ["ping", "-c", "4", "google.com"],
"env": ["env"],
"hostname": ["hostname"],
"reboot": ["sudo", "reboot"],
"restart_mysql": ["sudo", "systemctl", "restart", "mysql"],
"mysql_status": ["sudo", "systemctl", "status", "mysql"],
"repair_mysql": [
"sudo",
"mysqlcheck",
"--defaults-file=/root/.my.cnf",
"--auto-repair",
"--optimize",
"Test_seekright",
"tbl_site_anomaly"
],
"repair_mysql_db": [
"sudo",
"mysqlcheck",
"--defaults-file=/root/.my.cnf",
"--auto-repair",
"--optimize",
"Test_seekright"
],
"storage_devices": ["lsblk"],
"mounted_disks": ["df", "-hT"],
"netstat": ["ss", "-tulnp"]
}
}

288
deploy_agent.sh Normal file
View File

@@ -0,0 +1,288 @@
#!/usr/bin/env bash
# Exit immediately if any command fails
set -e
# Make sure the script is running with superuser privileges
if [ "$EUID" -ne 0 ]; then
echo "Error: This deployment script must be run as root (sudo)." >&2
exit 1
fi
# ====================================================
# ⚙️ CONFIGURATION: Set your Client ID here!
# Leave this blank ("") to let the script automatically use
# the local hostname of the machine.
# ====================================================
CLIENT_ID=""
# Check if set above; if not, check command line arguments, otherwise default to hostname
if [ -z "$CLIENT_ID" ]; then
CLIENT_ID="${1:-$(hostname)}"
fi
echo "🚀 Preparing SeekRight RMM Agent installation..."
echo "📍 Target Client ID: $CLIENT_ID"
# 1. Install System Dependencies
echo "🔄 Updating system package indexes and installing python dependencies..."
apt-get update -y
apt-get install -y python3 python3-pip python3-psutil
# 2. Create Destination Directories
echo "📁 Creating installation workspace..."
mkdir -p /opt/seekright-agent
chmod 755 /opt/seekright-agent
# 3. Dynamically Generate Python Agent File
echo "📝 Generating client agent file..."
cat << 'EOF' > /opt/seekright-agent/client_agent_prototype.py
import urllib.request
import json
import time
import subprocess
import psutil
import os
CLIENT_ID = "TEMPLATE_CLIENT_ID"
CENTRAL_API_URL = f"https://culprit-campfire-galore.ngrok-free.dev/api/get-command?client_id={CLIENT_ID}"
CENTRAL_TELEMETRY_URL = f"https://culprit-campfire-galore.ngrok-free.dev/api/telemetry?client_id={CLIENT_ID}"
platform_name = "nt" if os.name == "nt" else "posix"
CENTRAL_COMMANDS_URL = f"https://culprit-campfire-galore.ngrok-free.dev/api/get-commands?platform={platform_name}"
CACHED_COMMANDS = {}
def load_allowed_commands():
global CACHED_COMMANDS
try:
req = urllib.request.Request(CENTRAL_COMMANDS_URL)
with urllib.request.urlopen(req, timeout=3) as response:
commands = json.loads(response.read().decode())
if commands:
CACHED_COMMANDS = commands
return CACHED_COMMANDS
except Exception as e:
print(f" -> [!] Failed to load commands from central API: {e}")
if CACHED_COMMANDS:
print(" -> Returning last successfully cached whitelist.")
return CACHED_COMMANDS
return {"whoami": ["whoami"]}
ROCKET_CHAT_WEBHOOK = "https://text.seekright.com/hooks/6a0eb41ea88367bc6e101f0c/gvXfD8zSNRti43kEabqdE9tMCvNi9zAAoGfbao7cxcKbiW6R"
def send_rocket_chat_notification(title, message, color="#764FA5"):
try:
if not message:
message = "No output provided."
payload = {
"alias": f"Agent: {CLIENT_ID}",
"text": "Execution Report",
"attachments": [{
"title": title,
"text": message,
"color": color
}]
}
data = json.dumps(payload).encode('utf-8')
req = urllib.request.Request(ROCKET_CHAT_WEBHOOK, data=data, headers={'Content-Type': 'application/json'}, method='POST')
urllib.request.urlopen(req, timeout=3)
except Exception as e:
print(f" -> [!] Failed to send webhook: {e}")
def collect_and_send_telemetry():
try:
gpus = []
try:
result = subprocess.run(
["nvidia-smi", "--query-gpu=name,utilization.gpu,temperature.gpu,memory.used,memory.total,power.draw,power.limit,fan.speed", "--format=csv,noheader"],
capture_output=True, text=True
)
if result.returncode == 0:
for line in result.stdout.strip().split('\n'):
if line:
parts = [p.strip() for p in line.split(',')]
if len(parts) >= 8:
gpus.append({
"name": parts[0],
"utilization": parts[1],
"temp": parts[2] + "C",
"memory_used": parts[3],
"memory_total": parts[4],
"power_draw": parts[5],
"power_limit": parts[6],
"fan_speed": parts[7]
})
except FileNotFoundError:
pass
disks = []
for part in psutil.disk_partitions(all=False):
if 'loop' in part.device or 'loop' in part.fstype or part.fstype == '':
continue
if part.mountpoint.startswith('/snap') or part.fstype == 'squashfs':
continue
if part.mountpoint.startswith('/boot') or part.mountpoint.startswith('/mnt/wsl'):
continue
try:
usage = psutil.disk_usage(part.mountpoint)
disks.append({
"mount": part.mountpoint,
"total_gb": round(usage.total / (1024**3), 2),
"free_gb": round(usage.free / (1024**3), 2),
"percent": usage.percent
})
except PermissionError:
continue
payload = {
"cpu_percent": psutil.cpu_percent(interval=0.5),
"memory_percent": psutil.virtual_memory().percent,
"memory_total_gb": round(psutil.virtual_memory().total / (1024**3), 2),
"memory_free_gb": round(psutil.virtual_memory().available / (1024**3), 2),
"disks": disks,
"gpus": gpus
}
data = json.dumps(payload).encode('utf-8')
req = urllib.request.Request(CENTRAL_TELEMETRY_URL, data=data, headers={'Content-Type': 'application/json'}, method='POST')
urllib.request.urlopen(req, timeout=3)
print("[*] Telemetry pushed successfully.")
except Exception as e:
print(f"[!] Failed to push telemetry: {e}")
def send_command_result_to_server(command: str, returncode: int, stdout: str, stderr: str):
try:
base_url = CENTRAL_API_URL.split("/api/")[0]
url = f"{base_url}/api/command-result?client_id={CLIENT_ID}"
payload = {
"command": command,
"returncode": returncode,
"stdout": stdout,
"stderr": stderr
}
data = json.dumps(payload).encode('utf-8')
req = urllib.request.Request(
url,
data=data,
headers={'Content-Type': 'application/json'},
method='POST'
)
with urllib.request.urlopen(req, timeout=3) as response:
pass
print(" -> Sent command execution output to Central Server.")
except Exception as e:
print(f" -> [!] Failed to send command result to server: {e}")
def poll_server():
try:
print(f"[*] Checking for commands...")
req = urllib.request.Request(CENTRAL_API_URL)
with urllib.request.urlopen(req, timeout=3) as response:
data = json.loads(response.read().decode())
command_key = data.get("command", "none")
if command_key == "none":
print(" -> No pending commands.")
else:
allowed_commands = load_allowed_commands()
if command_key in allowed_commands:
print(f" -> Executing approved command: '{command_key}'")
safe_cmd_list = allowed_commands[command_key]
is_shutdown_or_restart = any(x in command_key.lower() for x in ["reboot", "restart", "kill_python"])
if is_shutdown_or_restart:
print(" -> Shutdown/restart command detected. Sending pre-execution success notification...")
send_rocket_chat_notification(
f"🔄 Executing: {command_key}",
f"System/agent is performing a scheduled action: {' '.join(safe_cmd_list)}\nConnection may drop temporarily.",
"#FFA500"
)
time.sleep(3)
try:
result = subprocess.run(safe_cmd_list, capture_output=True, text=True)
output = result.stdout.strip()
error_output = result.stderr.strip() if result.stderr else ""
# Post execution results to Central Server
send_command_result_to_server(command_key, result.returncode, output, error_output)
if not is_shutdown_or_restart:
if result.returncode == 0:
print(f" -> Output: {output}")
send_rocket_chat_notification(f"✅ Success: {command_key}", output, "#00FF00")
else:
print(f" -> Error: {error_output}")
send_rocket_chat_notification(f"❌ Failed: {command_key}", error_output, "#FF0000")
except Exception as ex:
print(f" -> [!] Execution Error: {ex}")
send_command_result_to_server(command_key, -1, "", str(ex))
if not is_shutdown_or_restart:
send_rocket_chat_notification(f"❌ Execution Error: {command_key}", str(ex), "#FF0000")
else:
warning_msg = f"Server requested unknown/malicious command: '{command_key}'. Ignored."
print(f" -> [!] SECURITY WARNING: {warning_msg}")
send_rocket_chat_notification("⚠️ SECURITY WARNING", warning_msg, "#FFA500")
except Exception as e:
print(f" -> [!] Failed to connect to central server: {e}")
if __name__ == "__main__":
print(f"Starting Agent for {CLIENT_ID}...")
try:
while True:
collect_and_send_telemetry()
poll_server()
time.sleep(10)
except KeyboardInterrupt:
print("\n[*] Agent stopped.")
EOF
# Inject the Client ID dynamically
sed -i "s/TEMPLATE_CLIENT_ID/$CLIENT_ID/g" /opt/seekright-agent/client_agent_prototype.py
chmod 644 /opt/seekright-agent/client_agent_prototype.py
# 4. Create Background systemd Service
echo "⚙️ Configuring background systemd daemon service..."
cat << 'EOF' > /etc/systemd/system/seekright-agent.service
[Unit]
Description=SeekRight RMM Background Agent
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
# -u enables unbuffered stdout output for direct real-time logging via journalctl
ExecStart=/usr/bin/python3 -u /opt/seekright-agent/client_agent_prototype.py
WorkingDirectory=/opt/seekright-agent
Restart=always
RestartSec=5
User=root
Group=root
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=seekright-agent
[Install]
WantedBy=multi-user.target
EOF
chmod 644 /etc/systemd/system/seekright-agent.service
# 5. Enable and Launch Service
echo "🔄 Reloading system daemons and starting agent service..."
systemctl daemon-reload
systemctl enable seekright-agent.service
systemctl restart seekright-agent.service
echo ""
echo "✨ ============================================= ✨"
echo "✅ SeekRight RMM Agent successfully deployed!"
echo "📍 Location: /opt/seekright-agent/client_agent_prototype.py"
echo "📋 System Service: seekright-agent.service"
echo "✨ ============================================= ✨"
echo ""
echo "📈 Displaying real-time execution logs (Press Ctrl+C to exit log view):"
echo "--------------------------------------------------------"
journalctl -u seekright-agent.service -n 15 -f

80
deploy_exporters.sh Normal file
View File

@@ -0,0 +1,80 @@
#!/bin/bash
set -e
WEBHOOK_URL="https://text.seekright.com/hooks/6a0eb41ea88367bc6e101f0c/gvXfD8zSNRti43kEabqdE9tMCvNi9zAAoGfbao7cxcKbiW6R"
HOSTNAME=$(hostname)
notify() {
curl -s -X POST "$WEBHOOK_URL" \
-H 'Content-Type: application/json' \
-d "{\"text\": \"$1\"}" > /dev/null 2>&1
}
trap 'notify "❌ *FAILED* on *${HOSTNAME}* — script crashed at line $LINENO. Check the machine manually."' ERR
echo "Starting Automated Exporter Installation..."
if [ "$EUID" -ne 0 ]; then echo "❌ Error: Please run this script with sudo."; exit 1; fi
NODE_VERSION="1.7.0"
GPU_VERSION="1.4.1"
echo "======================================"
echo "1. Installing Node Exporter v${NODE_VERSION}"
echo "======================================"
cd /tmp
wget -q https://github.com/prometheus/node_exporter/releases/download/v${NODE_VERSION}/node_exporter-${NODE_VERSION}.linux-amd64.tar.gz
tar -xf node_exporter-${NODE_VERSION}.linux-amd64.tar.gz
if ! id "node_exporter" &>/dev/null; then useradd --no-create-home --shell /bin/false node_exporter; fi
cp node_exporter-${NODE_VERSION}.linux-amd64/node_exporter /usr/local/bin/
chown node_exporter:node_exporter /usr/local/bin/node_exporter
cat << 'EOF' > /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now node_exporter
echo "✅ Node Exporter installed and running on port 9100!"
echo "======================================"
echo "2. Checking for Nvidia GPU..."
echo "======================================"
GPU_STATUS="No GPU detected"
if command -v nvidia-smi &> /dev/null; then
echo "Nvidia GPU detected! Installing Nvidia GPU Exporter v${GPU_VERSION}..."
wget -q https://github.com/utkuozdemir/nvidia_gpu_exporter/releases/download/v${GPU_VERSION}/nvidia_gpu_exporter_${GPU_VERSION}_linux_x86_64.tar.gz
tar -xf nvidia_gpu_exporter_${GPU_VERSION}_linux_x86_64.tar.gz
if ! id "nvidia_exporter" &>/dev/null; then useradd --no-create-home --shell /bin/false nvidia_exporter; fi
cp nvidia_gpu_exporter /usr/local/bin/
chown nvidia_exporter:nvidia_exporter /usr/local/bin/nvidia_gpu_exporter
cat << 'EOF' > /etc/systemd/system/nvidia_gpu_exporter.service
[Unit]
Description=Nvidia GPU Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=nvidia_exporter
Group=nvidia_exporter
Type=simple
ExecStart=/usr/local/bin/nvidia_gpu_exporter
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now nvidia_gpu_exporter
GPU_STATUS="✅ GPU Exporter running on port 9835"
echo "✅ Nvidia GPU Exporter installed and running on port 9835!"
else
echo "No Nvidia GPU detected. Skipping GPU Exporter installation."
fi
echo "======================================"
echo "🎉 Installation Complete!"
echo "Node Exporter is listening on port 9100"
echo "======================================"
notify "✅ *SUCCESS* on *${HOSTNAME}* | Node Exporter: :9100 | GPU Exporter: ${GPU_STATUS}"

629
instructions.md Normal file
View File

@@ -0,0 +1,629 @@
# PROMETHEUS SETUP INSTRUCTIONS FROM SCRATCH
_Follow these instructions sequentially to build a production-grade Prometheus and Node Exporter setup from scratch._
---
## PART 1: INSTALLING THE PROMETHEUS CENTRAL SERVER
**1. Download Prometheus**
```bash
wget https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz
```
**2. Extract Prometheus**
```bash
tar -xvf prometheus-2.51.2.linux-amd64.tar.gz
```
**3. Create a restricted system user for security**
```bash
sudo useradd --no-create-home --shell /bin/false prometheus
```
**4. Create the Configuration and Data folders**
```bash
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
```
**5. Hand ownership of those folders to the new user**
```bash
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
```
**6. Copy the main engine and spell-checker to the secure Linux bin folder**
```bash
sudo cp prometheus-2.51.2.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.51.2.linux-amd64/promtool /usr/local/bin/
```
**7. Hand ownership of the executables to the new user**
```bash
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
```
**8. Create the main configuration file**
```bash
sudo nano /etc/prometheus/prometheus.yml
```
_Paste this exact code into the file and save:_
```yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus_self_monitor"
static_configs:
- targets: ["localhost:9090"]
- job_name: "laptop_server_1"
static_configs:
- targets: ["localhost:9100"]
```
_Verify the YAML syntax is perfect so you don't crash the server:_
```bash
promtool check config /etc/prometheus/prometheus.yml
```
**9. Create the background service file**
```bash
sudo nano /etc/systemd/system/prometheus.service
```
_Paste this exact code into the file and save:_
```ini
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/
[Install]
WantedBy=multi-user.target
```
**10. Turn Prometheus on forever**
```bash
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
```
---
## PART 2: INSTALLING NODE EXPORTER (For hardware data)
**1. Download Node Exporter**
```bash
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
```
**2. Extract Node Exporter**
```bash
tar -xvf node_exporter-1.7.0.linux-amd64.tar.gz
```
**3. Create a restricted system user for security**
```bash
sudo useradd --no-create-home --shell /bin/false node_exporter
```
**4. Copy the main engine to the secure Linux bin folder**
```bash
sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
```
**5. Hand ownership of the executable to the new user**
```bash
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
```
**6. Create the background service file**
```bash
sudo nano /etc/systemd/system/node_exporter.service
```
_Paste this exact code into the file and save:_
```ini
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
```
**7. Turn Node Exporter on forever**
```bash
sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
```
_(You can verify everything is working by visiting `http://localhost:9090/targets` in your browser. Both targets should say UP)._
---
## PART 2.5: INSTALLING NVIDIA GPU EXPORTER (Only for servers with GPUs)
_Note: The server must already have Nvidia drivers installed so the `nvidia-smi` command works in the terminal._
**1. Download the Exporter**
```bash
wget https://github.com/utkuozdemir/nvidia_gpu_exporter/releases/download/v1.4.1/nvidia_gpu_exporter_1.4.1_linux_x86_64.tar.gz
```
**2. Extract it**
```bash
tar -xvf nvidia_gpu_exporter_1.4.1_linux_x86_64.tar.gz
```
**3. Create a restricted system user**
```bash
sudo useradd --no-create-home --shell /bin/false nvidia_exporter
```
**4. Copy the engine to the secure bin folder**
```bash
sudo cp nvidia_gpu_exporter /usr/local/bin/
```
**5. Hand ownership to the new user**
```bash
sudo chown nvidia_exporter:nvidia_exporter /usr/local/bin/nvidia_gpu_exporter
```
**6. Create the background service file**
```bash
sudo nano /etc/systemd/system/nvidia_gpu_exporter.service
```
_Paste this exact code into the file and save:_
```ini
[Unit]
Description=Nvidia GPU Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=nvidia_exporter
Group=nvidia_exporter
Type=simple
ExecStart=/usr/local/bin/nvidia_gpu_exporter
[Install]
WantedBy=multi-user.target
```
**7. Turn it on forever**
```bash
sudo systemctl daemon-reload
sudo systemctl start nvidia_gpu_exporter
sudo systemctl enable nvidia_gpu_exporter
```
_(Once running, the GPU Exporter listens on port `9835`. You must go back to the Central Server and add a new job in `prometheus.yml` targeting `[SERVER_IP]:9835`)._
---
## PART 3: ESSENTIAL PROMQL QUERIES
_Use these queries in the Prometheus search bar or Grafana to monitor hardware._
**1. RAM: Percentage Available**
```promql
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
```
**2. CPU: Percentage Actively Used**
_(Node Exporter tracks how long the CPU is "idle". We subtract the idle speed from 100% to get the active usage)._
```promql
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
```
**3. DISK (SSD/HDD): Percentage Free Space (Root Drive)**
```promql
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100
```
**4. NETWORK: Total Upload Speed in Megabytes/second (MB/s)**
```promql
rate(node_network_transmit_bytes_total[5m]) / 1024 / 1024
```
**5. HARDWARE: Server Uptime in Days**
```promql
(time() - node_boot_time_seconds) / 86400
```
**6. SERVERS: Is a server down? (1 = UP, 0 = DOWN)**
```promql
up
```
---
## PART 4: INSTALLING GRAFANA (THE DASHBOARD UI)
_Grafana runs on port 3000 by default. Use this to visualize Prometheus data._
**1. Download the Grafana security key:**
```bash
sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
```
**2. Add the official Grafana repository to Linux:**
```bash
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
```
**3. Install Grafana and start the service:**
```bash
sudo apt-get update
sudo apt-get install grafana
sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
```
---
## PART 5: GRAFANA DASHBOARD TEMPLATE IDs & DEBUGGING
When importing templates from `grafana.com/dashboards`, use these exact IDs:
- **Node Exporter Full (CPU/RAM/Network):** `1860`
- **Nvidia GPU Exporter (Temperature/VRAM):** `14574`
### How to Fix "Blank" or "No Data" Dashboards
1. **Variables Mismatch:** Look at the dropdown menus at the very top of the dashboard. If they say "N/A", click them and select your specific `job_name` (e.g. `sasi_gpu` instead of `nvidia_gpu_exporter`).
2. **PromQL Mismatch:** If the graph is still blank, click the `...` next to the graph title and hit **Edit**. Look at the raw PromQL code. Sometimes community dashboards hardcode filters like `{job="gpu"}`. Simply delete that strict filter so your specific data can flow through!
---
### 1. Browser Blocks HTTP Exporters
If your exporter is running (e.g., `curl http://localhost:9100/metrics` works in the terminal) but the browser says "Connection Refused", make sure your browser didn't automatically force `https://`. Exporters only run on plain `http://` by default!
### 2. Nvidia GPU Exporter Panic `[ms]` Error
If you use an outdated version of `nvidia_gpu_exporter` (like `v1.2.1`), the engine will instantly crash with a `panic: descriptor Desc{...}` error because newer Nvidia drivers output a unit called `[ms]`. Prometheus strictly forbids spaces and brackets in metric names!
**The Fix:** Always use `v1.4.1` or newer, as the developer hardcoded a fix for this exact regex bug.
_(Note: Node Exporter does NOT monitor GPUs. To monitor GPU temperatures and VRAM, you must install a separate exporter called `nvidia_smi_exporter`)._
---
## PART 6: INSTALLING ALERTMANAGER (For Slack/Email Notifications)
**1. Download and Extract Alertmanager**
```bash
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar -xvf alertmanager-0.27.0.linux-amd64.tar.gz
```
**2. Create a restricted user and folders**
```bash
sudo useradd --no-create-home --shell /bin/false alertmanager
sudo mkdir /etc/alertmanager
sudo mkdir /var/lib/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager
```
**3. Move the binary and hand over ownership**
```bash
sudo cp alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
```
**4. Create the Configuration File**
```bash
sudo nano /etc/alertmanager/alertmanager.yml
```
_Paste this code to route alerts to both Rocket.Chat and Gmail simultaneously:_
```yaml
global:
resolve_timeout: 5m
smtp_smarthost: "smtp.gmail.com:587"
smtp_from: "knmkaushik@gmail.com"
smtp_auth_username: "knmkaushik@gmail.com"
smtp_auth_password: "YOUR_APP_PASSWORD"
route:
receiver: "fallback-do-nothing"
group_wait: 10s
group_interval: 1m
repeat_interval: 5m
routes:
- receiver: "rocket-chat-alerts"
continue: true
- receiver: "email-alerts"
continue: true
receivers:
- name: "fallback-do-nothing"
- name: "rocket-chat-alerts"
webhook_configs:
- url: "YOUR_ROCKETCHAT_WEBHOOK_URL"
send_resolved: true
- name: "email-alerts"
email_configs:
- to: "kawsik97@gmail.com"
send_resolved: true
```
**5. Hand over ownership of the config file**
```bash
sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml
```
**6. Create the background service file**
```bash
sudo nano /etc/systemd/system/alertmanager.service
```
_Paste this exact code into the file and save:_
```ini
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager
[Install]
WantedBy=multi-user.target
```
**7. Turn Alertmanager on forever**
```bash
sudo systemctl daemon-reload
sudo systemctl start alertmanager
sudo systemctl enable alertmanager
```
**8. Configure Rocket.Chat Webhook Script**
Rocket.Chat natively only understands `{"text": "..."}` payloads, but Alertmanager sends structured JSON. To fix this, you must add a transformation script in the Rocket.Chat admin panel:
1. Go to your Rocket.Chat Admin Panel -> **Integrations** -> **Incoming**.
2. Create or Edit the incoming webhook for your channel (e.g., `#devops-alerts`).
3. Toggle **Script Enabled** to **ON**.
4. Paste the following into the Script box:
```javascript
class Script {
process_incoming_request({ request }) {
var content = request.content;
var status = content.status ? content.status.toUpperCase() : "UNKNOWN";
var alertName = "Unknown";
var instance = "";
var summary = "";
if (content.alerts && content.alerts[0]) {
var a = content.alerts[0];
if (a.labels) {
alertName = a.labels.alertname || "Unknown";
instance = a.labels.instance || "";
}
if (a.annotations) {
summary = a.annotations.summary || "";
}
}
var icon = status === "FIRING" ? ":red_circle:" : ":white_check_mark:";
var msg = icon + " *" + status + "* — " + alertName;
if (instance) {
msg = msg + " (" + instance + ")";
}
if (summary) {
msg = msg + "\n" + summary;
}
return {
content: {
text: msg,
},
};
}
}
```
5. Make sure the "Script Sandbox" is set to "Secure Sandbox".
6. Click **Save** and use the generated Webhook URL in your `alertmanager.yml`.
---
## PART 7: LINKING PROMETHEUS & ALERTMANAGER
Prometheus must be told that Alertmanager exists, and where the "Rules" are stored.
**1. Open the main Prometheus config**
```bash
sudo nano /etc/prometheus/prometheus.yml
```
**2. Update the `alerting` and `rule_files` blocks at the top:**
```yaml
alerting:
alertmanagers:
- static_configs:
- targets: ["localhost:9093"]
rule_files:
- "rules.yml"
```
_(Save and exit)._
**3. Create the Rules file (Where the Alert Logic lives):**
```bash
sudo nano /etc/prometheus/rules.yml
```
**4. Paste this exact rule and save:**
_(This logic says: If any server goes offline for more than 1 minute, fire a CRITICAL alert)._
```yaml
groups:
- name: hardware_alerts
rules:
- alert: ServerDown
expr: up == 0
for: 10s
labels:
severity: critical
annotations:
summary: "Server {{ $labels.instance }} is {{ if eq $value 0.0 }}DOWN{{ else }}UP{{ end }}!"
description: "Server {{ $labels.instance }} has been unreachable for more than 10 seconds."
```
**5. Hand over ownership and restart Prometheus:**
```bash
sudo chown prometheus:prometheus /etc/prometheus/rules.yml
sudo systemctl restart prometheus
```
---
## PART 8: MANAGING & DEBUGGING SERVICES (Crash, Reboot, & Maintenance)
Since we configured all components (Prometheus, Node Exporter, Grafana, Alertmanager) as Linux `systemd` services and used the `enable` command, **they will start automatically every time you boot or reboot your server.**
If you ever experience a crash, need to reboot, or want to test if services are running, use the following commands.
### 1. Check the Status of All Services at Once
If you just rebooted your server and want to ensure everything turned on properly, run this:
```bash
sudo systemctl status prometheus node_exporter grafana-server alertmanager
```
_Tip: If the output locks your terminal and shows `(END)` at the bottom, just press the `q` key on your keyboard to quit the reader mode and return to your prompt._
### 2. Restarting a Service After a Crash or Configuration Change
If you edit a configuration file (like `prometheus.yml` or `rules.yml`), the changes won't apply until you restart the service. If a service ever crashes and you want to force it to start fresh, use `restart`:
```bash
# Restart Prometheus
sudo systemctl restart prometheus
# Restart Alertmanager
sudo systemctl restart alertmanager
# Restart Grafana
sudo systemctl restart grafana-server
```
### 3. Turning Services On / Off Manually
If you need to intentionally stop a service to perform maintenance:
```bash
sudo systemctl stop prometheus
```
To start it back up:
```bash
sudo systemctl start prometheus
```
### 4. Viewing Error Logs (Debugging)
If a service says it is "failed" or "inactive" when you check its status, you can view its exact crash logs by checking the Linux `journalctl` (the master log book).
To see the last 50 logs for Prometheus (useful for finding YAML syntax errors):
```bash
sudo journalctl -u prometheus -n 50 --no-pager
```
To stream the logs live in real-time (press `Ctrl+C` to stop):
```bash
sudo journalctl -u prometheus -f
```

58
prometheus.yml.tmp Normal file
View File

@@ -0,0 +1,58 @@
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ["localhost:9093"]
rule_files:
- "rules.yml"
scrape_configs:
- job_name: "prometheus_self_monitor"
static_configs:
- targets: ["localhost:9090"]
- job_name: "rmm_central_bridge"
static_configs:
- targets: ["192.168.176.1:8000"] # WSL Gateway to the Windows host running FastAPI
# - job_name: "laptop_server_1"
# static_configs:
# - targets: ["localhost:9100"]
#
# - job_name: "desktop_ngrok_test"
# scheme: https
# static_configs:
# - targets: ["culprit-campfire-galore.ngrok-free.dev"]
#
# - job_name: "sasi_gpu"
# scheme: https
# static_configs:
# - targets: ["varnish-gonad-snowstorm.ngrok-free.dev"]
#
# - job_name: "sasi_cpu"
# scheme: https
# static_configs:
# - targets: ["statute-earlier-plastic-log.trycloudflare.com"]
#
# - job_name: 'node_exporter'
# static_configs:
# - targets:
# - '100.119.113.98:9100' # bs13
#
# # 1. Scrape the Remote CPUs via Cloudflare
# - job_name: 'cloudflare_cpus'
# scheme: https
# static_configs:
# - targets:
# - 'jagan-cpu.seekright.com'
# - 'kaushik-cpu.seekright.com'
# # 2. Scrape the Remote GPUs via Cloudflare
# - job_name: 'cloudflare_gpus'
# scheme: https
# static_configs:
# - targets:
# - 'jagan-gpu.seekright.com'

70
resume.md Normal file
View File

@@ -0,0 +1,70 @@
# RESUME INSTRUCTIONS FOR NEW IDE
Generated: 2026-05-11 14:44:00
## What I Am Doing
Learning Prometheus to monitor 6-7 remote servers globally.
Central server: [PENDING]
Remote servers: [PENDING]
Teaching method: Antigravity Learning with MERN/Express bridges.
## My Background
MERN fullstack, FastAPI, RAG systems.
Does NOT know Docker, docker-compose, or Linux basics (SSH, navigation).
Needs step-by-step guidance for any Linux/DevOps commands.
## My Teaching Rules — Follow All Of These
1. Zero Handout Policy
One line at a time. Never full configs.
Ask what I think it does before explaining.
2. Bridge Method First
Every concept: "In your Express world X. Here Y."
Never skip the bridge.
3. Socratic Method First
Ask before explaining. Hints before answers.
4. One Blank Slate Drill Per Concept
Recreate from memory once. Pass = move on.
5. Hands On Always
Every concept must be typed and run.
6. Line By Line Accountability
What does it do, Express equivalent, what breaks, what happens to 7 servers.
7. Concept Fingerprint Card
After every concept.
8. Real Servers Always
Connect every concept to the 7 remote servers.
9. Troubleshooting Mindset
What is most likely to go wrong on real servers.
10. End of Layer Revision Task — 5 parts.
11. Simple Language.
12. No Moving Forward Rule.
## Where I Am Right Now
Layer 4 — PromQL
## What To Do Right Now
Mastering Instant Vectors and Labels in PromQL.
## Setup Status
0 servers connected.
## Errors Currently Blocking
None.
## How To Start
Read cheatsheet.md for what I already know.
Read status.md for exact position and setup status.
Read tasks.md for what to do this session.
Greet me, tell me how many of my 7 servers are connected,
confirm current layer in one sentence,
continue from exactly where I stopped.

24
rmm-ui/.gitignore vendored Normal file
View File

@@ -0,0 +1,24 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
*.local
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?

16
rmm-ui/README.md Normal file
View File

@@ -0,0 +1,16 @@
# React + Vite
This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.
Currently, two official plugins are available:
- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Oxc](https://oxc.rs)
- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/)
## React Compiler
The React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation).
## Expanding the ESLint configuration
If you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project.

21
rmm-ui/eslint.config.js Normal file
View File

@@ -0,0 +1,21 @@
import js from '@eslint/js'
import globals from 'globals'
import reactHooks from 'eslint-plugin-react-hooks'
import reactRefresh from 'eslint-plugin-react-refresh'
import { defineConfig, globalIgnores } from 'eslint/config'
export default defineConfig([
globalIgnores(['dist']),
{
files: ['**/*.{js,jsx}'],
extends: [
js.configs.recommended,
reactHooks.configs.flat.recommended,
reactRefresh.configs.vite,
],
languageOptions: {
globals: globals.browser,
parserOptions: { ecmaFeatures: { jsx: true } },
},
},
])

13
rmm-ui/index.html Normal file
View File

@@ -0,0 +1,13 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>rmm-ui</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.jsx"></script>
</body>
</html>

3324
rmm-ui/package-lock.json generated Normal file
View File

File diff suppressed because it is too large Load Diff

31
rmm-ui/package.json Normal file
View File

@@ -0,0 +1,31 @@
{
"name": "rmm-ui",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"lint": "eslint .",
"preview": "vite preview"
},
"dependencies": {
"lucide-react": "^1.16.0",
"react": "^19.2.6",
"react-dom": "^19.2.6"
},
"devDependencies": {
"@eslint/js": "^10.0.1",
"@types/react": "^19.2.14",
"@types/react-dom": "^19.2.3",
"@vitejs/plugin-react": "^6.0.1",
"autoprefixer": "^10.5.0",
"eslint": "^10.3.0",
"eslint-plugin-react-hooks": "^7.1.1",
"eslint-plugin-react-refresh": "^0.5.2",
"globals": "^17.6.0",
"postcss": "^8.5.15",
"tailwindcss": "^3.4.19",
"vite": "^8.0.12"
}
}

6
rmm-ui/postcss.config.js Normal file
View File

@@ -0,0 +1,6 @@
export default {
plugins: {
tailwindcss: {},
autoprefixer: {},
},
}

1
rmm-ui/public/favicon.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 9.3 KiB

24
rmm-ui/public/icons.svg Normal file
View File

@@ -0,0 +1,24 @@
<svg xmlns="http://www.w3.org/2000/svg">
<symbol id="bluesky-icon" viewBox="0 0 16 17">
<g clip-path="url(#bluesky-clip)"><path fill="#08060d" d="M7.75 7.735c-.693-1.348-2.58-3.86-4.334-5.097-1.68-1.187-2.32-.981-2.74-.79C.188 2.065.1 2.812.1 3.251s.241 3.602.398 4.13c.52 1.744 2.367 2.333 4.07 2.145-2.495.37-4.71 1.278-1.805 4.512 3.196 3.309 4.38-.71 4.987-2.746.608 2.036 1.307 5.91 4.93 2.746 2.72-2.746.747-4.143-1.747-4.512 1.702.189 3.55-.4 4.07-2.145.156-.528.397-3.691.397-4.13s-.088-1.186-.575-1.406c-.42-.19-1.06-.395-2.741.79-1.755 1.24-3.64 3.752-4.334 5.099"/></g>
<defs><clipPath id="bluesky-clip"><path fill="#fff" d="M.1.85h15.3v15.3H.1z"/></clipPath></defs>
</symbol>
<symbol id="discord-icon" viewBox="0 0 20 19">
<path fill="#08060d" d="M16.224 3.768a14.5 14.5 0 0 0-3.67-1.153c-.158.286-.343.67-.47.976a13.5 13.5 0 0 0-4.067 0c-.128-.306-.317-.69-.476-.976A14.4 14.4 0 0 0 3.868 3.77C1.546 7.28.916 10.703 1.231 14.077a14.7 14.7 0 0 0 4.5 2.306q.545-.748.965-1.587a9.5 9.5 0 0 1-1.518-.74q.191-.14.372-.293c2.927 1.369 6.107 1.369 8.999 0q.183.152.372.294-.723.437-1.52.74.418.838.963 1.588a14.6 14.6 0 0 0 4.504-2.308c.37-3.911-.63-7.302-2.644-10.309m-9.13 8.234c-.878 0-1.599-.82-1.599-1.82 0-.998.705-1.82 1.6-1.82.894 0 1.614.82 1.599 1.82.001 1-.705 1.82-1.6 1.82m5.91 0c-.878 0-1.599-.82-1.599-1.82 0-.998.705-1.82 1.6-1.82.893 0 1.614.82 1.599 1.82 0 1-.706 1.82-1.6 1.82"/>
</symbol>
<symbol id="documentation-icon" viewBox="0 0 21 20">
<path fill="none" stroke="#aa3bff" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.35" d="m15.5 13.333 1.533 1.322c.645.555.967.833.967 1.178s-.322.623-.967 1.179L15.5 18.333m-3.333-5-1.534 1.322c-.644.555-.966.833-.966 1.178s.322.623.966 1.179l1.534 1.321"/>
<path fill="none" stroke="#aa3bff" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.35" d="M17.167 10.836v-4.32c0-1.41 0-2.117-.224-2.68-.359-.906-1.118-1.621-2.08-1.96-.599-.21-1.349-.21-2.848-.21-2.623 0-3.935 0-4.983.369-1.684.591-3.013 1.842-3.641 3.428C3 6.449 3 7.684 3 10.154v2.122c0 2.558 0 3.838.706 4.726q.306.383.713.671c.76.536 1.79.64 3.581.66"/>
<path fill="none" stroke="#aa3bff" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.35" d="M3 10a2.78 2.78 0 0 1 2.778-2.778c.555 0 1.209.097 1.748-.047.48-.129.854-.503.982-.982.145-.54.048-1.194.048-1.749a2.78 2.78 0 0 1 2.777-2.777"/>
</symbol>
<symbol id="github-icon" viewBox="0 0 19 19">
<path fill="#08060d" fill-rule="evenodd" d="M9.356 1.85C5.05 1.85 1.57 5.356 1.57 9.694a7.84 7.84 0 0 0 5.324 7.44c.387.079.528-.168.528-.376 0-.182-.013-.805-.013-1.454-2.165.467-2.616-.935-2.616-.935-.349-.91-.864-1.143-.864-1.143-.71-.48.051-.48.051-.48.787.051 1.2.805 1.2.805.695 1.194 1.817.857 2.268.649.064-.507.27-.857.49-1.052-1.728-.182-3.545-.857-3.545-3.87 0-.857.31-1.558.8-2.104-.078-.195-.349-1 .077-2.078 0 0 .657-.208 2.14.805a7.5 7.5 0 0 1 1.946-.26c.657 0 1.328.092 1.946.26 1.483-1.013 2.14-.805 2.14-.805.426 1.078.155 1.883.078 2.078.502.546.799 1.247.799 2.104 0 3.013-1.818 3.675-3.558 3.87.284.247.528.714.528 1.454 0 1.052-.012 1.896-.012 2.156 0 .208.142.455.528.377a7.84 7.84 0 0 0 5.324-7.441c.013-4.338-3.48-7.844-7.773-7.844" clip-rule="evenodd"/>
</symbol>
<symbol id="social-icon" viewBox="0 0 20 20">
<path fill="none" stroke="#aa3bff" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.35" d="M12.5 6.667a4.167 4.167 0 1 0-8.334 0 4.167 4.167 0 0 0 8.334 0"/>
<path fill="none" stroke="#aa3bff" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.35" d="M2.5 16.667a5.833 5.833 0 0 1 8.75-5.053m3.837.474.513 1.035c.07.144.257.282.414.309l.93.155c.596.1.736.536.307.965l-.723.73a.64.64 0 0 0-.152.531l.207.903c.164.715-.213.991-.84.618l-.872-.52a.63.63 0 0 0-.577 0l-.872.52c-.624.373-1.003.094-.84-.618l.207-.903a.64.64 0 0 0-.152-.532l-.723-.729c-.426-.43-.289-.864.306-.964l.93-.156a.64.64 0 0 0 .412-.31l.513-1.034c.28-.562.735-.562 1.012 0"/>
</symbol>
<symbol id="x-icon" viewBox="0 0 19 19">
<path fill="#08060d" fill-rule="evenodd" d="M1.893 1.98c.052.072 1.245 1.769 2.653 3.77l2.892 4.114c.183.261.333.48.333.486s-.068.089-.152.183l-.522.593-.765.867-3.597 4.087c-.375.426-.734.834-.798.905a1 1 0 0 0-.118.148c0 .01.236.017.664.017h.663l.729-.83c.4-.457.796-.906.879-.999a692 692 0 0 0 1.794-2.038c.034-.037.301-.34.594-.675l.551-.624.345-.392a7 7 0 0 1 .34-.374c.006 0 .93 1.306 2.052 2.903l2.084 2.965.045.063h2.275c1.87 0 2.273-.003 2.266-.021-.008-.02-1.098-1.572-3.894-5.547-2.013-2.862-2.28-3.246-2.273-3.266.008-.019.282-.332 2.085-2.38l2-2.274 1.567-1.782c.022-.028-.016-.03-.65-.03h-.674l-.3.342a871 871 0 0 1-1.782 2.025c-.067.075-.405.458-.75.852a100 100 0 0 1-.803.91c-.148.172-.299.344-.99 1.127-.304.343-.32.358-.345.327-.015-.019-.904-1.282-1.976-2.808L6.365 1.85H1.8zm1.782.91 8.078 11.294c.772 1.08 1.413 1.973 1.425 1.984.016.017.241.02 1.05.017l1.03-.004-2.694-3.766L7.796 5.75 5.722 2.852l-1.039-.004-1.039-.004z" clip-rule="evenodd"/>
</symbol>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 4.9 KiB

1
rmm-ui/src/App.css Normal file
View File

@@ -0,0 +1 @@
/* Cleared to avoid styling conflicts with index.css and Tailwind */

825
rmm-ui/src/App.jsx Normal file
View File

@@ -0,0 +1,825 @@
import React, { useState, useEffect, useRef } from 'react';
import {
Activity,
Cpu,
HardDrive,
Terminal,
Settings,
RefreshCw,
CheckCircle,
XCircle,
Zap,
Flame,
Server,
AlertTriangle,
Play
} from 'lucide-react';
const API_BASE = 'https://culprit-campfire-galore.ngrok-free.dev'; // Target FastAPI central server in production VM
// Secure API fetch wrapper to automatically bypass ngrok's transitional browser warning page
const apiFetch = (url, options = {}) => {
return fetch(url, {
...options,
headers: {
...options.headers,
'ngrok-skip-browser-warning': 'true'
}
});
};
function App() {
const [activeTab, setActiveTab] = useState('dashboard'); // 'dashboard' | 'editor'
const [clients, setClients] = useState({});
const [telemetry, setTelemetry] = useState({});
const [logs, setLogs] = useState({});
const [whitelist, setWhitelist] = useState({});
// Dispatcher form state
const [selectedClients, setSelectedClients] = useState([]);
const [selectedCommand, setSelectedCommand] = useState('');
const [manualClient, setManualClient] = useState('');
const [manualCommand, setManualCommand] = useState('');
// Whitelist config editor state
const [editorText, setEditorText] = useState('{\n "nt": {},\n "posix": {}\n}');
const [toast, setToast] = useState(null);
const [isSyncing, setIsSyncing] = useState(false);
const [filterClient, setFilterClient] = useState('all');
const terminalRef = useRef(null);
const showToast = (message, type = 'success') => {
setToast({ message, type });
setTimeout(() => setToast(null), 4000);
};
// 1. Initial configuration bootstrap
const fetchWhitelist = async () => {
try {
const response = await apiFetch(`${API_BASE}/api/get-raw-commands`);
if (response.ok) {
const data = await response.json();
setWhitelist(data);
setEditorText(JSON.stringify(data, null, 2));
}
} catch (err) {
console.error('Failed to load raw commands whitelist:', err);
}
};
// 2. Continuous telemetry & log sync polling
const syncFleetData = async () => {
setIsSyncing(true);
try {
// Sync active clients registry
const clientsRes = await apiFetch(`${API_BASE}/api/clients`);
if (clientsRes.ok) {
const clientsData = await clientsRes.json();
setClients(clientsData);
}
// Sync active telemetry details
const telRes = await apiFetch(`${API_BASE}/api/telemetry`);
if (telRes.ok) {
const telData = await telRes.json();
setTelemetry(telData);
}
// Sync circular history logs
const logsRes = await apiFetch(`${API_BASE}/api/logs`);
if (logsRes.ok) {
const logsData = await logsRes.json();
setLogs(logsData);
}
} catch (err) {
console.error('Network sync failure:', err);
} finally {
setIsSyncing(false);
}
};
useEffect(() => {
fetchWhitelist();
syncFleetData();
// Set up rapid 3-second fleet synchronization loop
const syncInterval = setInterval(syncFleetData, 3000);
return () => clearInterval(syncInterval);
}, []);
// Scroll terminal logs to bottom automatically when new actions populate,
// but only if the user is already looking at the latest logs (near the bottom)!
// If they scrolled up to inspect an error, we freeze the screen scroll.
useEffect(() => {
if (terminalRef.current) {
const { scrollTop, scrollHeight, clientHeight } = terminalRef.current;
// Define a 120px threshold close to the bottom
const isNearBottom = scrollHeight - scrollTop - clientHeight < 120;
const isFirstLoad = scrollTop === 0;
if (isNearBottom || isFirstLoad) {
terminalRef.current.scrollTop = scrollHeight;
}
}
}, [logs]);
// Dispatch command to client execution queue
const handleDispatch = async (e) => {
e.preventDefault();
// Resolve final list of targets
let finalClientsString = '';
if (manualClient.trim()) {
finalClientsString = manualClient.trim();
} else if (selectedClients.length > 0) {
finalClientsString = selectedClients.join(',');
}
const finalCommand = (manualCommand.trim() || selectedCommand).trim();
if (!finalClientsString) {
showToast('Please select target nodes or specify a manual name!', 'error');
return;
}
if (!finalCommand) {
showToast('Please specify a Whitelisted Operation Key!', 'error');
return;
}
try {
const response = await apiFetch(
`${API_BASE}/api/schedule-command?client_id=${encodeURIComponent(finalClientsString)}&command=${encodeURIComponent(finalCommand)}`,
{ method: 'POST' }
);
const data = await response.json();
if (response.ok) {
showToast(`Operation batch successfully dispatched to: ${finalClientsString}!`, 'success');
setManualCommand('');
setManualClient('');
setSelectedClients([]);
syncFleetData();
} else {
showToast(data.detail || 'Dispatch scheduling failed.', 'error');
}
} catch (err) {
showToast('Network error contacting central RMM server API.', 'error');
}
};
// Dynamic Whitelist Save Config
const handleSaveWhitelist = async () => {
try {
const parsedConfig = JSON.parse(editorText);
const response = await apiFetch(`${API_BASE}/api/save-commands`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(parsedConfig)
});
if (response.ok) {
showToast('Dynamic Commands Whitelist successfully deployed to cluster!', 'success');
fetchWhitelist(); // Reload whitelists
} else {
const errorData = await response.json();
showToast(`Failed to deploy: ${errorData.detail}`, 'error');
}
} catch (err) {
showToast('Invalid JSON structure! Please audit your config syntax.', 'error');
}
};
// Compile individual client arrays into a single global chronological timeline for terminal display
const getSortedTimeline = () => {
const timeline = [];
Object.entries(logs).forEach(([cid, entryList]) => {
// Apply system filter
if (filterClient !== 'all' && cid !== filterClient) {
return;
}
if (Array.isArray(entryList)) {
entryList.forEach((logItem) => {
timeline.push({
...logItem,
client_id: cid
});
});
}
});
// Sort ascending by ISO timestamp
return timeline.sort((a, b) => new Date(a.timestamp) - new Date(b.timestamp));
};
const sortedTimelineLogs = getSortedTimeline();
const activeClients = Object.values(clients).filter(c => c.active).length;
const totalClients = Object.keys(clients).length;
return (
<div className="min-h-screen pb-16 flex flex-col">
{/* Dynamic Toast Alerts */}
{toast && (
<div className={`fixed bottom-6 right-6 z-50 px-6 py-4 rounded-xl shadow-2xl flex items-center gap-3 border transition-all duration-300 transform translate-y-0 ${
toast.type === 'error'
? 'bg-rose-950/85 border-rose-500/30 text-rose-200'
: 'bg-emerald-950/85 border-emerald-500/30 text-emerald-200'
}`}>
{toast.type === 'error' ? <XCircle className="w-5 h-5 text-rose-400" /> : <CheckCircle className="w-5 h-5 text-emerald-400" />}
<span className="font-semibold text-sm">{toast.message}</span>
</div>
)}
{/* Modern Glass Header Section */}
<header className="sticky top-0 z-30 px-6 py-4 glass-panel border-b border-white/5 flex items-center justify-between">
<div className="flex items-center gap-3">
<div className="w-10 h-10 rounded-xl bg-gradient-to-tr from-violet-600 to-indigo-600 flex items-center justify-center shadow-lg shadow-violet-900/30">
<Zap className="w-5 h-5 text-white" />
</div>
<div>
<h1 className="font-bold text-lg tracking-tight text-white flex items-center gap-2">
SEEKRIGHT PULSE RMM
<span className="text-[10px] uppercase font-mono px-2 py-0.5 rounded bg-violet-500/10 border border-violet-500/20 text-violet-300">CORE-v1.0</span>
</h1>
<p className="text-xs text-slate-400 font-medium">Enterprise Global Fleet & Whitelist Agent Orchestrator</p>
</div>
</div>
{/* Action Controls & Navigation tabs */}
<div className="flex items-center gap-4">
<nav className="flex items-center gap-1.5 p-1 rounded-xl bg-slate-950/60 border border-white/5">
<button
onClick={() => setActiveTab('dashboard')}
className={`px-4 py-2 rounded-lg font-semibold text-xs transition-all ${
activeTab === 'dashboard'
? 'bg-gradient-to-r from-violet-600 to-indigo-600 text-white shadow-md'
: 'text-slate-400 hover:text-slate-200 hover:bg-white/5'
}`}
>
<Activity className="w-3.5 h-3.5 inline mr-1.5 -mt-0.5" />
Fleet Status
</button>
<button
onClick={() => setActiveTab('editor')}
className={`px-4 py-2 rounded-lg font-semibold text-xs transition-all ${
activeTab === 'editor'
? 'bg-gradient-to-r from-violet-600 to-indigo-600 text-white shadow-md'
: 'text-slate-400 hover:text-slate-200 hover:bg-white/5'
}`}
>
<Settings className="w-3.5 h-3.5 inline mr-1.5 -mt-0.5" />
Commands Whitelist
</button>
</nav>
{/* Sync indicator status badge */}
<div className="flex items-center gap-2 px-3 py-1.5 rounded-xl bg-slate-900/50 border border-white/5">
<span className={`w-2 h-2 rounded-full ${isSyncing ? 'bg-violet-400 animate-spin' : 'bg-emerald-400 pulse-glow-emerald'}`}></span>
<span className="text-xs font-mono text-slate-300">
API Status: <b className="text-emerald-400 font-bold">Online</b>
</span>
<button
onClick={syncFleetData}
title="Force Sync Fleet Vitals"
className="p-1 rounded hover:bg-white/5 text-slate-400 hover:text-slate-200 transition-colors"
>
<RefreshCw className={`w-3 h-3 ${isSyncing ? 'animate-spin text-violet-400' : ''}`} />
</button>
</div>
</div>
</header>
{/* Main Container */}
<main className="max-w-7xl w-full mx-auto px-6 mt-8 flex-grow grid grid-cols-1 lg:grid-cols-3 gap-8">
{activeTab === 'dashboard' ? (
<>
{/* Dashboard Vitals: Grid of Cards (Left/Center Column) */}
<div className="lg:col-span-2 space-y-6">
{/* Central Metrics Board */}
<div className="grid grid-cols-3 gap-4">
<div className="glass-panel p-4 rounded-2xl flex flex-col justify-between">
<span className="text-xs font-semibold text-slate-400">Total Systems</span>
<span className="text-2xl font-bold text-white mt-1">{totalClients}</span>
</div>
<div className="glass-panel p-4 rounded-2xl flex flex-col justify-between border-l-2 border-l-emerald-500">
<span className="text-xs font-semibold text-slate-400">🟢 Active Systems</span>
<span className="text-2xl font-bold text-emerald-400 mt-1">{activeClients}</span>
</div>
<div className="glass-panel p-4 rounded-2xl flex flex-col justify-between border-l-2 border-l-rose-500">
<span className="text-xs font-semibold text-slate-400">🔴 Offline Systems</span>
<span className="text-2xl font-bold text-rose-400 mt-1">{totalClients - activeClients}</span>
</div>
</div>
{/* Vitals Node Grid Cards */}
<div className="space-y-4">
<h2 className="text-sm font-bold text-white tracking-wide uppercase flex items-center gap-2">
<Server className="w-4 h-4 text-violet-400" />
Remote Fleet Systems Grid
</h2>
{Object.keys(clients).length === 0 ? (
<div className="glass-panel p-12 rounded-2xl text-center text-slate-400">
<AlertTriangle className="w-8 h-8 mx-auto text-yellow-400 mb-2 opacity-60" />
<p className="font-semibold text-sm">Searching for active RMM telemetry agents...</p>
<p className="text-xs text-slate-500 mt-1">Boot up an agent using deploy_agent.sh to register</p>
</div>
) : (
Object.entries(clients).map(([clientId, info]) => {
const clientTelemetry = telemetry[clientId];
const isOnline = info.active;
const lastSeenDate = info.last_seen ? new Date(info.last_seen).toLocaleString() : 'Never';
return (
<div
key={clientId}
className="glass-panel p-6 rounded-2xl hover:border-violet-500/40 hover:-translate-y-1 hover:shadow-[0_8px_30px_rgb(99,102,241,0.12)] transition-all duration-300 relative overflow-hidden group bg-gradient-to-b from-slate-900/30 to-slate-950/50"
>
{/* Status bar */}
<div className={`absolute top-0 left-0 right-0 h-1 ${isOnline ? 'bg-gradient-to-r from-emerald-500 to-teal-400' : 'bg-gradient-to-r from-rose-500 to-red-400'}`} />
{/* Title block */}
<div className="flex items-center justify-between mb-5">
<div className="flex items-center gap-3">
<div className={`w-3.5 h-3.5 rounded-full flex items-center justify-center ${isOnline ? 'bg-emerald-500/10' : 'bg-rose-500/10'}`}>
<span className="relative flex h-2 w-2">
{isOnline && <span className="animate-ping absolute inline-flex h-full w-full rounded-full bg-emerald-400 opacity-75"></span>}
<span className={`relative inline-flex rounded-full h-2 w-2 ${isOnline ? 'bg-emerald-500' : 'bg-rose-500'}`}></span>
</span>
</div>
<span className="font-extrabold text-white text-lg tracking-tight group-hover:text-violet-300 transition-colors">{clientId}</span>
<span className={`text-[9px] uppercase tracking-wider font-extrabold px-2 py-0.5 rounded ${
isOnline
? 'bg-emerald-500/10 border border-emerald-500/20 text-emerald-300'
: 'bg-rose-500/10 border border-rose-500/20 text-rose-300'
}`}>
{isOnline ? 'Active' : 'Offline'}
</span>
</div>
<div className="text-right">
<span className="text-[10px] text-slate-500 block">LAST CONTACT</span>
<span className="text-[11px] font-bold text-slate-300 font-mono">{lastSeenDate}</span>
</div>
</div>
{/* Telemetry charts row */}
{clientTelemetry ? (
<div className="space-y-5">
{/* CPU & RAM progress */}
<div className="grid grid-cols-2 gap-4">
{/* CPU Progress block */}
{(() => {
const cpu = clientTelemetry.cpu_percent || 0;
const isWarning = cpu > 80;
return (
<div className="p-3.5 bg-slate-950/40 border border-white/5 rounded-2xl flex flex-col justify-between">
<div className="flex justify-between items-center text-xs font-bold text-slate-400 mb-2">
<span className="flex items-center gap-1.5">
<Cpu className={`w-3.5 h-3.5 ${isWarning ? 'text-rose-400 animate-pulse' : 'text-violet-400'}`} />
CPU Load
</span>
<span className={`font-bold ${isWarning ? 'text-rose-400' : 'text-white'}`}>{cpu}%</span>
</div>
<div className="w-full h-2.5 rounded-full bg-slate-900 overflow-hidden relative border border-white/5">
<div
className={`h-full rounded-full transition-all duration-500 ${
isWarning
? 'bg-gradient-to-r from-rose-500 to-red-400'
: cpu > 60
? 'bg-gradient-to-r from-amber-500 to-yellow-400'
: 'bg-gradient-to-r from-violet-500 to-indigo-500'
}`}
style={{ width: `${cpu}%` }}
/>
</div>
{isWarning && (
<div className="text-[9px] text-rose-400 font-bold flex items-center gap-1 mt-1.5 animate-pulse">
<AlertTriangle className="w-2.5 h-2.5" /> High Utilization
</div>
)}
</div>
);
})()}
{/* Memory Progress block */}
{(() => {
const total = clientTelemetry.memory_total_gb || 0;
const free = clientTelemetry.memory_free_gb || 0;
const used = (total - free).toFixed(2);
const ram = clientTelemetry.memory_percent || 0;
const isWarning = ram > 85;
return (
<div className="p-3.5 bg-slate-950/40 border border-white/5 rounded-2xl flex flex-col justify-between">
<div className="flex justify-between items-center text-xs font-bold text-slate-400 mb-2">
<span className="flex items-center gap-1.5">
<Activity className={`w-3.5 h-3.5 ${isWarning ? 'text-rose-400 animate-pulse' : 'text-violet-400'}`} />
Virtual RAM
</span>
<span className={`font-bold ${isWarning ? 'text-rose-400' : 'text-white'}`}>{ram}%</span>
</div>
<div className="w-full h-2.5 rounded-full bg-slate-900 overflow-hidden relative border border-white/5">
<div
className={`h-full rounded-full transition-all duration-500 ${
isWarning
? 'bg-gradient-to-r from-rose-500 to-red-400'
: ram > 70
? 'bg-gradient-to-r from-amber-500 to-yellow-400'
: 'bg-gradient-to-r from-violet-500 to-indigo-500'
}`}
style={{ width: `${ram}%` }}
/>
</div>
<div className="flex flex-wrap justify-between text-[9px] text-slate-500 font-mono font-medium mt-2 border-t border-white/5 pt-1.5 font-bold gap-x-1">
<span>Used: <b className="text-slate-300">{used}G</b></span>
<span>Free: <b className="text-slate-300">{free}G</b></span>
<span>Total: <b className="text-slate-300">{total}G</b></span>
</div>
</div>
);
})()}
</div>
{/* GPU and Partition details */}
<div className="grid grid-cols-1 md:grid-cols-2 gap-4 pt-4 border-t border-white/5">
{/* Partition Storage */}
<div className="space-y-2">
<span className="text-[10px] uppercase font-extrabold tracking-wider text-slate-400 flex items-center gap-1.5">
<HardDrive className="w-3.5 h-3.5 text-indigo-400" /> Disk Drive Space
</span>
<div className="space-y-2.5">
{clientTelemetry.disks && clientTelemetry.disks.length > 0 ? (
clientTelemetry.disks.map((d, index) => {
const diskTotal = d.total_gb || 0;
const diskFree = d.free_gb || 0;
const diskUsed = (diskTotal - diskFree).toFixed(2);
return (
<div key={index} className="p-2.5 rounded-xl bg-slate-950/30 border border-white/5 space-y-1.5">
<div className="flex items-center justify-between text-xs">
<span className="font-semibold text-slate-300">Disk [ {d.mount} ]</span>
<span className="font-mono text-slate-400 text-[10px]">
Used: <b className="text-slate-300">{diskUsed}G</b> / {diskTotal}G
</span>
</div>
<div className="w-full h-1.5 rounded-full bg-slate-900 overflow-hidden">
<div
className={`h-full rounded-full ${d.percent > 85 ? 'bg-rose-500' : 'bg-indigo-500'}`}
style={{ width: `${d.percent}%` }}
/>
</div>
<div className="flex justify-between text-[8px] text-slate-500 font-mono">
<span>Free: <b className="text-slate-400">{diskFree} GB</b></span>
<span>{d.percent}% Used</span>
</div>
</div>
);
})
) : (
<span className="text-xs text-slate-500 block italic">No storage drives reported.</span>
)}
</div>
</div>
{/* Nvidia Core GPU Vitals */}
<div className="space-y-2">
<span className="text-[10px] uppercase font-extrabold tracking-wider text-slate-400 flex items-center gap-1.5">
<Flame className="w-3.5 h-3.5 text-rose-400 animate-pulse" /> Nvidia GPU Accelerator
</span>
<div className="space-y-1.5">
{clientTelemetry.gpus && clientTelemetry.gpus.length > 0 ? (
clientTelemetry.gpus.map((gpu, index) => (
<div key={index} className="p-2.5 rounded-xl bg-slate-950/40 border border-white/5 space-y-2">
<div className="flex justify-between items-center text-xs border-b border-white/5 pb-1">
<span className="font-bold text-slate-200 truncate max-w-[130px]">🎮 {gpu.name}</span>
<span className="text-violet-400 font-extrabold font-mono text-[11px]">Load: {gpu.utilization}</span>
</div>
<div className="grid grid-cols-2 gap-x-2 gap-y-1 text-[10px] text-slate-400 font-mono">
<div className="flex justify-between"><span>Temp:</span> <b className="text-slate-200 font-semibold">{gpu.temp}</b></div>
<div className="flex justify-between"><span>Fans:</span> <b className="text-slate-200 font-semibold">{gpu.fan_speed}</b></div>
<div className="col-span-2 flex justify-between border-t border-white/5 pt-1 mt-0.5">
<span>VRAM Used:</span>
<b className="text-slate-200 font-semibold">{gpu.memory_used} / {gpu.memory_total || 'N/A'}</b>
</div>
<div className="col-span-2 flex justify-between">
<span>Power Draw:</span>
<b className="text-slate-200 font-semibold">{gpu.power_draw} / {gpu.power_limit || 'N/A'}</b>
</div>
</div>
</div>
))
) : (
<div className="text-xs text-slate-500 py-3 text-center font-medium bg-slate-950/15 border border-white/5 rounded-xl italic">
No NVIDIA GPU hardware reported.
</div>
)}
</div>
</div>
</div>
</div>
) : (
<div className="text-center py-6 bg-slate-950/20 rounded-xl border border-white/5 my-4">
<p className="text-xs text-slate-400 font-medium italic">Telemetry check-in pending... Waiting for dynamic vitals transmission</p>
</div>
)}
</div>
);
})
)}
</div>
</div>
{/* Maintenance Command Dispatch deck (Right sidebar) */}
<div className="lg:col-span-1">
<div className="glass-panel p-6 rounded-2xl sticky top-24 border border-white/5 space-y-6">
<div className="border-b border-white/5 pb-3">
<h2 className="text-sm font-bold text-white uppercase tracking-wider flex items-center gap-2">
<Play className="w-4 h-4 text-violet-400 fill-violet-400" />
Maintenance Dispatcher
</h2>
<p className="text-xs text-slate-400 mt-0.5">Send a whitelisted command down to a node</p>
</div>
<form onSubmit={handleDispatch} className="space-y-4">
{/* Select target node check list */}
<div>
<label className="block text-xs font-semibold text-slate-400 uppercase tracking-wider mb-2 flex justify-between">
<span>Select Fleet Targets</span>
<div className="flex gap-2.5">
<button
type="button"
onClick={() => setSelectedClients(Object.keys(clients))}
className="text-[10px] text-violet-400 font-extrabold hover:text-violet-300 transition-colors uppercase tracking-wider"
>
All
</button>
<button
type="button"
onClick={() => setSelectedClients(Object.entries(clients).filter(([_, info]) => info.active).map(([id]) => id))}
className="text-[10px] text-emerald-400 font-extrabold hover:text-emerald-300 transition-colors uppercase tracking-wider"
>
Active
</button>
<button
type="button"
onClick={() => setSelectedClients([])}
className="text-[10px] text-slate-500 font-extrabold hover:text-slate-400 transition-colors uppercase tracking-wider"
>
None
</button>
</div>
</label>
<div className="grid grid-cols-2 gap-2 max-h-48 overflow-y-auto p-1 bg-slate-950/40 border border-white/5 rounded-xl">
{Object.entries(clients).length === 0 ? (
<span className="col-span-2 text-xs text-slate-500 italic text-center py-6 block font-medium">No registered nodes found.</span>
) : (
Object.entries(clients).map(([id, info]) => {
const isSelected = selectedClients.includes(id);
return (
<button
key={id}
type="button"
onClick={() => {
if (isSelected) {
setSelectedClients(selectedClients.filter(c => c !== id));
} else {
setSelectedClients([...selectedClients, id]);
}
}}
className={`p-2.5 rounded-xl border text-left transition-all duration-200 relative flex flex-col justify-between overflow-hidden ${
isSelected
? 'bg-gradient-to-br from-violet-600/15 to-indigo-600/15 border-violet-500/80 shadow-md shadow-violet-900/15 text-white'
: 'bg-slate-950/60 border-white/5 hover:border-white/15 text-slate-400 hover:text-slate-200'
}`}
>
{/* Glow element */}
{isSelected && <span className="absolute top-0 right-0 w-2 h-2 bg-violet-400 rounded-full blur-[1px] -mr-0.5 -mt-0.5" />}
<div className="flex items-center justify-between w-full gap-2">
<span className={`text-xs font-extrabold truncate ${isSelected ? 'text-violet-200' : 'text-slate-300'}`}>
{id}
</span>
<span className={`w-1.5 h-1.5 rounded-full shrink-0 ${info.active ? 'bg-emerald-400' : 'bg-rose-400'}`} />
</div>
<span className="text-[9px] font-mono text-slate-500 mt-1 font-semibold uppercase tracking-wider">
{info.active ? '🟢 Online' : '🔴 Offline'}
</span>
</button>
);
})
)}
</div>
</div>
{/* Manual Target target ID */}
<div>
<label className="block text-xs font-semibold text-slate-400 uppercase tracking-wider mb-1.5">Or Type Target Name manually (comma-separated for batch)</label>
<input
type="text"
placeholder="e.g. kawsik, client_1, client_2"
value={manualClient}
onChange={(e) => setManualClient(e.target.value)}
className="w-full bg-slate-950 border border-white/10 rounded-xl p-3 text-sm text-slate-300 font-semibold focus:outline-none focus:border-violet-500 placeholder-slate-600 transition-colors"
/>
</div>
{/* Select command key */}
<div>
<label className="block text-xs font-semibold text-slate-400 uppercase tracking-wider mb-1.5">Pick Whitelisted Operation</label>
<select
value={selectedCommand}
onChange={(e) => setSelectedCommand(e.target.value)}
className="w-full bg-slate-950 border border-white/10 rounded-xl p-3 text-sm text-slate-300 font-semibold focus:outline-none focus:border-violet-500 transition-colors"
>
<option value="">-- Choose Whitelisted Command --</option>
{Array.from(new Set([
...Object.keys(whitelist.nt || {}),
...Object.keys(whitelist.posix || {})
])).map((cmdKey) => (
<option key={cmdKey} value={cmdKey}>
{cmdKey}
</option>
))}
</select>
</div>
{/* Manual command command key */}
<div>
<label className="block text-xs font-semibold text-slate-400 uppercase tracking-wider mb-1.5">Or Type Operation Key manually</label>
<input
type="text"
placeholder="e.g. repair_mysql"
value={manualCommand}
onChange={(e) => setManualCommand(e.target.value)}
className="w-full bg-slate-950 border border-white/10 rounded-xl p-3 text-sm text-slate-300 font-semibold focus:outline-none focus:border-violet-500 placeholder-slate-600 transition-colors"
/>
</div>
{/* Fire execution button */}
<button
type="submit"
className="w-full bg-gradient-to-r from-violet-600 to-indigo-600 hover:from-violet-500 hover:to-indigo-500 text-white font-bold py-3 px-4 rounded-xl text-sm flex items-center justify-center gap-2 shadow-lg shadow-violet-950/45 hover:shadow-violet-900/50 hover:translate-y-[-1px] active:translate-y-[1px] transition-all"
>
<Zap className="w-4 h-4 fill-white" />
Dispatch Operation to Node
</button>
</form>
</div>
</div>
</>
) : (
/* Visual Whitelist JSON Configuration Editor */
<div className="col-span-3">
<div className="glass-panel p-6 rounded-2xl border border-white/5 space-y-6">
<div className="border-b border-white/5 pb-3">
<h2 className="text-sm font-bold text-white uppercase tracking-wider flex items-center gap-2">
<Settings className="w-4 h-4 text-violet-400" />
Visual Command Whitelist Editor
</h2>
<p className="text-xs text-slate-400 mt-0.5">
Directly configure permitted cluster arrays. Permitted operations are pushed dynamically to remote agents without any central restart required!
</p>
</div>
<div className="space-y-4">
<textarea
className="w-full h-[400px] bg-slate-950 border border-white/10 rounded-xl p-4 font-mono text-sm text-emerald-400 leading-relaxed focus:outline-none focus:border-violet-500 transition-colors focus:ring-1 focus:ring-violet-500/25"
spellCheck="false"
value={editorText}
onChange={(e) => setEditorText(e.target.value)}
/>
<div className="flex justify-end gap-3">
<button
onClick={fetchWhitelist}
className="px-5 py-2.5 rounded-xl border border-white/10 hover:bg-white/5 font-semibold text-xs text-slate-300 transition-colors"
>
Discard Changes
</button>
<button
onClick={handleSaveWhitelist}
className="px-6 py-2.5 rounded-xl bg-gradient-to-r from-violet-600 to-indigo-600 hover:from-violet-500 hover:to-indigo-500 text-white font-bold text-xs shadow-lg shadow-violet-950/40 hover:shadow-violet-900/50 hover:translate-y-[-1px] active:translate-y-[1px] transition-all"
>
Deploy updated JSON Configuration
</button>
</div>
</div>
</div>
</div>
)}
</main>
{/* Scrolling DevOps CLI output shell terminal */}
<footer className="max-w-7xl w-full mx-auto px-6 mt-8">
<div className="bg-slate-950/95 border border-white/5 rounded-2xl overflow-hidden shadow-2xl">
<div className="bg-slate-900/80 px-4 py-2 border-b border-white/5 flex items-center justify-between">
<div className="flex items-center gap-2">
<div className="flex gap-1.5">
<span className="w-2.5 h-2.5 rounded-full bg-rose-500/60 block"></span>
<span className="w-2.5 h-2.5 rounded-full bg-yellow-500/60 block"></span>
<span className="w-2.5 h-2.5 rounded-full bg-emerald-500/60 block"></span>
</div>
<span className="text-[11px] uppercase tracking-wider font-bold text-slate-400 font-mono ml-2 flex items-center gap-1.5">
<Terminal className="w-3.5 h-3.5 text-violet-400" />
Live Fleet Activity Logs & Webhook outputs
</span>
</div>
<div className="flex items-center gap-3">
<select
value={filterClient}
onChange={(e) => setFilterClient(e.target.value)}
className="bg-slate-950 border border-white/10 rounded-lg px-2.5 py-1 text-[11px] text-slate-300 font-semibold focus:outline-none focus:border-violet-500 transition-colors"
>
<option value="all">🔍 All Fleet Logs</option>
{Object.keys(clients).map((cid) => (
<option key={cid} value={cid}>
{cid} logs
</option>
))}
</select>
<span className="text-[10px] text-slate-500 font-mono font-medium">Terminal Shell v1.0</span>
</div>
</div>
<div ref={terminalRef} className="p-4 h-64 overflow-y-auto font-mono text-xs space-y-2.5 terminal-scroll bg-black/90">
{sortedTimelineLogs.length === 0 ? (
<span className="text-slate-500 block italic py-2">Waiting for check-in events to stream... No historical logs recorded yet.</span>
) : (
sortedTimelineLogs.map((log, index) => {
const formattedTime = new Date(log.timestamp).toLocaleTimeString();
if (log.type === 'telemetry') {
return (
<div key={index} className="text-slate-400 flex items-start gap-1 leading-relaxed">
<span className="text-slate-600 flex-shrink-0">[{formattedTime}]</span>
<span className="text-sky-400 flex-shrink-0 font-bold">INFO:</span>
<span>
Client <b className="text-white">{log.client_id}</b> check-in telemetry pushed. (CPU: {log.detail.cpu_percent}%, RAM: {log.detail.memory_percent}%)
</span>
</div>
);
}
if (log.type === 'command_scheduled') {
return (
<div key={index} className="text-yellow-200 flex items-start gap-1 leading-relaxed">
<span className="text-slate-600 flex-shrink-0">[{formattedTime}]</span>
<span className="text-yellow-500 flex-shrink-0 font-bold">DISPATCH:</span>
<span>
Operation <b className="text-yellow-400 font-semibold">"{log.detail.command}"</b> successfully queued for node <b className="text-white">{log.client_id}</b>.
</span>
</div>
);
}
if (log.type === 'command_polled') {
return (
<div key={index} className="text-orange-200/90 flex items-start gap-1 leading-relaxed">
<span className="text-slate-600 flex-shrink-0">[{formattedTime}]</span>
<span className="text-orange-400 flex-shrink-0 font-bold">POLL:</span>
<span>
Agent <b className="text-white">{log.client_id}</b> retrieved pending command <b className="text-orange-300 font-semibold">"{log.detail.command}"</b>. Starting execution...
</span>
</div>
);
}
if (log.type === 'command_result') {
const isSuccess = log.detail.returncode === 0;
return (
<div key={index} className="p-2.5 rounded-lg border bg-slate-950/70 border-white/5 space-y-1 my-1">
<div className="flex items-center justify-between">
<div className="flex items-center gap-1.5">
<span className="text-slate-600 font-mono">[{formattedTime}]</span>
<span className={`font-bold ${isSuccess ? 'text-emerald-400' : 'text-rose-400'}`}>
{isSuccess ? 'OUTCOME SUCCESS:' : 'OUTCOME FAILURE:'}
</span>
<span className="text-slate-200">
Command <b className="text-white">"{log.detail.command}"</b> on <b className="text-white">{log.client_id}</b> finished with code <b className="font-mono">{log.detail.returncode}</b>.
</span>
</div>
</div>
{/* Subprocess console stdout/stderr */}
{(log.detail.stdout || log.detail.stderr) && (
<pre className={`p-2 rounded bg-black/60 font-mono text-[11px] leading-normal overflow-x-auto ${isSuccess ? 'text-slate-300 border-l border-emerald-500/45' : 'text-rose-300 border-l border-rose-500/45'}`}>
{log.detail.stdout && <code className="block">{log.detail.stdout}</code>}
{log.detail.stderr && <code className="block text-rose-400 font-semibold">{log.detail.stderr}</code>}
</pre>
)}
</div>
);
}
return null;
})
)}
{/* Removed internal dummy anchor to avoid viewport jumping */}
</div>
</div>
</footer>
</div>
);
}
export default App;

BIN
rmm-ui/src/assets/hero.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

1
rmm-ui/src/assets/react.svg Normal file
View File

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" class="iconify iconify--logos" width="35.93" height="32" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 228"><path fill="#00D8FF" d="M210.483 73.824a171.49 171.49 0 0 0-8.24-2.597c.465-1.9.893-3.777 1.273-5.621c6.238-30.281 2.16-54.676-11.769-62.708c-13.355-7.7-35.196.329-57.254 19.526a171.23 171.23 0 0 0-6.375 5.848a155.866 155.866 0 0 0-4.241-3.917C100.759 3.829 77.587-4.822 63.673 3.233C50.33 10.957 46.379 33.89 51.995 62.588a170.974 170.974 0 0 0 1.892 8.48c-3.28.932-6.445 1.924-9.474 2.98C17.309 83.498 0 98.307 0 113.668c0 15.865 18.582 31.778 46.812 41.427a145.52 145.52 0 0 0 6.921 2.165a167.467 167.467 0 0 0-2.01 9.138c-5.354 28.2-1.173 50.591 12.134 58.266c13.744 7.926 36.812-.22 59.273-19.855a145.567 145.567 0 0 0 5.342-4.923a168.064 168.064 0 0 0 6.92 6.314c21.758 18.722 43.246 26.282 56.54 18.586c13.731-7.949 18.194-32.003 12.4-61.268a145.016 145.016 0 0 0-1.535-6.842c1.62-.48 3.21-.974 4.76-1.488c29.348-9.723 48.443-25.443 48.443-41.52c0-15.417-17.868-30.326-45.517-39.844Zm-6.365 70.984c-1.4.463-2.836.91-4.3 1.345c-3.24-10.257-7.612-21.163-12.963-32.432c5.106-11 9.31-21.767 12.459-31.957c2.619.758 5.16 1.557 7.61 2.4c23.69 8.156 38.14 20.213 38.14 29.504c0 9.896-15.606 22.743-40.946 31.14Zm-10.514 20.834c2.562 12.94 2.927 24.64 1.23 33.787c-1.524 8.219-4.59 13.698-8.382 15.893c-8.067 4.67-25.32-1.4-43.927-17.412a156.726 156.726 0 0 1-6.437-5.87c7.214-7.889 14.423-17.06 21.459-27.246c12.376-1.098 24.068-2.894 34.671-5.345a134.17 134.17 0 0 1 1.386 6.193ZM87.276 214.515c-7.882 2.783-14.16 2.863-17.955.675c-8.075-4.657-11.432-22.636-6.853-46.752a156.923 156.923 0 0 1 1.869-8.499c10.486 2.32 22.093 3.988 34.498 4.994c7.084 9.967 14.501 19.128 21.976 27.15a134.668 134.668 0 0 1-4.877 4.492c-9.933 8.682-19.886 14.842-28.658 17.94ZM50.35 144.747c-12.483-4.267-22.792-9.812-29.858-15.863c-6.35-5.437-9.555-10.836-9.555-15.216c0-9.322 13.897-21.212 37.076-29.293c2.813-.98 5.757-1.905 8.812-2.773c3.204 10.42 7.406 21.315 12.477 32.332c-5.137 11.18-9.399 22.249-12.634 32.792a134.718 134.718 0 0 1-6.318-1.979Zm12.378-84.26c-4.811-24.587-1.616-43.134 6.425-47.789c8.564-4.958 27.502 2.111 47.463 19.835a144.318 144.318 0 0 1 3.841 3.545c-7.438 7.987-14.787 17.08-21.808 26.988c-12.04 1.116-23.565 2.908-34.161 5.309a160.342 160.342 0 0 1-1.76-7.887Zm110.427 27.268a347.8 347.8 0 0 0-7.785-12.803c8.168 1.033 15.994 2.404 23.343 4.08c-2.206 7.072-4.956 14.465-8.193 22.045a381.151 381.151 0 0 0-7.365-13.322Zm-45.032-43.861c5.044 5.465 10.096 11.566 15.065 18.186a322.04 322.04 0 0 0-30.257-.006c4.974-6.559 10.069-12.652 15.192-18.18ZM82.802 87.83a323.167 323.167 0 0 0-7.227 13.238c-3.184-7.553-5.909-14.98-8.134-22.152c7.304-1.634 15.093-2.97 23.209-3.984a321.524 321.524 0 0 0-7.848 12.897Zm8.081 65.352c-8.385-.936-16.291-2.203-23.593-3.793c2.26-7.3 5.045-14.885 8.298-22.6a321.187 321.187 0 0 0 7.257 13.246c2.594 4.48 5.28 8.868 8.038 13.147Zm37.542 31.03c-5.184-5.592-10.354-11.779-15.403-18.433c4.902.192 9.899.29 14.978.29c5.218 0 10.376-.117 15.453-.343c-4.985 6.774-10.018 12.97-15.028 18.486Zm52.198-57.817c3.422 7.8 6.306 15.345 8.596 22.52c-7.422 1.694-15.436 3.058-23.88 4.071a382.417 382.417 0 0 0 7.859-13.026a347.403 347.403 0 0 0 7.425-13.565Zm-16.898 8.101a358.557 358.557 0 0 1-12.281 19.815a329.4 329.4 0 0 1-23.444.823c-7.967 0-15.716-.248-23.178-.732a310.202 310.202 0 0 1-12.513-19.846h.001a307.41 307.41 0 0 1-10.923-20.627a310.278 310.278 0 0 1 10.89-20.637l-.001.001a307.318 307.318 0 0 1 12.413-19.761c7.613-.576 15.42-.876 23.31-.876H128c7.926 0 15.743.303 23.354.883a329.357 329.357 0 0 1 12.335 19.695a358.489 358.489 0 0 1 11.036 20.54a329.472 329.472 0 0 1-11 20.722Zm22.56-122.124c8.572 4.944 11.906 24.881 6.52 51.026c-.344 1.668-.73 3.367-1.15 5.09c-10.622-2.452-22.155-4.275-34.23-5.408c-7.034-10.017-14.323-19.124-21.64-27.008a160.789 160.789 0 0 1 5.888-5.4c18.9-16.447 36.564-22.941 44.612-18.3ZM128 90.808c12.625 0 22.86 10.235 22.86 22.86s-10.235 22.86-22.86 22.86s-22.86-10.235-22.86-22.86s10.235-22.86 22.86-22.86Z"></path></svg>
Side by Side

After

Width:  |  Height:  |  Size: 4.0 KiB

1
rmm-ui/src/assets/vite.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 8.5 KiB

64
rmm-ui/src/index.css Normal file
View File

@@ -0,0 +1,64 @@
@import url('https://fonts.googleapis.com/css2?family=Outfit:wght@300;400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap');
@tailwind base;
@tailwind components;
@tailwind utilities;
body {
margin: 0;
font-family: 'Outfit', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Helvetica, Arial, sans-serif;
background-color: #080c14;
background-image: radial-gradient(circle at 12% 18%, rgba(139, 92, 246, 0.05) 0%, transparent 45%),
radial-gradient(circle at 88% 82%, rgba(59, 130, 246, 0.05) 0%, transparent 45%);
color: #f3f4f6;
min-height: 100vh;
overflow-x: hidden;
}
code {
font-family: 'JetBrains Mono', source-code-pro, Menlo, Monaco, Consolas, 'Courier New', monospace;
}
/* Custom scrollbars */
::-webkit-scrollbar {
width: 8px;
height: 8px;
}
::-webkit-scrollbar-track {
background: rgba(0, 0, 0, 0.2);
}
::-webkit-scrollbar-thumb {
background: rgba(255, 255, 255, 0.1);
border-radius: 4px;
}
::-webkit-scrollbar-thumb:hover {
background: rgba(255, 255, 255, 0.2);
}
/* Glassmorphism Panel styles */
.glass-panel {
background: rgba(13, 18, 30, 0.75);
backdrop-filter: blur(20px);
-webkit-backdrop-filter: blur(20px);
border: 1px solid rgba(255, 255, 255, 0.06);
box-shadow: 0 10px 40px 0 rgba(0, 0, 0, 0.45);
}
/* Smooth active pulses */
.pulse-glow-emerald {
box-shadow: 0 0 12px #10b981;
animation: pulse-emerald 2s infinite;
}
@keyframes pulse-emerald {
0% { transform: scale(0.92); opacity: 1; }
50% { transform: scale(1.15); opacity: 0.6; }
100% { transform: scale(0.92); opacity: 1; }
}
.terminal-scroll {
scroll-behavior: smooth;
}

10
rmm-ui/src/main.jsx Normal file
View File

@@ -0,0 +1,10 @@
import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client'
import './index.css'
import App from './App.jsx'
createRoot(document.getElementById('root')).render(
<StrictMode>
<App />
</StrictMode>,
)

12
rmm-ui/tailwind.config.js Normal file
View File

@@ -0,0 +1,12 @@
/** @type {import('tailwindcss').Config} */
export default {
content: [
"./index.html",
"./src/**/*.{js,ts,jsx,tsx}",
],
theme: {
extend: {},
},
plugins: [],
}

11
rmm-ui/vite.config.js Normal file
View File

@@ -0,0 +1,11 @@
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
// https://vite.dev/config/
export default defineConfig({
plugins: [react()],
server: {
host: 'localhost',
port: 8080,
},
})

295
setup_central_server.sh Normal file
View File

@@ -0,0 +1,295 @@
#!/bin/bash
set -e
# ==============================================================================
# CENTRAL SERVER AUTOMATED INSTALLATION SCRIPT
# Installs: Tailscale, NodeJS, MeshCentral, Prometheus, Alertmanager, Grafana
# ==============================================================================
# --- CONFIGURATION VARIABLES ---
SMTP_EMAIL="no-reply@seekright.com"
SMTP_PASSWORD="clzfywewriqxlvqv"
ADMIN_EMAIL="kaushik@seekright.com"
ROCKETCHAT_WEBHOOK="https://text.seekright.com/hooks/6a05a268a88367bc6e0fe318/aNjZzmLy2wB3R4BR7ZtoLtAZYyfNyfLAoQMszmhfBDveKrB"
PROM_VERSION="2.51.2"
AM_VERSION="0.27.0"
# -------------------------------
if [ "$EUID" -ne 0 ]; then
echo "❌ Error: Please run this script with sudo."
exit 1
fi
# Get the actual user who ran sudo, so we don't put MeshCentral in /root
ACTUAL_USER="${SUDO_USER:-root}"
ACTUAL_HOME=$(eval echo ~$ACTUAL_USER)
echo "======================================"
echo "🚀 Starting Central Server Setup..."
echo "======================================"
echo "1. System Preparation..."
apt-get update -yq
# Disable sleep and hibernate (critical for a server)
systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target >/dev/null 2>&1
echo "======================================"
echo "2. Installing Tailscale..."
echo "======================================"
if ! command -v tailscale &> /dev/null; then
curl -fsSL https://tailscale.com/install.sh | sh
fi
echo ""
echo "⚠️ TAILSCALE AUTHENTICATION REQUIRED ⚠️"
echo "A URL will appear below. Open it in your browser and log in."
echo "DO NOT PRESS Ctrl+C! The script will automatically continue after you log in."
echo ""
tailscale up
TAILSCALE_IP=$(tailscale ip -4)
echo "✅ Tailscale connected! Server IP: $TAILSCALE_IP"
echo "======================================"
echo "3. Installing NodeJS & MeshCentral..."
echo "======================================"
if ! command -v node &> /dev/null; then
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -yq nodejs
fi
# Install MeshCentral in the user's home directory
mkdir -p "$ACTUAL_HOME/meshcentral/meshcentral-data"
cd "$ACTUAL_HOME/meshcentral"
# Run npm install as the actual user to avoid root permission issues
sudo -u "$ACTUAL_USER" npm install meshcentral
# Create MeshCentral config
cat > "$ACTUAL_HOME/meshcentral/meshcentral-data/config.json" << EOF
{
"settings": {
"cert": "$TAILSCALE_IP",
"port": 4430,
"redirport": 4431
},
"domains": {
"": {
"title": "My RMM"
}
}
}
EOF
chown -R "$ACTUAL_USER:$ACTUAL_USER" "$ACTUAL_HOME/meshcentral"
# Create MeshCentral Systemd Service
cat > /etc/systemd/system/meshcentral.service << EOF
[Unit]
Description=MeshCentral RMM Server
After=network.target
[Service]
Type=simple
User=$ACTUAL_USER
WorkingDirectory=$ACTUAL_HOME/meshcentral
ExecStart=/usr/bin/node $ACTUAL_HOME/meshcentral/node_modules/meshcentral
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now meshcentral
echo "✅ MeshCentral installed and running on port 4430!"
echo "======================================"
echo "4. Installing Prometheus..."
echo "======================================"
cd /tmp
wget -q https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
tar -xf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
mv prometheus-${PROM_VERSION}.linux-amd64/prometheus /usr/local/bin/
mv prometheus-${PROM_VERSION}.linux-amd64/promtool /usr/local/bin/
mkdir -p /etc/prometheus /var/lib/prometheus
# Prometheus config
cat > /etc/prometheus/prometheus.yml << 'EOF'
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ['localhost:9093']
rule_files:
- "/etc/prometheus/rules.yml"
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
EOF
# Alert Rules
cat > /etc/prometheus/rules.yml << 'EOF'
groups:
- name: server_alerts
rules:
- alert: ServerDown
expr: up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Server {{ $labels.instance }} is DOWN"
description: "{{ $labels.instance }} has been unreachable for over 1 minute."
- alert: HighCPU
expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU on {{ $labels.instance }}"
- alert: HighRAM
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
for: 5m
labels:
severity: warning
annotations:
summary: "High RAM on {{ $labels.instance }}"
- alert: LowDisk
expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
for: 5m
labels:
severity: warning
annotations:
summary: "Low disk on {{ $labels.instance }}"
EOF
# Prometheus Systemd Service
cat > /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Monitoring
After=network.target
[Service]
User=root
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--storage.tsdb.retention.time=30d \
--storage.tsdb.retention.size=15GB \
--web.listen-address=0.0.0.0:9090 \
--web.enable-lifecycle
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now prometheus
echo "✅ Prometheus installed and running on port 9090!"
echo "======================================"
echo "5. Installing Alertmanager..."
echo "======================================"
cd /tmp
wget -q https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz
tar -xf alertmanager-${AM_VERSION}.linux-amd64.tar.gz
mv alertmanager-${AM_VERSION}.linux-amd64/alertmanager /usr/local/bin/
mv alertmanager-${AM_VERSION}.linux-amd64/amtool /usr/local/bin/
mkdir -p /etc/alertmanager /var/lib/alertmanager
# Alertmanager config (Injecting variables)
cat > /etc/alertmanager/alertmanager.yml << EOF
global:
resolve_timeout: 5m
smtp_smarthost: "smtp.gmail.com:587"
smtp_from: "${SMTP_EMAIL}"
smtp_auth_username: "${SMTP_EMAIL}"
smtp_auth_password: "${SMTP_PASSWORD}"
route:
receiver: "fallback-do-nothing"
group_wait: 10s
group_interval: 1m
repeat_interval: 1m
routes:
- receiver: "rocket-chat-alerts"
continue: true
- receiver: "email-alerts"
continue: true
receivers:
- name: "fallback-do-nothing"
- name: "rocket-chat-alerts"
webhook_configs:
- url: "${ROCKETCHAT_WEBHOOK}"
send_resolved: true
- name: "email-alerts"
email_configs:
- to: "${ADMIN_EMAIL}"
send_resolved: true
EOF
# Alertmanager Systemd Service
cat > /etc/systemd/system/alertmanager.service << 'EOF'
[Unit]
Description=Alertmanager
After=network.target
[Service]
User=root
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yml \
--storage.path=/var/lib/alertmanager
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now alertmanager
echo "✅ Alertmanager installed and running on port 9093!"
echo "======================================"
echo "6. Installing Grafana..."
echo "======================================"
if ! command -v grafana-server &> /dev/null; then
apt-get install -y apt-transport-https software-properties-common wget
mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | tee /etc/apt/sources.list.d/grafana.list
apt-get update -yq
apt-get install -yq grafana
fi
systemctl enable --now grafana-server
echo "✅ Grafana installed and running on port 3000!"
echo "======================================"
echo "🔍 Verifying All Services..."
echo "======================================"
sleep 3
echo "Service Status:"
systemctl is-active tailscaled meshcentral prometheus alertmanager grafana-server | sed 's/^/ - /'
echo "======================================"
echo "🎉 Central Server Installation Complete!"
echo "======================================"
echo "Access your services:"
echo "• MeshCentral: https://$TAILSCALE_IP:4430"
echo "• Grafana: http://$TAILSCALE_IP:3000 (login: admin/admin)"
echo "• Prometheus: http://$TAILSCALE_IP:9090"
echo "• Alerts: http://$TAILSCALE_IP:9093"
echo "======================================"

168
status.md Normal file
View File

@@ -0,0 +1,168 @@
# PROMETHEUS LEARNING STATUS
Last Updated: 2026-05-11 14:44:00
Current IDE: VS Code
## Goal
Monitor 7 remote Linux servers from one central server.
*Currently in testing phase using 2 Windows systems (Laptop & Desktop).*
Central server: [Windows Laptop/Desktop - PENDING IP]
Remote servers:
- Test Server 1 (Windows): [PENDING IP]
- Prod Servers 1-7 (Linux): [PENDING IPs]
## Current Position
Layer: Layer 4 — PromQL
Concept: Basic Syntax and Instant Vectors
Drill: not done
## Setup Status
- [ ] Prometheus installed on central server
- [ ] Prometheus UI accessible at port 9090
- [ ] Grafana installed on central server
- [ ] Grafana UI accessible at port 3000
- [ ] Alertmanager installed on central server
- [ ] Node Exporter on server 1: [PENDING]
- [ ] Node Exporter on server 2: [PENDING]
- [ ] Node Exporter on server 3: [PENDING]
- [ ] Node Exporter on server 4: [PENDING]
- [ ] Node Exporter on server 5: [PENDING]
- [ ] Node Exporter on server 6: [PENDING]
- [ ] Node Exporter on server 7: [PENDING]
- [ ] All 7 servers showing UP in Prometheus Targets
- [ ] Node Exporter Full dashboard imported in Grafana
- [ ] Alert rules configured and showing in Prometheus
- [ ] Alert notification received successfully
## Layer Progress
### Layer 1 — Mental Model
- [x] Pull vs push model
- [x] Time series data concept
- [x] Four components understood
- [x] Architecture drawn for 7 servers
- [x] Four metric types
Revision Task: [x] DONE
### Layer 2 — Installing Prometheus
- [x] File system structure
- [x] Download and extract
- [x] prometheus user created
- [x] Directories and permissions
- [x] Binaries in correct location
- [x] First prometheus.yml written
- [x] systemd service created
- [x] Prometheus running and accessible
Revision Task: [x] DONE
### Layer 3 — Node Exporter
- [x] What Node Exporter does
- [x] /metrics format understood
- [x] Installation on remote server
- [ ] systemd service for node_exporter
- [ ] /metrics verified with curl
- [ ] Firewall configured correctly
- [x] Connectivity from central server tested
- [x] Server added to prometheus.yml
- [x] Target showing UP in Prometheus
- [ ] Repeated for all 7 servers
Revision Task: [x] DONE
### Layer 4 — PromQL
- [x] Instant vs range vectors
- [x] Label selectors
- [x] rate() function
- [x] CPU query
- [x] RAM query
- [x] Disk query
- [x] Network query
- [x] Up/Down query
- [x] Aggregation operators
- [ ] Expression browser used
Revision Task: [ ] NOT DONE
### Layer 5 — Grafana
- [x] Installation
- [x] Prometheus data source added
- [x] Dashboard 1860 imported
- [ ] Custom dashboard created
- [ ] Variables configured
- [ ] 6 panels built
Revision Task: [ ] NOT DONE
### Layer 6 — Alertmanager
- [ ] Installation
- [ ] alertmanager.yml configured
- [ ] Email or Slack receiver set up
- [ ] Alert rules file created
- [ ] ServerDown alert written
- [ ] HighCPU alert written
- [ ] DiskSpaceLow alert written
- [ ] HighMemory alert written
- [ ] Alertmanager linked to Prometheus
- [ ] Test alert received
Revision Task: [ ] NOT DONE
### Layer 7 — Config Deep Dive
- [ ] scrape_configs deep dive
- [ ] Labels mastered
- [ ] Relabeling understood
- [ ] Recording rules
- [ ] promtool validation
Revision Task: [ ] NOT DONE
### Layer 8 — Security
- [ ] TLS for Prometheus UI
- [ ] Basic auth configured
- [ ] nginx reverse proxy
- [ ] Node Exporter access restricted
- [ ] Storage retention configured
- [ ] Backup strategy in place
Revision Task: [ ] NOT DONE
### Layer 9 — Docker Setup
- [ ] docker-compose.yml written
- [ ] Volumes configured
- [ ] Stack running in Docker
- [ ] Node Exporter Docker vs bare metal decision
Revision Task: [ ] NOT DONE
### Layer 10 — Advanced Features
- [ ] file_sd_configs
- [ ] Cloud service discovery concept
- [ ] Pushgateway concept
- [ ] Additional exporters
- [ ] Custom exporter in Python
- [ ] Federation concept
- [ ] Long-term storage concept
Revision Task: [ ] NOT DONE
### Layer 11 — Troubleshooting
- [ ] Target DOWN debugging steps
- [ ] Metrics wrong debugging
- [ ] Disk usage management
- [ ] Memory management
- [ ] Alert not firing debugging
- [ ] Grafana no data debugging
- [ ] Config validation process
Revision Task: [ ] NOT DONE
### Layer 12 — Apex Project
- [ ] Full setup from scratch — complete
## My Weak Spots
[None yet]
## Bridge Map Learned
- Pull Model -> Central server running `setInterval()` with Axios GET requests to 7 dumb servers.
- Time Series Data -> `metric_name{labels} value timestamp` (Labels are like MongoDB fields).
- Architecture -> Remote: Node Exporter. Central: Prometheus, Grafana, Alertmanager. PULL direction.
- Linux Folders -> `/etc` (configs/.env), `/var/lib` (MongoDB storage), `/usr/local/bin` (node.exe executable).
## Common Errors Encountered
[None yet]
## Spaced Repetition Queue
[None yet]
## IDE Transfer Log
[None yet]

31
tasks.md Normal file
View File

@@ -0,0 +1,31 @@
# PROMETHEUS LEARNING TASKS
Last Updated: 2026-05-11 14:44:00
## RIGHT NOW
- [ ] Learn how to download binaries via the Linux terminal (`wget`).
## SERVERS TO CONNECT
[ ] Server 1: [PENDING] — Node Exporter status
[ ] Server 2: [PENDING] — Node Exporter status
[ ] Server 3: [PENDING] — Node Exporter status
[ ] Server 4: [PENDING] — Node Exporter status
[ ] Server 5: [PENDING] — Node Exporter status
[ ] Server 6: [PENDING] — Node Exporter status
[ ] Server 7: [PENDING] — Node Exporter status
## UP NEXT — Current Layer Remaining
Layer 4 — PromQL
- Layer 4 Revision Task
## Pending Drills
[None yet]
## Errors To Resolve
[None yet]
## Revision Tasks Due
[None yet]
## Session Goals
- Initialize learning workspace
- Complete Layer 1: Prometheus Mental Model

34
test_install.sh Normal file
View File

@@ -0,0 +1,34 @@
#!/bin/bash
set -e
echo "Starting Dummy Installation Test..."
if [ "$EUID" -ne 0 ]; then
echo "❌ Error: Please run this script with sudo."
exit 1
fi
echo "======================================"
echo "1. Installing test tools ('cowsay' and 'htop')..."
echo "======================================"
# Update package list silently and install tools
apt-get update -yq
apt-get install -yq cowsay htop
echo "======================================"
echo "2. Verifying Installation..."
echo "======================================"
if command -v htop &> /dev/null; then
echo "✅ 'htop' installed successfully! (Type 'htop' in the terminal to see a cool task manager)"
fi
if command -v /usr/games/cowsay &> /dev/null; then
echo "✅ 'cowsay' installed successfully!"
/usr/games/cowsay "Hello from the automated script! If you see this, the remote execution test was a massive success!"
fi
echo "======================================"
echo "🎉 Test Complete!"
echo "To clean up and remove these test tools later, just run:"
echo "sudo apt-get remove --purge -y cowsay htop"
echo "======================================"