146 lines
9.4 KiB
Markdown
146 lines
9.4 KiB
Markdown
# Global Monitoring Architecture — Full Roadmap
|
|
|
|
## What We Are Building (The Big Picture)
|
|
|
|
A fully automated, zero-touch monitoring system that:
|
|
|
|
- Tracks CPU, RAM, Disk, GPU health of 100+ client computers across the world.
|
|
- Sends instant alerts to Rocket.Chat and Email when a machine goes offline or overheats.
|
|
- Lets you remotely execute scripts on any or all machines without touching them physically.
|
|
- Scales from 10 to 1000 machines without changing the core architecture.
|
|
|
|
---
|
|
|
|
## The Complete Architecture Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ CLIENT MACHINES (100+) │
|
|
│ Each machine runs 2 silent background agents: │
|
|
│ ┌──────────────────────┐ ┌──────────────────────────┐ │
|
|
│ │ Node Exporter :9100 │ │ MeshCentral Agent │ │
|
|
│ │ (Hardware Metrics) │ │ (RMM Command Channel) │ │
|
|
│ └──────────────────────┘ └──────────────────────────┘ │
|
|
└──────────────────────────────────────┬──────────────────────────────┘
|
|
│ Both agents "call home"
|
|
│ through client's firewall
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ CLOUDFLARE (Free Global Relay) │
|
|
│ Acts as a secure public-facing entry point for your private server │
|
|
│ Gives your desktop a real public URL (e.g. rmm.yourcompany.com) │
|
|
└──────────────────────────────────────┬──────────────────────────────┘
|
|
│ Cloudflare Tunnel
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ YOUR CENTRAL SERVER (Always-On Linux) │
|
|
│ │
|
|
│ ┌─────────────────┐ ┌────────────────┐ ┌──────────────────────┐ │
|
|
│ │ MeshCentral │ │ Prometheus │ │ Grafana Dashboard │ │
|
|
│ │ RMM Dashboard │ │ :9090 │ │ :3000 │ │
|
|
│ │ │ │ │ │ │ │
|
|
│ │ - See all 100 │ │ - Scrapes │ │ - Visual graphs of │ │
|
|
│ │ machines │ │ metrics │ │ CPU/RAM/GPU │ │
|
|
│ │ - Open remote │ │ from all │ │ - Real-time and │ │
|
|
│ │ terminals │ │ clients │ │ historical data │ │
|
|
│ │ - Push scripts │ │ - Evaluates │ └──────────────────────┘ │
|
|
│ │ to all │ │ alert rules │ │
|
|
│ └─────────────────┘ └───────┬────────┘ ┌──────────────────────┐ │
|
|
│ │ │ Alertmanager │ │
|
|
│ └──────────►│ :9093 │ │
|
|
│ │ │ │
|
|
│ │ Routes alerts to: │ │
|
|
│ │ - Rocket.Chat │ │
|
|
│ │ - Gmail │ │
|
|
│ └──────────────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## What Software is Required (and Why)
|
|
|
|
### On Every Client Machine (100+ remote desktops)
|
|
|
|
| Software | Purpose | Cost |
|
|
| ----------------------- | ------------------------------------------------------------------------------------- | ---- |
|
|
| **Node Exporter** | Exposes CPU, RAM, Disk, Network metrics on port 9100 | Free |
|
|
| **Nvidia GPU Exporter** | Exposes GPU temp and VRAM metrics on port 9835 (only if GPU exists) | Free |
|
|
| **MeshCentral Agent** | Maintains a silent reverse tunnel back to your RMM server for remote script execution | Free |
|
|
|
|
### On Your Central Server (Always-On Linux Machine)
|
|
|
|
| Software | Purpose | Cost |
|
|
| ------------------------------------- | ------------------------------------------------------------------------------ | ---- |
|
|
| **MeshCentral** | RMM dashboard — deploy scripts to all clients in one click | Free |
|
|
| **Cloudflare Tunnel (`cloudflared`)** | Creates a secure public URL for your local server without touching your router | Free |
|
|
| **Prometheus** | Scrapes and stores time-series hardware metrics from all clients | Free |
|
|
| **Alertmanager** | Receives firing alerts from Prometheus and routes to Rocket.Chat/Gmail | Free |
|
|
| **Grafana** | Connects to Prometheus and draws real-time dashboards | Free |
|
|
|
|
---
|
|
|
|
## The Build Phases (In Order)
|
|
|
|
### Phase 1: Central Server Foundation ✅ DONE
|
|
|
|
- Install Prometheus, Alertmanager, Grafana on central server.
|
|
- Configure `alertmanager.yml` with Rocket.Chat webhook and Gmail SMTP.
|
|
- Write `rules.yml` with the `ServerDown` alert.
|
|
|
|
### Phase 2: Remote Agent Automation ✅ DONE
|
|
|
|
- Write `deploy_exporters.sh` to silently install Node Exporter and GPU Exporter.
|
|
- Test the script on a remote machine via RustDesk.
|
|
|
|
### Phase 3: MeshCentral RMM Setup (CURRENT PHASE)
|
|
|
|
- Install NodeJS on your central Linux server.
|
|
- Install and configure MeshCentral.
|
|
- Expose it to the internet via a Cloudflare Tunnel.
|
|
- Install the MeshCentral Agent on one test machine.
|
|
- Verify you can push a script remotely from the web dashboard.
|
|
|
|
### Phase 4: Networking — Connect Prometheus to Remote Clients
|
|
|
|
- Install Tailscale (mesh VPN) on the Central Server and all 100 client machines.
|
|
- Each client gets a stable private IP (e.g., `100.85.x.x`).
|
|
- Add all 100 Tailscale IPs to `prometheus.yml` as scrape targets.
|
|
|
|
### Phase 5: Scale and Polish
|
|
|
|
- Add all 100 machines to MeshCentral.
|
|
- Use MeshCentral's "Run Script on Group" feature to deploy `deploy_exporters.sh` to all machines simultaneously.
|
|
- Import Grafana dashboard templates (Node Exporter ID: `1860`, GPU ID: `14574`).
|
|
- Set up Grafana Alerting for CPU/RAM thresholds.
|
|
|
|
---
|
|
|
|
## WSL vs Production Linux — What Works Where
|
|
|
|
| Feature | WSL (Testing) | Native Linux (Production) |
|
|
| ----------------------------------------- | ---------------------------- | ------------------------- |
|
|
| Install MeshCentral | ✅ Yes | ✅ Yes |
|
|
| Open MeshCentral in browser | ✅ Yes | ✅ Yes |
|
|
| Install MeshCentral Agent on same machine | ✅ Yes | ✅ Yes |
|
|
| Expose to internet via Cloudflare Tunnel | ⚠️ Tricky (WSL networking) | ✅ Easy |
|
|
| Run as background service on reboot | ❌ WSL doesn't auto-start | ✅ Yes (systemd) |
|
|
| Accept real remote client connections | ⚠️ Possible with workarounds | ✅ Yes |
|
|
|
|
**Conclusion:** WSL is perfect for Phase 3 learning and testing. You can install MeshCentral, learn the dashboard, test pushing scripts locally, and understand how everything works. When you are ready to go live with real clients, you simply run the same commands on a native Linux machine and it works perfectly.
|
|
|
|
---
|
|
|
|
## Immediate Next Step
|
|
|
|
Install NodeJS on your WSL terminal and deploy MeshCentral locally.
|
|
|
|
```bash
|
|
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
|
|
sudo apt-get install -y nodejs
|
|
sudo npm install -g meshcentral
|
|
node /usr/lib/node_modules/meshcentral
|
|
```
|
|
|
|
Then open `http://localhost:4430` in your browser to see the MeshCentral dashboard!
|