RMM-UI-BE/09_central_system_setup.md

Central Monitoring System — Complete Setup

Prometheus + Grafana + Alertmanager + systemd Process Management

Run every command below on your central Linux server (native Linux, not WSL).

---

Prerequisites

• Ubuntu 22.04 or 24.04 LTS (native, always-on machine)
• You have already installed Tailscale and MeshCentral (see 08_central_server_production_setup.md)
• Your Tailscale IP is noted (run tailscale ip -4 to get it)
• You are running all commands as your normal user (use sudo where shown)

---

Part 1: Prometheus

Step 1 — Download and Install

cd /tmp
PROM_VERSION="2.51.2"
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
tar xvf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
cd prometheus-${PROM_VERSION}.linux-amd64

sudo mv prometheus promtool /usr/local/bin/
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo mv prometheus.yml consoles console_libraries /etc/prometheus/

Verify:

prometheus --version
# Should print: prometheus, version 2.51.2

Step 2 — Write the Config File

sudo tee /etc/prometheus/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

rule_files:
  - "/etc/prometheus/rules.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Add each client machine's Tailscale IP here after connecting them
  # - job_name: 'node_exporter'
  #   static_configs:
  #     - targets:
  #         - '100.x.x.x:9100'   # machine name
  #         - '100.x.x.x:9100'   # machine name
EOF

Step 3 — Write Alert Rules

sudo tee /etc/prometheus/rules.yml << 'EOF'
groups:
  - name: server_alerts
    rules:
      - alert: ServerDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Server {{ $labels.instance }} is DOWN"
          description: "{{ $labels.instance }} has been unreachable for over 1 minute."

      - alert: HighCPU
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "CPU usage is above 85% for 5 minutes."

      - alert: HighRAM
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High RAM on {{ $labels.instance }}"
          description: "RAM usage is above 90% for 5 minutes."

      - alert: LowDisk
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk on {{ $labels.instance }}"
          description: "Root disk has less than 15% free space."

      - alert: HighGPUTemp
        expr: nvidia_smi_temperature_gpu > 85
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "GPU overheating on {{ $labels.instance }}"
          description: "GPU temperature is above 85°C."
EOF

Step 4 — Create systemd Service

sudo tee /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Monitoring
Documentation=https://prometheus.io/docs/
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=10GB \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-lifecycle
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Expected: active (running)

Verify in browser: http://localhost:9090

Step 5 — Adding a New Client to Prometheus

After connecting a client machine via Tailscale and deploying Node Exporter:

sudo nano /etc/prometheus/prometheus.yml

Add the machine under scrape_configs:

- job_name: "node_exporter"
  static_configs:
    - targets:
        - "100.x.x.x:9100" # machine hostname

Reload without restarting (since --web.enable-lifecycle is set):

curl -X POST http://localhost:9090/-/reload

Check http://localhost:9090/targets — new machine should appear green.

---

Part 2: Alertmanager

Step 1 — Download and Install

cd /tmp
AM_VERSION="0.27.0"
wget https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz
tar xvf alertmanager-${AM_VERSION}.linux-amd64.tar.gz
cd alertmanager-${AM_VERSION}.linux-amd64

sudo mv alertmanager amtool /usr/local/bin/
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager

Step 2 — Configure Alertmanager

sudo tee /etc/alertmanager/alertmanager.yml << 'EOF'
global:
  resolve_timeout: 5m
  smtp_smarthost: "smtp.gmail.com:587"
  smtp_from: "no-reply@seekright.com"
  smtp_auth_username: "no-reply@seekright.com"
  smtp_auth_password: "clzfywewriqxlvqv"

route:
  receiver: "fallback-do-nothing"
  group_wait: 10s        # ← ADD THIS
  group_interval: 1m     # ← ADD THIS (must be ≤ repeat_interval)
  repeat_interval: 1m    # now this actually works
  routes:
    - receiver: "rocket-chat-alerts"
      continue: true
    - receiver: "email-alerts"
      continue: true

receivers:
  - name: "fallback-do-nothing"

  - name: "rocket-chat-alerts"
    webhook_configs:                             # Changed from slack_configs
      - url: "https://text.seekright.com/hooks/6a05a268a88367bc6e0fe318/aNjZzmLy2wB3R4BR7ZtoLtAZYyfNyfLAoQMszmhfBDveKrB>
        send_resolved: true
  - name: "email-alerts"
EOF

Replace:

• YOUR_GMAIL@gmail.com — your Gmail address
• YOUR_GMAIL_APP_PASSWORD — Gmail App Password (not your login password — create one at myaccount.google.com → Security → App Passwords)
• YOUR_ROCKETCHAT_WEBHOOK_URL — your Rocket.Chat incoming webhook URL

Step 3 — Create systemd Service

sudo tee /etc/systemd/system/alertmanager.service << 'EOF'
[Unit]
Description=Alertmanager
Documentation=https://prometheus.io/docs/alerting/alertmanager/
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=0.0.0.0:9093
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager

Verify in browser: http://localhost:9093

---

Part 3: Grafana

Step 1 — Install via apt (Official Repo)

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Verify in browser: http://localhost:3000 Default login: admin / admin — change password immediately on first login.

Step 2 — Connect Grafana to Prometheus

1. Go to Connections → Data Sources → Add data source
2. Choose Prometheus
3. Set URL: http://localhost:9090
4. Click Save & Test — must show "Data source is working"

Step 3 — Import Dashboards

Go to Dashboards → Import and import these IDs one by one:

Dashboard ID	What It Shows
1860	Node Exporter Full — CPU, RAM, Disk, Network for all machines
14574	NVIDIA GPU Metrics — GPU temp, VRAM, utilization

---

Part 4: Process Management Reference

Check all service statuses at once

sudo systemctl status prometheus alertmanager grafana-server

Start / Stop / Restart any service

sudo systemctl start prometheus
sudo systemctl stop prometheus
sudo systemctl restart prometheus

sudo systemctl start alertmanager
sudo systemctl stop alertmanager
sudo systemctl restart alertmanager

sudo systemctl start grafana-server
sudo systemctl stop grafana-server
sudo systemctl restart grafana-server

Check live logs for any service

sudo journalctl -u prometheus -f
sudo journalctl -u alertmanager -f
sudo journalctl -u grafana-server -f

Reload Prometheus config without restarting

curl -X POST http://localhost:9090/-/reload
# Only works because we set --web.enable-lifecycle

Check all listening ports

ss -tlnp | grep -E "9090|9093|3000|4430"

Check all enabled services (confirms they survive reboot)

sudo systemctl is-enabled prometheus alertmanager grafana-server
# All three must print: enabled

---

Part 5: Final Verification Checklist

Run this after complete setup to confirm everything is working:

echo "=== Service Status ===" && \
sudo systemctl is-active prometheus alertmanager grafana-server && \
echo "=== Prometheus Targets ===" && \
curl -s http://localhost:9090/api/v1/targets | python3 -c "import sys,json; [print(t['labels']['job'], t['health']) for t in json.load(sys.stdin)['data']['activeTargets']]" && \
echo "=== Ports Listening ===" && \
ss -tlnp | grep -E "9090|9093|3000"

Expected output:

=== Service Status ===
active
active
active
=== Prometheus Targets ===
prometheus up
node_exporter up
=== Ports Listening ===
LISTEN  *:9090   prometheus
LISTEN  *:9093   alertmanager
LISTEN  *:3000   grafana

---

Quick Reference

Service	Port	Config File	Log Command
Prometheus	9090	/etc/prometheus/prometheus.yml	journalctl -u prometheus -f
Alertmanager	9093	/etc/alertmanager/alertmanager.yml	journalctl -u alertmanager -f
Grafana	3000	/etc/grafana/grafana.ini	journalctl -u grafana-server -f
MeshCentral	4430	~/meshcentral/meshcentral-data/config.json	journalctl -u meshcentral -f
Tailscale	—	—	tailscale status