RMM-UI-BE/08_central_server_production_setup.md

Central Server Production Setup

Fresh Linux Machine → Full Monitoring + RMM Stack

---

What This Document Covers

Setting up your central server from a fresh Ubuntu/PopOS Linux install with:

• Tailscale — private network backbone connecting all machines
• MeshCentral — RMM dashboard for remote management of all client machines
• Prometheus — metrics collection engine
• Alertmanager — alert routing to Rocket.Chat and Gmail
• Grafana — visualization dashboards

---

Requirements

Requirement	Minimum	Recommended
OS	Ubuntu 22.04 / 24.04 LTS	Ubuntu 24.04 LTS (native, NOT WSL)
RAM	2 GB	4 GB
Disk	20 GB	50 GB
CPU	2 cores	4 cores
Network	Always-on internet	Always-on internet
Power	No sleep/hibernate	UPS recommended

⚠️ WSL is for testing only. For production, use a native Linux machine (bare metal or VM). WSL does not auto-start services on Windows boot and has double-NAT networking issues.

---

Accounts You Need Before Starting

1. Tailscale Account (Free)

• Go to: https://tailscale.com
• Click Get Started
• Sign in with Google or GitHub — whichever you use on all machines
• You do not need to configure anything on the website — just create the account
• Free plan supports up to 100 machines, which is exactly what we need

2. Tailscale is the ONLY external account required

MeshCentral, Prometheus, Alertmanager, and Grafana are all self-hosted on your machine. No cloud accounts needed for any of them.

---

Part 1: System Preparation

Update the system

sudo apt-get update && sudo apt-get upgrade -y

Disable sleep and hibernate (critical for a server)

sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

Create a dedicated user for running services (optional but recommended)

sudo useradd -r -s /bin/false prometheus
sudo useradd -r -s /bin/false alertmanager

---

Part 2: Install Tailscale

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

⚠️ A URL will appear. Open it in a browser and log in with your Tailscale account. DO NOT press Ctrl+C — wait for Success.

Get your server's stable Tailscale IP:

tailscale ip -4
# Example: 100.66.205.2  ← save this, you will use it everywhere

Verify Tailscale starts on boot (it does by default):

sudo systemctl is-enabled tailscaled
# Should print: enabled

---

Part 3: Install NodeJS and MeshCentral

Install NodeJS v20

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs
node --version   # Should show v20.x.x

Install MeshCentral locally

mkdir ~/meshcentral
cd ~/meshcentral
npm install meshcentral

⚠️ Do NOT use sudo npm install -g meshcentral — permission errors will occur.

Configure MeshCentral

Replace YOUR_TAILSCALE_IP with the IP from Part 2:

mkdir -p ~/meshcentral/meshcentral-data
cat > ~/meshcentral/meshcentral-data/config.json << 'EOF'
{
  "settings": {
    "cert": "YOUR_TAILSCALE_IP",
    "port": 4430,
    "redirport": 4431
  },
  "domains": {
    "": {
      "title": "My RMM"
    }
  }
}
EOF

Test MeshCentral starts correctly

cd ~/meshcentral && node node_modules/meshcentral

Expected output:

MeshCentral HTTP redirection server running on port 4431.
MeshCentral v1.1.x, Hybrid (LAN + WAN) mode.
MeshCentral HTTPS server running on YOUR_TAILSCALE_IP:4430.

Open browser at https://localhost:4430 — accept the certificate warning. Create your admin account — the first account created is automatically the Site Administrator. Then press Ctrl+C to stop it. We will run it as a service next.

Run MeshCentral as a systemd service (auto-starts on reboot)

cat > /tmp/meshcentral.service << EOF
[Unit]
Description=MeshCentral RMM Server
After=network.target

[Service]
Type=simple
User=$USER
WorkingDirectory=$HOME/meshcentral
ExecStart=/usr/bin/node $HOME/meshcentral/node_modules/meshcentral
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo mv /tmp/meshcentral.service /etc/systemd/system/meshcentral.service
sudo systemctl daemon-reload
sudo systemctl enable meshcentral
sudo systemctl start meshcentral
sudo systemctl status meshcentral

Verify it is running:

sudo systemctl status meshcentral
# Must show: active (running)

---

Part 4: Install Prometheus

Download and install

cd /tmp
PROM_VERSION="2.51.2"
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
tar xvf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
sudo mv prometheus-${PROM_VERSION}.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-${PROM_VERSION}.linux-amd64/promtool /usr/local/bin/
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo mv prometheus-${PROM_VERSION}.linux-amd64/prometheus.yml /etc/prometheus/
sudo mv prometheus-${PROM_VERSION}.linux-amd64/consoles /etc/prometheus/
sudo mv prometheus-${PROM_VERSION}.linux-amd64/console_libraries /etc/prometheus/

Configure prometheus.yml

sudo nano /etc/prometheus/prometheus.yml

Replace the entire file contents with:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

rule_files:
  - "/etc/prometheus/rules.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Add client machines here as you connect them via Tailscale
  # - job_name: 'node_exporter'
  #   static_configs:
  #     - targets:
  #       - '100.119.113.98:9100'   # bs13
  #       - '100.76.222.82:9100'    # pop-os / byfar

Create alert rules file

sudo tee /etc/prometheus/rules.yml << 'EOF'
groups:
  - name: server_alerts
    rules:
      - alert: ServerDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Server {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} has been unreachable for more than 1 minute."

      - alert: HighCPU
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"

      - alert: HighRAM
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High RAM on {{ $labels.instance }}"

      - alert: LowDisk
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk on {{ $labels.instance }}"
EOF

Run Prometheus as a systemd service

sudo tee /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Monitoring
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=10GB \
  --web.listen-address=0.0.0.0:9090
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Verify: Open http://localhost:9090 in your browser — you should see the Prometheus UI.

---

Part 5: Install Alertmanager

Download and install

cd /tmp
AM_VERSION="0.27.0"
wget https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz
tar xvf alertmanager-${AM_VERSION}.linux-amd64.tar.gz
sudo mv alertmanager-${AM_VERSION}.linux-amd64/alertmanager /usr/local/bin/
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager

Configure Alertmanager

sudo tee /etc/alertmanager/alertmanager.yml << 'EOF'
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'default'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'YOUR_ROCKETCHAT_WEBHOOK_URL'
        send_resolved: true
EOF

Replace YOUR_ROCKETCHAT_WEBHOOK_URL with your actual Rocket.Chat webhook URL.

Run as systemd service

sudo tee /etc/systemd/system/alertmanager.service << 'EOF'
[Unit]
Description=Alertmanager
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager

---

Part 6: Install Grafana

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Verify: Open http://localhost:3000 — default login is admin / admin. Change the password on first login.

Connect Grafana to Prometheus

1. Go to Connections → Data Sources → Add data source
2. Select Prometheus
3. Set URL to http://localhost:9090
4. Click Save & Test — should show "Data source is working"

Import recommended dashboards

• Go to Dashboards → Import
• Enter ID 1860 → Load → Import (Node Exporter Full)
• Enter ID 14574 → Load → Import (NVIDIA GPU Metrics)

---

Part 7: Verify All Services Are Running

sudo systemctl status meshcentral prometheus alertmanager grafana-server tailscaled

All five should show active (running).

Quick port check:

ss -tlnp | grep -E "4430|9090|9093|3000"

Expected:

LISTEN  *:4430   → MeshCentral
LISTEN  *:9090   → Prometheus
LISTEN  *:9093   → Alertmanager
LISTEN  *:3000   → Grafana

---

Part 8: Adding Client Machines

For each new client machine, see 07_connect_remote_machine.md.

After the client is connected to MeshCentral and Tailscale, add its Tailscale IP to Prometheus:

sudo nano /etc/prometheus/prometheus.yml

Add under scrape_configs:

  - job_name: 'node_exporter'
    static_configs:
      - targets:
          - '100.x.x.x:9100'    # machine name

Reload Prometheus without restarting:

curl -X POST http://localhost:9090/-/reload

---

Part 9: Production Checklist

• Native Linux (not WSL) — services survive reboots
• All 5 services enabled with systemctl enable
• Sleep/hibernate disabled
• MeshCentral admin account created and saved
• Tailscale logged in and showing Connected
• Grafana default password changed
• At least one client machine connected and showing in Prometheus targets
• At least one Grafana dashboard imported and showing data
• Test alert fired and received in Rocket.Chat

---

Quick Reference

Service	Port	URL
MeshCentral	4430	https://localhost:4430
Prometheus	9090	http://localhost:9090
Alertmanager	9093	http://localhost:9093
Grafana	3000	http://localhost:3000
Tailscale	—	tailscale status

Command	Purpose
sudo systemctl restart meshcentral	Restart RMM server
sudo systemctl restart prometheus	Restart metrics collector
curl -X POST http://localhost:9090/-/reload	Reload Prometheus config without restart
tailscale ip -4	Get this machine's Tailscale IP
tailscale status	See all connected machines