RMM-UI-BE/08_central_server_production_setup.md

# Central Server Production Setup
## Fresh Linux Machine → Full Monitoring + RMM Stack

---

## What This Document Covers

Setting up your central server from a fresh Ubuntu/PopOS Linux install with:
- **Tailscale** — private network backbone connecting all machines
- **MeshCentral** — RMM dashboard for remote management of all client machines
- **Prometheus** — metrics collection engine
- **Alertmanager** — alert routing to Rocket.Chat and Gmail
- **Grafana** — visualization dashboards

---

## Requirements

| Requirement | Minimum | Recommended |
|---|---|---|
| OS | Ubuntu 22.04 / 24.04 LTS | Ubuntu 24.04 LTS (native, NOT WSL) |
| RAM | 2 GB | 4 GB |
| Disk | 20 GB | 50 GB |
| CPU | 2 cores | 4 cores |
| Network | Always-on internet | Always-on internet |
| Power | No sleep/hibernate | UPS recommended |

> ⚠️ **WSL is for testing only.** For production, use a native Linux machine (bare metal or VM).
> WSL does not auto-start services on Windows boot and has double-NAT networking issues.

---

## Accounts You Need Before Starting

### 1. Tailscale Account (Free)
- Go to: **https://tailscale.com**
- Click **Get Started**
- Sign in with Google or GitHub — whichever you use on all machines
- You do not need to configure anything on the website — just create the account
- Free plan supports up to 100 machines, which is exactly what we need

### 2. Tailscale is the ONLY external account required
MeshCentral, Prometheus, Alertmanager, and Grafana are all self-hosted on your machine.
No cloud accounts needed for any of them.

---

## Part 1: System Preparation

### Update the system
```bash
sudo apt-get update && sudo apt-get upgrade -y
```

### Disable sleep and hibernate (critical for a server)
```bash
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
```

### Create a dedicated user for running services (optional but recommended)
```bash
sudo useradd -r -s /bin/false prometheus
sudo useradd -r -s /bin/false alertmanager
```

---

## Part 2: Install Tailscale

```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
```

> ⚠️ A URL will appear. Open it in a browser and log in with your Tailscale account.
> **DO NOT press Ctrl+C** — wait for `Success.`

Get your server's stable Tailscale IP:
```bash
tailscale ip -4
# Example: 100.66.205.2  ← save this, you will use it everywhere
```

Verify Tailscale starts on boot (it does by default):
```bash
sudo systemctl is-enabled tailscaled
# Should print: enabled
```

---

## Part 3: Install NodeJS and MeshCentral

### Install NodeJS v20
```bash
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs
node --version   # Should show v20.x.x
```

### Install MeshCentral locally
```bash
mkdir ~/meshcentral
cd ~/meshcentral
npm install meshcentral
```

> ⚠️ Do NOT use `sudo npm install -g meshcentral` — permission errors will occur.

### Configure MeshCentral
Replace `YOUR_TAILSCALE_IP` with the IP from Part 2:
```bash
mkdir -p ~/meshcentral/meshcentral-data
cat > ~/meshcentral/meshcentral-data/config.json << 'EOF'
{
  "settings": {
    "cert": "YOUR_TAILSCALE_IP",
    "port": 4430,
    "redirport": 4431
  },
  "domains": {
    "": {
      "title": "My RMM"
    }
  }
}
EOF
```

### Test MeshCentral starts correctly
```bash
cd ~/meshcentral && node node_modules/meshcentral
```

Expected output:
```
MeshCentral HTTP redirection server running on port 4431.
MeshCentral v1.1.x, Hybrid (LAN + WAN) mode.
MeshCentral HTTPS server running on YOUR_TAILSCALE_IP:4430.
```

Open browser at `https://localhost:4430` — accept the certificate warning.
**Create your admin account** — the first account created is automatically the Site Administrator.
Then press **Ctrl+C** to stop it. We will run it as a service next.

### Run MeshCentral as a systemd service (auto-starts on reboot)
```bash
cat > /tmp/meshcentral.service << EOF
[Unit]
Description=MeshCentral RMM Server
After=network.target

[Service]
Type=simple
User=$USER
WorkingDirectory=$HOME/meshcentral
ExecStart=/usr/bin/node $HOME/meshcentral/node_modules/meshcentral
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo mv /tmp/meshcentral.service /etc/systemd/system/meshcentral.service
sudo systemctl daemon-reload
sudo systemctl enable meshcentral
sudo systemctl start meshcentral
sudo systemctl status meshcentral
```

Verify it is running:
```bash
sudo systemctl status meshcentral
# Must show: active (running)
```

---

## Part 4: Install Prometheus

### Download and install
```bash
cd /tmp
PROM_VERSION="2.51.2"
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
tar xvf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
sudo mv prometheus-${PROM_VERSION}.linux-amd64/prometheus /usr/local/bin/
sudo mv prometheus-${PROM_VERSION}.linux-amd64/promtool /usr/local/bin/
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo mv prometheus-${PROM_VERSION}.linux-amd64/prometheus.yml /etc/prometheus/
sudo mv prometheus-${PROM_VERSION}.linux-amd64/consoles /etc/prometheus/
sudo mv prometheus-${PROM_VERSION}.linux-amd64/console_libraries /etc/prometheus/
```

### Configure prometheus.yml
```bash
sudo nano /etc/prometheus/prometheus.yml
```

Replace the entire file contents with:
```yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['localhost:9093']

rule_files:
  - "/etc/prometheus/rules.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Add client machines here as you connect them via Tailscale
  # - job_name: 'node_exporter'
  #   static_configs:
  #     - targets:
  #       - '100.119.113.98:9100'   # bs13
  #       - '100.76.222.82:9100'    # pop-os / byfar
```

### Create alert rules file
```bash
sudo tee /etc/prometheus/rules.yml << 'EOF'
groups:
  - name: server_alerts
    rules:
      - alert: ServerDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Server {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} has been unreachable for more than 1 minute."

      - alert: HighCPU
        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"

      - alert: HighRAM
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High RAM on {{ $labels.instance }}"

      - alert: LowDisk
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk on {{ $labels.instance }}"
EOF
```

### Run Prometheus as a systemd service
```bash
sudo tee /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Monitoring
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d \
  --storage.tsdb.retention.size=10GB \
  --web.listen-address=0.0.0.0:9090
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus
```

Verify: Open `http://localhost:9090` in your browser — you should see the Prometheus UI.

---

## Part 5: Install Alertmanager

### Download and install
```bash
cd /tmp
AM_VERSION="0.27.0"
wget https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz
tar xvf alertmanager-${AM_VERSION}.linux-amd64.tar.gz
sudo mv alertmanager-${AM_VERSION}.linux-amd64/alertmanager /usr/local/bin/
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
```

### Configure Alertmanager
```bash
sudo tee /etc/alertmanager/alertmanager.yml << 'EOF'
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'default'

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'YOUR_ROCKETCHAT_WEBHOOK_URL'
        send_resolved: true
EOF
```
Replace `YOUR_ROCKETCHAT_WEBHOOK_URL` with your actual Rocket.Chat webhook URL.

### Run as systemd service
```bash
sudo tee /etc/systemd/system/alertmanager.service << 'EOF'
[Unit]
Description=Alertmanager
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager
```

---

## Part 6: Install Grafana

```bash
sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server
```

Verify: Open `http://localhost:3000` — default login is `admin` / `admin`. Change the password on first login.

### Connect Grafana to Prometheus
1. Go to **Connections → Data Sources → Add data source**
2. Select **Prometheus**
3. Set URL to `http://localhost:9090`
4. Click **Save & Test** — should show "Data source is working"

### Import recommended dashboards
- Go to **Dashboards → Import**
- Enter ID `1860` → Load → Import (Node Exporter Full)
- Enter ID `14574` → Load → Import (NVIDIA GPU Metrics)

---

## Part 7: Verify All Services Are Running

```bash
sudo systemctl status meshcentral prometheus alertmanager grafana-server tailscaled
```

All five should show `active (running)`.

Quick port check:
```bash
ss -tlnp | grep -E "4430|9090|9093|3000"
```

Expected:
```
LISTEN  *:4430   → MeshCentral
LISTEN  *:9090   → Prometheus
LISTEN  *:9093   → Alertmanager
LISTEN  *:3000   → Grafana
```

---

## Part 8: Adding Client Machines

For each new client machine, see `07_connect_remote_machine.md`.

After the client is connected to MeshCentral and Tailscale, add its Tailscale IP to Prometheus:

```bash
sudo nano /etc/prometheus/prometheus.yml
```

Add under `scrape_configs`:
```yaml
  - job_name: 'node_exporter'
    static_configs:
      - targets:
          - '100.x.x.x:9100'    # machine name
```

Reload Prometheus without restarting:
```bash
curl -X POST http://localhost:9090/-/reload
```

---

## Part 9: Production Checklist

- [ ] Native Linux (not WSL) — services survive reboots
- [ ] All 5 services enabled with `systemctl enable`
- [ ] Sleep/hibernate disabled
- [ ] MeshCentral admin account created and saved
- [ ] Tailscale logged in and showing `Connected`
- [ ] Grafana default password changed
- [ ] At least one client machine connected and showing in Prometheus targets
- [ ] At least one Grafana dashboard imported and showing data
- [ ] Test alert fired and received in Rocket.Chat

---

## Quick Reference

| Service | Port | URL |
|---|---|---|
| MeshCentral | 4430 | `https://localhost:4430` |
| Prometheus | 9090 | `http://localhost:9090` |
| Alertmanager | 9093 | `http://localhost:9093` |
| Grafana | 3000 | `http://localhost:3000` |
| Tailscale | — | `tailscale status` |

| Command | Purpose |
|---|---|
| `sudo systemctl restart meshcentral` | Restart RMM server |
| `sudo systemctl restart prometheus` | Restart metrics collector |
| `curl -X POST http://localhost:9090/-/reload` | Reload Prometheus config without restart |
| `tailscale ip -4` | Get this machine's Tailscale IP |
| `tailscale status` | See all connected machines |