RMM-UI-BE/02_central_server_setup.md

02 - Central Server Setup

Follow these instructions to build the "Brain" of your monitoring architecture. All of these components (Prometheus, Grafana, and Alertmanager) will be installed on your central server.

---

1. INSTALLING PROMETHEUS (The Database & Scraper)

1. Download Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz

2. Extract Prometheus

tar -xvf prometheus-2.51.2.linux-amd64.tar.gz

3. Create a restricted system user for security

sudo useradd --no-create-home --shell /bin/false prometheus

4. Create the Configuration and Data folders

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus

5. Hand ownership of those folders to the new user

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

6. Copy the main engine and spell-checker to the secure Linux bin folder

sudo cp prometheus-2.51.2.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.51.2.linux-amd64/promtool /usr/local/bin/

7. Hand ownership of the executables to the new user

sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

8. Create the main configuration file

sudo nano /etc/prometheus/prometheus.yml

Paste this exact code into the file and save:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus_self_monitor"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "laptop_server_1"
    static_configs:
      - targets: ["localhost:9100"]

Verify the YAML syntax is perfect so you don't crash the server:

promtool check config /etc/prometheus/prometheus.yml

9. Create the background service file

sudo nano /etc/systemd/system/prometheus.service

Paste this exact code into the file and save:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/

[Install]
WantedBy=multi-user.target

10. Turn Prometheus on forever

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

---

2. INSTALLING ALERTMANAGER (For Notifications)

1. Download and Extract Alertmanager

wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar -xvf alertmanager-0.27.0.linux-amd64.tar.gz

2. Create a restricted user and folders

sudo useradd --no-create-home --shell /bin/false alertmanager
sudo mkdir /etc/alertmanager
sudo mkdir /var/lib/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager

3. Move the binary and hand over ownership

sudo cp alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager

4. Create the Configuration File

sudo nano /etc/alertmanager/alertmanager.yml

Paste this code to route alerts to both Rocket.Chat and Gmail simultaneously:

global:
  resolve_timeout: 5m
  smtp_smarthost: "smtp.gmail.com:587"
  smtp_from: "knmkaushik@gmail.com"
  smtp_auth_username: "knmkaushik@gmail.com"
  smtp_auth_password: "YOUR_APP_PASSWORD"

route:
  receiver: "fallback-do-nothing"
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 5m
  routes:
    - receiver: "rocket-chat-alerts"
      continue: true
    - receiver: "email-alerts"
      continue: true

receivers:
  - name: "fallback-do-nothing"

  - name: "rocket-chat-alerts"
    webhook_configs:
      - url: "YOUR_ROCKETCHAT_WEBHOOK_URL"
        send_resolved: true

  - name: "email-alerts"
    email_configs:
      - to: "kawsik97@gmail.com"
        send_resolved: true

5. Check syntax, then hand over ownership of the config file

amtool check-config /etc/alertmanager/alertmanager.yml
sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml

6. Create the background service file

sudo nano /etc/systemd/system/alertmanager.service

Paste this exact code into the file and save:

[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager

[Install]
WantedBy=multi-user.target

7. Turn Alertmanager on forever

sudo systemctl daemon-reload
sudo systemctl start alertmanager
sudo systemctl enable alertmanager

8. Configure Rocket.Chat Webhook Script Rocket.Chat natively only understands {"text": "..."} payloads, but Alertmanager sends structured JSON. To fix this, you must add a transformation script in the Rocket.Chat admin panel:

1. Go to your Rocket.Chat Admin Panel -> Integrations -> Incoming.
2. Create or Edit the incoming webhook for your channel (e.g., #devops-alerts).
3. Toggle Script Enabled to ON.
4. Paste the following into the Script box:

class Script {
  process_incoming_request({ request }) {
    var content = request.content;
    var status = content.status ? content.status.toUpperCase() : 'UNKNOWN';
    var alertName = 'Unknown';
    var instance = '';
    var summary = '';

    if (content.alerts && content.alerts[0]) {
      var a = content.alerts[0];
      if (a.labels) {
        alertName = a.labels.alertname || 'Unknown';
        instance = a.labels.instance || '';
      }
      if (a.annotations) {
        summary = a.annotations.summary || '';
      }
    }

    var icon = status === 'FIRING' ? ':red_circle:' : ':white_check_mark:';
    var msg = icon + ' *' + status + '* — ' + alertName;
    if (instance) { msg = msg + ' (' + instance + ')'; }
    if (summary)  { msg = msg + '\n' + summary; }

    return {
      content: {
        text: msg
      }
    };
  }
}

5. Make sure the "Script Sandbox" is set to "Secure Sandbox".
6. Click Save and use the generated Webhook URL in your alertmanager.yml.

---

3. LINKING PROMETHEUS & ALERTMANAGER

Prometheus must be told that Alertmanager exists, and where the "Rules" are stored.

1. Open the main Prometheus config

sudo nano /etc/prometheus/prometheus.yml

2. Update the alerting and rule_files blocks at the top:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

rule_files:
  - "rules.yml"

(Save and exit).

3. Create the Rules file (Where the Alert Logic lives):

sudo nano /etc/prometheus/rules.yml

4. Paste this exact rule and save: (This logic says: If any server goes offline for more than 10 seconds, fire a CRITICAL alert).

groups:
  - name: hardware_alerts
    rules:
      - alert: ServerDown
        expr: up == 0
        for: 10s
        labels:
          severity: critical
        annotations:
          summary: "Server {{ $labels.instance }} is {{ if eq $value 0.0 }}DOWN{{ else }}UP{{ end }}!"
          description: "Server {{ $labels.instance }} has been unreachable for more than 10 seconds."

5. Hand over ownership and restart Prometheus:

sudo chown prometheus:prometheus /etc/prometheus/rules.yml
sudo systemctl restart prometheus

---

4. INSTALLING GRAFANA (THE DASHBOARD UI)

Grafana runs on port 3000 by default. Use this to visualize Prometheus data.

1. Download the Grafana security key:

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

2. Add the official Grafana repository to Linux:

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

3. Install Grafana and start the service:

sudo apt-get update
sudo apt-get install grafana
sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

You have now successfully built the Central Server! Proceed to 03_remote_node_setup.md to add your remote computers to the dashboard.