# 02 - Central Server Setup

_Follow these instructions to build the "Brain" of your monitoring architecture. All of these components (Prometheus, Grafana, and Alertmanager) will be installed on your central server._

---

## 1. INSTALLING PROMETHEUS (The Database & Scraper)

**1. Download Prometheus**
```bash
wget https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz
```

**2. Extract Prometheus**
```bash
tar -xvf prometheus-2.51.2.linux-amd64.tar.gz
```

**3. Create a restricted system user for security**
```bash
sudo useradd --no-create-home --shell /bin/false prometheus
```

**4. Create the Configuration and Data folders**
```bash
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
```

**5. Hand ownership of those folders to the new user**
```bash
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
```

**6. Copy the main engine and spell-checker to the secure Linux bin folder**
```bash
sudo cp prometheus-2.51.2.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.51.2.linux-amd64/promtool /usr/local/bin/
```

**7. Hand ownership of the executables to the new user**
```bash
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
```

**8. Create the main configuration file**
```bash
sudo nano /etc/prometheus/prometheus.yml
```

_Paste this exact code into the file and save:_
```yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus_self_monitor"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "laptop_server_1"
    static_configs:
      - targets: ["localhost:9100"]
```

_Verify the YAML syntax is perfect so you don't crash the server:_
```bash
promtool check config /etc/prometheus/prometheus.yml
```

**9. Create the background service file**
```bash
sudo nano /etc/systemd/system/prometheus.service
```

_Paste this exact code into the file and save:_
```ini
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/

[Install]
WantedBy=multi-user.target
```

**10. Turn Prometheus on forever**
```bash
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
```

---

## 2. INSTALLING ALERTMANAGER (For Notifications)

**1. Download and Extract Alertmanager**
```bash
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar -xvf alertmanager-0.27.0.linux-amd64.tar.gz
```

**2. Create a restricted user and folders**
```bash
sudo useradd --no-create-home --shell /bin/false alertmanager
sudo mkdir /etc/alertmanager
sudo mkdir /var/lib/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager
```

**3. Move the binary and hand over ownership**
```bash
sudo cp alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
```

**4. Create the Configuration File**
```bash
sudo nano /etc/alertmanager/alertmanager.yml
```

_Paste this code to route alerts to both Rocket.Chat and Gmail simultaneously:_
```yaml
global:
  resolve_timeout: 5m
  smtp_smarthost: "smtp.gmail.com:587"
  smtp_from: "knmkaushik@gmail.com"
  smtp_auth_username: "knmkaushik@gmail.com"
  smtp_auth_password: "YOUR_APP_PASSWORD"

route:
  receiver: "fallback-do-nothing"
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 5m
  routes:
    - receiver: "rocket-chat-alerts"
      continue: true
    - receiver: "email-alerts"
      continue: true

receivers:
  - name: "fallback-do-nothing"

  - name: "rocket-chat-alerts"
    webhook_configs:
      - url: "YOUR_ROCKETCHAT_WEBHOOK_URL"
        send_resolved: true

  - name: "email-alerts"
    email_configs:
      - to: "kawsik97@gmail.com"
        send_resolved: true
```

**5. Check syntax, then hand over ownership of the config file**
```bash
amtool check-config /etc/alertmanager/alertmanager.yml
sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml
```

**6. Create the background service file**
```bash
sudo nano /etc/systemd/system/alertmanager.service
```

_Paste this exact code into the file and save:_
```ini
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager

[Install]
WantedBy=multi-user.target
```

**7. Turn Alertmanager on forever**
```bash
sudo systemctl daemon-reload
sudo systemctl start alertmanager
sudo systemctl enable alertmanager
```

**8. Configure Rocket.Chat Webhook Script**
Rocket.Chat natively only understands `{"text": "..."}` payloads, but Alertmanager sends structured JSON. To fix this, you must add a transformation script in the Rocket.Chat admin panel:

1. Go to your Rocket.Chat Admin Panel -> **Integrations** -> **Incoming**.
2. Create or Edit the incoming webhook for your channel (e.g., `#devops-alerts`).
3. Toggle **Script Enabled** to **ON**.
4. Paste the following into the Script box:
```javascript
class Script {
  process_incoming_request({ request }) {
    var content = request.content;
    var status = content.status ? content.status.toUpperCase() : 'UNKNOWN';
    var alertName = 'Unknown';
    var instance = '';
    var summary = '';

    if (content.alerts && content.alerts[0]) {
      var a = content.alerts[0];
      if (a.labels) {
        alertName = a.labels.alertname || 'Unknown';
        instance = a.labels.instance || '';
      }
      if (a.annotations) {
        summary = a.annotations.summary || '';
      }
    }

    var icon = status === 'FIRING' ? ':red_circle:' : ':white_check_mark:';
    var msg = icon + ' *' + status + '* — ' + alertName;
    if (instance) { msg = msg + ' (' + instance + ')'; }
    if (summary)  { msg = msg + '\n' + summary; }

    return {
      content: {
        text: msg
      }
    };
  }
}
```
5. Make sure the "Script Sandbox" is set to "Secure Sandbox".
6. Click **Save** and use the generated Webhook URL in your `alertmanager.yml`.

---

## 3. LINKING PROMETHEUS & ALERTMANAGER

Prometheus must be told that Alertmanager exists, and where the "Rules" are stored.

**1. Open the main Prometheus config**
```bash
sudo nano /etc/prometheus/prometheus.yml
```

**2. Update the `alerting` and `rule_files` blocks at the top:**
```yaml
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

rule_files:
  - "rules.yml"
```
_(Save and exit)._

**3. Create the Rules file (Where the Alert Logic lives):**
```bash
sudo nano /etc/prometheus/rules.yml
```

**4. Paste this exact rule and save:**
_(This logic says: If any server goes offline for more than 10 seconds, fire a CRITICAL alert)._
```yaml
groups:
  - name: hardware_alerts
    rules:
      - alert: ServerDown
        expr: up == 0
        for: 10s
        labels:
          severity: critical
        annotations:
          summary: "Server {{ $labels.instance }} is {{ if eq $value 0.0 }}DOWN{{ else }}UP{{ end }}!"
          description: "Server {{ $labels.instance }} has been unreachable for more than 10 seconds."
```

**5. Hand over ownership and restart Prometheus:**
```bash
sudo chown prometheus:prometheus /etc/prometheus/rules.yml
sudo systemctl restart prometheus
```

---

## 4. INSTALLING GRAFANA (THE DASHBOARD UI)

_Grafana runs on port 3000 by default. Use this to visualize Prometheus data._

**1. Download the Grafana security key:**
```bash
sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
```

**2. Add the official Grafana repository to Linux:**
```bash
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
```

**3. Install Grafana and start the service:**
```bash
sudo apt-get update
sudo apt-get install grafana
sudo systemctl daemon-reload
sudo systemctl start grafana-server
sudo systemctl enable grafana-server
```

> **You have now successfully built the Central Server! Proceed to `03_remote_node_setup.md` to add your remote computers to the dashboard.**