initial commit
This commit is contained in:
kawsik
323
02_central_server_setup.md
Normal file
323
02_central_server_setup.md
Normal file
@@ -0,0 +1,323 @@
|
||||
# 02 - Central Server Setup
|
||||
|
||||
_Follow these instructions to build the "Brain" of your monitoring architecture. All of these components (Prometheus, Grafana, and Alertmanager) will be installed on your central server._
|
||||
|
||||
---
|
||||
|
||||
## 1. INSTALLING PROMETHEUS (The Database & Scraper)
|
||||
|
||||
**1. Download Prometheus**
|
||||
```bash
|
||||
wget https://github.com/prometheus/prometheus/releases/download/v2.51.2/prometheus-2.51.2.linux-amd64.tar.gz
|
||||
```
|
||||
|
||||
**2. Extract Prometheus**
|
||||
```bash
|
||||
tar -xvf prometheus-2.51.2.linux-amd64.tar.gz
|
||||
```
|
||||
|
||||
**3. Create a restricted system user for security**
|
||||
```bash
|
||||
sudo useradd --no-create-home --shell /bin/false prometheus
|
||||
```
|
||||
|
||||
**4. Create the Configuration and Data folders**
|
||||
```bash
|
||||
sudo mkdir /etc/prometheus
|
||||
sudo mkdir /var/lib/prometheus
|
||||
```
|
||||
|
||||
**5. Hand ownership of those folders to the new user**
|
||||
```bash
|
||||
sudo chown prometheus:prometheus /etc/prometheus
|
||||
sudo chown prometheus:prometheus /var/lib/prometheus
|
||||
```
|
||||
|
||||
**6. Copy the main engine and spell-checker to the secure Linux bin folder**
|
||||
```bash
|
||||
sudo cp prometheus-2.51.2.linux-amd64/prometheus /usr/local/bin/
|
||||
sudo cp prometheus-2.51.2.linux-amd64/promtool /usr/local/bin/
|
||||
```
|
||||
|
||||
**7. Hand ownership of the executables to the new user**
|
||||
```bash
|
||||
sudo chown prometheus:prometheus /usr/local/bin/prometheus
|
||||
sudo chown prometheus:prometheus /usr/local/bin/promtool
|
||||
```
|
||||
|
||||
**8. Create the main configuration file**
|
||||
```bash
|
||||
sudo nano /etc/prometheus/prometheus.yml
|
||||
```
|
||||
|
||||
_Paste this exact code into the file and save:_
|
||||
```yaml
|
||||
global:
|
||||
scrape_interval: 15s
|
||||
evaluation_interval: 15s
|
||||
|
||||
scrape_configs:
|
||||
- job_name: "prometheus_self_monitor"
|
||||
static_configs:
|
||||
- targets: ["localhost:9090"]
|
||||
|
||||
- job_name: "laptop_server_1"
|
||||
static_configs:
|
||||
- targets: ["localhost:9100"]
|
||||
```
|
||||
|
||||
_Verify the YAML syntax is perfect so you don't crash the server:_
|
||||
```bash
|
||||
promtool check config /etc/prometheus/prometheus.yml
|
||||
```
|
||||
|
||||
**9. Create the background service file**
|
||||
```bash
|
||||
sudo nano /etc/systemd/system/prometheus.service
|
||||
```
|
||||
|
||||
_Paste this exact code into the file and save:_
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Prometheus
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
User=prometheus
|
||||
Group=prometheus
|
||||
Type=simple
|
||||
ExecStart=/usr/local/bin/prometheus \
|
||||
--config.file /etc/prometheus/prometheus.yml \
|
||||
--storage.tsdb.path /var/lib/prometheus/
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
**10. Turn Prometheus on forever**
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl start prometheus
|
||||
sudo systemctl enable prometheus
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. INSTALLING ALERTMANAGER (For Notifications)
|
||||
|
||||
**1. Download and Extract Alertmanager**
|
||||
```bash
|
||||
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
|
||||
tar -xvf alertmanager-0.27.0.linux-amd64.tar.gz
|
||||
```
|
||||
|
||||
**2. Create a restricted user and folders**
|
||||
```bash
|
||||
sudo useradd --no-create-home --shell /bin/false alertmanager
|
||||
sudo mkdir /etc/alertmanager
|
||||
sudo mkdir /var/lib/alertmanager
|
||||
sudo chown alertmanager:alertmanager /var/lib/alertmanager
|
||||
```
|
||||
|
||||
**3. Move the binary and hand over ownership**
|
||||
```bash
|
||||
sudo cp alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
|
||||
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
|
||||
```
|
||||
|
||||
**4. Create the Configuration File**
|
||||
```bash
|
||||
sudo nano /etc/alertmanager/alertmanager.yml
|
||||
```
|
||||
|
||||
_Paste this code to route alerts to both Rocket.Chat and Gmail simultaneously:_
|
||||
```yaml
|
||||
global:
|
||||
resolve_timeout: 5m
|
||||
smtp_smarthost: "smtp.gmail.com:587"
|
||||
smtp_from: "knmkaushik@gmail.com"
|
||||
smtp_auth_username: "knmkaushik@gmail.com"
|
||||
smtp_auth_password: "YOUR_APP_PASSWORD"
|
||||
|
||||
route:
|
||||
receiver: "fallback-do-nothing"
|
||||
group_wait: 10s
|
||||
group_interval: 1m
|
||||
repeat_interval: 5m
|
||||
routes:
|
||||
- receiver: "rocket-chat-alerts"
|
||||
continue: true
|
||||
- receiver: "email-alerts"
|
||||
continue: true
|
||||
|
||||
receivers:
|
||||
- name: "fallback-do-nothing"
|
||||
|
||||
- name: "rocket-chat-alerts"
|
||||
webhook_configs:
|
||||
- url: "YOUR_ROCKETCHAT_WEBHOOK_URL"
|
||||
send_resolved: true
|
||||
|
||||
- name: "email-alerts"
|
||||
email_configs:
|
||||
- to: "kawsik97@gmail.com"
|
||||
send_resolved: true
|
||||
```
|
||||
|
||||
**5. Check syntax, then hand over ownership of the config file**
|
||||
```bash
|
||||
amtool check-config /etc/alertmanager/alertmanager.yml
|
||||
sudo chown alertmanager:alertmanager /etc/alertmanager/alertmanager.yml
|
||||
```
|
||||
|
||||
**6. Create the background service file**
|
||||
```bash
|
||||
sudo nano /etc/systemd/system/alertmanager.service
|
||||
```
|
||||
|
||||
_Paste this exact code into the file and save:_
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Alertmanager
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
|
||||
[Service]
|
||||
User=alertmanager
|
||||
Group=alertmanager
|
||||
Type=simple
|
||||
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
**7. Turn Alertmanager on forever**
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl start alertmanager
|
||||
sudo systemctl enable alertmanager
|
||||
```
|
||||
|
||||
**8. Configure Rocket.Chat Webhook Script**
|
||||
Rocket.Chat natively only understands `{"text": "..."}` payloads, but Alertmanager sends structured JSON. To fix this, you must add a transformation script in the Rocket.Chat admin panel:
|
||||
|
||||
1. Go to your Rocket.Chat Admin Panel -> **Integrations** -> **Incoming**.
|
||||
2. Create or Edit the incoming webhook for your channel (e.g., `#devops-alerts`).
|
||||
3. Toggle **Script Enabled** to **ON**.
|
||||
4. Paste the following into the Script box:
|
||||
```javascript
|
||||
class Script {
|
||||
process_incoming_request({ request }) {
|
||||
var content = request.content;
|
||||
var status = content.status ? content.status.toUpperCase() : 'UNKNOWN';
|
||||
var alertName = 'Unknown';
|
||||
var instance = '';
|
||||
var summary = '';
|
||||
|
||||
if (content.alerts && content.alerts[0]) {
|
||||
var a = content.alerts[0];
|
||||
if (a.labels) {
|
||||
alertName = a.labels.alertname || 'Unknown';
|
||||
instance = a.labels.instance || '';
|
||||
}
|
||||
if (a.annotations) {
|
||||
summary = a.annotations.summary || '';
|
||||
}
|
||||
}
|
||||
|
||||
var icon = status === 'FIRING' ? ':red_circle:' : ':white_check_mark:';
|
||||
var msg = icon + ' *' + status + '* — ' + alertName;
|
||||
if (instance) { msg = msg + ' (' + instance + ')'; }
|
||||
if (summary) { msg = msg + '\n' + summary; }
|
||||
|
||||
return {
|
||||
content: {
|
||||
text: msg
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
5. Make sure the "Script Sandbox" is set to "Secure Sandbox".
|
||||
6. Click **Save** and use the generated Webhook URL in your `alertmanager.yml`.
|
||||
|
||||
---
|
||||
|
||||
## 3. LINKING PROMETHEUS & ALERTMANAGER
|
||||
|
||||
Prometheus must be told that Alertmanager exists, and where the "Rules" are stored.
|
||||
|
||||
**1. Open the main Prometheus config**
|
||||
```bash
|
||||
sudo nano /etc/prometheus/prometheus.yml
|
||||
```
|
||||
|
||||
**2. Update the `alerting` and `rule_files` blocks at the top:**
|
||||
```yaml
|
||||
alerting:
|
||||
alertmanagers:
|
||||
- static_configs:
|
||||
- targets: ["localhost:9093"]
|
||||
|
||||
rule_files:
|
||||
- "rules.yml"
|
||||
```
|
||||
_(Save and exit)._
|
||||
|
||||
**3. Create the Rules file (Where the Alert Logic lives):**
|
||||
```bash
|
||||
sudo nano /etc/prometheus/rules.yml
|
||||
```
|
||||
|
||||
**4. Paste this exact rule and save:**
|
||||
_(This logic says: If any server goes offline for more than 10 seconds, fire a CRITICAL alert)._
|
||||
```yaml
|
||||
groups:
|
||||
- name: hardware_alerts
|
||||
rules:
|
||||
- alert: ServerDown
|
||||
expr: up == 0
|
||||
for: 10s
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Server {{ $labels.instance }} is {{ if eq $value 0.0 }}DOWN{{ else }}UP{{ end }}!"
|
||||
description: "Server {{ $labels.instance }} has been unreachable for more than 10 seconds."
|
||||
```
|
||||
|
||||
**5. Hand over ownership and restart Prometheus:**
|
||||
```bash
|
||||
sudo chown prometheus:prometheus /etc/prometheus/rules.yml
|
||||
sudo systemctl restart prometheus
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. INSTALLING GRAFANA (THE DASHBOARD UI)
|
||||
|
||||
_Grafana runs on port 3000 by default. Use this to visualize Prometheus data._
|
||||
|
||||
**1. Download the Grafana security key:**
|
||||
```bash
|
||||
sudo apt-get install -y apt-transport-https software-properties-common wget
|
||||
sudo mkdir -p /etc/apt/keyrings/
|
||||
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
|
||||
```
|
||||
|
||||
**2. Add the official Grafana repository to Linux:**
|
||||
```bash
|
||||
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
|
||||
```
|
||||
|
||||
**3. Install Grafana and start the service:**
|
||||
```bash
|
||||
sudo apt-get update
|
||||
sudo apt-get install grafana
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl start grafana-server
|
||||
sudo systemctl enable grafana-server
|
||||
```
|
||||
|
||||
> **You have now successfully built the Central Server! Proceed to `03_remote_node_setup.md` to add your remote computers to the dashboard.**
|
||||
Reference in New Issue
Block a user