127 lines
5.0 KiB
Markdown
127 lines
5.0 KiB
Markdown
# 04 - Operations and Debugging
|
|
|
|
_This document serves as your day-to-day playbook for maintaining the monitoring infrastructure, writing queries, building dashboards, and debugging system crashes._
|
|
|
|
---
|
|
|
|
## 1. ESSENTIAL PROMQL QUERIES
|
|
|
|
_Use these queries in the Prometheus search bar (port 9090) or inside Grafana panels to monitor your hardware._
|
|
|
|
**1. RAM: Percentage Available**
|
|
```promql
|
|
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
|
|
```
|
|
|
|
**2. CPU: Percentage Actively Used**
|
|
_(Node Exporter tracks how long the CPU is "idle". We subtract the idle speed from 100% to get the active usage)._
|
|
```promql
|
|
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
|
|
```
|
|
|
|
**3. DISK (SSD/HDD): Percentage Free Space (Root Drive)**
|
|
```promql
|
|
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100
|
|
```
|
|
|
|
**4. NETWORK: Total Upload Speed in Megabytes/second (MB/s)**
|
|
```promql
|
|
rate(node_network_transmit_bytes_total[5m]) / 1024 / 1024
|
|
```
|
|
|
|
**5. HARDWARE: Server Uptime in Days**
|
|
```promql
|
|
(time() - node_boot_time_seconds) / 86400
|
|
```
|
|
|
|
**6. SERVERS: Is a server down? (1 = UP, 0 = DOWN)**
|
|
```promql
|
|
up
|
|
```
|
|
|
|
---
|
|
|
|
## 2. GRAFANA DASHBOARD TEMPLATES
|
|
|
|
When importing templates from `grafana.com/dashboards`, use these exact IDs for pre-built, beautiful hardware graphs:
|
|
|
|
- **Node Exporter Full (CPU/RAM/Network/Disk):** `1860`
|
|
- **Nvidia GPU Exporter (Temperature/VRAM/Power):** `14574`
|
|
|
|
### How to Fix "Blank" or "No Data" Dashboards
|
|
1. **Variables Mismatch:** Look at the dropdown menus at the very top of the dashboard. If they say "N/A", click them and select your specific `job_name` (e.g. `remote_gpu_server`).
|
|
2. **PromQL Mismatch:** If a specific graph is still blank, click the `...` next to the graph title and hit **Edit**. Look at the raw PromQL code. Sometimes community dashboards hardcode filters like `{job="gpu"}`. Simply delete that strict filter so your specific data can flow through!
|
|
|
|
---
|
|
|
|
## 3. MANAGING LINUX SERVICES (Crash, Reboot, & Maintenance)
|
|
|
|
Because we configured all components (Prometheus, Node Exporter, Grafana, Alertmanager) as Linux `systemd` services and used the `enable` command, **they will start automatically every time you boot or reboot your server.**
|
|
|
|
### 1. Check the Status of All Services at Once
|
|
If you just rebooted your server and want to ensure everything turned on properly, run this:
|
|
```bash
|
|
sudo systemctl status prometheus node_exporter grafana-server alertmanager
|
|
```
|
|
_Tip: If the output locks your terminal and shows `(END)` at the bottom, just press the `q` key on your keyboard to quit the reader mode and return to your prompt._
|
|
|
|
### 2. Restarting a Service After a Configuration Change
|
|
If you edit a configuration file (like `prometheus.yml` or `alertmanager.yml`), the changes won't apply until you restart the service. If a service ever crashes and you want to force it to start fresh, use `restart`:
|
|
```bash
|
|
# Restart Prometheus
|
|
sudo systemctl restart prometheus
|
|
|
|
# Restart Alertmanager
|
|
sudo systemctl restart alertmanager
|
|
|
|
# Restart Grafana
|
|
sudo systemctl restart grafana-server
|
|
```
|
|
|
|
### 3. Turning Services On / Off Manually
|
|
If you need to intentionally stop a service to perform maintenance (or trigger a fake alert):
|
|
```bash
|
|
sudo systemctl stop node_exporter
|
|
```
|
|
To start it back up:
|
|
```bash
|
|
sudo systemctl start node_exporter
|
|
```
|
|
|
|
---
|
|
|
|
## 4. DEBUGGING CRASHES WITH LOGS
|
|
|
|
If a service says it is "failed" or "inactive" when you check its status, you can view its exact crash logs by checking the Linux `journalctl` (the master log book).
|
|
|
|
**To see the last 50 logs for Alertmanager (useful for debugging 500 errors or webhook failures):**
|
|
```bash
|
|
sudo journalctl -u alertmanager -n 50 --no-pager
|
|
```
|
|
|
|
**To stream the logs live in real-time (press `Ctrl+C` to stop):**
|
|
```bash
|
|
sudo journalctl -u alertmanager -f
|
|
```
|
|
|
|
*(You can replace `alertmanager` with `prometheus`, `node_exporter`, or `grafana-server` to view logs for any other component).*
|
|
|
|
---
|
|
|
|
## 5. COMMON TROUBLESHOOTING
|
|
|
|
### 1. Browser Blocks HTTP Exporters
|
|
If your exporter is running (e.g., `curl http://localhost:9100/metrics` works in the terminal) but the browser says "Connection Refused", make sure your browser didn't automatically force `https://`. Exporters only run on plain `http://` by default!
|
|
|
|
### 2. "field repeat_interval already set" Error in Alertmanager
|
|
Alertmanager will crash on startup if you have duplicate keys in the YAML file (like declaring `repeat_interval` twice in the `route` block). Always run the syntax checker before restarting:
|
|
```bash
|
|
amtool check-config /etc/alertmanager/alertmanager.yml
|
|
```
|
|
|
|
### 3. Rocket.Chat 500 Internal Server Error
|
|
If `journalctl` shows Alertmanager successfully trying to send an alert but getting a `500 Internal Server Error`, the issue is on the Rocket.Chat server. Make sure:
|
|
1. The **"Allow to overwrite destination channel"** toggle is ON in the webhook settings.
|
|
2. The transformation script (from `02_central_server_setup.md`) is successfully pasted into the Rocket.Chat integration and **Script Enabled** is ON.
|
|
3. "Script Sandbox" is set to "Secure Sandbox".
|