# 04 - Operations and Debugging

_This document serves as your day-to-day playbook for maintaining the monitoring infrastructure, writing queries, building dashboards, and debugging system crashes._

---

## 1. ESSENTIAL PROMQL QUERIES

_Use these queries in the Prometheus search bar (port 9090) or inside Grafana panels to monitor your hardware._

**1. RAM: Percentage Available**
```promql
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
```

**2. CPU: Percentage Actively Used**
_(Node Exporter tracks how long the CPU is "idle". We subtract the idle speed from 100% to get the active usage)._
```promql
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
```

**3. DISK (SSD/HDD): Percentage Free Space (Root Drive)**
```promql
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100
```

**4. NETWORK: Total Upload Speed in Megabytes/second (MB/s)**
```promql
rate(node_network_transmit_bytes_total[5m]) / 1024 / 1024
```

**5. HARDWARE: Server Uptime in Days**
```promql
(time() - node_boot_time_seconds) / 86400
```

**6. SERVERS: Is a server down? (1 = UP, 0 = DOWN)**
```promql
up
```

---

## 2. GRAFANA DASHBOARD TEMPLATES

When importing templates from `grafana.com/dashboards`, use these exact IDs for pre-built, beautiful hardware graphs:

- **Node Exporter Full (CPU/RAM/Network/Disk):** `1860`
- **Nvidia GPU Exporter (Temperature/VRAM/Power):** `14574`

### How to Fix "Blank" or "No Data" Dashboards
1. **Variables Mismatch:** Look at the dropdown menus at the very top of the dashboard. If they say "N/A", click them and select your specific `job_name` (e.g. `remote_gpu_server`).
2. **PromQL Mismatch:** If a specific graph is still blank, click the `...` next to the graph title and hit **Edit**. Look at the raw PromQL code. Sometimes community dashboards hardcode filters like `{job="gpu"}`. Simply delete that strict filter so your specific data can flow through!

---

## 3. MANAGING LINUX SERVICES (Crash, Reboot, & Maintenance)

Because we configured all components (Prometheus, Node Exporter, Grafana, Alertmanager) as Linux `systemd` services and used the `enable` command, **they will start automatically every time you boot or reboot your server.**

### 1. Check the Status of All Services at Once
If you just rebooted your server and want to ensure everything turned on properly, run this:
```bash
sudo systemctl status prometheus node_exporter grafana-server alertmanager
```
_Tip: If the output locks your terminal and shows `(END)` at the bottom, just press the `q` key on your keyboard to quit the reader mode and return to your prompt._

### 2. Restarting a Service After a Configuration Change
If you edit a configuration file (like `prometheus.yml` or `alertmanager.yml`), the changes won't apply until you restart the service. If a service ever crashes and you want to force it to start fresh, use `restart`:
```bash
# Restart Prometheus
sudo systemctl restart prometheus

# Restart Alertmanager
sudo systemctl restart alertmanager

# Restart Grafana
sudo systemctl restart grafana-server
```

### 3. Turning Services On / Off Manually
If you need to intentionally stop a service to perform maintenance (or trigger a fake alert):
```bash
sudo systemctl stop node_exporter
```
To start it back up:
```bash
sudo systemctl start node_exporter
```

---

## 4. DEBUGGING CRASHES WITH LOGS

If a service says it is "failed" or "inactive" when you check its status, you can view its exact crash logs by checking the Linux `journalctl` (the master log book).

**To see the last 50 logs for Alertmanager (useful for debugging 500 errors or webhook failures):**
```bash
sudo journalctl -u alertmanager -n 50 --no-pager
```

**To stream the logs live in real-time (press `Ctrl+C` to stop):**
```bash
sudo journalctl -u alertmanager -f
```

*(You can replace `alertmanager` with `prometheus`, `node_exporter`, or `grafana-server` to view logs for any other component).*

---

## 5. COMMON TROUBLESHOOTING

### 1. Browser Blocks HTTP Exporters
If your exporter is running (e.g., `curl http://localhost:9100/metrics` works in the terminal) but the browser says "Connection Refused", make sure your browser didn't automatically force `https://`. Exporters only run on plain `http://` by default!

### 2. "field repeat_interval already set" Error in Alertmanager
Alertmanager will crash on startup if you have duplicate keys in the YAML file (like declaring `repeat_interval` twice in the `route` block). Always run the syntax checker before restarting:
```bash
amtool check-config /etc/alertmanager/alertmanager.yml
```

### 3. Rocket.Chat 500 Internal Server Error
If `journalctl` shows Alertmanager successfully trying to send an alert but getting a `500 Internal Server Error`, the issue is on the Rocket.Chat server. Make sure:
1. The **"Allow to overwrite destination channel"** toggle is ON in the webhook settings.
2. The transformation script (from `02_central_server_setup.md`) is successfully pasted into the Rocket.Chat integration and **Script Enabled** is ON. 
3. "Script Sandbox" is set to "Secure Sandbox".