5.0 KiB
04 - Operations and Debugging
This document serves as your day-to-day playbook for maintaining the monitoring infrastructure, writing queries, building dashboards, and debugging system crashes.
1. ESSENTIAL PROMQL QUERIES
Use these queries in the Prometheus search bar (port 9090) or inside Grafana panels to monitor your hardware.
1. RAM: Percentage Available
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
2. CPU: Percentage Actively Used (Node Exporter tracks how long the CPU is "idle". We subtract the idle speed from 100% to get the active usage).
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
3. DISK (SSD/HDD): Percentage Free Space (Root Drive)
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100
4. NETWORK: Total Upload Speed in Megabytes/second (MB/s)
rate(node_network_transmit_bytes_total[5m]) / 1024 / 1024
5. HARDWARE: Server Uptime in Days
(time() - node_boot_time_seconds) / 86400
6. SERVERS: Is a server down? (1 = UP, 0 = DOWN)
up
2. GRAFANA DASHBOARD TEMPLATES
When importing templates from grafana.com/dashboards, use these exact IDs for pre-built, beautiful hardware graphs:
- Node Exporter Full (CPU/RAM/Network/Disk):
1860 - Nvidia GPU Exporter (Temperature/VRAM/Power):
14574
How to Fix "Blank" or "No Data" Dashboards
- Variables Mismatch: Look at the dropdown menus at the very top of the dashboard. If they say "N/A", click them and select your specific
job_name(e.g.remote_gpu_server). - PromQL Mismatch: If a specific graph is still blank, click the
...next to the graph title and hit Edit. Look at the raw PromQL code. Sometimes community dashboards hardcode filters like{job="gpu"}. Simply delete that strict filter so your specific data can flow through!
3. MANAGING LINUX SERVICES (Crash, Reboot, & Maintenance)
Because we configured all components (Prometheus, Node Exporter, Grafana, Alertmanager) as Linux systemd services and used the enable command, they will start automatically every time you boot or reboot your server.
1. Check the Status of All Services at Once
If you just rebooted your server and want to ensure everything turned on properly, run this:
sudo systemctl status prometheus node_exporter grafana-server alertmanager
Tip: If the output locks your terminal and shows (END) at the bottom, just press the q key on your keyboard to quit the reader mode and return to your prompt.
2. Restarting a Service After a Configuration Change
If you edit a configuration file (like prometheus.yml or alertmanager.yml), the changes won't apply until you restart the service. If a service ever crashes and you want to force it to start fresh, use restart:
# Restart Prometheus
sudo systemctl restart prometheus
# Restart Alertmanager
sudo systemctl restart alertmanager
# Restart Grafana
sudo systemctl restart grafana-server
3. Turning Services On / Off Manually
If you need to intentionally stop a service to perform maintenance (or trigger a fake alert):
sudo systemctl stop node_exporter
To start it back up:
sudo systemctl start node_exporter
4. DEBUGGING CRASHES WITH LOGS
If a service says it is "failed" or "inactive" when you check its status, you can view its exact crash logs by checking the Linux journalctl (the master log book).
To see the last 50 logs for Alertmanager (useful for debugging 500 errors or webhook failures):
sudo journalctl -u alertmanager -n 50 --no-pager
To stream the logs live in real-time (press Ctrl+C to stop):
sudo journalctl -u alertmanager -f
(You can replace alertmanager with prometheus, node_exporter, or grafana-server to view logs for any other component).
5. COMMON TROUBLESHOOTING
1. Browser Blocks HTTP Exporters
If your exporter is running (e.g., curl http://localhost:9100/metrics works in the terminal) but the browser says "Connection Refused", make sure your browser didn't automatically force https://. Exporters only run on plain http:// by default!
2. "field repeat_interval already set" Error in Alertmanager
Alertmanager will crash on startup if you have duplicate keys in the YAML file (like declaring repeat_interval twice in the route block). Always run the syntax checker before restarting:
amtool check-config /etc/alertmanager/alertmanager.yml
3. Rocket.Chat 500 Internal Server Error
If journalctl shows Alertmanager successfully trying to send an alert but getting a 500 Internal Server Error, the issue is on the Rocket.Chat server. Make sure:
- The "Allow to overwrite destination channel" toggle is ON in the webhook settings.
- The transformation script (from
02_central_server_setup.md) is successfully pasted into the Rocket.Chat integration and Script Enabled is ON. - "Script Sandbox" is set to "Secure Sandbox".