169 lines
4.7 KiB
Markdown
169 lines
4.7 KiB
Markdown
# PROMETHEUS LEARNING STATUS
|
|
Last Updated: 2026-05-11 14:44:00
|
|
Current IDE: VS Code
|
|
|
|
## Goal
|
|
Monitor 7 remote Linux servers from one central server.
|
|
*Currently in testing phase using 2 Windows systems (Laptop & Desktop).*
|
|
Central server: [Windows Laptop/Desktop - PENDING IP]
|
|
Remote servers:
|
|
- Test Server 1 (Windows): [PENDING IP]
|
|
- Prod Servers 1-7 (Linux): [PENDING IPs]
|
|
|
|
## Current Position
|
|
Layer: Layer 4 — PromQL
|
|
Concept: Basic Syntax and Instant Vectors
|
|
Drill: not done
|
|
|
|
## Setup Status
|
|
- [ ] Prometheus installed on central server
|
|
- [ ] Prometheus UI accessible at port 9090
|
|
- [ ] Grafana installed on central server
|
|
- [ ] Grafana UI accessible at port 3000
|
|
- [ ] Alertmanager installed on central server
|
|
- [ ] Node Exporter on server 1: [PENDING]
|
|
- [ ] Node Exporter on server 2: [PENDING]
|
|
- [ ] Node Exporter on server 3: [PENDING]
|
|
- [ ] Node Exporter on server 4: [PENDING]
|
|
- [ ] Node Exporter on server 5: [PENDING]
|
|
- [ ] Node Exporter on server 6: [PENDING]
|
|
- [ ] Node Exporter on server 7: [PENDING]
|
|
- [ ] All 7 servers showing UP in Prometheus Targets
|
|
- [ ] Node Exporter Full dashboard imported in Grafana
|
|
- [ ] Alert rules configured and showing in Prometheus
|
|
- [ ] Alert notification received successfully
|
|
|
|
## Layer Progress
|
|
|
|
### Layer 1 — Mental Model
|
|
- [x] Pull vs push model
|
|
- [x] Time series data concept
|
|
- [x] Four components understood
|
|
- [x] Architecture drawn for 7 servers
|
|
- [x] Four metric types
|
|
Revision Task: [x] DONE
|
|
|
|
### Layer 2 — Installing Prometheus
|
|
- [x] File system structure
|
|
- [x] Download and extract
|
|
- [x] prometheus user created
|
|
- [x] Directories and permissions
|
|
- [x] Binaries in correct location
|
|
- [x] First prometheus.yml written
|
|
- [x] systemd service created
|
|
- [x] Prometheus running and accessible
|
|
Revision Task: [x] DONE
|
|
|
|
### Layer 3 — Node Exporter
|
|
- [x] What Node Exporter does
|
|
- [x] /metrics format understood
|
|
- [x] Installation on remote server
|
|
- [ ] systemd service for node_exporter
|
|
- [ ] /metrics verified with curl
|
|
- [ ] Firewall configured correctly
|
|
- [x] Connectivity from central server tested
|
|
- [x] Server added to prometheus.yml
|
|
- [x] Target showing UP in Prometheus
|
|
- [ ] Repeated for all 7 servers
|
|
Revision Task: [x] DONE
|
|
|
|
### Layer 4 — PromQL
|
|
- [x] Instant vs range vectors
|
|
- [x] Label selectors
|
|
- [x] rate() function
|
|
- [x] CPU query
|
|
- [x] RAM query
|
|
- [x] Disk query
|
|
- [x] Network query
|
|
- [x] Up/Down query
|
|
- [x] Aggregation operators
|
|
- [ ] Expression browser used
|
|
Revision Task: [ ] NOT DONE
|
|
|
|
### Layer 5 — Grafana
|
|
- [x] Installation
|
|
- [x] Prometheus data source added
|
|
- [x] Dashboard 1860 imported
|
|
- [ ] Custom dashboard created
|
|
- [ ] Variables configured
|
|
- [ ] 6 panels built
|
|
Revision Task: [ ] NOT DONE
|
|
|
|
### Layer 6 — Alertmanager
|
|
- [ ] Installation
|
|
- [ ] alertmanager.yml configured
|
|
- [ ] Email or Slack receiver set up
|
|
- [ ] Alert rules file created
|
|
- [ ] ServerDown alert written
|
|
- [ ] HighCPU alert written
|
|
- [ ] DiskSpaceLow alert written
|
|
- [ ] HighMemory alert written
|
|
- [ ] Alertmanager linked to Prometheus
|
|
- [ ] Test alert received
|
|
Revision Task: [ ] NOT DONE
|
|
|
|
### Layer 7 — Config Deep Dive
|
|
- [ ] scrape_configs deep dive
|
|
- [ ] Labels mastered
|
|
- [ ] Relabeling understood
|
|
- [ ] Recording rules
|
|
- [ ] promtool validation
|
|
Revision Task: [ ] NOT DONE
|
|
|
|
### Layer 8 — Security
|
|
- [ ] TLS for Prometheus UI
|
|
- [ ] Basic auth configured
|
|
- [ ] nginx reverse proxy
|
|
- [ ] Node Exporter access restricted
|
|
- [ ] Storage retention configured
|
|
- [ ] Backup strategy in place
|
|
Revision Task: [ ] NOT DONE
|
|
|
|
### Layer 9 — Docker Setup
|
|
- [ ] docker-compose.yml written
|
|
- [ ] Volumes configured
|
|
- [ ] Stack running in Docker
|
|
- [ ] Node Exporter Docker vs bare metal decision
|
|
Revision Task: [ ] NOT DONE
|
|
|
|
### Layer 10 — Advanced Features
|
|
- [ ] file_sd_configs
|
|
- [ ] Cloud service discovery concept
|
|
- [ ] Pushgateway concept
|
|
- [ ] Additional exporters
|
|
- [ ] Custom exporter in Python
|
|
- [ ] Federation concept
|
|
- [ ] Long-term storage concept
|
|
Revision Task: [ ] NOT DONE
|
|
|
|
### Layer 11 — Troubleshooting
|
|
- [ ] Target DOWN debugging steps
|
|
- [ ] Metrics wrong debugging
|
|
- [ ] Disk usage management
|
|
- [ ] Memory management
|
|
- [ ] Alert not firing debugging
|
|
- [ ] Grafana no data debugging
|
|
- [ ] Config validation process
|
|
Revision Task: [ ] NOT DONE
|
|
|
|
### Layer 12 — Apex Project
|
|
- [ ] Full setup from scratch — complete
|
|
|
|
## My Weak Spots
|
|
[None yet]
|
|
|
|
## Bridge Map Learned
|
|
- Pull Model -> Central server running `setInterval()` with Axios GET requests to 7 dumb servers.
|
|
- Time Series Data -> `metric_name{labels} value timestamp` (Labels are like MongoDB fields).
|
|
- Architecture -> Remote: Node Exporter. Central: Prometheus, Grafana, Alertmanager. PULL direction.
|
|
- Linux Folders -> `/etc` (configs/.env), `/var/lib` (MongoDB storage), `/usr/local/bin` (node.exe executable).
|
|
|
|
## Common Errors Encountered
|
|
[None yet]
|
|
|
|
## Spaced Repetition Queue
|
|
[None yet]
|
|
|
|
## IDE Transfer Log
|
|
[None yet]
|