4.7 KiB
4.7 KiB
PROMETHEUS LEARNING STATUS
Last Updated: 2026-05-11 14:44:00 Current IDE: VS Code
Goal
Monitor 7 remote Linux servers from one central server. Currently in testing phase using 2 Windows systems (Laptop & Desktop). Central server: [Windows Laptop/Desktop - PENDING IP] Remote servers:
- Test Server 1 (Windows): [PENDING IP]
- Prod Servers 1-7 (Linux): [PENDING IPs]
Current Position
Layer: Layer 4 — PromQL Concept: Basic Syntax and Instant Vectors Drill: not done
Setup Status
- Prometheus installed on central server
- Prometheus UI accessible at port 9090
- Grafana installed on central server
- Grafana UI accessible at port 3000
- Alertmanager installed on central server
- Node Exporter on server 1: [PENDING]
- Node Exporter on server 2: [PENDING]
- Node Exporter on server 3: [PENDING]
- Node Exporter on server 4: [PENDING]
- Node Exporter on server 5: [PENDING]
- Node Exporter on server 6: [PENDING]
- Node Exporter on server 7: [PENDING]
- All 7 servers showing UP in Prometheus Targets
- Node Exporter Full dashboard imported in Grafana
- Alert rules configured and showing in Prometheus
- Alert notification received successfully
Layer Progress
Layer 1 — Mental Model
- Pull vs push model
- Time series data concept
- Four components understood
- Architecture drawn for 7 servers
- Four metric types Revision Task: [x] DONE
Layer 2 — Installing Prometheus
- File system structure
- Download and extract
- prometheus user created
- Directories and permissions
- Binaries in correct location
- First prometheus.yml written
- systemd service created
- Prometheus running and accessible Revision Task: [x] DONE
Layer 3 — Node Exporter
- What Node Exporter does
- /metrics format understood
- Installation on remote server
- systemd service for node_exporter
- /metrics verified with curl
- Firewall configured correctly
- Connectivity from central server tested
- Server added to prometheus.yml
- Target showing UP in Prometheus
- Repeated for all 7 servers Revision Task: [x] DONE
Layer 4 — PromQL
- Instant vs range vectors
- Label selectors
- rate() function
- CPU query
- RAM query
- Disk query
- Network query
- Up/Down query
- Aggregation operators
- Expression browser used Revision Task: [ ] NOT DONE
Layer 5 — Grafana
- Installation
- Prometheus data source added
- Dashboard 1860 imported
- Custom dashboard created
- Variables configured
- 6 panels built Revision Task: [ ] NOT DONE
Layer 6 — Alertmanager
- Installation
- alertmanager.yml configured
- Email or Slack receiver set up
- Alert rules file created
- ServerDown alert written
- HighCPU alert written
- DiskSpaceLow alert written
- HighMemory alert written
- Alertmanager linked to Prometheus
- Test alert received Revision Task: [ ] NOT DONE
Layer 7 — Config Deep Dive
- scrape_configs deep dive
- Labels mastered
- Relabeling understood
- Recording rules
- promtool validation Revision Task: [ ] NOT DONE
Layer 8 — Security
- TLS for Prometheus UI
- Basic auth configured
- nginx reverse proxy
- Node Exporter access restricted
- Storage retention configured
- Backup strategy in place Revision Task: [ ] NOT DONE
Layer 9 — Docker Setup
- docker-compose.yml written
- Volumes configured
- Stack running in Docker
- Node Exporter Docker vs bare metal decision Revision Task: [ ] NOT DONE
Layer 10 — Advanced Features
- file_sd_configs
- Cloud service discovery concept
- Pushgateway concept
- Additional exporters
- Custom exporter in Python
- Federation concept
- Long-term storage concept Revision Task: [ ] NOT DONE
Layer 11 — Troubleshooting
- Target DOWN debugging steps
- Metrics wrong debugging
- Disk usage management
- Memory management
- Alert not firing debugging
- Grafana no data debugging
- Config validation process Revision Task: [ ] NOT DONE
Layer 12 — Apex Project
- Full setup from scratch — complete
My Weak Spots
[None yet]
Bridge Map Learned
- Pull Model -> Central server running
setInterval()with Axios GET requests to 7 dumb servers. - Time Series Data ->
metric_name{labels} value timestamp(Labels are like MongoDB fields). - Architecture -> Remote: Node Exporter. Central: Prometheus, Grafana, Alertmanager. PULL direction.
- Linux Folders ->
/etc(configs/.env),/var/lib(MongoDB storage),/usr/local/bin(node.exe executable).
Common Errors Encountered
[None yet]
Spaced Repetition Queue
[None yet]
IDE Transfer Log
[None yet]