# PROMETHEUS LEARNING STATUS
Last Updated: 2026-05-11 14:44:00
Current IDE: VS Code

## Goal
Monitor 7 remote Linux servers from one central server.
*Currently in testing phase using 2 Windows systems (Laptop & Desktop).*
Central server: [Windows Laptop/Desktop - PENDING IP]
Remote servers: 
- Test Server 1 (Windows): [PENDING IP]
- Prod Servers 1-7 (Linux): [PENDING IPs]

## Current Position
Layer: Layer 4 — PromQL
Concept: Basic Syntax and Instant Vectors
Drill: not done

## Setup Status
- [ ] Prometheus installed on central server
- [ ] Prometheus UI accessible at port 9090
- [ ] Grafana installed on central server
- [ ] Grafana UI accessible at port 3000
- [ ] Alertmanager installed on central server
- [ ] Node Exporter on server 1: [PENDING]
- [ ] Node Exporter on server 2: [PENDING]
- [ ] Node Exporter on server 3: [PENDING]
- [ ] Node Exporter on server 4: [PENDING]
- [ ] Node Exporter on server 5: [PENDING]
- [ ] Node Exporter on server 6: [PENDING]
- [ ] Node Exporter on server 7: [PENDING]
- [ ] All 7 servers showing UP in Prometheus Targets
- [ ] Node Exporter Full dashboard imported in Grafana
- [ ] Alert rules configured and showing in Prometheus
- [ ] Alert notification received successfully

## Layer Progress

### Layer 1 — Mental Model
- [x] Pull vs push model
- [x] Time series data concept
- [x] Four components understood
- [x] Architecture drawn for 7 servers
- [x] Four metric types
Revision Task: [x] DONE

### Layer 2 — Installing Prometheus
- [x] File system structure
- [x] Download and extract
- [x] prometheus user created
- [x] Directories and permissions
- [x] Binaries in correct location
- [x] First prometheus.yml written
- [x] systemd service created
- [x] Prometheus running and accessible
Revision Task: [x] DONE

### Layer 3 — Node Exporter
- [x] What Node Exporter does
- [x] /metrics format understood
- [x] Installation on remote server
- [ ] systemd service for node_exporter
- [ ] /metrics verified with curl
- [ ] Firewall configured correctly
- [x] Connectivity from central server tested
- [x] Server added to prometheus.yml
- [x] Target showing UP in Prometheus
- [ ] Repeated for all 7 servers
Revision Task: [x] DONE

### Layer 4 — PromQL
- [x] Instant vs range vectors
- [x] Label selectors
- [x] rate() function
- [x] CPU query
- [x] RAM query
- [x] Disk query
- [x] Network query
- [x] Up/Down query
- [x] Aggregation operators
- [ ] Expression browser used
Revision Task: [ ] NOT DONE

### Layer 5 — Grafana
- [x] Installation
- [x] Prometheus data source added
- [x] Dashboard 1860 imported
- [ ] Custom dashboard created
- [ ] Variables configured
- [ ] 6 panels built
Revision Task: [ ] NOT DONE

### Layer 6 — Alertmanager
- [ ] Installation
- [ ] alertmanager.yml configured
- [ ] Email or Slack receiver set up
- [ ] Alert rules file created
- [ ] ServerDown alert written
- [ ] HighCPU alert written
- [ ] DiskSpaceLow alert written
- [ ] HighMemory alert written
- [ ] Alertmanager linked to Prometheus
- [ ] Test alert received
Revision Task: [ ] NOT DONE

### Layer 7 — Config Deep Dive
- [ ] scrape_configs deep dive
- [ ] Labels mastered
- [ ] Relabeling understood
- [ ] Recording rules
- [ ] promtool validation
Revision Task: [ ] NOT DONE

### Layer 8 — Security
- [ ] TLS for Prometheus UI
- [ ] Basic auth configured
- [ ] nginx reverse proxy
- [ ] Node Exporter access restricted
- [ ] Storage retention configured
- [ ] Backup strategy in place
Revision Task: [ ] NOT DONE

### Layer 9 — Docker Setup
- [ ] docker-compose.yml written
- [ ] Volumes configured
- [ ] Stack running in Docker
- [ ] Node Exporter Docker vs bare metal decision
Revision Task: [ ] NOT DONE

### Layer 10 — Advanced Features
- [ ] file_sd_configs
- [ ] Cloud service discovery concept
- [ ] Pushgateway concept
- [ ] Additional exporters
- [ ] Custom exporter in Python
- [ ] Federation concept
- [ ] Long-term storage concept
Revision Task: [ ] NOT DONE

### Layer 11 — Troubleshooting
- [ ] Target DOWN debugging steps
- [ ] Metrics wrong debugging
- [ ] Disk usage management
- [ ] Memory management
- [ ] Alert not firing debugging
- [ ] Grafana no data debugging
- [ ] Config validation process
Revision Task: [ ] NOT DONE

### Layer 12 — Apex Project
- [ ] Full setup from scratch — complete

## My Weak Spots
[None yet]

## Bridge Map Learned
- Pull Model -> Central server running `setInterval()` with Axios GET requests to 7 dumb servers.
- Time Series Data -> `metric_name{labels} value timestamp` (Labels are like MongoDB fields).
- Architecture -> Remote: Node Exporter. Central: Prometheus, Grafana, Alertmanager. PULL direction.
- Linux Folders -> `/etc` (configs/.env), `/var/lib` (MongoDB storage), `/usr/local/bin` (node.exe executable).

## Common Errors Encountered
[None yet]

## Spaced Repetition Queue
[None yet]

## IDE Transfer Log
[None yet]