91 lines
4.1 KiB
Markdown
91 lines
4.1 KiB
Markdown
# The "Push" Architecture (Prometheus Remote Write)
|
|
|
|
When a client refuses to use VPNs or Outbound Tunnels (like Cloudflare), we must use the **Push Architecture**. In this model, the remote client acts like a normal web browser. It wakes up every 15 seconds, packages the CPU/GPU data, and securely "uploads" it over HTTPS to your Central Server.
|
|
|
|
## 1. The Architecture Flow Diagram
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph "Paranoid Client Network (100% Locked Down)"
|
|
NE[Node Exporter<br>Port 9100]
|
|
GE[GPU Exporter<br>Port 9835]
|
|
PA[Prometheus Agent<br>Scrapes local ports]
|
|
|
|
NE -.->|Scraped locally| PA
|
|
GE -.->|Scraped locally| PA
|
|
end
|
|
|
|
subgraph "The Public Internet"
|
|
HTTPS[HTTPS POST Request<br>Username & Password]
|
|
end
|
|
|
|
subgraph "Your Central Office Network"
|
|
ROUTER[Office Router<br>Port Forward 443]
|
|
NGINX[Nginx Reverse Proxy<br>Checks Password & SSL]
|
|
PROM[Central Prometheus<br>Port 9090 Receiver]
|
|
GRAF[Grafana<br>Port 3000]
|
|
end
|
|
|
|
PA ===>|Pushes Data Out| HTTPS
|
|
HTTPS ===>|Arrives at Public IP| ROUTER
|
|
ROUTER ===>|Routes to vm3| NGINX
|
|
NGINX ===>|If Password OK| PROM
|
|
PROM ===>|Stores Data| GRAF
|
|
```
|
|
|
|
## 2. What Needs to be Installed?
|
|
|
|
### On the Central Server (`vm3`)
|
|
Your Central Server acts as the public "mailbox" receiving data from 100 clients.
|
|
* **Prometheus:** Standard installation, but we must turn on the "Receiver" switch.
|
|
* **Nginx:** A highly secure web server. Prometheus does not have built-in passwords for receiving data. We put Nginx in front of Prometheus to act as a bouncer. If the client doesn't provide the correct username/password, Nginx blocks them.
|
|
* **Certbot (Let's Encrypt):** A tool that generates a free SSL certificate so the data travels over secure HTTPS.
|
|
|
|
### On the Remote Client
|
|
The client's firewall remains completely closed. No inbound connections are allowed.
|
|
* **Node & GPU Exporters:** These run normally, gathering hardware data locally.
|
|
* **Prometheus Agent:** We install a tiny version of Prometheus called "Agent Mode". It does not store data. Its only job is to scrape the local Node Exporter and push the data out to the internet.
|
|
|
|
---
|
|
|
|
## 3. How the Configuration Happens (Step-by-Step)
|
|
|
|
### Step 1: Central Router Setup
|
|
You log into your physical Cisco/ISP router at your main office. You create a Port Forwarding rule: **Route all internet traffic on Port 443 to the local IP of `vm3` (e.g., `192.168.1.199`).**
|
|
|
|
### Step 2: Nginx Bouncer Setup (Central Server)
|
|
On `vm3`, you install Nginx and create a password file (`.htpasswd`). You configure Nginx with this rule:
|
|
> *"If a request comes in on HTTPS (443), check if they have the password. If yes, forward the data to Prometheus on local Port 9090."*
|
|
|
|
### Step 3: Prometheus Receiver (Central Server)
|
|
By default, Prometheus is a "Pull" system. We edit the `prometheus.service` systemd file and add the startup flag: `--web.enable-remote-write-receiver`. This tells Prometheus it is allowed to catch incoming pushed data.
|
|
|
|
### Step 4: The Agent Setup (Remote Client)
|
|
On the remote client, we install the Prometheus binary, but we run it with a special configuration file (`agent.yml`).
|
|
|
|
The configuration looks exactly like this:
|
|
```yaml
|
|
# 1. Gather data from the local machine
|
|
scrape_configs:
|
|
- job_name: 'client_hardware'
|
|
static_configs:
|
|
- targets: ['localhost:9100', 'localhost:9835']
|
|
|
|
# 2. Push the data out to the Central Server
|
|
remote_write:
|
|
- url: "https://metrics.seekright.com/api/v1/write"
|
|
basic_auth:
|
|
username: "admin"
|
|
password: "SuperSecretPassword123"
|
|
```
|
|
|
|
## 4. The Final Result
|
|
1. The Prometheus Agent on the client wakes up.
|
|
2. It looks at `localhost:9100` and sees "CPU is at 45%".
|
|
3. It packages that string into a tiny web request.
|
|
4. It attaches the username `admin` and the password `SuperSecretPassword123` to the header.
|
|
5. It fires the request out across the public internet to `https://metrics.seekright.com/api/v1/write`.
|
|
6. Your office router catches it and sends it to `vm3`.
|
|
7. Nginx on `vm3` checks the password, says "Access Granted", and hands the CPU data to Prometheus.
|
|
8. You see the CPU data appear on Grafana.
|