The "Push" Architecture (Prometheus Remote Write)

When a client refuses to use VPNs or Outbound Tunnels (like Cloudflare), we must use the Push Architecture. In this model, the remote client acts like a normal web browser. It wakes up every 15 seconds, packages the CPU/GPU data, and securely "uploads" it over HTTPS to your Central Server.

1. The Architecture Flow Diagram

graph LR
    subgraph "Paranoid Client Network (100% Locked Down)"
        NE[Node Exporter<br>Port 9100]
        GE[GPU Exporter<br>Port 9835]
        PA[Prometheus Agent<br>Scrapes local ports]
        
        NE -.->|Scraped locally| PA
        GE -.->|Scraped locally| PA
    end

    subgraph "The Public Internet"
        HTTPS[HTTPS POST Request<br>Username & Password]
    end

    subgraph "Your Central Office Network"
        ROUTER[Office Router<br>Port Forward 443]
        NGINX[Nginx Reverse Proxy<br>Checks Password & SSL]
        PROM[Central Prometheus<br>Port 9090 Receiver]
        GRAF[Grafana<br>Port 3000]
    end

    PA ===>|Pushes Data Out| HTTPS
    HTTPS ===>|Arrives at Public IP| ROUTER
    ROUTER ===>|Routes to vm3| NGINX
    NGINX ===>|If Password OK| PROM
    PROM ===>|Stores Data| GRAF

2. What Needs to be Installed?

On the Central Server (`vm3`)

Your Central Server acts as the public "mailbox" receiving data from 100 clients.

Prometheus: Standard installation, but we must turn on the "Receiver" switch.
Nginx: A highly secure web server. Prometheus does not have built-in passwords for receiving data. We put Nginx in front of Prometheus to act as a bouncer. If the client doesn't provide the correct username/password, Nginx blocks them.
Certbot (Let's Encrypt): A tool that generates a free SSL certificate so the data travels over secure HTTPS.

On the Remote Client

The client's firewall remains completely closed. No inbound connections are allowed.

Node & GPU Exporters: These run normally, gathering hardware data locally.
Prometheus Agent: We install a tiny version of Prometheus called "Agent Mode". It does not store data. Its only job is to scrape the local Node Exporter and push the data out to the internet.

3. How the Configuration Happens (Step-by-Step)

Step 1: Central Router Setup

You log into your physical Cisco/ISP router at your main office. You create a Port Forwarding rule: Route all internet traffic on Port 443 to the local IP of vm3 (e.g., 192.168.1.199).

Step 2: Nginx Bouncer Setup (Central Server)

On vm3, you install Nginx and create a password file (.htpasswd). You configure Nginx with this rule:

"If a request comes in on HTTPS (443), check if they have the password. If yes, forward the data to Prometheus on local Port 9090."

Step 3: Prometheus Receiver (Central Server)

By default, Prometheus is a "Pull" system. We edit the prometheus.service systemd file and add the startup flag: --web.enable-remote-write-receiver. This tells Prometheus it is allowed to catch incoming pushed data.

Step 4: The Agent Setup (Remote Client)

On the remote client, we install the Prometheus binary, but we run it with a special configuration file (agent.yml).

The configuration looks exactly like this:

# 1. Gather data from the local machine
scrape_configs:
  - job_name: 'client_hardware'
    static_configs:
      - targets: ['localhost:9100', 'localhost:9835']

# 2. Push the data out to the Central Server
remote_write:
  - url: "https://metrics.seekright.com/api/v1/write"
    basic_auth:
      username: "admin"
      password: "SuperSecretPassword123"

4. The Final Result

The Prometheus Agent on the client wakes up.
It looks at localhost:9100 and sees "CPU is at 45%".
It packages that string into a tiny web request.
It attaches the username admin and the password SuperSecretPassword123 to the header.
It fires the request out across the public internet to https://metrics.seekright.com/api/v1/write.
Your office router catches it and sends it to vm3.
Nginx on vm3 checks the password, says "Access Granted", and hands the CPU data to Prometheus.
You see the CPU data appear on Grafana.

4.1 KiB Raw Blame History