initial commit

2026-06-02 10:37:57 +05:30
commit b4acb8d1df
49 changed files with 15350 additions and 0 deletions
--- a/08_central_server_production_setup.md
+++ b/08_central_server_production_setup.md
@@ -0,0 +1,469 @@
+# Central Server Production Setup
+## Fresh Linux Machine → Full Monitoring + RMM Stack
+
+---
+
+## What This Document Covers
+
+Setting up your central server from a fresh Ubuntu/PopOS Linux install with:
+- **Tailscale** — private network backbone connecting all machines
+- **MeshCentral** — RMM dashboard for remote management of all client machines
+- **Prometheus** — metrics collection engine
+- **Alertmanager** — alert routing to Rocket.Chat and Gmail
+- **Grafana** — visualization dashboards
+
+---
+
+## Requirements
+
+| Requirement | Minimum | Recommended |
+|---|---|---|
+| OS | Ubuntu 22.04 / 24.04 LTS | Ubuntu 24.04 LTS (native, NOT WSL) |
+| RAM | 2 GB | 4 GB |
+| Disk | 20 GB | 50 GB |
+| CPU | 2 cores | 4 cores |
+| Network | Always-on internet | Always-on internet |
+| Power | No sleep/hibernate | UPS recommended |
+
+> ⚠️ **WSL is for testing only.** For production, use a native Linux machine (bare metal or VM).
+> WSL does not auto-start services on Windows boot and has double-NAT networking issues.
+
+---
+
+## Accounts You Need Before Starting
+
+### 1. Tailscale Account (Free)
+- Go to: **https://tailscale.com**
+- Click **Get Started**
+- Sign in with Google or GitHub — whichever you use on all machines
+- You do not need to configure anything on the website — just create the account
+- Free plan supports up to 100 machines, which is exactly what we need
+
+### 2. Tailscale is the ONLY external account required
+MeshCentral, Prometheus, Alertmanager, and Grafana are all self-hosted on your machine.
+No cloud accounts needed for any of them.
+
+---
+
+## Part 1: System Preparation
+
+### Update the system
+```bash
+sudo apt-get update && sudo apt-get upgrade -y
+```
+
+### Disable sleep and hibernate (critical for a server)
+```bash
+sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
+```
+
+### Create a dedicated user for running services (optional but recommended)
+```bash
+sudo useradd -r -s /bin/false prometheus
+sudo useradd -r -s /bin/false alertmanager
+```
+
+---
+
+## Part 2: Install Tailscale
+
+```bash
+curl -fsSL https://tailscale.com/install.sh | sh
+sudo tailscale up
+```
+
+> ⚠️ A URL will appear. Open it in a browser and log in with your Tailscale account.
+> **DO NOT press Ctrl+C** — wait for `Success.`
+
+Get your server's stable Tailscale IP:
+```bash
+tailscale ip -4
+# Example: 100.66.205.2  ← save this, you will use it everywhere
+```
+
+Verify Tailscale starts on boot (it does by default):
+```bash
+sudo systemctl is-enabled tailscaled
+# Should print: enabled
+```
+
+---
+
+## Part 3: Install NodeJS and MeshCentral
+
+### Install NodeJS v20
+```bash
+curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
+sudo apt-get install -y nodejs
+node --version   # Should show v20.x.x
+```
+
+### Install MeshCentral locally
+```bash
+mkdir ~/meshcentral
+cd ~/meshcentral
+npm install meshcentral
+```
+
+> ⚠️ Do NOT use `sudo npm install -g meshcentral` — permission errors will occur.
+
+### Configure MeshCentral
+Replace `YOUR_TAILSCALE_IP` with the IP from Part 2:
+```bash
+mkdir -p ~/meshcentral/meshcentral-data
+cat > ~/meshcentral/meshcentral-data/config.json << 'EOF'
+{
+  "settings": {
+    "cert": "YOUR_TAILSCALE_IP",
+    "port": 4430,
+    "redirport": 4431
+  },
+  "domains": {
+    "": {
+      "title": "My RMM"
+    }
+  }
+}
+EOF
+```
+
+### Test MeshCentral starts correctly
+```bash
+cd ~/meshcentral && node node_modules/meshcentral
+```
+
+Expected output:
+```
+MeshCentral HTTP redirection server running on port 4431.
+MeshCentral v1.1.x, Hybrid (LAN + WAN) mode.
+MeshCentral HTTPS server running on YOUR_TAILSCALE_IP:4430.
+```
+
+Open browser at `https://localhost:4430` — accept the certificate warning.
+**Create your admin account** — the first account created is automatically the Site Administrator.
+Then press **Ctrl+C** to stop it. We will run it as a service next.
+
+### Run MeshCentral as a systemd service (auto-starts on reboot)
+```bash
+cat > /tmp/meshcentral.service << EOF
+[Unit]
+Description=MeshCentral RMM Server
+After=network.target
+
+[Service]
+Type=simple
+User=$USER
+WorkingDirectory=$HOME/meshcentral
+ExecStart=/usr/bin/node $HOME/meshcentral/node_modules/meshcentral
+Restart=always
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+sudo mv /tmp/meshcentral.service /etc/systemd/system/meshcentral.service
+sudo systemctl daemon-reload
+sudo systemctl enable meshcentral
+sudo systemctl start meshcentral
+sudo systemctl status meshcentral
+```
+
+Verify it is running:
+```bash
+sudo systemctl status meshcentral
+# Must show: active (running)
+```
+
+---
+
+## Part 4: Install Prometheus
+
+### Download and install
+```bash
+cd /tmp
+PROM_VERSION="2.51.2"
+wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VERSION}/prometheus-${PROM_VERSION}.linux-amd64.tar.gz
+tar xvf prometheus-${PROM_VERSION}.linux-amd64.tar.gz
+sudo mv prometheus-${PROM_VERSION}.linux-amd64/prometheus /usr/local/bin/
+sudo mv prometheus-${PROM_VERSION}.linux-amd64/promtool /usr/local/bin/
+sudo mkdir -p /etc/prometheus /var/lib/prometheus
+sudo mv prometheus-${PROM_VERSION}.linux-amd64/prometheus.yml /etc/prometheus/
+sudo mv prometheus-${PROM_VERSION}.linux-amd64/consoles /etc/prometheus/
+sudo mv prometheus-${PROM_VERSION}.linux-amd64/console_libraries /etc/prometheus/
+```
+
+### Configure prometheus.yml
+```bash
+sudo nano /etc/prometheus/prometheus.yml
+```
+
+Replace the entire file contents with:
+```yaml
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+alerting:
+  alertmanagers:
+    - static_configs:
+        - targets: ['localhost:9093']
+
+rule_files:
+  - "/etc/prometheus/rules.yml"
+
+scrape_configs:
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: ['localhost:9090']
+
+  # Add client machines here as you connect them via Tailscale
+  # - job_name: 'node_exporter'
+  #   static_configs:
+  #     - targets:
+  #       - '100.119.113.98:9100'   # bs13
+  #       - '100.76.222.82:9100'    # pop-os / byfar
+```
+
+### Create alert rules file
+```bash
+sudo tee /etc/prometheus/rules.yml << 'EOF'
+groups:
+  - name: server_alerts
+    rules:
+      - alert: ServerDown
+        expr: up == 0
+        for: 1m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Server {{ $labels.instance }} is down"
+          description: "{{ $labels.instance }} has been unreachable for more than 1 minute."
+
+      - alert: HighCPU
+        expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High CPU on {{ $labels.instance }}"
+
+      - alert: HighRAM
+        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "High RAM on {{ $labels.instance }}"
+
+      - alert: LowDisk
+        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Low disk on {{ $labels.instance }}"
+EOF
+```
+
+### Run Prometheus as a systemd service
+```bash
+sudo tee /etc/systemd/system/prometheus.service << 'EOF'
+[Unit]
+Description=Prometheus Monitoring
+After=network.target
+
+[Service]
+User=root
+ExecStart=/usr/local/bin/prometheus \
+  --config.file=/etc/prometheus/prometheus.yml \
+  --storage.tsdb.path=/var/lib/prometheus \
+  --storage.tsdb.retention.time=30d \
+  --storage.tsdb.retention.size=10GB \
+  --web.listen-address=0.0.0.0:9090
+Restart=always
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+sudo systemctl daemon-reload
+sudo systemctl enable prometheus
+sudo systemctl start prometheus
+sudo systemctl status prometheus
+```
+
+Verify: Open `http://localhost:9090` in your browser — you should see the Prometheus UI.
+
+---
+
+## Part 5: Install Alertmanager
+
+### Download and install
+```bash
+cd /tmp
+AM_VERSION="0.27.0"
+wget https://github.com/prometheus/alertmanager/releases/download/v${AM_VERSION}/alertmanager-${AM_VERSION}.linux-amd64.tar.gz
+tar xvf alertmanager-${AM_VERSION}.linux-amd64.tar.gz
+sudo mv alertmanager-${AM_VERSION}.linux-amd64/alertmanager /usr/local/bin/
+sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
+```
+
+### Configure Alertmanager
+```bash
+sudo tee /etc/alertmanager/alertmanager.yml << 'EOF'
+global:
+  resolve_timeout: 5m
+
+route:
+  group_by: ['alertname']
+  group_wait: 30s
+  group_interval: 5m
+  repeat_interval: 1h
+  receiver: 'default'
+
+receivers:
+  - name: 'default'
+    webhook_configs:
+      - url: 'YOUR_ROCKETCHAT_WEBHOOK_URL'
+        send_resolved: true
+EOF
+```
+Replace `YOUR_ROCKETCHAT_WEBHOOK_URL` with your actual Rocket.Chat webhook URL.
+
+### Run as systemd service
+```bash
+sudo tee /etc/systemd/system/alertmanager.service << 'EOF'
+[Unit]
+Description=Alertmanager
+After=network.target
+
+[Service]
+User=root
+ExecStart=/usr/local/bin/alertmanager \
+  --config.file=/etc/alertmanager/alertmanager.yml \
+  --storage.path=/var/lib/alertmanager
+Restart=always
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+sudo systemctl daemon-reload
+sudo systemctl enable alertmanager
+sudo systemctl start alertmanager
+sudo systemctl status alertmanager
+```
+
+---
+
+## Part 6: Install Grafana
+
+```bash
+sudo apt-get install -y apt-transport-https software-properties-common wget
+sudo mkdir -p /etc/apt/keyrings/
+wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
+echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
+sudo apt-get update
+sudo apt-get install -y grafana
+
+sudo systemctl daemon-reload
+sudo systemctl enable grafana-server
+sudo systemctl start grafana-server
+sudo systemctl status grafana-server
+```
+
+Verify: Open `http://localhost:3000` — default login is `admin` / `admin`. Change the password on first login.
+
+### Connect Grafana to Prometheus
+1. Go to **Connections → Data Sources → Add data source**
+2. Select **Prometheus**
+3. Set URL to `http://localhost:9090`
+4. Click **Save & Test** — should show "Data source is working"
+
+### Import recommended dashboards
+- Go to **Dashboards → Import**
+- Enter ID `1860` → Load → Import (Node Exporter Full)
+- Enter ID `14574` → Load → Import (NVIDIA GPU Metrics)
+
+---
+
+## Part 7: Verify All Services Are Running
+
+```bash
+sudo systemctl status meshcentral prometheus alertmanager grafana-server tailscaled
+```
+
+All five should show `active (running)`.
+
+Quick port check:
+```bash
+ss -tlnp | grep -E "4430|9090|9093|3000"
+```
+
+Expected:
+```
+LISTEN  *:4430   → MeshCentral
+LISTEN  *:9090   → Prometheus
+LISTEN  *:9093   → Alertmanager
+LISTEN  *:3000   → Grafana
+```
+
+---
+
+## Part 8: Adding Client Machines
+
+For each new client machine, see `07_connect_remote_machine.md`.
+
+After the client is connected to MeshCentral and Tailscale, add its Tailscale IP to Prometheus:
+
+```bash
+sudo nano /etc/prometheus/prometheus.yml
+```
+
+Add under `scrape_configs`:
+```yaml
+  - job_name: 'node_exporter'
+    static_configs:
+      - targets:
+          - '100.x.x.x:9100'    # machine name
+```
+
+Reload Prometheus without restarting:
+```bash
+curl -X POST http://localhost:9090/-/reload
+```
+
+---
+
+## Part 9: Production Checklist
+
+- [ ] Native Linux (not WSL) — services survive reboots
+- [ ] All 5 services enabled with `systemctl enable`
+- [ ] Sleep/hibernate disabled
+- [ ] MeshCentral admin account created and saved
+- [ ] Tailscale logged in and showing `Connected`
+- [ ] Grafana default password changed
+- [ ] At least one client machine connected and showing in Prometheus targets
+- [ ] At least one Grafana dashboard imported and showing data
+- [ ] Test alert fired and received in Rocket.Chat
+
+---
+
+## Quick Reference
+
+| Service | Port | URL |
+|---|---|---|
+| MeshCentral | 4430 | `https://localhost:4430` |
+| Prometheus | 9090 | `http://localhost:9090` |
+| Alertmanager | 9093 | `http://localhost:9093` |
+| Grafana | 3000 | `http://localhost:3000` |
+| Tailscale | — | `tailscale status` |
+
+| Command | Purpose |
+|---|---|
+| `sudo systemctl restart meshcentral` | Restart RMM server |
+| `sudo systemctl restart prometheus` | Restart metrics collector |
+| `curl -X POST http://localhost:9090/-/reload` | Reload Prometheus config without restart |
+| `tailscale ip -4` | Get this machine's Tailscale IP |
+| `tailscale status` | See all connected machines |