RMM-UI-BE/06_meshcentral_setup_guide.md

# MeshCentral RMM — Complete Setup Guide
## From Zero to Connected (With All Challenges Documented)

---

## What is MeshCentral and Why We Need It

MeshCentral is a free, open-source Remote Monitoring and Management (RMM) tool. It lets you see all your remote machines in one browser dashboard, open root terminals, transfer files, and push scripts to 100 machines simultaneously — without using RustDesk or asking for passwords every time.

**The Goal:** Install MeshCentral on the central server once. Install the lightweight agent on each client machine once. After that, manage everything from a web browser forever.

---

## Architecture Overview

```
Client Machine (anywhere in the world)
  └── MeshCentral Agent (background service, runs as root)
  └── Tailscale (mesh VPN, gives stable 100.x.x.x IP)
        │
        │ Direct encrypted tunnel (no middleman)
        ▼
Central Server (your always-on Linux machine / WSL)
  └── MeshCentral Server (NodeJS app, port 4430)
  └── Tailscale (same mesh VPN, stable 100.x.x.x IP)
```

---

## Part 1: Setting Up the Central Server (Your Laptop/Desktop)

### Prerequisites
- Linux machine or WSL2 on Windows
- NodeJS v20+
- Tailscale account (free at tailscale.com)

### Step 1: Install NodeJS
```bash
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs
```
Verify: `node --version` should show v20.x.x

### Step 2: Install MeshCentral Locally
```bash
mkdir ~/meshcentral
cd ~/meshcentral
npm install meshcentral
```

> **CRITICAL:** Do NOT use `sudo npm install -g meshcentral`.
> Installing globally puts MeshCentral in `/usr/lib/` which is root-owned.
> MeshCentral will fail with `EACCES: permission denied, mkdir '/usr/lib/meshcentral-data'`.
> Always install locally in your home directory.

### Step 3: Install Tailscale on the Central Server
```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
```

> **CRITICAL — DO NOT PRESS Ctrl+C:**
> When `sudo tailscale up` runs, it prints an authentication URL and then WAITS.
> You MUST open that URL in a browser and log in FIRST.
> Only after successful login will the terminal print `Success.` and return to the prompt.
> If you press Ctrl+C before authenticating, Tailscale will be installed but NOT logged in.
> You will have to run `sudo tailscale up` again and get a NEW URL to authenticate.

Open the authentication URL in a browser. Log in with Google/GitHub. After the terminal prints `Success.`, get your stable IP:
```bash
tailscale ip -4
# Example output: 100.66.205.2
```
Save this IP — it is your permanent MeshCentral server address.

### Step 4: Configure MeshCentral
```bash
cat > ~/meshcentral/meshcentral-data/config.json << 'EOF'
{
  "settings": {
    "cert": "YOUR_TAILSCALE_IP",
    "port": 4430,
    "redirport": 4431
  },
  "domains": {
    "": {
      "title": "My RMM"
    }
  }
}
EOF
```
Replace `YOUR_TAILSCALE_IP` with your actual Tailscale IP (e.g., `100.66.205.2`).

### Step 5: Start MeshCentral
```bash
cd ~/meshcentral && node node_modules/meshcentral
```

Expected healthy startup output:
```
MeshCentral HTTP redirection server running on port 4431.
MeshCentral v1.1.59, Hybrid (LAN + WAN) mode.
MeshCentral HTTPS server running on 100.66.205.2:4430.
```

### Step 6: Access the Dashboard
Open your browser and go to:
```
https://localhost:4430
```
Accept the certificate warning (self-signed cert is normal).
**The first account you create automatically becomes the Site Administrator.**

### Step 7: Create a Device Group
In the dashboard:
- Click **My Devices**
- Click **Add Device Group**
- Name it (e.g., `Client Machines`)
- Click OK

---

## Part 2: Connecting a New Client Machine

Repeat these steps for every new machine you want to manage.

### Step 1: Install Tailscale on the Client Machine
```bash
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
```
Open the authentication URL. Log in with the **SAME Tailscale account** as the server. This is critical — both must be on the same account to communicate.

Verify connection:
```bash
tailscale ip -4
# Should give a 100.x.x.x IP
ping 100.66.205.2  # Should reach the server
```

### Step 2: Get the Agent Install Command
In the MeshCentral browser:
- Go to **My Devices**
- Click on your Device Group
- Click **Add Agent**
- Select **Linux / BSD**
- Copy the generated command

The command will look like:
```bash
(wget "https://100.66.205.2:4430/meshagents?script=1" --no-check-certificate -O ./meshinstall.sh) && chmod 755 ./meshinstall.sh && sudo ./meshinstall.sh https://100.66.205.2:4430 'YOUR_GROUP_KEY'
```

### Step 3: Run the Agent Install Command on the Client
On the client machine terminal:
```bash
cd /tmp && (wget "https://100.66.205.2:4430/meshagents?script=1" --no-check-certificate -O ./meshinstall.sh) && chmod 755 ./meshinstall.sh && sudo ./meshinstall.sh https://100.66.205.2:4430 'YOUR_GROUP_KEY'
```

Expected output:
```
Downloading agent #6...
Agent downloaded.
...Installing service [DONE]
-> Starting service... [OK]
```

### Step 4: Verify the Agent is Running
```bash
sudo systemctl status meshagent
```
Should show `active (running)`.

### Step 5: Confirm in the Dashboard
The machine should appear in MeshCentral within 10 seconds showing **"Agent, Powered"** with a blue refresh icon.

---

## Part 3: Using MeshCentral to Deploy Scripts

### Running a Script on One Machine
1. Go to **My Devices**
2. Right-click the machine → **Terminal** (for interactive use)
   OR
3. Check the checkbox next to the machine
4. Click **Group Action** → **Run commands**
5. Select **Linux/BSD/macOS Command Shell**
6. Select **Run as agent** (runs as root on Linux)
7. Select **Commands from file**
8. Click **Choose file** → select your `.sh` script
9. Click **OK**

### Running a Script on ALL 100 Machines Simultaneously
1. Click **Select All** at the top
2. Click **Group Action** → **Run commands**
3. Upload your script and click OK
4. Watch the output for each machine stream in simultaneously

---

## Part 4: Challenges and Issues Faced (With Solutions)

### Issue 1: Permission Denied on First Run
**Error:**
```
Error: EACCES: permission denied, mkdir '/usr/lib/meshcentral-data'
```
**Cause:** MeshCentral was installed globally with `sudo npm install -g`, placing it in the root-owned `/usr/lib/` directory.

**Fix:** Install locally without sudo:
```bash
mkdir ~/meshcentral && cd ~/meshcentral
npm install meshcentral
node node_modules/meshcentral
```

---

### Issue 2: Permission Denied When Saving Script in nano
**Error:** `[ Error writing test_install.sh: Permission denied ]`

**Cause:** Terminal was sitting in a restricted system folder (like `/` or `/root`).

**Fix:** Always create files in `/tmp` which allows all users to write:
```bash
cd /tmp
nano test_install.sh
```

---

### Issue 3: Script Fails with "command not found" After Clipboard Paste
**Error:** `test_install.sh: line 27: Update: command not found`

**Cause:** RustDesk/AnyDesk clipboard dropped the `#` symbol from comments. Linux interpreted the English word "Update" as a command.

**Fix:** Remove ALL comments and blank lines from scripts before pasting over remote desktop:
```bash
# Comments like this get corrupted when pasted via RustDesk
# Solution: Strip the script to pure commands only
```

---

### Issue 4: MeshCentral Running in LAN Mode Only
**Symptom:** MeshCentral startup shows `LAN mode` — agents from other networks cannot connect.

**Cause:** No `cert` was set in `config.json`. MeshCentral defaults to LAN-only mode.

**Fix:** Set the `cert` field in `config.json` to the server's public hostname or Tailscale IP:
```json
{
  "settings": {
    "cert": "100.66.205.2",
    "port": 4430
  }
}
```
Server will restart in `Hybrid (LAN + WAN) mode`.

---

### Issue 5: Agent Stuck in SYN-SENT State (Port Mismatch)
**Symptom:** Agent connects to `104.16.x.x:1025` and hangs forever.

**Cause:** MeshCentral was running on port 1025. The `.msh` file embedded port 1025 in the server URL. But Cloudflare Quick Tunnels only listen on port 443. The connection attempt to `:1025` on Cloudflare was silently blocked.

**Diagnosis:**
```bash
sudo ss -tp | grep meshagent
# Shows: SYN-SENT ... 104.16.x.x:1025
```

**Fix (with Cloudflare):** Use `setcap` to allow NodeJS to bind to port 443, and enable `tlsoffload` in config so the agent connects on the standard HTTPS port.

**Better Fix:** Switch to Tailscale (see Issue 6).

---

### Issue 6: Certificate Hash Mismatch with Cloudflare Tunnel
**Error:**
```
Agent bad web cert hash (Agent:9969c2afd9 != Server:a364860c2f)
holding connection (106.51.126.27:43912)
```

**Cause:** Cloudflare "terminates TLS" — it acts as a middleman and presents its own TLS certificate to the agent instead of MeshCentral's certificate. The agent's `.msh` file contains MeshCentral's certificate fingerprint. When the agent sees Cloudflare's certificate instead, the fingerprints don't match and the agent refuses to connect.

**Why this cannot be easily fixed with Cloudflare Quick Tunnels:** The only way to fix it properly is to use a real Cloudflare account with a named tunnel and an Origin Certificate — so MeshCentral uses the same certificate that Cloudflare presents externally.

**The Clean Fix: Use Tailscale Instead**

Tailscale does not terminate TLS. It creates a direct encrypted tunnel between machines without intercepting TLS certificates. The agent connects directly to MeshCentral and sees its real certificate — the fingerprint always matches.

**Setup:**
```bash
# On server
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up  # Login with Google/GitHub
tailscale ip -4    # Note this IP (e.g., 100.66.205.2)

# On client
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up  # DO NOT PRESS Ctrl+C — wait for browser login then Success.
```

> **CRITICAL — DO NOT PRESS Ctrl+C during `sudo tailscale up`:**
> The command prints a URL and waits. Open the URL in ANY browser (even on your laptop).
> Log in with the SAME account. The terminal will then print `Success.` on its own.
> If you press Ctrl+C too early, the machine will show as logged out.
> Run `tailscale status` to check — if it says `Logged out`, run `sudo tailscale up` again.

Both machines get stable `100.x.x.x` IPs. Update MeshCentral config to use the Tailscale IP. Agent connects cleanly.

---

### Issue 7: Windows Browser Cannot Reach WSL Tailscale IP
**Symptom:** Browser shows `ERR_CONNECTION_TIMED_OUT` when visiting `https://100.66.205.2:4430`

**Cause:** Windows cannot directly route to the Tailscale IP assigned to WSL. The Tailscale IP is only reachable from other Tailscale machines on the network, not from the Windows host itself.

**Fix:** From your own laptop's Windows browser, always use `localhost` to access WSL services:
```
https://localhost:4430  ← Use this from your laptop browser
https://100.x.x.x:4430 ← Use this from OTHER machines on Tailscale
```

---

### Issue 8: MeshCentral Terminal vs Regular Terminal Permissions
**Symptom:** Running `apt-get install` in the regular terminal gives `Permission denied`.

**Cause:** The regular terminal runs as the normal user (`kawsik`). The MeshCentral Terminal runs as `root` because the agent service itself runs as root.

**Rule:**
- Regular terminal: needs `sudo` for system commands
- MeshCentral Terminal: already root, never needs `sudo`

---

### Issue 9: Tailscale Shows "Logged Out" After Ctrl+C
**Symptom:** `tailscale status` shows `Logged out` even after running `sudo tailscale up`.

**Cause:** The user pressed Ctrl+C while `sudo tailscale up` was waiting for browser authentication. This cancelled the authentication process before it completed. Tailscale is installed but not authenticated.

**Fix:** Simply run `sudo tailscale up` again. A new authentication URL will be printed. Open it in any browser (including your laptop's browser), log in, and wait for `Success.` to appear in the terminal. Never press Ctrl+C while the URL is shown.

```bash
sudo tailscale up
# Opens URL → open it in browser → log in → wait for Success. → done
tailscale status  # Should now show Connected
```

---

### Issue 10: Native Linux Client Cannot Ping WSL Tailscale IP (100% Packet Loss)
**Symptom:** `ping 100.66.205.2` from a native Linux client (like PopOS) gives 100% packet loss. The Tailscale dashboard shows both machines as Connected but they cannot communicate.

**Cause:** The MeshCentral server's Tailscale is installed inside **WSL2**, which is behind double NAT — the home router AND Windows' internal NAT. Other WSL machines (like `bs13`) can reach it through Tailscale relay servers. But native Linux machines sometimes cannot punch through both NAT layers to reach a WSL Tailscale node.

**Diagnosis:**
```bash
tailscale status          # Check if machine is actually online
tailscale ping 100.66.205.2  # Tailscale's own connectivity check
```

**Fix Option 1 (Simple):** Make sure Tailscale on the client is actually authenticated (run `tailscale status` — if it says `Logged out`, re-authenticate with `sudo tailscale up`).

**Fix Option 2 (Production):** Install Tailscale on the **Windows host** instead of WSL, giving it a proper host-level IP without double NAT. All clients can then reach the Windows Tailscale IP directly.

**Fix Option 3 (Production):** Move MeshCentral to a cloud VPS (Oracle Free Tier, etc.) so it has a real public IP with no NAT issues at all.

---

### Issue 11: RustDesk "Failed to get capturer display info" on COSMIC Desktop (PopOS 24.04)
**Symptom:** RustDesk connects, password accepted, but immediately shows `Failed to get capturer display info`.

**Cause:** PopOS 24.04 uses the **COSMIC desktop environment** which is Wayland-only. There is no X11 option. RustDesk has limited support for Wayland/COSMIC screen capture.

**Root Cause Detail:** `echo $XDG_SESSION_TYPE` returns empty when run over SSH because no display session is attached to the SSH terminal. Running `loginctl list-sessions` showed session on `tty1` — a text console, not a graphical session. The machine was sitting at the COSMIC login screen with no user logged into the GUI.

**Identification Commands:**
```bash
cat /etc/X11/default-display-manager  # Shows: /usr/bin/cosmic-greeter
echo $XDG_SESSION_TYPE                # Empty = no display in this terminal
loginctl list-sessions                # Shows TTY type (tty1 = text, not GUI)
```

**Fix:** Enable auto-login for COSMIC so the machine automatically logs into the desktop on boot:
```bash
sudo mkdir -p /etc/sddm.conf.d
sudo nano /etc/sddm.conf.d/autologin.conf
```
Add:
```ini
[Autologin]
User=byfar
Session=cosmic
```
Then reboot. After reboot, COSMIC logs in automatically and RustDesk can capture the screen.

**Alternative:** Skip RustDesk entirely. Use MeshCentral's built-in **Terminal** (root shell) and **Web-VNC** features for remote access instead. These work regardless of display server type.

---

## Part 5: Quick Reference — Adding Machine #N to MeshCentral

This is the exact 3-command sequence to run on any new Linux client machine:

```bash
# Step 1: Install Tailscale
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
# ⚠️  A URL will appear — open it in your laptop browser and log in with the SAME account
# ⚠️  DO NOT press Ctrl+C — wait until the terminal prints Success.
tailscale status   # Must show Connected before proceeding

# Step 2: Install MeshCentral Agent
cd /tmp && (wget "https://100.66.205.2:4430/meshagents?script=1" --no-check-certificate -O ./meshinstall.sh) && chmod 755 ./meshinstall.sh && sudo ./meshinstall.sh https://100.66.205.2:4430 'YOUR_GROUP_KEY'

# Step 3: Verify
sudo systemctl status meshagent
```

The machine will appear in the MeshCentral dashboard within 10 seconds.

> **If the agent install hangs connecting:** Run `tailscale status` — if it shows `Logged out`, the Tailscale session expired. Run `sudo tailscale up` again, authenticate in the browser, then retry the agent install.

---

## Part 6: Production Checklist

- [ ] Central server is always-on (disable sleep/hibernate)
- [ ] MeshCentral starts automatically on server reboot (configure as systemd service)
- [ ] Tailscale starts automatically on all machines (enabled by default after install)
- [ ] All client machines enrolled in Tailscale with the same account
- [ ] MeshCentral agent enabled as systemd service on all clients (`systemctl enable meshagent`)
- [ ] Node Exporter deployed and running on all clients (port 9100)
- [ ] All Tailscale IPs added to `prometheus.yml` as scrape targets