# Deploying SAM3 via HuggingFace, then going fully offline

The SAM3 backend uses Meta's `facebook/sam3` weights via HuggingFace. The
flow is **online once** (to download + cache the weights) → **then
air-gapped** (server runs without ever talking to HuggingFace again).
This doc walks through both phases and how to verify your box is actually
offline at the end.

---

## Phase 0 — prerequisites

On the target PC:

- The `sam2-backend/` folder, copied or cloned, with its `.venv`
  containing `torch`, `transformers>=4.57`, `huggingface_hub`, and the
  rest of `requirements.txt`. See [`sam2-backend/README.md`](../sam2-backend/README.md)
  for the full install.
- A HuggingFace account that has been **granted access to
  `facebook/sam3`** — gated. Request access at
  https://huggingface.co/facebook/sam3 (usually approved within minutes).
- A read token from https://huggingface.co/settings/tokens.

---

## Phase 1 — first-time online download

The first call to `Sam3TrackerModel.from_pretrained("facebook/sam3")`
streams the weights from HF and writes them to
`~/.cache/huggingface/hub/`. Everything afterward can be served from that
cache with no network.

### 1.1 Authenticate

```bash
cd /home/<you>/sam2-backend
source .venv/bin/activate
hf auth login                 # paste your read token
hf auth whoami                # confirm the right account
```

Token is stored at `~/.cache/huggingface/token`. It's a plain file with
`chmod 600`.

### 1.2 Set offline flag to OFF for the first start

In `run.sh`, the relevant lines are:
```bash
export HF_HUB_DISABLE_TELEMETRY="${HF_HUB_DISABLE_TELEMETRY:-1}"
export DO_NOT_TRACK="${DO_NOT_TRACK:-1}"
export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-0}"        # ← MUST be 0 first time
export TRANSFORMERS_OFFLINE="${TRANSFORMERS_OFFLINE:-0}"   # ← MUST be 0 first time
```

For a one-off override, just don't pass the env vars:
```bash
./run.sh
```

### 1.3 Make ONE successful predict call

Either from the Tauri app (Settings → pick `SAM 3 · tracker (HuggingFace)` →
draw a bbox), or directly:

```bash
curl -s -X POST "http://localhost:9090/predict?model=sam3_tracker" \
  -H "Content-Type: application/json" \
  --data '@some_test_payload.json'
```

A successful response means the weights are now fully cached. Confirm:

```bash
ls -la ~/.cache/huggingface/hub/models--facebook--sam3/
# expect a snapshots/<hash>/ subdir with model.safetensors and configs
du -sh ~/.cache/huggingface/hub/models--facebook--sam3/
# ~3-4 GB for SAM3
```

### 1.4 Optional — pre-warm without a real predict

If you want to download but not yet wire up Tauri, do it from a Python
shell:

```bash
python3 - <<'EOF'
from transformers import Sam3TrackerModel, Sam3TrackerProcessor
Sam3TrackerProcessor.from_pretrained("facebook/sam3")
Sam3TrackerModel.from_pretrained("facebook/sam3")
print("cached.")
EOF
```

Same effect — fills `~/.cache/huggingface/hub/`.

---

## Phase 2 — flip to fully offline

Once the cache is populated, set the offline flags so the server never
tries to revalidate metadata or check for updates.

### 2.1 Edit `run.sh`

Change the defaults to `1`:

```bash
export HF_HUB_DISABLE_TELEMETRY="${HF_HUB_DISABLE_TELEMETRY:-1}"
export DO_NOT_TRACK="${DO_NOT_TRACK:-1}"
export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-1}"           # ← 1
export TRANSFORMERS_OFFLINE="${TRANSFORMERS_OFFLINE:-1}"   # ← 1
```

Or, if you're running via systemd, add to your unit file:

```ini
[Service]
Environment=HF_HUB_DISABLE_TELEMETRY=1
Environment=DO_NOT_TRACK=1
Environment=HF_HUB_OFFLINE=1
Environment=TRANSFORMERS_OFFLINE=1
```

Then `sudo systemctl daemon-reload && sudo systemctl restart sam-backend`.

### 2.2 Restart and verify

```bash
./run.sh
```

You should see clean startup logs **without** any `httpx HTTP Request`
lines hitting `huggingface.co`. The `Loading weights: 685/685 [00:00<00:00]`
line confirms a cache hit (instant load, no download).

### 2.3 Pin a custom cache location (optional)

If you want the cache somewhere predictable (for backup, rsync to other
boxes, or to keep `$HOME` clean), set `HF_HOME` in `run.sh`:

```bash
export HF_HOME=/var/cache/sam-backend/huggingface
```

Move the existing cache once:
```bash
mv ~/.cache/huggingface /var/cache/sam-backend/
```

Subsequent reads come from there.

---

## Verifying you're really offline

Two-stage check:

### Step 1 — disconnect & restart

Easiest sanity test: pull the network cable (or `sudo ip link set eth0 down`),
restart the server, hit `/predict` from the Tauri app on the same box.
If it returns a polygon, you're truly offline.

### Step 2 — packet-level proof

If you can't disconnect (e.g., other services need the network), watch
for any HF traffic during normal use:

```bash
sudo tcpdump -nn -i any \
  'host huggingface.co or host cdn-lfs.huggingface.co' \
  2>/dev/null
```

Restart the backend, hit `/predict` ten times from the Tauri app. With
the OFFLINE flags set you should see **zero packets**.

---

## Moving the cache to a different machine

Useful when you want to deploy to a new box without the first-run
download (e.g., setting up a second annotation station behind the same
firewall).

```bash
# On the source machine:
tar czf hf-sam3-cache.tar.gz -C ~/.cache huggingface
scp hf-sam3-cache.tar.gz target-box:/tmp/

# On the target machine:
mkdir -p ~/.cache
tar xzf /tmp/hf-sam3-cache.tar.gz -C ~/.cache
```

Then start the new backend with `HF_HUB_OFFLINE=1` from day one — it'll
load from the imported cache without ever calling out.

---

## Common gotchas

| Symptom                                                                  | Cause / fix                                                                 |
|--------------------------------------------------------------------------|-----------------------------------------------------------------------------|
| `Cannot find the requested files in the cached path` while OFFLINE=1     | Cache is incomplete or in a different `HF_HOME`. Re-run online once.        |
| `gated repo, access denied`                                               | Token doesn't have `facebook/sam3` access. Accept gate on the model page, then `hf auth login` again. |
| Many `HEAD https://huggingface.co/...` lines on startup despite OFFLINE=1 | One of the two env flags is missing. Both `HF_HUB_OFFLINE=1` AND `TRANSFORMERS_OFFLINE=1` are required. |
| `Loading weights: ...` takes minutes even with cache                     | The cache file is on a slow disk (NFS, HDD). Move it to local SSD via `HF_HOME`. |
| Token leaked into logs                                                   | `hf auth login` only echoes your username, never the token. The token at `~/.cache/huggingface/token` is the only place it's stored. |

---

## What HF still sees in offline mode

**Nothing.** With both `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`:

- `from_pretrained` reads only from `~/.cache/huggingface/hub/` — no API
  calls, no HEAD revalidation pings.
- Telemetry is independently off via `HF_HUB_DISABLE_TELEMETRY=1` /
  `DO_NOT_TRACK=1`.
- Inference itself never touches the network — your images, bboxes, and
  resulting masks stay on the box regardless of online/offline mode.

The only thing HF "knows" about you, ever, is "this account downloaded
`facebook/sam3` once on date X" — recorded server-side at the moment
`from_pretrained` first ran in Phase 1.