Files

ravi 413f06fb97 performance improvement

2026-05-28 14:00:38 +05:30

7.2 KiB

Raw Blame History

Deploying SAM3 via HuggingFace, then going fully offline

The SAM3 backend uses Meta's facebook/sam3 weights via HuggingFace. The flow is online once (to download + cache the weights) → then air-gapped (server runs without ever talking to HuggingFace again). This doc walks through both phases and how to verify your box is actually offline at the end.

Phase 0 — prerequisites

On the target PC:

The sam2-backend/ folder, copied or cloned, with its .venv containing torch, transformers>=4.57, huggingface_hub, and the rest of requirements.txt. See sam2-backend/README.md for the full install.
A HuggingFace account that has been granted access to facebook/sam3 — gated. Request access at https://huggingface.co/facebook/sam3 (usually approved within minutes).
A read token from https://huggingface.co/settings/tokens.

Phase 1 — first-time online download

The first call to Sam3TrackerModel.from_pretrained("facebook/sam3") streams the weights from HF and writes them to ~/.cache/huggingface/hub/. Everything afterward can be served from that cache with no network.

1.1 Authenticate

cd /home/<you>/sam2-backend
source .venv/bin/activate
hf auth login                 # paste your read token
hf auth whoami                # confirm the right account

Token is stored at ~/.cache/huggingface/token. It's a plain file with chmod 600.

1.2 Set offline flag to OFF for the first start

In run.sh, the relevant lines are:

export HF_HUB_DISABLE_TELEMETRY="${HF_HUB_DISABLE_TELEMETRY:-1}"
export DO_NOT_TRACK="${DO_NOT_TRACK:-1}"
export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-0}"        # ← MUST be 0 first time
export TRANSFORMERS_OFFLINE="${TRANSFORMERS_OFFLINE:-0}"   # ← MUST be 0 first time

For a one-off override, just don't pass the env vars:

./run.sh

1.3 Make ONE successful predict call

Either from the Tauri app (Settings → pick SAM 3 · tracker (HuggingFace) → draw a bbox), or directly:

curl -s -X POST "http://localhost:9090/predict?model=sam3_tracker" \
  -H "Content-Type: application/json" \
  --data '@some_test_payload.json'

A successful response means the weights are now fully cached. Confirm:

ls -la ~/.cache/huggingface/hub/models--facebook--sam3/
# expect a snapshots/<hash>/ subdir with model.safetensors and configs
du -sh ~/.cache/huggingface/hub/models--facebook--sam3/
# ~3-4 GB for SAM3

1.4 Optional — pre-warm without a real predict

If you want to download but not yet wire up Tauri, do it from a Python shell:

python3 - <<'EOF'
from transformers import Sam3TrackerModel, Sam3TrackerProcessor
Sam3TrackerProcessor.from_pretrained("facebook/sam3")
Sam3TrackerModel.from_pretrained("facebook/sam3")
print("cached.")
EOF

Same effect — fills ~/.cache/huggingface/hub/.

Phase 2 — flip to fully offline

Once the cache is populated, set the offline flags so the server never tries to revalidate metadata or check for updates.

2.1 Edit `run.sh`

Change the defaults to 1:

export HF_HUB_DISABLE_TELEMETRY="${HF_HUB_DISABLE_TELEMETRY:-1}"
export DO_NOT_TRACK="${DO_NOT_TRACK:-1}"
export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-1}"           # ← 1
export TRANSFORMERS_OFFLINE="${TRANSFORMERS_OFFLINE:-1}"   # ← 1

Or, if you're running via systemd, add to your unit file:

[Service]
Environment=HF_HUB_DISABLE_TELEMETRY=1
Environment=DO_NOT_TRACK=1
Environment=HF_HUB_OFFLINE=1
Environment=TRANSFORMERS_OFFLINE=1

Then sudo systemctl daemon-reload && sudo systemctl restart sam-backend.

2.2 Restart and verify

./run.sh

You should see clean startup logs without any httpx HTTP Request lines hitting huggingface.co. The Loading weights: 685/685 [00:00<00:00] line confirms a cache hit (instant load, no download).

2.3 Pin a custom cache location (optional)

If you want the cache somewhere predictable (for backup, rsync to other boxes, or to keep $HOME clean), set HF_HOME in run.sh:

export HF_HOME=/var/cache/sam-backend/huggingface

Move the existing cache once:

mv ~/.cache/huggingface /var/cache/sam-backend/

Subsequent reads come from there.

Verifying you're really offline

Two-stage check:

Step 1 — disconnect & restart

Easiest sanity test: pull the network cable (or sudo ip link set eth0 down), restart the server, hit /predict from the Tauri app on the same box. If it returns a polygon, you're truly offline.

Step 2 — packet-level proof

If you can't disconnect (e.g., other services need the network), watch for any HF traffic during normal use:

sudo tcpdump -nn -i any \
  'host huggingface.co or host cdn-lfs.huggingface.co' \
  2>/dev/null

Restart the backend, hit /predict ten times from the Tauri app. With the OFFLINE flags set you should see zero packets.

Moving the cache to a different machine

Useful when you want to deploy to a new box without the first-run download (e.g., setting up a second annotation station behind the same firewall).

# On the source machine:
tar czf hf-sam3-cache.tar.gz -C ~/.cache huggingface
scp hf-sam3-cache.tar.gz target-box:/tmp/

# On the target machine:
mkdir -p ~/.cache
tar xzf /tmp/hf-sam3-cache.tar.gz -C ~/.cache

Then start the new backend with HF_HUB_OFFLINE=1 from day one — it'll load from the imported cache without ever calling out.

Common gotchas

Symptom	Cause / fix
`Cannot find the requested files in the cached path` while OFFLINE=1	Cache is incomplete or in a different `HF_HOME`. Re-run online once.
`gated repo, access denied`	Token doesn't have `facebook/sam3` access. Accept gate on the model page, then `hf auth login` again.
Many `HEAD https://huggingface.co/...` lines on startup despite OFFLINE=1	One of the two env flags is missing. Both `HF_HUB_OFFLINE=1` AND `TRANSFORMERS_OFFLINE=1` are required.
`Loading weights: ...` takes minutes even with cache	The cache file is on a slow disk (NFS, HDD). Move it to local SSD via `HF_HOME`.
Token leaked into logs	`hf auth login` only echoes your username, never the token. The token at `~/.cache/huggingface/token` is the only place it's stored.

What HF still sees in offline mode

Nothing. With both HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1:

from_pretrained reads only from ~/.cache/huggingface/hub/ — no API calls, no HEAD revalidation pings.
Telemetry is independently off via HF_HUB_DISABLE_TELEMETRY=1 / DO_NOT_TRACK=1.
Inference itself never touches the network — your images, bboxes, and resulting masks stay on the box regardless of online/offline mode.

The only thing HF "knows" about you, ever, is "this account downloaded facebook/sam3 once on date X" — recorded server-side at the moment from_pretrained first ran in Phase 1.

7.2 KiB Raw Blame History