SR-Label-SamTool/sam2-backend

SAM2 Backend

Standalone HTTP server that runs Meta's SAM-family image predictors and speaks the Label-Studio-ML /predict protocol the SAM-Tool Tauri app already uses. Can host locally or on any remote PC on your network — point the Tauri app at http://<host>:9090 from Settings.

Ships with adapters for both SAM 2 / 2.1 and SAM 3. The registry (models.yaml) is pluggable — new families just drop an adapter file into adapters/ and register in adapters/__init__.py.

---

Layout

sam2-backend/
├── server.py            FastAPI app — /, /models, /predict
├── registry.py          Model spec loader + lazy-load cache
├── adapters/
│   ├── base.py          Adapter contract
│   ├── sam2_adapter.py  SAM2 implementation
│   └── __init__.py      Family → adapter class map
├── models.yaml          Which models exist + where their checkpoints live
├── requirements.txt     HTTP server deps (Torch + SAM2 installed separately)
├── run.sh               Convenience launcher
└── README.md

---

Install (one-time, on the host PC)

1. Python environment

cd sam2-backend
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

2. PyTorch (choose one)

Pick the build that matches your GPU/CPU — do NOT rely on the generic torch pulled in transitively by SAM2.

# CUDA 12.1 (typical modern NVIDIA)
pip install torch==2.4.* torchvision==0.19.* --index-url https://download.pytorch.org/whl/cu121

# CPU-only (works but inference is ~50× slower)
pip install torch==2.4.* torchvision==0.19.* --index-url https://download.pytorch.org/whl/cpu

3. SAM 2 package

The SAM2 source is already in this repo at ../sam2/sam2/. Install it as an editable package:

pip install -e ../sam2/sam2

If you see RuntimeError: You're likely running Python from the parent directory of the sam2 repository, you started Python from the wrong directory — always run server.py from sam2-backend/ (not from the repo root).

4. SAM 3 package (optional — only if you want SAM3 models)

# Upstream repo (adjust if you're using a fork):
pip install "sam3 @ git+https://github.com/facebookresearch/sam3"
# For the HuggingFace tracker path (optional, used by `config: tracker`):
pip install transformers

The SAM3 image-predictor path uses the .pt file in ../sam3_weights/sam3.pt — already referenced by models.yaml. No additional downloads needed for the default SAM3 entry.

5. The rest

pip install -r requirements.txt

6. Checkpoints

SAM 2 / 2.1 — present at ../sam2/sam2/checkpoints/sam2.1_hiera_{tiny,small,base_plus,large}.pt. If any are missing, download them with:

cd ../sam2/sam2/checkpoints
./download_ckpts.sh

SAM 3 — present at ../files/sam3_weights/sam3.pt.

models.yaml points at all of these by default. Edit the file to add, remove, or relocate entries.

---

Run

# From sam2-backend/
./run.sh                          # defaults: 0.0.0.0:9090, auto device
HOST=127.0.0.1 PORT=9090 ./run.sh # localhost only
DEVICE=cuda:1 ./run.sh            # pin a specific GPU

# or directly
python server.py --host 0.0.0.0 --port 9090 --device auto

First request for a given model triggers a one-time load (few seconds). By default the previously-loaded model is evicted on switch to keep GPU memory bounded. Set SAM_BACKEND_KEEP_ALL=1 to keep them all resident.

---

Smoke-test

curl -s http://localhost:9090/          # {"status":"ok",...}
curl -s http://localhost:9090/models    # {"default":"...","models":[...]}

From the Tauri app: open Settings, set the SAM URL to http://<host>:9090, enable SAM, pick a model, save. Drawing a bbox should now round-trip through SAM2 and come back as a polygon.

---

API

GET /

{"status":"ok","device":"cuda","models":4,"default":"sam2.1_hiera_small"}

GET /models

{
  "default": "sam2.1_hiera_small",
  "models": [
    {"id":"sam2.1_hiera_tiny","name":"SAM 2.1 · Hiera Tiny",
     "family":"sam2","available":true,
     "checkpoint":"/abs/path/sam2.1_hiera_tiny.pt"},
    ...
  ]
}

POST /predict?model=<id>

Body: Label-Studio-ML task + bbox prompt (see server.py docstring). Optional ?model=<id> query selects which model. Omit to use the default.

Response: LS-ML results[].result[].value.points as percentages.

---

SAM 3 notes

Two config modes are supported per entry in models.yaml:

config	Loader	Checkpoint	Notes
image_predictor	sam3.model_builder.build_sam3_image_model	Local .pt file	Default. Box = visual exemplar; picks mask with highest IoU to user box.
tracker	transformers.Sam3TrackerModel.from_pretrained("facebook/sam3")	HF model id (set remote: true in yaml)	Cleaner single-box → single-mask. Requires transformers + first-run network access.

The shipped default SAM3 entry uses image_predictor + the local weights. To enable the tracker path, uncomment the sam3_tracker block in models.yaml.

---

Adding another model family later

1. Drop a new file into adapters/, e.g. adapters/sam4_adapter.py, that implements BaseAdapter.
2. Register it in adapters/__init__.py:

   ADAPTERS = {"sam2": Sam2Adapter, "sam3": Sam3Adapter, "sam4": Sam4Adapter}
   

3. Add entries to models.yaml with family: sam4.
4. Restart the server.

No changes to server.py or registry.py should be needed.