SR-Label-SamTool/sam2-backend/README.md

# SAM2 Backend

Standalone HTTP server that runs Meta's SAM-family image predictors and
speaks the Label-Studio-ML `/predict` protocol the SAM-Tool Tauri app
already uses. Can host locally or on any remote PC on your network — point
the Tauri app at `http://<host>:9090` from Settings.

Ships with adapters for both **SAM 2 / 2.1** and **SAM 3**. The registry
(`models.yaml`) is pluggable — new families just drop an adapter file into
`adapters/` and register in `adapters/__init__.py`.

---

## Layout

```
sam2-backend/
├── server.py            FastAPI app — /, /models, /predict
├── registry.py          Model spec loader + lazy-load cache
├── adapters/
│   ├── base.py          Adapter contract
│   ├── sam2_adapter.py  SAM2 implementation
│   └── __init__.py      Family → adapter class map
├── models.yaml          Which models exist + where their checkpoints live
├── requirements.txt     HTTP server deps (Torch + SAM2 installed separately)
├── run.sh               Convenience launcher
└── README.md
```

---

## Install (one-time, on the host PC)

### 1. Python environment

```bash
cd sam2-backend
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
```

### 2. PyTorch (choose one)

Pick the build that matches your GPU/CPU — do NOT rely on the generic
`torch` pulled in transitively by SAM2.

```bash
# CUDA 12.1 (typical modern NVIDIA)
pip install torch==2.4.* torchvision==0.19.* --index-url https://download.pytorch.org/whl/cu121

# CPU-only (works but inference is ~50× slower)
pip install torch==2.4.* torchvision==0.19.* --index-url https://download.pytorch.org/whl/cpu
```

### 3. SAM 2 package

The SAM2 source is already in this repo at `../sam2/sam2/`. Install it as
an editable package:

```bash
pip install -e ../sam2/sam2
```

> If you see `RuntimeError: You're likely running Python from the parent
> directory of the sam2 repository`, you started Python from the *wrong*
> directory — always run `server.py` from `sam2-backend/` (not from the
> repo root).

### 4. SAM 3 package (optional — only if you want SAM3 models)

```bash
# Upstream repo (adjust if you're using a fork):
pip install "sam3 @ git+https://github.com/facebookresearch/sam3"
# For the HuggingFace tracker path (optional, used by `config: tracker`):
pip install transformers
```

The SAM3 image-predictor path uses the `.pt` file in `../sam3_weights/sam3.pt`
— already referenced by `models.yaml`. No additional downloads needed for
the default SAM3 entry.

### 5. The rest

```bash
pip install -r requirements.txt
```

### 6. Checkpoints

**SAM 2 / 2.1** — present at `../sam2/sam2/checkpoints/sam2.1_hiera_{tiny,small,base_plus,large}.pt`.
If any are missing, download them with:

```bash
cd ../sam2/sam2/checkpoints
./download_ckpts.sh
```

**SAM 3** — present at `../files/sam3_weights/sam3.pt`.

`models.yaml` points at all of these by default. Edit the file to add,
remove, or relocate entries.

---

## Run

```bash
# From sam2-backend/
./run.sh                          # defaults: 0.0.0.0:9090, auto device
HOST=127.0.0.1 PORT=9090 ./run.sh # localhost only
DEVICE=cuda:1 ./run.sh            # pin a specific GPU

# or directly
python server.py --host 0.0.0.0 --port 9090 --device auto
```

First request for a given model triggers a one-time load (few seconds). By
default the previously-loaded model is evicted on switch to keep GPU memory
bounded. Set `SAM_BACKEND_KEEP_ALL=1` to keep them all resident.

---

## Smoke-test

```bash
curl -s http://localhost:9090/          # {"status":"ok",...}
curl -s http://localhost:9090/models    # {"default":"...","models":[...]}
```

From the Tauri app: open **Settings**, set the SAM URL to
`http://<host>:9090`, enable SAM, pick a model, save. Drawing a bbox should
now round-trip through SAM2 and come back as a polygon.

---

## API

### `GET /`
```json
{"status":"ok","device":"cuda","models":4,"default":"sam2.1_hiera_small"}
```

### `GET /models`
```json
{
  "default": "sam2.1_hiera_small",
  "models": [
    {"id":"sam2.1_hiera_tiny","name":"SAM 2.1 · Hiera Tiny",
     "family":"sam2","available":true,
     "checkpoint":"/abs/path/sam2.1_hiera_tiny.pt"},
    ...
  ]
}
```

### `POST /predict?model=<id>`
Body: Label-Studio-ML task + bbox prompt (see `server.py` docstring).
Optional `?model=<id>` query selects which model. Omit to use the default.

Response: LS-ML `results[].result[].value.points` as percentages.

---

## SAM 3 notes

Two `config` modes are supported per entry in `models.yaml`:

| `config`          | Loader                                                          | Checkpoint                                 | Notes                                                                 |
|-------------------|-----------------------------------------------------------------|--------------------------------------------|-----------------------------------------------------------------------|
| `image_predictor` | `sam3.model_builder.build_sam3_image_model`                     | Local `.pt` file                           | Default. Box = visual exemplar; picks mask with highest IoU to user box. |
| `tracker`         | `transformers.Sam3TrackerModel.from_pretrained("facebook/sam3")` | HF model id (set `remote: true` in yaml) | Cleaner single-box → single-mask. Requires `transformers` + first-run network access. |

The shipped default SAM3 entry uses `image_predictor` + the local weights.
To enable the tracker path, uncomment the `sam3_tracker` block in
`models.yaml`.

---

## Adding another model family later

1. Drop a new file into `adapters/`, e.g. `adapters/sam4_adapter.py`, that
   implements `BaseAdapter`.
2. Register it in `adapters/__init__.py`:
   ```python
   ADAPTERS = {"sam2": Sam2Adapter, "sam3": Sam3Adapter, "sam4": Sam4Adapter}
   ```
3. Add entries to `models.yaml` with `family: sam4`.
4. Restart the server.

No changes to `server.py` or `registry.py` should be needed.