first version
This commit is contained in:
191
sam2-backend/README.md
Normal file
191
sam2-backend/README.md
Normal file
@@ -0,0 +1,191 @@
|
||||
# SAM2 Backend
|
||||
|
||||
Standalone HTTP server that runs Meta's SAM-family image predictors and
|
||||
speaks the Label-Studio-ML `/predict` protocol the SAM-Tool Tauri app
|
||||
already uses. Can host locally or on any remote PC on your network — point
|
||||
the Tauri app at `http://<host>:9090` from Settings.
|
||||
|
||||
Ships with adapters for both **SAM 2 / 2.1** and **SAM 3**. The registry
|
||||
(`models.yaml`) is pluggable — new families just drop an adapter file into
|
||||
`adapters/` and register in `adapters/__init__.py`.
|
||||
|
||||
---
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
sam2-backend/
|
||||
├── server.py FastAPI app — /, /models, /predict
|
||||
├── registry.py Model spec loader + lazy-load cache
|
||||
├── adapters/
|
||||
│ ├── base.py Adapter contract
|
||||
│ ├── sam2_adapter.py SAM2 implementation
|
||||
│ └── __init__.py Family → adapter class map
|
||||
├── models.yaml Which models exist + where their checkpoints live
|
||||
├── requirements.txt HTTP server deps (Torch + SAM2 installed separately)
|
||||
├── run.sh Convenience launcher
|
||||
└── README.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Install (one-time, on the host PC)
|
||||
|
||||
### 1. Python environment
|
||||
|
||||
```bash
|
||||
cd sam2-backend
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install --upgrade pip
|
||||
```
|
||||
|
||||
### 2. PyTorch (choose one)
|
||||
|
||||
Pick the build that matches your GPU/CPU — do NOT rely on the generic
|
||||
`torch` pulled in transitively by SAM2.
|
||||
|
||||
```bash
|
||||
# CUDA 12.1 (typical modern NVIDIA)
|
||||
pip install torch==2.4.* torchvision==0.19.* --index-url https://download.pytorch.org/whl/cu121
|
||||
|
||||
# CPU-only (works but inference is ~50× slower)
|
||||
pip install torch==2.4.* torchvision==0.19.* --index-url https://download.pytorch.org/whl/cpu
|
||||
```
|
||||
|
||||
### 3. SAM 2 package
|
||||
|
||||
The SAM2 source is already in this repo at `../sam2/sam2/`. Install it as
|
||||
an editable package:
|
||||
|
||||
```bash
|
||||
pip install -e ../sam2/sam2
|
||||
```
|
||||
|
||||
> If you see `RuntimeError: You're likely running Python from the parent
|
||||
> directory of the sam2 repository`, you started Python from the *wrong*
|
||||
> directory — always run `server.py` from `sam2-backend/` (not from the
|
||||
> repo root).
|
||||
|
||||
### 4. SAM 3 package (optional — only if you want SAM3 models)
|
||||
|
||||
```bash
|
||||
# Upstream repo (adjust if you're using a fork):
|
||||
pip install "sam3 @ git+https://github.com/facebookresearch/sam3"
|
||||
# For the HuggingFace tracker path (optional, used by `config: tracker`):
|
||||
pip install transformers
|
||||
```
|
||||
|
||||
The SAM3 image-predictor path uses the `.pt` file in `../sam3_weights/sam3.pt`
|
||||
— already referenced by `models.yaml`. No additional downloads needed for
|
||||
the default SAM3 entry.
|
||||
|
||||
### 5. The rest
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 6. Checkpoints
|
||||
|
||||
**SAM 2 / 2.1** — present at `../sam2/sam2/checkpoints/sam2.1_hiera_{tiny,small,base_plus,large}.pt`.
|
||||
If any are missing, download them with:
|
||||
|
||||
```bash
|
||||
cd ../sam2/sam2/checkpoints
|
||||
./download_ckpts.sh
|
||||
```
|
||||
|
||||
**SAM 3** — present at `../files/sam3_weights/sam3.pt`.
|
||||
|
||||
`models.yaml` points at all of these by default. Edit the file to add,
|
||||
remove, or relocate entries.
|
||||
|
||||
---
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
# From sam2-backend/
|
||||
./run.sh # defaults: 0.0.0.0:9090, auto device
|
||||
HOST=127.0.0.1 PORT=9090 ./run.sh # localhost only
|
||||
DEVICE=cuda:1 ./run.sh # pin a specific GPU
|
||||
|
||||
# or directly
|
||||
python server.py --host 0.0.0.0 --port 9090 --device auto
|
||||
```
|
||||
|
||||
First request for a given model triggers a one-time load (few seconds). By
|
||||
default the previously-loaded model is evicted on switch to keep GPU memory
|
||||
bounded. Set `SAM_BACKEND_KEEP_ALL=1` to keep them all resident.
|
||||
|
||||
---
|
||||
|
||||
## Smoke-test
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:9090/ # {"status":"ok",...}
|
||||
curl -s http://localhost:9090/models # {"default":"...","models":[...]}
|
||||
```
|
||||
|
||||
From the Tauri app: open **Settings**, set the SAM URL to
|
||||
`http://<host>:9090`, enable SAM, pick a model, save. Drawing a bbox should
|
||||
now round-trip through SAM2 and come back as a polygon.
|
||||
|
||||
---
|
||||
|
||||
## API
|
||||
|
||||
### `GET /`
|
||||
```json
|
||||
{"status":"ok","device":"cuda","models":4,"default":"sam2.1_hiera_small"}
|
||||
```
|
||||
|
||||
### `GET /models`
|
||||
```json
|
||||
{
|
||||
"default": "sam2.1_hiera_small",
|
||||
"models": [
|
||||
{"id":"sam2.1_hiera_tiny","name":"SAM 2.1 · Hiera Tiny",
|
||||
"family":"sam2","available":true,
|
||||
"checkpoint":"/abs/path/sam2.1_hiera_tiny.pt"},
|
||||
...
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### `POST /predict?model=<id>`
|
||||
Body: Label-Studio-ML task + bbox prompt (see `server.py` docstring).
|
||||
Optional `?model=<id>` query selects which model. Omit to use the default.
|
||||
|
||||
Response: LS-ML `results[].result[].value.points` as percentages.
|
||||
|
||||
---
|
||||
|
||||
## SAM 3 notes
|
||||
|
||||
Two `config` modes are supported per entry in `models.yaml`:
|
||||
|
||||
| `config` | Loader | Checkpoint | Notes |
|
||||
|-------------------|-----------------------------------------------------------------|--------------------------------------------|-----------------------------------------------------------------------|
|
||||
| `image_predictor` | `sam3.model_builder.build_sam3_image_model` | Local `.pt` file | Default. Box = visual exemplar; picks mask with highest IoU to user box. |
|
||||
| `tracker` | `transformers.Sam3TrackerModel.from_pretrained("facebook/sam3")` | HF model id (set `remote: true` in yaml) | Cleaner single-box → single-mask. Requires `transformers` + first-run network access. |
|
||||
|
||||
The shipped default SAM3 entry uses `image_predictor` + the local weights.
|
||||
To enable the tracker path, uncomment the `sam3_tracker` block in
|
||||
`models.yaml`.
|
||||
|
||||
---
|
||||
|
||||
## Adding another model family later
|
||||
|
||||
1. Drop a new file into `adapters/`, e.g. `adapters/sam4_adapter.py`, that
|
||||
implements `BaseAdapter`.
|
||||
2. Register it in `adapters/__init__.py`:
|
||||
```python
|
||||
ADAPTERS = {"sam2": Sam2Adapter, "sam3": Sam3Adapter, "sam4": Sam4Adapter}
|
||||
```
|
||||
3. Add entries to `models.yaml` with `family: sam4`.
|
||||
4. Restart the server.
|
||||
|
||||
No changes to `server.py` or `registry.py` should be needed.
|
||||
Reference in New Issue
Block a user