# SAM 3 local segmentation server

A small FastAPI server that wraps Meta's **SAM 3** (released Nov 19, 2025) so your local annotation tool can send a bbox and get back a segmentation mask.

## What it does

Two endpoints, both take an image + bbox:

| Endpoint | Behavior | Use for |
|---|---|---|
| `POST /segment/box` | Returns the mask of the **one** object inside your bbox. Uses the SAM 3 tracker (SAM2-style PVS). | Classic annotation — user draws a box around one object, you want that one mask. |
| `POST /segment/exemplar` | Treats the bbox as an **exemplar** and returns masks for **all similar** objects in the image. | Auto-propagation — label one thing, get all instances. |

Mask is returned as a base64 PNG, plus optional **COCO RLE** and **polygon contour** for easy integration with annotation-tool formats (CVAT, Label Studio, COCO JSON, etc).

## Setup

### 1. Prerequisites
- Python 3.12+
- PyTorch 2.7+ with CUDA 12.6 (CPU works too, just slow)
- A CUDA-capable GPU is strongly recommended — SAM 3 is 848M params

### 2. Request access to weights
SAM 3 weights are gated on Hugging Face. Request access at https://huggingface.co/facebook/sam3, wait for approval, then:
```bash
hf auth login
```

### 3. Install
```bash
# SAM 3 itself
git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .

# Server deps
pip install fastapi uvicorn pillow numpy requests opencv-python-headless pycocotools
# Optional but recommended — gives the clean tracker API used by /segment/box
pip install "transformers>=4.57"
```

### 4. Run
```bash
python sam3_server.py --host 0.0.0.0 --port 8000
# or on CPU:
python sam3_server.py --device cpu
```

Check it's alive:
```bash
curl http://localhost:8000/health
```

## Calling it from your annotation tool

### Python
See `client_example.py`. Minimal version:
```python
import base64, requests
img_b64 = base64.b64encode(open("img.jpg","rb").read()).decode()
r = requests.post("http://localhost:8000/segment/box", json={
    "image": img_b64,
    "bbox":  [120, 80, 540, 420],    # [x1, y1, x2, y2] pixels
})
data = r.json()
# data["mask"]    -> base64 PNG mask
# data["polygon"] -> [[x,y], ...] contour
# data["rle"]     -> COCO RLE dict
```

### JavaScript (browser annotation tools)
```js
const imgB64 = /* base64 of your image */;
const res = await fetch("http://localhost:8000/segment/box", {
  method: "POST",
  headers: {"Content-Type": "application/json"},
  body: JSON.stringify({ image: imgB64, bbox: [x1, y1, x2, y2] })
});
const { mask, polygon, rle, score } = await res.json();
// Draw `mask` (PNG) or `polygon` onto your canvas
```

CORS is enabled for `*`, so browser clients work out of the box — lock this down for production.

## Request/response reference

**Request body** (JSON):
```json
{
  "image":     "<base64 PNG/JPG>",   // OR image_path OR image_url
  "bbox":      [x1, y1, x2, y2],     // absolute pixels, xyxy
  "multimask_output": false,         // optional, /segment/box only — returns 3 candidates
  "return_polygon": true,
  "return_rle": true
}
```

**Response**:
```json
{
  "mask":     "<base64 PNG, 0/255 single channel>",
  "score":    0.97,
  "bbox_out": [x1, y1, x2, y2],
  "rle":      {"size": [H, W], "counts": "..."},
  "polygon":  [[x, y], ...],
  "extra_masks": null
}
```

## Notes on the two endpoints

The SAM 3 reference repo's `Sam3Processor.set_box_prompt` is actually a **concept/exemplar** prompt — it finds *all* objects similar to what's in the box. For a bbox-to-single-mask annotation workflow you specifically want the **tracker** (`Sam3Tracker`), which is SAM 3's SAM2-compatible PVS head. That's what `/segment/box` uses when `transformers` is installed.

If `transformers` isn't available, `/segment/box` falls back to the exemplar predictor and picks the returned instance whose predicted bbox has the highest IoU with your input — this usually gives you the right object but can pick a similar neighbor in cluttered scenes. Install `transformers` for best results.

## Performance tips
- Keep the process alive; model loading is the slow part (~10–20s).
- For batch annotation, call `/segment/box` sequentially — `set_image` caches the image embedding in the session state.
- On a single RTX 4090, per-request latency for a 1024px image is typically **~150–300ms** once warm.
- For large images, consider resizing on the client to ≤2048px on the long side; SAM 3 resizes internally anyway.