121 lines
4.3 KiB
Markdown
121 lines
4.3 KiB
Markdown
# SAM 3 local segmentation server
|
||
|
||
A small FastAPI server that wraps Meta's **SAM 3** (released Nov 19, 2025) so your local annotation tool can send a bbox and get back a segmentation mask.
|
||
|
||
## What it does
|
||
|
||
Two endpoints, both take an image + bbox:
|
||
|
||
| Endpoint | Behavior | Use for |
|
||
|---|---|---|
|
||
| `POST /segment/box` | Returns the mask of the **one** object inside your bbox. Uses the SAM 3 tracker (SAM2-style PVS). | Classic annotation — user draws a box around one object, you want that one mask. |
|
||
| `POST /segment/exemplar` | Treats the bbox as an **exemplar** and returns masks for **all similar** objects in the image. | Auto-propagation — label one thing, get all instances. |
|
||
|
||
Mask is returned as a base64 PNG, plus optional **COCO RLE** and **polygon contour** for easy integration with annotation-tool formats (CVAT, Label Studio, COCO JSON, etc).
|
||
|
||
## Setup
|
||
|
||
### 1. Prerequisites
|
||
- Python 3.12+
|
||
- PyTorch 2.7+ with CUDA 12.6 (CPU works too, just slow)
|
||
- A CUDA-capable GPU is strongly recommended — SAM 3 is 848M params
|
||
|
||
### 2. Request access to weights
|
||
SAM 3 weights are gated on Hugging Face. Request access at https://huggingface.co/facebook/sam3, wait for approval, then:
|
||
```bash
|
||
hf auth login
|
||
```
|
||
|
||
### 3. Install
|
||
```bash
|
||
# SAM 3 itself
|
||
git clone https://github.com/facebookresearch/sam3.git
|
||
cd sam3
|
||
pip install -e .
|
||
|
||
# Server deps
|
||
pip install fastapi uvicorn pillow numpy requests opencv-python-headless pycocotools
|
||
# Optional but recommended — gives the clean tracker API used by /segment/box
|
||
pip install "transformers>=4.57"
|
||
```
|
||
|
||
### 4. Run
|
||
```bash
|
||
python sam3_server.py --host 0.0.0.0 --port 8000
|
||
# or on CPU:
|
||
python sam3_server.py --device cpu
|
||
```
|
||
|
||
Check it's alive:
|
||
```bash
|
||
curl http://localhost:8000/health
|
||
```
|
||
|
||
## Calling it from your annotation tool
|
||
|
||
### Python
|
||
See `client_example.py`. Minimal version:
|
||
```python
|
||
import base64, requests
|
||
img_b64 = base64.b64encode(open("img.jpg","rb").read()).decode()
|
||
r = requests.post("http://localhost:8000/segment/box", json={
|
||
"image": img_b64,
|
||
"bbox": [120, 80, 540, 420], # [x1, y1, x2, y2] pixels
|
||
})
|
||
data = r.json()
|
||
# data["mask"] -> base64 PNG mask
|
||
# data["polygon"] -> [[x,y], ...] contour
|
||
# data["rle"] -> COCO RLE dict
|
||
```
|
||
|
||
### JavaScript (browser annotation tools)
|
||
```js
|
||
const imgB64 = /* base64 of your image */;
|
||
const res = await fetch("http://localhost:8000/segment/box", {
|
||
method: "POST",
|
||
headers: {"Content-Type": "application/json"},
|
||
body: JSON.stringify({ image: imgB64, bbox: [x1, y1, x2, y2] })
|
||
});
|
||
const { mask, polygon, rle, score } = await res.json();
|
||
// Draw `mask` (PNG) or `polygon` onto your canvas
|
||
```
|
||
|
||
CORS is enabled for `*`, so browser clients work out of the box — lock this down for production.
|
||
|
||
## Request/response reference
|
||
|
||
**Request body** (JSON):
|
||
```json
|
||
{
|
||
"image": "<base64 PNG/JPG>", // OR image_path OR image_url
|
||
"bbox": [x1, y1, x2, y2], // absolute pixels, xyxy
|
||
"multimask_output": false, // optional, /segment/box only — returns 3 candidates
|
||
"return_polygon": true,
|
||
"return_rle": true
|
||
}
|
||
```
|
||
|
||
**Response**:
|
||
```json
|
||
{
|
||
"mask": "<base64 PNG, 0/255 single channel>",
|
||
"score": 0.97,
|
||
"bbox_out": [x1, y1, x2, y2],
|
||
"rle": {"size": [H, W], "counts": "..."},
|
||
"polygon": [[x, y], ...],
|
||
"extra_masks": null
|
||
}
|
||
```
|
||
|
||
## Notes on the two endpoints
|
||
|
||
The SAM 3 reference repo's `Sam3Processor.set_box_prompt` is actually a **concept/exemplar** prompt — it finds *all* objects similar to what's in the box. For a bbox-to-single-mask annotation workflow you specifically want the **tracker** (`Sam3Tracker`), which is SAM 3's SAM2-compatible PVS head. That's what `/segment/box` uses when `transformers` is installed.
|
||
|
||
If `transformers` isn't available, `/segment/box` falls back to the exemplar predictor and picks the returned instance whose predicted bbox has the highest IoU with your input — this usually gives you the right object but can pick a similar neighbor in cluttered scenes. Install `transformers` for best results.
|
||
|
||
## Performance tips
|
||
- Keep the process alive; model loading is the slow part (~10–20s).
|
||
- For batch annotation, call `/segment/box` sequentially — `set_image` caches the image embedding in the session state.
|
||
- On a single RTX 4090, per-request latency for a 1024px image is typically **~150–300ms** once warm.
|
||
- For large images, consider resizing on the client to ≤2048px on the long side; SAM 3 resizes internally anyway.
|