SR-Label-SamTool/files

SAM 3 local segmentation server

A small FastAPI server that wraps Meta's SAM 3 (released Nov 19, 2025) so your local annotation tool can send a bbox and get back a segmentation mask.

What it does

Two endpoints, both take an image + bbox:

Endpoint	Behavior	Use for
POST /segment/box	Returns the mask of the one object inside your bbox. Uses the SAM 3 tracker (SAM2-style PVS).	Classic annotation — user draws a box around one object, you want that one mask.
POST /segment/exemplar	Treats the bbox as an exemplar and returns masks for all similar objects in the image.	Auto-propagation — label one thing, get all instances.

Mask is returned as a base64 PNG, plus optional COCO RLE and polygon contour for easy integration with annotation-tool formats (CVAT, Label Studio, COCO JSON, etc).

Setup

1. Prerequisites

• Python 3.12+
• PyTorch 2.7+ with CUDA 12.6 (CPU works too, just slow)
• A CUDA-capable GPU is strongly recommended — SAM 3 is 848M params

2. Request access to weights

SAM 3 weights are gated on Hugging Face. Request access at https://huggingface.co/facebook/sam3, wait for approval, then:

hf auth login

3. Install

# SAM 3 itself
git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .

# Server deps
pip install fastapi uvicorn pillow numpy requests opencv-python-headless pycocotools
# Optional but recommended — gives the clean tracker API used by /segment/box
pip install "transformers>=4.57"

4. Run

python sam3_server.py --host 0.0.0.0 --port 8000
# or on CPU:
python sam3_server.py --device cpu

Check it's alive:

curl http://localhost:8000/health

Calling it from your annotation tool

Python

See client_example.py. Minimal version:

import base64, requests
img_b64 = base64.b64encode(open("img.jpg","rb").read()).decode()
r = requests.post("http://localhost:8000/segment/box", json={
    "image": img_b64,
    "bbox":  [120, 80, 540, 420],    # [x1, y1, x2, y2] pixels
})
data = r.json()
# data["mask"]    -> base64 PNG mask
# data["polygon"] -> [[x,y], ...] contour
# data["rle"]     -> COCO RLE dict

JavaScript (browser annotation tools)

const imgB64 = /* base64 of your image */;
const res = await fetch("http://localhost:8000/segment/box", {
  method: "POST",
  headers: {"Content-Type": "application/json"},
  body: JSON.stringify({ image: imgB64, bbox: [x1, y1, x2, y2] })
});
const { mask, polygon, rle, score } = await res.json();
// Draw `mask` (PNG) or `polygon` onto your canvas

CORS is enabled for *, so browser clients work out of the box — lock this down for production.

Request/response reference

Request body (JSON):

{
  "image":     "<base64 PNG/JPG>",   // OR image_path OR image_url
  "bbox":      [x1, y1, x2, y2],     // absolute pixels, xyxy
  "multimask_output": false,         // optional, /segment/box only — returns 3 candidates
  "return_polygon": true,
  "return_rle": true
}

Response:

{
  "mask":     "<base64 PNG, 0/255 single channel>",
  "score":    0.97,
  "bbox_out": [x1, y1, x2, y2],
  "rle":      {"size": [H, W], "counts": "..."},
  "polygon":  [[x, y], ...],
  "extra_masks": null
}

Notes on the two endpoints

The SAM 3 reference repo's Sam3Processor.set_box_prompt is actually a concept/exemplar prompt — it finds all objects similar to what's in the box. For a bbox-to-single-mask annotation workflow you specifically want the tracker (Sam3Tracker), which is SAM 3's SAM2-compatible PVS head. That's what /segment/box uses when transformers is installed.

If transformers isn't available, /segment/box falls back to the exemplar predictor and picks the returned instance whose predicted bbox has the highest IoU with your input — this usually gives you the right object but can pick a similar neighbor in cluttered scenes. Install transformers for best results.

Performance tips

• Keep the process alive; model loading is the slow part (~10–20s).
• For batch annotation, call /segment/box sequentially — set_image caches the image embedding in the session state.
• On a single RTX 4090, per-request latency for a 1024px image is typically ~150–300ms once warm.
• For large images, consider resizing on the client to ≤2048px on the long side; SAM 3 resizes internally anyway.