first version

files/README.md

@@ -0,0 +1,120 @@
+# SAM 3 local segmentation server
+
+A small FastAPI server that wraps Meta's **SAM 3** (released Nov 19, 2025) so your local annotation tool can send a bbox and get back a segmentation mask.
+
+## What it does
+
+Two endpoints, both take an image + bbox:
+
+| Endpoint | Behavior | Use for |
+|---|---|---|
+| `POST /segment/box` | Returns the mask of the **one** object inside your bbox. Uses the SAM 3 tracker (SAM2-style PVS). | Classic annotation — user draws a box around one object, you want that one mask. |
+| `POST /segment/exemplar` | Treats the bbox as an **exemplar** and returns masks for **all similar** objects in the image. | Auto-propagation — label one thing, get all instances. |
+
+Mask is returned as a base64 PNG, plus optional **COCO RLE** and **polygon contour** for easy integration with annotation-tool formats (CVAT, Label Studio, COCO JSON, etc).
+
+## Setup
+
+### 1. Prerequisites
+- Python 3.12+
+- PyTorch 2.7+ with CUDA 12.6 (CPU works too, just slow)
+- A CUDA-capable GPU is strongly recommended — SAM 3 is 848M params
+
+### 2. Request access to weights
+SAM 3 weights are gated on Hugging Face. Request access at https://huggingface.co/facebook/sam3, wait for approval, then:
+```bash
+hf auth login
+```
+
+### 3. Install
+```bash
+# SAM 3 itself
+git clone https://github.com/facebookresearch/sam3.git
+cd sam3
+pip install -e .
+
+# Server deps
+pip install fastapi uvicorn pillow numpy requests opencv-python-headless pycocotools
+# Optional but recommended — gives the clean tracker API used by /segment/box
+pip install "transformers>=4.57"
+```
+
+### 4. Run
+```bash
+python sam3_server.py --host 0.0.0.0 --port 8000
+# or on CPU:
+python sam3_server.py --device cpu
+```
+
+Check it's alive:
+```bash
+curl http://localhost:8000/health
+```
+
+## Calling it from your annotation tool
+
+### Python
+See `client_example.py`. Minimal version:
+```python
+import base64, requests
+img_b64 = base64.b64encode(open("img.jpg","rb").read()).decode()
+r = requests.post("http://localhost:8000/segment/box", json={
+    "image": img_b64,
+    "bbox":  [120, 80, 540, 420],    # [x1, y1, x2, y2] pixels
+})
+data = r.json()
+# data["mask"]    -> base64 PNG mask
+# data["polygon"] -> [[x,y], ...] contour
+# data["rle"]     -> COCO RLE dict
+```
+
+### JavaScript (browser annotation tools)
+```js
+const imgB64 = /* base64 of your image */;
+const res = await fetch("http://localhost:8000/segment/box", {
+  method: "POST",
+  headers: {"Content-Type": "application/json"},
+  body: JSON.stringify({ image: imgB64, bbox: [x1, y1, x2, y2] })
+});
+const { mask, polygon, rle, score } = await res.json();
+// Draw `mask` (PNG) or `polygon` onto your canvas
+```
+
+CORS is enabled for `*`, so browser clients work out of the box — lock this down for production.
+
+## Request/response reference
+
+**Request body** (JSON):
+```json
+{
+  "image":     "<base64 PNG/JPG>",   // OR image_path OR image_url
+  "bbox":      [x1, y1, x2, y2],     // absolute pixels, xyxy
+  "multimask_output": false,         // optional, /segment/box only — returns 3 candidates
+  "return_polygon": true,
+  "return_rle": true
+}
+```
+
+**Response**:
+```json
+{
+  "mask":     "<base64 PNG, 0/255 single channel>",
+  "score":    0.97,
+  "bbox_out": [x1, y1, x2, y2],
+  "rle":      {"size": [H, W], "counts": "..."},
+  "polygon":  [[x, y], ...],
+  "extra_masks": null
+}
+```
+
+## Notes on the two endpoints
+
+The SAM 3 reference repo's `Sam3Processor.set_box_prompt` is actually a **concept/exemplar** prompt — it finds *all* objects similar to what's in the box. For a bbox-to-single-mask annotation workflow you specifically want the **tracker** (`Sam3Tracker`), which is SAM 3's SAM2-compatible PVS head. That's what `/segment/box` uses when `transformers` is installed.
+
+If `transformers` isn't available, `/segment/box` falls back to the exemplar predictor and picks the returned instance whose predicted bbox has the highest IoU with your input — this usually gives you the right object but can pick a similar neighbor in cluttered scenes. Install `transformers` for best results.
+
+## Performance tips
+- Keep the process alive; model loading is the slow part (~10–20s).
+- For batch annotation, call `/segment/box` sequentially — `set_image` caches the image embedding in the session state.
+- On a single RTX 4090, per-request latency for a 1024px image is typically **~150–300ms** once warm.
+- For large images, consider resizing on the client to ≤2048px on the long side; SAM 3 resizes internally anyway.