# SAM-Tool

Desktop video-frame extraction + image annotation tool, with optional
SAM 2 / SAM 3 segmentation backend. Built to replace an earlier
Python/Tkinter prototype (`sam_tool_v2.py`) for YOLO-style training
dataset prep.

```
┌──────────────────┐        HTTP        ┌──────────────────┐
│   sam-tool-tauri │ ◀───────────────▶ │   sam2-backend   │
│ (Tauri/Rust + UI)│   /predict, etc   │ (FastAPI + GPU)  │
└──────────────────┘                    └──────────────────┘
        │                                        │
        ▼                                        ▼
   local images +                         SAM 2.x / SAM 3
   sibling JSONs                          weights on disk
```

The two halves can run on the same machine or separate boxes — point the
desktop app at any reachable backend URL from **Settings**.

---

## Layout

```
SAM-Tool/
├── sam-tool-tauri/         # Desktop app (Tauri 2, React 18, TS, Vite)
│   ├── src/                # Frontend
│   ├── src-tauri/          # Rust commands (video, decoder, IPC)
│   └── package.json        # `npm run tauri dev` / `tauri build`
│
├── sam2-backend/           # FastAPI HTTP server (SAM 2 + SAM 3)
│   ├── server.py           # /, /models, /predict
│   ├── registry.py         # Lazy-load + single-resident cache
│   ├── adapters/
│   │   ├── sam2_adapter.py
│   │   ├── sam3_adapter.py
│   │   └── utils.py        # mask → polygon helper
│   ├── models.yaml         # Model registry (paths + family)
│   ├── run.sh              # Convenience launcher (env vars)
│   └── README.md           # Install + run on the server side
│
├── sam2/sam2/              # Meta SAM 2 source + checkpoints (vendored)
├── files/sam3_weights/     # SAM 3 weights (3.4 GB sam3.pt)
├── files/sam3_server.py    # Reference SAM3 server (used as a guide)
│
├── sam_tool_v2.py          # Original Python/Tkinter tool (legacy)
├── sam_annotation_test/    # Sample data — dashcam MP4 + custom SAM JSON
│
├── docs/                   # ← you are here
│   ├── POLYGON_SMOOTHING.md   How to tune mask→polygon output
│   ├── HF_DEPLOY_OFFLINE.md   First-online-then-airgapped HF deploy
│   └── BUILD_RELEASE.md       Building the Tauri app for release
│
└── CLAUDE.md               # Project rules (think → simplify → surgical)
```

---

## Quick start (single-machine dev)

### 1. SAM backend

```bash
cd sam2-backend
python3 -m venv .venv && source .venv/bin/activate
pip install --upgrade pip

# Pick the right Torch wheel for your GPU (driver 570+ → cu128):
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
# CPU-only fallback (slow):
# pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

# SAM 2 source + deps
pip install -e ../sam2/sam2
pip install -r requirements.txt

# Optional — SAM 3 support (needs HF account + accepted gating)
git clone https://github.com/facebookresearch/sam3.git ..
pip install -e ../sam3
pip install "transformers>=4.57"
hf auth login              # paste read token from https://huggingface.co/settings/tokens

./run.sh                   # binds 0.0.0.0:9090, auto-picks cuda/cpu
```

Smoke-test:
```bash
curl http://localhost:9090/         # {"status":"ok",...}
curl http://localhost:9090/models   # registered models
```

Full backend setup details: [`sam2-backend/README.md`](sam2-backend/README.md).

### 2. Tauri desktop app

```bash
cd sam-tool-tauri
npm install
npm run tauri dev                 # hot-reload dev mode

# Or for a real build / installer:
npm run tauri build               # see docs/BUILD_RELEASE.md
```

In the running app:
1. **Login** — pick or type a username (no password — used for annotation
   author stamping).
2. **Extract mode** — Upload a video → optionally `Import annotations…`
   (COCO / SAM JSON) → click `choose output folder…` (or just open a
   video; the default `<video>_extracted/` folder is auto-adopted) →
   scrub timeline, press **E** to extract, **U** to undo, **[ … ]** for
   bulk range.
3. **Annotate mode** — Open the extracted folder. Draw bboxes (B) /
   polygons (P), label them via the picker, switch to select (S) to
   move/resize/reclass. Settings → enable SAM, pick a model, get
   polygons back instead of bboxes.

### 3. Deploying to another PC

Backend → see the install steps in [`sam2-backend/README.md`](sam2-backend/README.md)
("install on a different PC" section).

Desktop app → [`docs/BUILD_RELEASE.md`](docs/BUILD_RELEASE.md) covers
producing `.deb` / AppImage installers.

---

## Feature highlights

### Extract mode
- Persistent `ffmpeg -f image2pipe` subprocess + LRU cache + prefetch
  worker → smooth playback up to ~2× on 1080p.
- Per-video default output folder (`<video>_extracted/`) auto-restored on
  reopen — extracted-frame state and "EXTRACTED" tint reflected
  immediately.
- Timeline ticks mark every previously-extracted frame at a glance.
- COCO + custom SAM JSON import, smart overlay with per-class colours,
  Review mode auto-pauses on annotated frames.
- `Space` play/pause · `←→` ±1 frame · `Ctrl+←→` ±10 · `↑↓` speed 0.25–4×
  · `H` overlay · `E` extract · `U` undo · `[ … ]` bulk range.

### Annotate mode
- Sibling JSON per image with `source_video` field — annotations know
  where they came from.
- Bbox + polygon tools, vertex drag (rAF-throttled), bbox handles, undo
  (40 deep) + redo, Tab-reclass, inline class creation in the picker.
- Multi-select image list (Shift-range, Ctrl-toggle) + bulk delete with
  two-step confirmation.
- A/D image navigation, +/- zoom, cursor-anchored Ctrl+wheel zoom.
- SAM mode round-trip with live phase indicator showing model + inference
  time (auto-fades after 3 s).
- Auto-switch to **select** mode after committing each new shape.

### SAM backend
- Pluggable model registry — add SAM-3 / SAM-4 / etc. by dropping an
  adapter into `adapters/` and one entry into `models.yaml`.
- Lazy-load with single-resident eviction (set
  `SAM_BACKEND_KEEP_ALL=1` to keep all loaded if you have VRAM).
- Speaks the Label-Studio-ML `/predict` protocol the Tauri client
  already used — drop-in compatible.
- Local SAM 3 image-predictor + HuggingFace SAM 3 tracker both supported.
- Privacy-first: telemetry off by default, fully air-gappable after the
  first weight download — see
  [`docs/HF_DEPLOY_OFFLINE.md`](docs/HF_DEPLOY_OFFLINE.md).

---

## Documentation index

| Doc                                                | What it covers                                              |
|----------------------------------------------------|-------------------------------------------------------------|
| [docs/POLYGON_SMOOTHING.md](docs/POLYGON_SMOOTHING.md) | Tune mask→polygon: point count, kernel size, smoothing on/off |
| [docs/HF_DEPLOY_OFFLINE.md](docs/HF_DEPLOY_OFFLINE.md) | Download HF SAM 3 weights, then run fully offline           |
| [docs/BUILD_RELEASE.md](docs/BUILD_RELEASE.md)       | Build the Tauri app for production (deb / AppImage / binary) |
| [sam2-backend/README.md](sam2-backend/README.md)     | Backend install, run, API reference, model-registry details |
| [CLAUDE.md](CLAUDE.md)                               | Project rules — Think Before Coding · Simplicity First · Surgical Changes · Goal-Driven Execution |

---

## Stack

- **Desktop:** Tauri 2 shell, Rust backend, React 18 + TypeScript, Vite,
  WebKit2GTK 2.50.
- **Backend:** Python 3.10+, FastAPI, uvicorn, Pillow, NumPy, OpenCV
  (headless), PyTorch ≥ 2.4 with matching CUDA, optional `transformers ≥ 4.57`.
- **Models:** Meta SAM 2 / SAM 2.1 (4 Hiera variants), Meta SAM 3.
- **Target OS:** Linux (Ubuntu 22.04 / Pop!_OS), AMD APU + NVIDIA RTX
  GPUs tested.

---

## Conventions

From [CLAUDE.md](CLAUDE.md):

- **Think Before Coding** — wrong assumptions are the most expensive bugs.
- **Simplicity First** — fewer moving parts beats clever abstractions.
- **Surgical Changes** — touch what the task needs, leave the rest.
- **Goal-Driven Execution** — verify the user-facing behaviour, not just
  the code path.

A few specific rules learned the hard way:
- `React.StrictMode` double-invokes setState updaters; don't put side
  effects (like `queueMicrotask` calls or save requests) inside them.
  Use refs + explicit calls instead.
- No `crypto.randomUUID()` without a fallback — webview contexts
  sometimes lack it; use the `uuid()` helper.
- Output folders must live **outside** `sam-tool-tauri/` — the dev
  watcher restarts the app on any file change inside the project tree.
- Backend telemetry stays **off** by default; HF cache moves to a known
  `HF_HOME` for explicit deployment.