SAM-Tool

Desktop video-frame extraction + image annotation tool, with optional SAM 2 / SAM 3 segmentation backend. Built to replace an earlier Python/Tkinter prototype (sam_tool_v2.py) for YOLO-style training dataset prep.

┌──────────────────┐        HTTP        ┌──────────────────┐
│   sam-tool-tauri │ ◀───────────────▶ │   sam2-backend   │
│ (Tauri/Rust + UI)│   /predict, etc   │ (FastAPI + GPU)  │
└──────────────────┘                    └──────────────────┘
        │                                        │
        ▼                                        ▼
   local images +                         SAM 2.x / SAM 3
   sibling JSONs                          weights on disk

The two halves can run on the same machine or separate boxes — point the desktop app at any reachable backend URL from Settings.

---

Layout

SAM-Tool/
├── sam-tool-tauri/         # Desktop app (Tauri 2, React 18, TS, Vite)
│   ├── src/                # Frontend
│   ├── src-tauri/          # Rust commands (video, decoder, IPC)
│   └── package.json        # `npm run tauri dev` / `tauri build`
│
├── sam2-backend/           # FastAPI HTTP server (SAM 2 + SAM 3)
│   ├── server.py           # /, /models, /predict
│   ├── registry.py         # Lazy-load + single-resident cache
│   ├── adapters/
│   │   ├── sam2_adapter.py
│   │   ├── sam3_adapter.py
│   │   └── utils.py        # mask → polygon helper
│   ├── models.yaml         # Model registry (paths + family)
│   ├── run.sh              # Convenience launcher (env vars)
│   └── README.md           # Install + run on the server side
│
├── sam2/sam2/              # Meta SAM 2 source + checkpoints (vendored)
├── files/sam3_weights/     # SAM 3 weights (3.4 GB sam3.pt)
├── files/sam3_server.py    # Reference SAM3 server (used as a guide)
│
├── sam_tool_v2.py          # Original Python/Tkinter tool (legacy)
├── sam_annotation_test/    # Sample data — dashcam MP4 + custom SAM JSON
│
├── docs/                   # ← you are here
│   ├── POLYGON_SMOOTHING.md   How to tune mask→polygon output
│   ├── HF_DEPLOY_OFFLINE.md   First-online-then-airgapped HF deploy
│   └── BUILD_RELEASE.md       Building the Tauri app for release
│
└── CLAUDE.md               # Project rules (think → simplify → surgical)

---

Quick start (single-machine dev)

1. SAM backend

cd sam2-backend
python3 -m venv .venv && source .venv/bin/activate
pip install --upgrade pip

# Pick the right Torch wheel for your GPU (driver 570+ → cu128):
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
# CPU-only fallback (slow):
# pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu

# SAM 2 source + deps
pip install -e ../sam2/sam2
pip install -r requirements.txt

# Optional — SAM 3 support (needs HF account + accepted gating)
git clone https://github.com/facebookresearch/sam3.git ..
pip install -e ../sam3
pip install "transformers>=4.57"
hf auth login              # paste read token from https://huggingface.co/settings/tokens

./run.sh                   # binds 0.0.0.0:9090, auto-picks cuda/cpu

Smoke-test:

curl http://localhost:9090/         # {"status":"ok",...}
curl http://localhost:9090/models   # registered models

Full backend setup details: sam2-backend/README.md.

2. Tauri desktop app

cd sam-tool-tauri
npm install
npm run tauri dev                 # hot-reload dev mode

# Or for a real build / installer:
npm run tauri build               # see docs/BUILD_RELEASE.md

In the running app:

1. Login — pick or type a username (no password — used for annotation author stamping).
2. Extract mode — Upload a video → optionally Import annotations… (COCO / SAM JSON) → click choose output folder… (or just open a video; the default <video>_extracted/ folder is auto-adopted) → scrub timeline, press E to extract, U to undo, [ … ] for bulk range.
3. Annotate mode — Open the extracted folder. Draw bboxes (B) / polygons (P), label them via the picker, switch to select (S) to move/resize/reclass. Settings → enable SAM, pick a model, get polygons back instead of bboxes.

3. Deploying to another PC

Backend → see the install steps in sam2-backend/README.md ("install on a different PC" section).

Desktop app → docs/BUILD_RELEASE.md covers producing .deb / AppImage installers.

---

Feature highlights

Extract mode

• Persistent ffmpeg -f image2pipe subprocess + LRU cache + prefetch worker → smooth playback up to ~2× on 1080p.
• Per-video default output folder (<video>_extracted/) auto-restored on reopen — extracted-frame state and "EXTRACTED" tint reflected immediately.
• Timeline ticks mark every previously-extracted frame at a glance.
• COCO + custom SAM JSON import, smart overlay with per-class colours, Review mode auto-pauses on annotated frames.
• Space play/pause · ←→ ±1 frame · Ctrl+←→ ±10 · ↑↓ speed 0.25–4× · H overlay · E extract · U undo · [ … ] bulk range.

Annotate mode

• Sibling JSON per image with source_video field — annotations know where they came from.
• Bbox + polygon tools, vertex drag (rAF-throttled), bbox handles, undo (40 deep) + redo, Tab-reclass, inline class creation in the picker.
• Multi-select image list (Shift-range, Ctrl-toggle) + bulk delete with two-step confirmation.
• A/D image navigation, +/- zoom, cursor-anchored Ctrl+wheel zoom.
• SAM mode round-trip with live phase indicator showing model + inference time (auto-fades after 3 s).
• Auto-switch to select mode after committing each new shape.

SAM backend

• Pluggable model registry — add SAM-3 / SAM-4 / etc. by dropping an adapter into adapters/ and one entry into models.yaml.
• Lazy-load with single-resident eviction (set SAM_BACKEND_KEEP_ALL=1 to keep all loaded if you have VRAM).
• Speaks the Label-Studio-ML /predict protocol the Tauri client already used — drop-in compatible.
• Local SAM 3 image-predictor + HuggingFace SAM 3 tracker both supported.
• Privacy-first: telemetry off by default, fully air-gappable after the first weight download — see docs/HF_DEPLOY_OFFLINE.md.

---

Documentation index

Doc	What it covers
docs/POLYGON_SMOOTHING.md	Tune mask→polygon: point count, kernel size, smoothing on/off
docs/HF_DEPLOY_OFFLINE.md	Download HF SAM 3 weights, then run fully offline
docs/BUILD_RELEASE.md	Build the Tauri app for production (deb / AppImage / binary)
sam2-backend/README.md	Backend install, run, API reference, model-registry details
CLAUDE.md	Project rules — Think Before Coding · Simplicity First · Surgical Changes · Goal-Driven Execution

---

Stack

• Desktop: Tauri 2 shell, Rust backend, React 18 + TypeScript, Vite, WebKit2GTK 2.50.
• Backend: Python 3.10+, FastAPI, uvicorn, Pillow, NumPy, OpenCV (headless), PyTorch ≥ 2.4 with matching CUDA, optional transformers ≥ 4.57.
• Models: Meta SAM 2 / SAM 2.1 (4 Hiera variants), Meta SAM 3.
• Target OS: Linux (Ubuntu 22.04 / Pop!_OS), AMD APU + NVIDIA RTX GPUs tested.

---

Conventions

From CLAUDE.md:

• Think Before Coding — wrong assumptions are the most expensive bugs.
• Simplicity First — fewer moving parts beats clever abstractions.
• Surgical Changes — touch what the task needs, leave the rest.
• Goal-Driven Execution — verify the user-facing behaviour, not just the code path.

A few specific rules learned the hard way:

• React.StrictMode double-invokes setState updaters; don't put side effects (like queueMicrotask calls or save requests) inside them. Use refs + explicit calls instead.
• No crypto.randomUUID() without a fallback — webview contexts sometimes lack it; use the uuid() helper.
• Output folders must live outside sam-tool-tauri/ — the dev watcher restarts the app on any file change inside the project tree.
• Backend telemetry stays off by default; HF cache moves to a known HF_HOME for explicit deployment.