214 lines
8.9 KiB
Markdown
214 lines
8.9 KiB
Markdown
# SAM-Tool
|
||
|
||
Desktop video-frame extraction + image annotation tool, with optional
|
||
SAM 2 / SAM 3 segmentation backend. Built to replace an earlier
|
||
Python/Tkinter prototype (`sam_tool_v2.py`) for YOLO-style training
|
||
dataset prep.
|
||
|
||
```
|
||
┌──────────────────┐ HTTP ┌──────────────────┐
|
||
│ sam-tool-tauri │ ◀───────────────▶ │ sam2-backend │
|
||
│ (Tauri/Rust + UI)│ /predict, etc │ (FastAPI + GPU) │
|
||
└──────────────────┘ └──────────────────┘
|
||
│ │
|
||
▼ ▼
|
||
local images + SAM 2.x / SAM 3
|
||
sibling JSONs weights on disk
|
||
```
|
||
|
||
The two halves can run on the same machine or separate boxes — point the
|
||
desktop app at any reachable backend URL from **Settings**.
|
||
|
||
---
|
||
|
||
## Layout
|
||
|
||
```
|
||
SAM-Tool/
|
||
├── sam-tool-tauri/ # Desktop app (Tauri 2, React 18, TS, Vite)
|
||
│ ├── src/ # Frontend
|
||
│ ├── src-tauri/ # Rust commands (video, decoder, IPC)
|
||
│ └── package.json # `npm run tauri dev` / `tauri build`
|
||
│
|
||
├── sam2-backend/ # FastAPI HTTP server (SAM 2 + SAM 3)
|
||
│ ├── server.py # /, /models, /predict
|
||
│ ├── registry.py # Lazy-load + single-resident cache
|
||
│ ├── adapters/
|
||
│ │ ├── sam2_adapter.py
|
||
│ │ ├── sam3_adapter.py
|
||
│ │ └── utils.py # mask → polygon helper
|
||
│ ├── models.yaml # Model registry (paths + family)
|
||
│ ├── run.sh # Convenience launcher (env vars)
|
||
│ └── README.md # Install + run on the server side
|
||
│
|
||
├── sam2/sam2/ # Meta SAM 2 source + checkpoints (vendored)
|
||
├── files/sam3_weights/ # SAM 3 weights (3.4 GB sam3.pt)
|
||
├── files/sam3_server.py # Reference SAM3 server (used as a guide)
|
||
│
|
||
├── sam_tool_v2.py # Original Python/Tkinter tool (legacy)
|
||
├── sam_annotation_test/ # Sample data — dashcam MP4 + custom SAM JSON
|
||
│
|
||
├── docs/ # ← you are here
|
||
│ ├── POLYGON_SMOOTHING.md How to tune mask→polygon output
|
||
│ ├── HF_DEPLOY_OFFLINE.md First-online-then-airgapped HF deploy
|
||
│ └── BUILD_RELEASE.md Building the Tauri app for release
|
||
│
|
||
└── CLAUDE.md # Project rules (think → simplify → surgical)
|
||
```
|
||
|
||
---
|
||
|
||
## Quick start (single-machine dev)
|
||
|
||
### 1. SAM backend
|
||
|
||
```bash
|
||
cd sam2-backend
|
||
python3 -m venv .venv && source .venv/bin/activate
|
||
pip install --upgrade pip
|
||
|
||
# Pick the right Torch wheel for your GPU (driver 570+ → cu128):
|
||
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
|
||
# CPU-only fallback (slow):
|
||
# pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
|
||
|
||
# SAM 2 source + deps
|
||
pip install -e ../sam2/sam2
|
||
pip install -r requirements.txt
|
||
|
||
# Optional — SAM 3 support (needs HF account + accepted gating)
|
||
git clone https://github.com/facebookresearch/sam3.git ..
|
||
pip install -e ../sam3
|
||
pip install "transformers>=4.57"
|
||
hf auth login # paste read token from https://huggingface.co/settings/tokens
|
||
|
||
./run.sh # binds 0.0.0.0:9090, auto-picks cuda/cpu
|
||
```
|
||
|
||
Smoke-test:
|
||
```bash
|
||
curl http://localhost:9090/ # {"status":"ok",...}
|
||
curl http://localhost:9090/models # registered models
|
||
```
|
||
|
||
Full backend setup details: [`sam2-backend/README.md`](sam2-backend/README.md).
|
||
|
||
### 2. Tauri desktop app
|
||
|
||
```bash
|
||
cd sam-tool-tauri
|
||
npm install
|
||
npm run tauri dev # hot-reload dev mode
|
||
|
||
# Or for a real build / installer:
|
||
npm run tauri build # see docs/BUILD_RELEASE.md
|
||
```
|
||
|
||
In the running app:
|
||
1. **Login** — pick or type a username (no password — used for annotation
|
||
author stamping).
|
||
2. **Extract mode** — Upload a video → optionally `Import annotations…`
|
||
(COCO / SAM JSON) → click `choose output folder…` (or just open a
|
||
video; the default `<video>_extracted/` folder is auto-adopted) →
|
||
scrub timeline, press **E** to extract, **U** to undo, **[ … ]** for
|
||
bulk range.
|
||
3. **Annotate mode** — Open the extracted folder. Draw bboxes (B) /
|
||
polygons (P), label them via the picker, switch to select (S) to
|
||
move/resize/reclass. Settings → enable SAM, pick a model, get
|
||
polygons back instead of bboxes.
|
||
|
||
### 3. Deploying to another PC
|
||
|
||
Backend → see the install steps in [`sam2-backend/README.md`](sam2-backend/README.md)
|
||
("install on a different PC" section).
|
||
|
||
Desktop app → [`docs/BUILD_RELEASE.md`](docs/BUILD_RELEASE.md) covers
|
||
producing `.deb` / AppImage installers.
|
||
|
||
---
|
||
|
||
## Feature highlights
|
||
|
||
### Extract mode
|
||
- Persistent `ffmpeg -f image2pipe` subprocess + LRU cache + prefetch
|
||
worker → smooth playback up to ~2× on 1080p.
|
||
- Per-video default output folder (`<video>_extracted/`) auto-restored on
|
||
reopen — extracted-frame state and "EXTRACTED" tint reflected
|
||
immediately.
|
||
- Timeline ticks mark every previously-extracted frame at a glance.
|
||
- COCO + custom SAM JSON import, smart overlay with per-class colours,
|
||
Review mode auto-pauses on annotated frames.
|
||
- `Space` play/pause · `←→` ±1 frame · `Ctrl+←→` ±10 · `↑↓` speed 0.25–4×
|
||
· `H` overlay · `E` extract · `U` undo · `[ … ]` bulk range.
|
||
|
||
### Annotate mode
|
||
- Sibling JSON per image with `source_video` field — annotations know
|
||
where they came from.
|
||
- Bbox + polygon tools, vertex drag (rAF-throttled), bbox handles, undo
|
||
(40 deep) + redo, Tab-reclass, inline class creation in the picker.
|
||
- Multi-select image list (Shift-range, Ctrl-toggle) + bulk delete with
|
||
two-step confirmation.
|
||
- A/D image navigation, +/- zoom, cursor-anchored Ctrl+wheel zoom.
|
||
- SAM mode round-trip with live phase indicator showing model + inference
|
||
time (auto-fades after 3 s).
|
||
- Auto-switch to **select** mode after committing each new shape.
|
||
|
||
### SAM backend
|
||
- Pluggable model registry — add SAM-3 / SAM-4 / etc. by dropping an
|
||
adapter into `adapters/` and one entry into `models.yaml`.
|
||
- Lazy-load with single-resident eviction (set
|
||
`SAM_BACKEND_KEEP_ALL=1` to keep all loaded if you have VRAM).
|
||
- Speaks the Label-Studio-ML `/predict` protocol the Tauri client
|
||
already used — drop-in compatible.
|
||
- Local SAM 3 image-predictor + HuggingFace SAM 3 tracker both supported.
|
||
- Privacy-first: telemetry off by default, fully air-gappable after the
|
||
first weight download — see
|
||
[`docs/HF_DEPLOY_OFFLINE.md`](docs/HF_DEPLOY_OFFLINE.md).
|
||
|
||
---
|
||
|
||
## Documentation index
|
||
|
||
| Doc | What it covers |
|
||
|----------------------------------------------------|-------------------------------------------------------------|
|
||
| [docs/POLYGON_SMOOTHING.md](docs/POLYGON_SMOOTHING.md) | Tune mask→polygon: point count, kernel size, smoothing on/off |
|
||
| [docs/HF_DEPLOY_OFFLINE.md](docs/HF_DEPLOY_OFFLINE.md) | Download HF SAM 3 weights, then run fully offline |
|
||
| [docs/BUILD_RELEASE.md](docs/BUILD_RELEASE.md) | Build the Tauri app for production (deb / AppImage / binary) |
|
||
| [sam2-backend/README.md](sam2-backend/README.md) | Backend install, run, API reference, model-registry details |
|
||
| [CLAUDE.md](CLAUDE.md) | Project rules — Think Before Coding · Simplicity First · Surgical Changes · Goal-Driven Execution |
|
||
|
||
---
|
||
|
||
## Stack
|
||
|
||
- **Desktop:** Tauri 2 shell, Rust backend, React 18 + TypeScript, Vite,
|
||
WebKit2GTK 2.50.
|
||
- **Backend:** Python 3.10+, FastAPI, uvicorn, Pillow, NumPy, OpenCV
|
||
(headless), PyTorch ≥ 2.4 with matching CUDA, optional `transformers ≥ 4.57`.
|
||
- **Models:** Meta SAM 2 / SAM 2.1 (4 Hiera variants), Meta SAM 3.
|
||
- **Target OS:** Linux (Ubuntu 22.04 / Pop!_OS), AMD APU + NVIDIA RTX
|
||
GPUs tested.
|
||
|
||
---
|
||
|
||
## Conventions
|
||
|
||
From [CLAUDE.md](CLAUDE.md):
|
||
|
||
- **Think Before Coding** — wrong assumptions are the most expensive bugs.
|
||
- **Simplicity First** — fewer moving parts beats clever abstractions.
|
||
- **Surgical Changes** — touch what the task needs, leave the rest.
|
||
- **Goal-Driven Execution** — verify the user-facing behaviour, not just
|
||
the code path.
|
||
|
||
A few specific rules learned the hard way:
|
||
- `React.StrictMode` double-invokes setState updaters; don't put side
|
||
effects (like `queueMicrotask` calls or save requests) inside them.
|
||
Use refs + explicit calls instead.
|
||
- No `crypto.randomUUID()` without a fallback — webview contexts
|
||
sometimes lack it; use the `uuid()` helper.
|
||
- Output folders must live **outside** `sam-tool-tauri/` — the dev
|
||
watcher restarts the app on any file change inside the project tree.
|
||
- Backend telemetry stays **off** by default; HF cache moves to a known
|
||
`HF_HOME` for explicit deployment.
|