first version
This commit is contained in:
213
README.md
Normal file
213
README.md
Normal file
@@ -0,0 +1,213 @@
|
||||
# SAM-Tool
|
||||
|
||||
Desktop video-frame extraction + image annotation tool, with optional
|
||||
SAM 2 / SAM 3 segmentation backend. Built to replace an earlier
|
||||
Python/Tkinter prototype (`sam_tool_v2.py`) for YOLO-style training
|
||||
dataset prep.
|
||||
|
||||
```
|
||||
┌──────────────────┐ HTTP ┌──────────────────┐
|
||||
│ sam-tool-tauri │ ◀───────────────▶ │ sam2-backend │
|
||||
│ (Tauri/Rust + UI)│ /predict, etc │ (FastAPI + GPU) │
|
||||
└──────────────────┘ └──────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
local images + SAM 2.x / SAM 3
|
||||
sibling JSONs weights on disk
|
||||
```
|
||||
|
||||
The two halves can run on the same machine or separate boxes — point the
|
||||
desktop app at any reachable backend URL from **Settings**.
|
||||
|
||||
---
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
SAM-Tool/
|
||||
├── sam-tool-tauri/ # Desktop app (Tauri 2, React 18, TS, Vite)
|
||||
│ ├── src/ # Frontend
|
||||
│ ├── src-tauri/ # Rust commands (video, decoder, IPC)
|
||||
│ └── package.json # `npm run tauri dev` / `tauri build`
|
||||
│
|
||||
├── sam2-backend/ # FastAPI HTTP server (SAM 2 + SAM 3)
|
||||
│ ├── server.py # /, /models, /predict
|
||||
│ ├── registry.py # Lazy-load + single-resident cache
|
||||
│ ├── adapters/
|
||||
│ │ ├── sam2_adapter.py
|
||||
│ │ ├── sam3_adapter.py
|
||||
│ │ └── utils.py # mask → polygon helper
|
||||
│ ├── models.yaml # Model registry (paths + family)
|
||||
│ ├── run.sh # Convenience launcher (env vars)
|
||||
│ └── README.md # Install + run on the server side
|
||||
│
|
||||
├── sam2/sam2/ # Meta SAM 2 source + checkpoints (vendored)
|
||||
├── files/sam3_weights/ # SAM 3 weights (3.4 GB sam3.pt)
|
||||
├── files/sam3_server.py # Reference SAM3 server (used as a guide)
|
||||
│
|
||||
├── sam_tool_v2.py # Original Python/Tkinter tool (legacy)
|
||||
├── sam_annotation_test/ # Sample data — dashcam MP4 + custom SAM JSON
|
||||
│
|
||||
├── docs/ # ← you are here
|
||||
│ ├── POLYGON_SMOOTHING.md How to tune mask→polygon output
|
||||
│ ├── HF_DEPLOY_OFFLINE.md First-online-then-airgapped HF deploy
|
||||
│ └── BUILD_RELEASE.md Building the Tauri app for release
|
||||
│
|
||||
└── CLAUDE.md # Project rules (think → simplify → surgical)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick start (single-machine dev)
|
||||
|
||||
### 1. SAM backend
|
||||
|
||||
```bash
|
||||
cd sam2-backend
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install --upgrade pip
|
||||
|
||||
# Pick the right Torch wheel for your GPU (driver 570+ → cu128):
|
||||
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
|
||||
# CPU-only fallback (slow):
|
||||
# pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
|
||||
|
||||
# SAM 2 source + deps
|
||||
pip install -e ../sam2/sam2
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Optional — SAM 3 support (needs HF account + accepted gating)
|
||||
git clone https://github.com/facebookresearch/sam3.git ..
|
||||
pip install -e ../sam3
|
||||
pip install "transformers>=4.57"
|
||||
hf auth login # paste read token from https://huggingface.co/settings/tokens
|
||||
|
||||
./run.sh # binds 0.0.0.0:9090, auto-picks cuda/cpu
|
||||
```
|
||||
|
||||
Smoke-test:
|
||||
```bash
|
||||
curl http://localhost:9090/ # {"status":"ok",...}
|
||||
curl http://localhost:9090/models # registered models
|
||||
```
|
||||
|
||||
Full backend setup details: [`sam2-backend/README.md`](sam2-backend/README.md).
|
||||
|
||||
### 2. Tauri desktop app
|
||||
|
||||
```bash
|
||||
cd sam-tool-tauri
|
||||
npm install
|
||||
npm run tauri dev # hot-reload dev mode
|
||||
|
||||
# Or for a real build / installer:
|
||||
npm run tauri build # see docs/BUILD_RELEASE.md
|
||||
```
|
||||
|
||||
In the running app:
|
||||
1. **Login** — pick or type a username (no password — used for annotation
|
||||
author stamping).
|
||||
2. **Extract mode** — Upload a video → optionally `Import annotations…`
|
||||
(COCO / SAM JSON) → click `choose output folder…` (or just open a
|
||||
video; the default `<video>_extracted/` folder is auto-adopted) →
|
||||
scrub timeline, press **E** to extract, **U** to undo, **[ … ]** for
|
||||
bulk range.
|
||||
3. **Annotate mode** — Open the extracted folder. Draw bboxes (B) /
|
||||
polygons (P), label them via the picker, switch to select (S) to
|
||||
move/resize/reclass. Settings → enable SAM, pick a model, get
|
||||
polygons back instead of bboxes.
|
||||
|
||||
### 3. Deploying to another PC
|
||||
|
||||
Backend → see the install steps in [`sam2-backend/README.md`](sam2-backend/README.md)
|
||||
("install on a different PC" section).
|
||||
|
||||
Desktop app → [`docs/BUILD_RELEASE.md`](docs/BUILD_RELEASE.md) covers
|
||||
producing `.deb` / AppImage installers.
|
||||
|
||||
---
|
||||
|
||||
## Feature highlights
|
||||
|
||||
### Extract mode
|
||||
- Persistent `ffmpeg -f image2pipe` subprocess + LRU cache + prefetch
|
||||
worker → smooth playback up to ~2× on 1080p.
|
||||
- Per-video default output folder (`<video>_extracted/`) auto-restored on
|
||||
reopen — extracted-frame state and "EXTRACTED" tint reflected
|
||||
immediately.
|
||||
- Timeline ticks mark every previously-extracted frame at a glance.
|
||||
- COCO + custom SAM JSON import, smart overlay with per-class colours,
|
||||
Review mode auto-pauses on annotated frames.
|
||||
- `Space` play/pause · `←→` ±1 frame · `Ctrl+←→` ±10 · `↑↓` speed 0.25–4×
|
||||
· `H` overlay · `E` extract · `U` undo · `[ … ]` bulk range.
|
||||
|
||||
### Annotate mode
|
||||
- Sibling JSON per image with `source_video` field — annotations know
|
||||
where they came from.
|
||||
- Bbox + polygon tools, vertex drag (rAF-throttled), bbox handles, undo
|
||||
(40 deep) + redo, Tab-reclass, inline class creation in the picker.
|
||||
- Multi-select image list (Shift-range, Ctrl-toggle) + bulk delete with
|
||||
two-step confirmation.
|
||||
- A/D image navigation, +/- zoom, cursor-anchored Ctrl+wheel zoom.
|
||||
- SAM mode round-trip with live phase indicator showing model + inference
|
||||
time (auto-fades after 3 s).
|
||||
- Auto-switch to **select** mode after committing each new shape.
|
||||
|
||||
### SAM backend
|
||||
- Pluggable model registry — add SAM-3 / SAM-4 / etc. by dropping an
|
||||
adapter into `adapters/` and one entry into `models.yaml`.
|
||||
- Lazy-load with single-resident eviction (set
|
||||
`SAM_BACKEND_KEEP_ALL=1` to keep all loaded if you have VRAM).
|
||||
- Speaks the Label-Studio-ML `/predict` protocol the Tauri client
|
||||
already used — drop-in compatible.
|
||||
- Local SAM 3 image-predictor + HuggingFace SAM 3 tracker both supported.
|
||||
- Privacy-first: telemetry off by default, fully air-gappable after the
|
||||
first weight download — see
|
||||
[`docs/HF_DEPLOY_OFFLINE.md`](docs/HF_DEPLOY_OFFLINE.md).
|
||||
|
||||
---
|
||||
|
||||
## Documentation index
|
||||
|
||||
| Doc | What it covers |
|
||||
|----------------------------------------------------|-------------------------------------------------------------|
|
||||
| [docs/POLYGON_SMOOTHING.md](docs/POLYGON_SMOOTHING.md) | Tune mask→polygon: point count, kernel size, smoothing on/off |
|
||||
| [docs/HF_DEPLOY_OFFLINE.md](docs/HF_DEPLOY_OFFLINE.md) | Download HF SAM 3 weights, then run fully offline |
|
||||
| [docs/BUILD_RELEASE.md](docs/BUILD_RELEASE.md) | Build the Tauri app for production (deb / AppImage / binary) |
|
||||
| [sam2-backend/README.md](sam2-backend/README.md) | Backend install, run, API reference, model-registry details |
|
||||
| [CLAUDE.md](CLAUDE.md) | Project rules — Think Before Coding · Simplicity First · Surgical Changes · Goal-Driven Execution |
|
||||
|
||||
---
|
||||
|
||||
## Stack
|
||||
|
||||
- **Desktop:** Tauri 2 shell, Rust backend, React 18 + TypeScript, Vite,
|
||||
WebKit2GTK 2.50.
|
||||
- **Backend:** Python 3.10+, FastAPI, uvicorn, Pillow, NumPy, OpenCV
|
||||
(headless), PyTorch ≥ 2.4 with matching CUDA, optional `transformers ≥ 4.57`.
|
||||
- **Models:** Meta SAM 2 / SAM 2.1 (4 Hiera variants), Meta SAM 3.
|
||||
- **Target OS:** Linux (Ubuntu 22.04 / Pop!_OS), AMD APU + NVIDIA RTX
|
||||
GPUs tested.
|
||||
|
||||
---
|
||||
|
||||
## Conventions
|
||||
|
||||
From [CLAUDE.md](CLAUDE.md):
|
||||
|
||||
- **Think Before Coding** — wrong assumptions are the most expensive bugs.
|
||||
- **Simplicity First** — fewer moving parts beats clever abstractions.
|
||||
- **Surgical Changes** — touch what the task needs, leave the rest.
|
||||
- **Goal-Driven Execution** — verify the user-facing behaviour, not just
|
||||
the code path.
|
||||
|
||||
A few specific rules learned the hard way:
|
||||
- `React.StrictMode` double-invokes setState updaters; don't put side
|
||||
effects (like `queueMicrotask` calls or save requests) inside them.
|
||||
Use refs + explicit calls instead.
|
||||
- No `crypto.randomUUID()` without a fallback — webview contexts
|
||||
sometimes lack it; use the `uuid()` helper.
|
||||
- Output folders must live **outside** `sam-tool-tauri/` — the dev
|
||||
watcher restarts the app on any file change inside the project tree.
|
||||
- Backend telemetry stays **off** by default; HF cache moves to a known
|
||||
`HF_HOME` for explicit deployment.
|
||||
Reference in New Issue
Block a user