From 413f06fb971a672d3cb9a6bacd90cc33d8a61bb6 Mon Sep 17 00:00:00 2001 From: ravi Date: Thu, 28 May 2026 14:00:38 +0530 Subject: [PATCH] performance improvement --- sam-tool-tauri/docs/BUILD_RELEASE.md | 226 +++++++++++++ sam-tool-tauri/docs/HF_DEPLOY_OFFLINE.md | 226 +++++++++++++ .../docs/LINUX_VIDEO_PLAYBACK_CHECKLIST.md | 137 ++++++++ sam-tool-tauri/docs/POLYGON_SMOOTHING.md | 164 ++++++++++ sam-tool-tauri/src-tauri/src/decoder.rs | 180 ++++++++++- sam-tool-tauri/src-tauri/src/lib.rs | 19 +- sam-tool-tauri/src-tauri/src/settings.rs | 18 ++ sam-tool-tauri/src/App.tsx | 2 +- .../src/components/extract/FrameCanvas.tsx | 5 +- .../src/components/shell/SettingsModal.tsx | 44 +++ sam-tool-tauri/src/ipc.ts | 4 +- sam-tool-tauri/src/modes/ExtractMode.tsx | 304 ++++++++++++++---- sam-tool-tauri/src/types.ts | 4 + 13 files changed, 1254 insertions(+), 79 deletions(-) create mode 100644 sam-tool-tauri/docs/BUILD_RELEASE.md create mode 100644 sam-tool-tauri/docs/HF_DEPLOY_OFFLINE.md create mode 100644 sam-tool-tauri/docs/LINUX_VIDEO_PLAYBACK_CHECKLIST.md create mode 100644 sam-tool-tauri/docs/POLYGON_SMOOTHING.md diff --git a/sam-tool-tauri/docs/BUILD_RELEASE.md b/sam-tool-tauri/docs/BUILD_RELEASE.md new file mode 100644 index 0000000..56acb3f --- /dev/null +++ b/sam-tool-tauri/docs/BUILD_RELEASE.md @@ -0,0 +1,226 @@ +# Building the Tauri app for release (instead of `dev` mode) + +Day-to-day you've been running the desktop app via `npm run tauri dev` — +that's the dev server: hot-reload, attached browser devtools, slower +binary, and it stays bound to your terminal. For real use (or shipping to +another machine), you build a standalone executable + installer. + +--- + +## TL;DR + +```bash +cd sam-tool-tauri +npm install # only on first run / after pulling deps +npm run tauri build # produces a release binary + installer +``` + +Output lands under +`sam-tool-tauri/src-tauri/target/release/bundle/`. On Linux you'll find: + +- `bundle/deb/sam-tool-tauri__amd64.deb` — installable on + Debian/Ubuntu hosts (`sudo dpkg -i …`). +- `bundle/appimage/sam-tool-tauri__amd64.AppImage` — single + portable file, `chmod +x` and run. +- `bundle/rpm/…` if rpm tooling is present. + +Plus the raw binary at +`src-tauri/target/release/sam-tool-tauri` (no installer, just `./run` it). + +--- + +## Prerequisites (Linux) + +If you've only ever run `tauri dev`, you might be missing the system +libraries the bundler needs. On Ubuntu / Debian / Pop!_OS: + +```bash +sudo apt update +sudo apt install -y \ + libwebkit2gtk-4.1-dev \ + libgtk-3-dev \ + libayatana-appindicator3-dev \ + librsvg2-dev \ + patchelf \ + build-essential \ + curl wget file \ + libssl-dev pkg-config +``` + +You also need a recent Rust toolchain (Tauri 2 requires `rustc >= 1.77`): + +```bash +rustup update stable +rustc --version # confirm >= 1.77 +``` + +For AppImage bundling specifically, the Tauri CLI auto-fetches the +AppImage tools on first build. The first run will be slower; subsequent +builds reuse the cache. + +--- + +## Build profiles + +Two ways to produce a release binary, with different tradeoffs: + +### A. `npm run tauri build` — full installer build + +```bash +npm run tauri build +``` + +What it does: +1. Runs `vite build` → produces optimized JS/CSS in `dist/`. +2. Compiles `src-tauri/` Rust crate in `--release` mode. +3. Bundles everything into platform-native installers under + `bundle/`. + +Pros: production-grade, optimised, signable, distributable. +Cons: slow first time (Rust release builds + bundler downloads). + +### B. `cargo build --release` — binary only, skip installers + +If you only want the executable (e.g. for quick testing on a sibling +machine on the same OS), drop the bundling step: + +```bash +cd sam-tool-tauri +npm run build # bundles the frontend → dist/ +cd src-tauri +cargo build --release # binary at target/release/sam-tool-tauri +``` + +The resulting binary expects `dist/` from the same checkout to be +present (Tauri 2 embeds frontend assets, so once compiled it's +self-contained). Faster than the full bundle, but no `.deb`/`.AppImage`. + +### C. Debug build — fast compile, big binary + +For iterating on Rust changes without a full release re-compile: + +```bash +cd src-tauri +cargo build # debug profile — fast compile, big slow binary +./target/debug/sam-tool-tauri +``` + +This is essentially what `tauri dev` runs under the hood, minus the dev +server. + +--- + +## Where the artefacts go + +``` +sam-tool-tauri/ +├── dist/ ← bundled frontend (vite build) +└── src-tauri/target/ + ├── debug/sam-tool-tauri ← debug binary + └── release/ + ├── sam-tool-tauri ← optimised binary + └── bundle/ + ├── deb/sam-tool-tauri__amd64.deb + ├── appimage/sam-tool-tauri__amd64.AppImage + └── rpm/… (if `rpm` tools are installed) +``` + +`bundle/deb/` and `bundle/appimage/` are the things you'd ship to other +machines. + +--- + +## Versioning + +The version string comes from +[`src-tauri/tauri.conf.json`](../sam-tool-tauri/src-tauri/tauri.conf.json) → +`"version": "..."`. Bump it before each release; the value is baked +into both the binary and the installer filenames. + +--- + +## Smoke-test the release build locally + +```bash +# Run the binary directly: +./src-tauri/target/release/sam-tool-tauri + +# Or install the .deb and launch from your app menu: +sudo dpkg -i src-tauri/target/release/bundle/deb/sam-tool-tauri_*_amd64.deb +sam-tool-tauri # binary on $PATH after dpkg + +# Or run the AppImage portably: +chmod +x src-tauri/target/release/bundle/appimage/sam-tool-tauri_*.AppImage +./src-tauri/target/release/bundle/appimage/sam-tool-tauri_*.AppImage +``` + +The release build behaves exactly like `dev` (same UI, same SAM +integration), with these differences: + +- No devtools by default. To re-enable in a release build, edit + `src-tauri/tauri.conf.json` → `"app.windows[0].devtools": true`, + rebuild. +- Faster cold start, smaller memory footprint, optimised JS bundle. +- React `StrictMode` still doubles state-updaters in dev — but that's a + React-side flag, not Tauri's. The stripped-down behaviour comes from + Vite's production mode (`process.env.NODE_ENV === 'production'`). + +--- + +## Distributing to another PC + +The user only needs **one** of the bundles: + +**Debian/Ubuntu:** +```bash +scp src-tauri/target/release/bundle/deb/sam-tool-tauri_*_amd64.deb \ + target-host:/tmp/ +ssh target-host "sudo dpkg -i /tmp/sam-tool-tauri_*_amd64.deb" +``` + +**Anywhere with glibc:** +```bash +scp src-tauri/target/release/bundle/appimage/sam-tool-tauri_*.AppImage \ + target-host:/home/user/ +ssh target-host "chmod +x ~/sam-tool-tauri_*.AppImage && ~/sam-tool-tauri_*.AppImage" +``` + +The target host needs `webkit2gtk` runtime (most desktops already have +it) and `ffmpeg` if they want the Extract mode to decode video. The +sibling SAM2 backend and weights are *not* bundled into the desktop app — +those live separately and are reached over HTTP, configurable from +**Settings → SAM backend URL**. + +--- + +## Common build errors + +| Error | Fix | +|--------------------------------------------------------------|-------------------------------------------------------------------------------------------| +| `error: linker 'cc' not found` | `sudo apt install build-essential` | +| `failed to find webkit2gtk-4.1` / `gobject-introspection` | `sudo apt install libwebkit2gtk-4.1-dev libgtk-3-dev` (see prerequisites) | +| `failed to bundle .deb` | Missing `dpkg-deb`. `sudo apt install dpkg` | +| AppImage step hangs on first build | Tauri downloading appimagetool; wait or pre-fetch under `~/.cache/tauri/` | +| `Could not resolve "@tauri-apps/api"` | `npm install` again in `sam-tool-tauri/` | +| Build OK on dev machine, fails on Ubuntu 22.04 | webkit2gtk-4.0 vs 4.1 mismatch — Tauri 2 needs 4.1; older Ubuntus may need a backport PPA | + +--- + +## Cross-compilation (Windows / macOS targets) + +Out of scope for this repo (Linux-only per CLAUDE.md), but for +completeness — Tauri supports it via extra rust targets and a Windows or +macOS host machine. See the upstream Tauri docs: +https://tauri.app/distribute/ + +--- + +## Recap — when to run what + +| You want to… | Command | +|---------------------------------------------|-------------------------------------------------| +| Iterate on UI (hot reload) | `npm run tauri dev` | +| Iterate on Rust quickly | `cd src-tauri && cargo build && ./target/debug/sam-tool-tauri` | +| Make a release executable, no installer | `npm run build && cd src-tauri && cargo build --release` | +| Make installers (.deb / AppImage) | `npm run tauri build` | +| Inspect production frontend without Tauri | `npm run preview` (browser-only, no native APIs)| diff --git a/sam-tool-tauri/docs/HF_DEPLOY_OFFLINE.md b/sam-tool-tauri/docs/HF_DEPLOY_OFFLINE.md new file mode 100644 index 0000000..b6863f4 --- /dev/null +++ b/sam-tool-tauri/docs/HF_DEPLOY_OFFLINE.md @@ -0,0 +1,226 @@ +# Deploying SAM3 via HuggingFace, then going fully offline + +The SAM3 backend uses Meta's `facebook/sam3` weights via HuggingFace. The +flow is **online once** (to download + cache the weights) → **then +air-gapped** (server runs without ever talking to HuggingFace again). +This doc walks through both phases and how to verify your box is actually +offline at the end. + +--- + +## Phase 0 — prerequisites + +On the target PC: + +- The `sam2-backend/` folder, copied or cloned, with its `.venv` + containing `torch`, `transformers>=4.57`, `huggingface_hub`, and the + rest of `requirements.txt`. See [`sam2-backend/README.md`](../sam2-backend/README.md) + for the full install. +- A HuggingFace account that has been **granted access to + `facebook/sam3`** — gated. Request access at + https://huggingface.co/facebook/sam3 (usually approved within minutes). +- A read token from https://huggingface.co/settings/tokens. + +--- + +## Phase 1 — first-time online download + +The first call to `Sam3TrackerModel.from_pretrained("facebook/sam3")` +streams the weights from HF and writes them to +`~/.cache/huggingface/hub/`. Everything afterward can be served from that +cache with no network. + +### 1.1 Authenticate + +```bash +cd /home//sam2-backend +source .venv/bin/activate +hf auth login # paste your read token +hf auth whoami # confirm the right account +``` + +Token is stored at `~/.cache/huggingface/token`. It's a plain file with +`chmod 600`. + +### 1.2 Set offline flag to OFF for the first start + +In `run.sh`, the relevant lines are: +```bash +export HF_HUB_DISABLE_TELEMETRY="${HF_HUB_DISABLE_TELEMETRY:-1}" +export DO_NOT_TRACK="${DO_NOT_TRACK:-1}" +export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-0}" # ← MUST be 0 first time +export TRANSFORMERS_OFFLINE="${TRANSFORMERS_OFFLINE:-0}" # ← MUST be 0 first time +``` + +For a one-off override, just don't pass the env vars: +```bash +./run.sh +``` + +### 1.3 Make ONE successful predict call + +Either from the Tauri app (Settings → pick `SAM 3 · tracker (HuggingFace)` → +draw a bbox), or directly: + +```bash +curl -s -X POST "http://localhost:9090/predict?model=sam3_tracker" \ + -H "Content-Type: application/json" \ + --data '@some_test_payload.json' +``` + +A successful response means the weights are now fully cached. Confirm: + +```bash +ls -la ~/.cache/huggingface/hub/models--facebook--sam3/ +# expect a snapshots// subdir with model.safetensors and configs +du -sh ~/.cache/huggingface/hub/models--facebook--sam3/ +# ~3-4 GB for SAM3 +``` + +### 1.4 Optional — pre-warm without a real predict + +If you want to download but not yet wire up Tauri, do it from a Python +shell: + +```bash +python3 - <<'EOF' +from transformers import Sam3TrackerModel, Sam3TrackerProcessor +Sam3TrackerProcessor.from_pretrained("facebook/sam3") +Sam3TrackerModel.from_pretrained("facebook/sam3") +print("cached.") +EOF +``` + +Same effect — fills `~/.cache/huggingface/hub/`. + +--- + +## Phase 2 — flip to fully offline + +Once the cache is populated, set the offline flags so the server never +tries to revalidate metadata or check for updates. + +### 2.1 Edit `run.sh` + +Change the defaults to `1`: + +```bash +export HF_HUB_DISABLE_TELEMETRY="${HF_HUB_DISABLE_TELEMETRY:-1}" +export DO_NOT_TRACK="${DO_NOT_TRACK:-1}" +export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-1}" # ← 1 +export TRANSFORMERS_OFFLINE="${TRANSFORMERS_OFFLINE:-1}" # ← 1 +``` + +Or, if you're running via systemd, add to your unit file: + +```ini +[Service] +Environment=HF_HUB_DISABLE_TELEMETRY=1 +Environment=DO_NOT_TRACK=1 +Environment=HF_HUB_OFFLINE=1 +Environment=TRANSFORMERS_OFFLINE=1 +``` + +Then `sudo systemctl daemon-reload && sudo systemctl restart sam-backend`. + +### 2.2 Restart and verify + +```bash +./run.sh +``` + +You should see clean startup logs **without** any `httpx HTTP Request` +lines hitting `huggingface.co`. The `Loading weights: 685/685 [00:00<00:00]` +line confirms a cache hit (instant load, no download). + +### 2.3 Pin a custom cache location (optional) + +If you want the cache somewhere predictable (for backup, rsync to other +boxes, or to keep `$HOME` clean), set `HF_HOME` in `run.sh`: + +```bash +export HF_HOME=/var/cache/sam-backend/huggingface +``` + +Move the existing cache once: +```bash +mv ~/.cache/huggingface /var/cache/sam-backend/ +``` + +Subsequent reads come from there. + +--- + +## Verifying you're really offline + +Two-stage check: + +### Step 1 — disconnect & restart + +Easiest sanity test: pull the network cable (or `sudo ip link set eth0 down`), +restart the server, hit `/predict` from the Tauri app on the same box. +If it returns a polygon, you're truly offline. + +### Step 2 — packet-level proof + +If you can't disconnect (e.g., other services need the network), watch +for any HF traffic during normal use: + +```bash +sudo tcpdump -nn -i any \ + 'host huggingface.co or host cdn-lfs.huggingface.co' \ + 2>/dev/null +``` + +Restart the backend, hit `/predict` ten times from the Tauri app. With +the OFFLINE flags set you should see **zero packets**. + +--- + +## Moving the cache to a different machine + +Useful when you want to deploy to a new box without the first-run +download (e.g., setting up a second annotation station behind the same +firewall). + +```bash +# On the source machine: +tar czf hf-sam3-cache.tar.gz -C ~/.cache huggingface +scp hf-sam3-cache.tar.gz target-box:/tmp/ + +# On the target machine: +mkdir -p ~/.cache +tar xzf /tmp/hf-sam3-cache.tar.gz -C ~/.cache +``` + +Then start the new backend with `HF_HUB_OFFLINE=1` from day one — it'll +load from the imported cache without ever calling out. + +--- + +## Common gotchas + +| Symptom | Cause / fix | +|--------------------------------------------------------------------------|-----------------------------------------------------------------------------| +| `Cannot find the requested files in the cached path` while OFFLINE=1 | Cache is incomplete or in a different `HF_HOME`. Re-run online once. | +| `gated repo, access denied` | Token doesn't have `facebook/sam3` access. Accept gate on the model page, then `hf auth login` again. | +| Many `HEAD https://huggingface.co/...` lines on startup despite OFFLINE=1 | One of the two env flags is missing. Both `HF_HUB_OFFLINE=1` AND `TRANSFORMERS_OFFLINE=1` are required. | +| `Loading weights: ...` takes minutes even with cache | The cache file is on a slow disk (NFS, HDD). Move it to local SSD via `HF_HOME`. | +| Token leaked into logs | `hf auth login` only echoes your username, never the token. The token at `~/.cache/huggingface/token` is the only place it's stored. | + +--- + +## What HF still sees in offline mode + +**Nothing.** With both `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`: + +- `from_pretrained` reads only from `~/.cache/huggingface/hub/` — no API + calls, no HEAD revalidation pings. +- Telemetry is independently off via `HF_HUB_DISABLE_TELEMETRY=1` / + `DO_NOT_TRACK=1`. +- Inference itself never touches the network — your images, bboxes, and + resulting masks stay on the box regardless of online/offline mode. + +The only thing HF "knows" about you, ever, is "this account downloaded +`facebook/sam3` once on date X" — recorded server-side at the moment +`from_pretrained` first ran in Phase 1. diff --git a/sam-tool-tauri/docs/LINUX_VIDEO_PLAYBACK_CHECKLIST.md b/sam-tool-tauri/docs/LINUX_VIDEO_PLAYBACK_CHECKLIST.md new file mode 100644 index 0000000..d2d89b7 --- /dev/null +++ b/sam-tool-tauri/docs/LINUX_VIDEO_PLAYBACK_CHECKLIST.md @@ -0,0 +1,137 @@ +# Linux Video Playback Checklist + +Use this checklist on Linux machines where video playback or frame stepping feels +slow or sluggish. + +## Baseline Packages + +Install or update the common runtime and diagnostic tools: + +```bash +sudo apt update +sudo apt full-upgrade +sudo apt install ffmpeg ffprobe vainfo vdpauinfo mesa-utils vulkan-tools +``` + +Make sure the user has access to GPU render devices: + +```bash +sudo usermod -aG render,video $USER +``` + +Log out and back in after changing groups. + +## Tauri / WebKit Runtime + +For Debian or Ubuntu systems: + +```bash +sudo apt install libwebkit2gtk-4.1-0 libgtk-3-0 libayatana-appindicator3-1 +``` + +## Intel GPU + +For newer Intel integrated GPUs: + +```bash +sudo apt install intel-media-va-driver intel-media-va-driver-non-free va-driver-all +``` + +If VAAPI fails, try launching with: + +```bash +LIBVA_DRIVER_NAME=iHD ./sam-tool +``` + +For older Intel GPUs: + +```bash +sudo apt install i965-va-driver +LIBVA_DRIVER_NAME=i965 ./sam-tool +``` + +## AMD GPU + +Use Mesa VAAPI / VDPAU drivers: + +```bash +sudo apt install mesa-va-drivers mesa-vdpau-drivers va-driver-all +``` + +In the app, prefer `VAAPI`. + +## NVIDIA GPU + +Use the proprietary NVIDIA driver, not Nouveau, for best decode support: + +```bash +sudo ubuntu-drivers autoinstall +sudo reboot +``` + +After reboot: + +```bash +nvidia-smi +ffmpeg -hide_banner -hwaccels +``` + +In the app, prefer `CUDA`. Try `VDPAU` only as a fallback. + +## GStreamer / WebKit Video Stack + +These packages are useful for WebKit video support, especially if native +`