performance improvement
This commit is contained in:
226
sam-tool-tauri/docs/BUILD_RELEASE.md
Normal file
226
sam-tool-tauri/docs/BUILD_RELEASE.md
Normal file
@@ -0,0 +1,226 @@
|
||||
# Building the Tauri app for release (instead of `dev` mode)
|
||||
|
||||
Day-to-day you've been running the desktop app via `npm run tauri dev` —
|
||||
that's the dev server: hot-reload, attached browser devtools, slower
|
||||
binary, and it stays bound to your terminal. For real use (or shipping to
|
||||
another machine), you build a standalone executable + installer.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
```bash
|
||||
cd sam-tool-tauri
|
||||
npm install # only on first run / after pulling deps
|
||||
npm run tauri build # produces a release binary + installer
|
||||
```
|
||||
|
||||
Output lands under
|
||||
`sam-tool-tauri/src-tauri/target/release/bundle/`. On Linux you'll find:
|
||||
|
||||
- `bundle/deb/sam-tool-tauri_<version>_amd64.deb` — installable on
|
||||
Debian/Ubuntu hosts (`sudo dpkg -i …`).
|
||||
- `bundle/appimage/sam-tool-tauri_<version>_amd64.AppImage` — single
|
||||
portable file, `chmod +x` and run.
|
||||
- `bundle/rpm/…` if rpm tooling is present.
|
||||
|
||||
Plus the raw binary at
|
||||
`src-tauri/target/release/sam-tool-tauri` (no installer, just `./run` it).
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites (Linux)
|
||||
|
||||
If you've only ever run `tauri dev`, you might be missing the system
|
||||
libraries the bundler needs. On Ubuntu / Debian / Pop!_OS:
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install -y \
|
||||
libwebkit2gtk-4.1-dev \
|
||||
libgtk-3-dev \
|
||||
libayatana-appindicator3-dev \
|
||||
librsvg2-dev \
|
||||
patchelf \
|
||||
build-essential \
|
||||
curl wget file \
|
||||
libssl-dev pkg-config
|
||||
```
|
||||
|
||||
You also need a recent Rust toolchain (Tauri 2 requires `rustc >= 1.77`):
|
||||
|
||||
```bash
|
||||
rustup update stable
|
||||
rustc --version # confirm >= 1.77
|
||||
```
|
||||
|
||||
For AppImage bundling specifically, the Tauri CLI auto-fetches the
|
||||
AppImage tools on first build. The first run will be slower; subsequent
|
||||
builds reuse the cache.
|
||||
|
||||
---
|
||||
|
||||
## Build profiles
|
||||
|
||||
Two ways to produce a release binary, with different tradeoffs:
|
||||
|
||||
### A. `npm run tauri build` — full installer build
|
||||
|
||||
```bash
|
||||
npm run tauri build
|
||||
```
|
||||
|
||||
What it does:
|
||||
1. Runs `vite build` → produces optimized JS/CSS in `dist/`.
|
||||
2. Compiles `src-tauri/` Rust crate in `--release` mode.
|
||||
3. Bundles everything into platform-native installers under
|
||||
`bundle/`.
|
||||
|
||||
Pros: production-grade, optimised, signable, distributable.
|
||||
Cons: slow first time (Rust release builds + bundler downloads).
|
||||
|
||||
### B. `cargo build --release` — binary only, skip installers
|
||||
|
||||
If you only want the executable (e.g. for quick testing on a sibling
|
||||
machine on the same OS), drop the bundling step:
|
||||
|
||||
```bash
|
||||
cd sam-tool-tauri
|
||||
npm run build # bundles the frontend → dist/
|
||||
cd src-tauri
|
||||
cargo build --release # binary at target/release/sam-tool-tauri
|
||||
```
|
||||
|
||||
The resulting binary expects `dist/` from the same checkout to be
|
||||
present (Tauri 2 embeds frontend assets, so once compiled it's
|
||||
self-contained). Faster than the full bundle, but no `.deb`/`.AppImage`.
|
||||
|
||||
### C. Debug build — fast compile, big binary
|
||||
|
||||
For iterating on Rust changes without a full release re-compile:
|
||||
|
||||
```bash
|
||||
cd src-tauri
|
||||
cargo build # debug profile — fast compile, big slow binary
|
||||
./target/debug/sam-tool-tauri
|
||||
```
|
||||
|
||||
This is essentially what `tauri dev` runs under the hood, minus the dev
|
||||
server.
|
||||
|
||||
---
|
||||
|
||||
## Where the artefacts go
|
||||
|
||||
```
|
||||
sam-tool-tauri/
|
||||
├── dist/ ← bundled frontend (vite build)
|
||||
└── src-tauri/target/
|
||||
├── debug/sam-tool-tauri ← debug binary
|
||||
└── release/
|
||||
├── sam-tool-tauri ← optimised binary
|
||||
└── bundle/
|
||||
├── deb/sam-tool-tauri_<v>_amd64.deb
|
||||
├── appimage/sam-tool-tauri_<v>_amd64.AppImage
|
||||
└── rpm/… (if `rpm` tools are installed)
|
||||
```
|
||||
|
||||
`bundle/deb/` and `bundle/appimage/` are the things you'd ship to other
|
||||
machines.
|
||||
|
||||
---
|
||||
|
||||
## Versioning
|
||||
|
||||
The version string comes from
|
||||
[`src-tauri/tauri.conf.json`](../sam-tool-tauri/src-tauri/tauri.conf.json) →
|
||||
`"version": "..."`. Bump it before each release; the value is baked
|
||||
into both the binary and the installer filenames.
|
||||
|
||||
---
|
||||
|
||||
## Smoke-test the release build locally
|
||||
|
||||
```bash
|
||||
# Run the binary directly:
|
||||
./src-tauri/target/release/sam-tool-tauri
|
||||
|
||||
# Or install the .deb and launch from your app menu:
|
||||
sudo dpkg -i src-tauri/target/release/bundle/deb/sam-tool-tauri_*_amd64.deb
|
||||
sam-tool-tauri # binary on $PATH after dpkg
|
||||
|
||||
# Or run the AppImage portably:
|
||||
chmod +x src-tauri/target/release/bundle/appimage/sam-tool-tauri_*.AppImage
|
||||
./src-tauri/target/release/bundle/appimage/sam-tool-tauri_*.AppImage
|
||||
```
|
||||
|
||||
The release build behaves exactly like `dev` (same UI, same SAM
|
||||
integration), with these differences:
|
||||
|
||||
- No devtools by default. To re-enable in a release build, edit
|
||||
`src-tauri/tauri.conf.json` → `"app.windows[0].devtools": true`,
|
||||
rebuild.
|
||||
- Faster cold start, smaller memory footprint, optimised JS bundle.
|
||||
- React `StrictMode` still doubles state-updaters in dev — but that's a
|
||||
React-side flag, not Tauri's. The stripped-down behaviour comes from
|
||||
Vite's production mode (`process.env.NODE_ENV === 'production'`).
|
||||
|
||||
---
|
||||
|
||||
## Distributing to another PC
|
||||
|
||||
The user only needs **one** of the bundles:
|
||||
|
||||
**Debian/Ubuntu:**
|
||||
```bash
|
||||
scp src-tauri/target/release/bundle/deb/sam-tool-tauri_*_amd64.deb \
|
||||
target-host:/tmp/
|
||||
ssh target-host "sudo dpkg -i /tmp/sam-tool-tauri_*_amd64.deb"
|
||||
```
|
||||
|
||||
**Anywhere with glibc:**
|
||||
```bash
|
||||
scp src-tauri/target/release/bundle/appimage/sam-tool-tauri_*.AppImage \
|
||||
target-host:/home/user/
|
||||
ssh target-host "chmod +x ~/sam-tool-tauri_*.AppImage && ~/sam-tool-tauri_*.AppImage"
|
||||
```
|
||||
|
||||
The target host needs `webkit2gtk` runtime (most desktops already have
|
||||
it) and `ffmpeg` if they want the Extract mode to decode video. The
|
||||
sibling SAM2 backend and weights are *not* bundled into the desktop app —
|
||||
those live separately and are reached over HTTP, configurable from
|
||||
**Settings → SAM backend URL**.
|
||||
|
||||
---
|
||||
|
||||
## Common build errors
|
||||
|
||||
| Error | Fix |
|
||||
|--------------------------------------------------------------|-------------------------------------------------------------------------------------------|
|
||||
| `error: linker 'cc' not found` | `sudo apt install build-essential` |
|
||||
| `failed to find webkit2gtk-4.1` / `gobject-introspection` | `sudo apt install libwebkit2gtk-4.1-dev libgtk-3-dev` (see prerequisites) |
|
||||
| `failed to bundle .deb` | Missing `dpkg-deb`. `sudo apt install dpkg` |
|
||||
| AppImage step hangs on first build | Tauri downloading appimagetool; wait or pre-fetch under `~/.cache/tauri/` |
|
||||
| `Could not resolve "@tauri-apps/api"` | `npm install` again in `sam-tool-tauri/` |
|
||||
| Build OK on dev machine, fails on Ubuntu 22.04 | webkit2gtk-4.0 vs 4.1 mismatch — Tauri 2 needs 4.1; older Ubuntus may need a backport PPA |
|
||||
|
||||
---
|
||||
|
||||
## Cross-compilation (Windows / macOS targets)
|
||||
|
||||
Out of scope for this repo (Linux-only per CLAUDE.md), but for
|
||||
completeness — Tauri supports it via extra rust targets and a Windows or
|
||||
macOS host machine. See the upstream Tauri docs:
|
||||
https://tauri.app/distribute/
|
||||
|
||||
---
|
||||
|
||||
## Recap — when to run what
|
||||
|
||||
| You want to… | Command |
|
||||
|---------------------------------------------|-------------------------------------------------|
|
||||
| Iterate on UI (hot reload) | `npm run tauri dev` |
|
||||
| Iterate on Rust quickly | `cd src-tauri && cargo build && ./target/debug/sam-tool-tauri` |
|
||||
| Make a release executable, no installer | `npm run build && cd src-tauri && cargo build --release` |
|
||||
| Make installers (.deb / AppImage) | `npm run tauri build` |
|
||||
| Inspect production frontend without Tauri | `npm run preview` (browser-only, no native APIs)|
|
||||
226
sam-tool-tauri/docs/HF_DEPLOY_OFFLINE.md
Normal file
226
sam-tool-tauri/docs/HF_DEPLOY_OFFLINE.md
Normal file
@@ -0,0 +1,226 @@
|
||||
# Deploying SAM3 via HuggingFace, then going fully offline
|
||||
|
||||
The SAM3 backend uses Meta's `facebook/sam3` weights via HuggingFace. The
|
||||
flow is **online once** (to download + cache the weights) → **then
|
||||
air-gapped** (server runs without ever talking to HuggingFace again).
|
||||
This doc walks through both phases and how to verify your box is actually
|
||||
offline at the end.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — prerequisites
|
||||
|
||||
On the target PC:
|
||||
|
||||
- The `sam2-backend/` folder, copied or cloned, with its `.venv`
|
||||
containing `torch`, `transformers>=4.57`, `huggingface_hub`, and the
|
||||
rest of `requirements.txt`. See [`sam2-backend/README.md`](../sam2-backend/README.md)
|
||||
for the full install.
|
||||
- A HuggingFace account that has been **granted access to
|
||||
`facebook/sam3`** — gated. Request access at
|
||||
https://huggingface.co/facebook/sam3 (usually approved within minutes).
|
||||
- A read token from https://huggingface.co/settings/tokens.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — first-time online download
|
||||
|
||||
The first call to `Sam3TrackerModel.from_pretrained("facebook/sam3")`
|
||||
streams the weights from HF and writes them to
|
||||
`~/.cache/huggingface/hub/`. Everything afterward can be served from that
|
||||
cache with no network.
|
||||
|
||||
### 1.1 Authenticate
|
||||
|
||||
```bash
|
||||
cd /home/<you>/sam2-backend
|
||||
source .venv/bin/activate
|
||||
hf auth login # paste your read token
|
||||
hf auth whoami # confirm the right account
|
||||
```
|
||||
|
||||
Token is stored at `~/.cache/huggingface/token`. It's a plain file with
|
||||
`chmod 600`.
|
||||
|
||||
### 1.2 Set offline flag to OFF for the first start
|
||||
|
||||
In `run.sh`, the relevant lines are:
|
||||
```bash
|
||||
export HF_HUB_DISABLE_TELEMETRY="${HF_HUB_DISABLE_TELEMETRY:-1}"
|
||||
export DO_NOT_TRACK="${DO_NOT_TRACK:-1}"
|
||||
export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-0}" # ← MUST be 0 first time
|
||||
export TRANSFORMERS_OFFLINE="${TRANSFORMERS_OFFLINE:-0}" # ← MUST be 0 first time
|
||||
```
|
||||
|
||||
For a one-off override, just don't pass the env vars:
|
||||
```bash
|
||||
./run.sh
|
||||
```
|
||||
|
||||
### 1.3 Make ONE successful predict call
|
||||
|
||||
Either from the Tauri app (Settings → pick `SAM 3 · tracker (HuggingFace)` →
|
||||
draw a bbox), or directly:
|
||||
|
||||
```bash
|
||||
curl -s -X POST "http://localhost:9090/predict?model=sam3_tracker" \
|
||||
-H "Content-Type: application/json" \
|
||||
--data '@some_test_payload.json'
|
||||
```
|
||||
|
||||
A successful response means the weights are now fully cached. Confirm:
|
||||
|
||||
```bash
|
||||
ls -la ~/.cache/huggingface/hub/models--facebook--sam3/
|
||||
# expect a snapshots/<hash>/ subdir with model.safetensors and configs
|
||||
du -sh ~/.cache/huggingface/hub/models--facebook--sam3/
|
||||
# ~3-4 GB for SAM3
|
||||
```
|
||||
|
||||
### 1.4 Optional — pre-warm without a real predict
|
||||
|
||||
If you want to download but not yet wire up Tauri, do it from a Python
|
||||
shell:
|
||||
|
||||
```bash
|
||||
python3 - <<'EOF'
|
||||
from transformers import Sam3TrackerModel, Sam3TrackerProcessor
|
||||
Sam3TrackerProcessor.from_pretrained("facebook/sam3")
|
||||
Sam3TrackerModel.from_pretrained("facebook/sam3")
|
||||
print("cached.")
|
||||
EOF
|
||||
```
|
||||
|
||||
Same effect — fills `~/.cache/huggingface/hub/`.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — flip to fully offline
|
||||
|
||||
Once the cache is populated, set the offline flags so the server never
|
||||
tries to revalidate metadata or check for updates.
|
||||
|
||||
### 2.1 Edit `run.sh`
|
||||
|
||||
Change the defaults to `1`:
|
||||
|
||||
```bash
|
||||
export HF_HUB_DISABLE_TELEMETRY="${HF_HUB_DISABLE_TELEMETRY:-1}"
|
||||
export DO_NOT_TRACK="${DO_NOT_TRACK:-1}"
|
||||
export HF_HUB_OFFLINE="${HF_HUB_OFFLINE:-1}" # ← 1
|
||||
export TRANSFORMERS_OFFLINE="${TRANSFORMERS_OFFLINE:-1}" # ← 1
|
||||
```
|
||||
|
||||
Or, if you're running via systemd, add to your unit file:
|
||||
|
||||
```ini
|
||||
[Service]
|
||||
Environment=HF_HUB_DISABLE_TELEMETRY=1
|
||||
Environment=DO_NOT_TRACK=1
|
||||
Environment=HF_HUB_OFFLINE=1
|
||||
Environment=TRANSFORMERS_OFFLINE=1
|
||||
```
|
||||
|
||||
Then `sudo systemctl daemon-reload && sudo systemctl restart sam-backend`.
|
||||
|
||||
### 2.2 Restart and verify
|
||||
|
||||
```bash
|
||||
./run.sh
|
||||
```
|
||||
|
||||
You should see clean startup logs **without** any `httpx HTTP Request`
|
||||
lines hitting `huggingface.co`. The `Loading weights: 685/685 [00:00<00:00]`
|
||||
line confirms a cache hit (instant load, no download).
|
||||
|
||||
### 2.3 Pin a custom cache location (optional)
|
||||
|
||||
If you want the cache somewhere predictable (for backup, rsync to other
|
||||
boxes, or to keep `$HOME` clean), set `HF_HOME` in `run.sh`:
|
||||
|
||||
```bash
|
||||
export HF_HOME=/var/cache/sam-backend/huggingface
|
||||
```
|
||||
|
||||
Move the existing cache once:
|
||||
```bash
|
||||
mv ~/.cache/huggingface /var/cache/sam-backend/
|
||||
```
|
||||
|
||||
Subsequent reads come from there.
|
||||
|
||||
---
|
||||
|
||||
## Verifying you're really offline
|
||||
|
||||
Two-stage check:
|
||||
|
||||
### Step 1 — disconnect & restart
|
||||
|
||||
Easiest sanity test: pull the network cable (or `sudo ip link set eth0 down`),
|
||||
restart the server, hit `/predict` from the Tauri app on the same box.
|
||||
If it returns a polygon, you're truly offline.
|
||||
|
||||
### Step 2 — packet-level proof
|
||||
|
||||
If you can't disconnect (e.g., other services need the network), watch
|
||||
for any HF traffic during normal use:
|
||||
|
||||
```bash
|
||||
sudo tcpdump -nn -i any \
|
||||
'host huggingface.co or host cdn-lfs.huggingface.co' \
|
||||
2>/dev/null
|
||||
```
|
||||
|
||||
Restart the backend, hit `/predict` ten times from the Tauri app. With
|
||||
the OFFLINE flags set you should see **zero packets**.
|
||||
|
||||
---
|
||||
|
||||
## Moving the cache to a different machine
|
||||
|
||||
Useful when you want to deploy to a new box without the first-run
|
||||
download (e.g., setting up a second annotation station behind the same
|
||||
firewall).
|
||||
|
||||
```bash
|
||||
# On the source machine:
|
||||
tar czf hf-sam3-cache.tar.gz -C ~/.cache huggingface
|
||||
scp hf-sam3-cache.tar.gz target-box:/tmp/
|
||||
|
||||
# On the target machine:
|
||||
mkdir -p ~/.cache
|
||||
tar xzf /tmp/hf-sam3-cache.tar.gz -C ~/.cache
|
||||
```
|
||||
|
||||
Then start the new backend with `HF_HUB_OFFLINE=1` from day one — it'll
|
||||
load from the imported cache without ever calling out.
|
||||
|
||||
---
|
||||
|
||||
## Common gotchas
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
|--------------------------------------------------------------------------|-----------------------------------------------------------------------------|
|
||||
| `Cannot find the requested files in the cached path` while OFFLINE=1 | Cache is incomplete or in a different `HF_HOME`. Re-run online once. |
|
||||
| `gated repo, access denied` | Token doesn't have `facebook/sam3` access. Accept gate on the model page, then `hf auth login` again. |
|
||||
| Many `HEAD https://huggingface.co/...` lines on startup despite OFFLINE=1 | One of the two env flags is missing. Both `HF_HUB_OFFLINE=1` AND `TRANSFORMERS_OFFLINE=1` are required. |
|
||||
| `Loading weights: ...` takes minutes even with cache | The cache file is on a slow disk (NFS, HDD). Move it to local SSD via `HF_HOME`. |
|
||||
| Token leaked into logs | `hf auth login` only echoes your username, never the token. The token at `~/.cache/huggingface/token` is the only place it's stored. |
|
||||
|
||||
---
|
||||
|
||||
## What HF still sees in offline mode
|
||||
|
||||
**Nothing.** With both `HF_HUB_OFFLINE=1` and `TRANSFORMERS_OFFLINE=1`:
|
||||
|
||||
- `from_pretrained` reads only from `~/.cache/huggingface/hub/` — no API
|
||||
calls, no HEAD revalidation pings.
|
||||
- Telemetry is independently off via `HF_HUB_DISABLE_TELEMETRY=1` /
|
||||
`DO_NOT_TRACK=1`.
|
||||
- Inference itself never touches the network — your images, bboxes, and
|
||||
resulting masks stay on the box regardless of online/offline mode.
|
||||
|
||||
The only thing HF "knows" about you, ever, is "this account downloaded
|
||||
`facebook/sam3` once on date X" — recorded server-side at the moment
|
||||
`from_pretrained` first ran in Phase 1.
|
||||
137
sam-tool-tauri/docs/LINUX_VIDEO_PLAYBACK_CHECKLIST.md
Normal file
137
sam-tool-tauri/docs/LINUX_VIDEO_PLAYBACK_CHECKLIST.md
Normal file
@@ -0,0 +1,137 @@
|
||||
# Linux Video Playback Checklist
|
||||
|
||||
Use this checklist on Linux machines where video playback or frame stepping feels
|
||||
slow or sluggish.
|
||||
|
||||
## Baseline Packages
|
||||
|
||||
Install or update the common runtime and diagnostic tools:
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt full-upgrade
|
||||
sudo apt install ffmpeg ffprobe vainfo vdpauinfo mesa-utils vulkan-tools
|
||||
```
|
||||
|
||||
Make sure the user has access to GPU render devices:
|
||||
|
||||
```bash
|
||||
sudo usermod -aG render,video $USER
|
||||
```
|
||||
|
||||
Log out and back in after changing groups.
|
||||
|
||||
## Tauri / WebKit Runtime
|
||||
|
||||
For Debian or Ubuntu systems:
|
||||
|
||||
```bash
|
||||
sudo apt install libwebkit2gtk-4.1-0 libgtk-3-0 libayatana-appindicator3-1
|
||||
```
|
||||
|
||||
## Intel GPU
|
||||
|
||||
For newer Intel integrated GPUs:
|
||||
|
||||
```bash
|
||||
sudo apt install intel-media-va-driver intel-media-va-driver-non-free va-driver-all
|
||||
```
|
||||
|
||||
If VAAPI fails, try launching with:
|
||||
|
||||
```bash
|
||||
LIBVA_DRIVER_NAME=iHD ./sam-tool
|
||||
```
|
||||
|
||||
For older Intel GPUs:
|
||||
|
||||
```bash
|
||||
sudo apt install i965-va-driver
|
||||
LIBVA_DRIVER_NAME=i965 ./sam-tool
|
||||
```
|
||||
|
||||
## AMD GPU
|
||||
|
||||
Use Mesa VAAPI / VDPAU drivers:
|
||||
|
||||
```bash
|
||||
sudo apt install mesa-va-drivers mesa-vdpau-drivers va-driver-all
|
||||
```
|
||||
|
||||
In the app, prefer `VAAPI`.
|
||||
|
||||
## NVIDIA GPU
|
||||
|
||||
Use the proprietary NVIDIA driver, not Nouveau, for best decode support:
|
||||
|
||||
```bash
|
||||
sudo ubuntu-drivers autoinstall
|
||||
sudo reboot
|
||||
```
|
||||
|
||||
After reboot:
|
||||
|
||||
```bash
|
||||
nvidia-smi
|
||||
ffmpeg -hide_banner -hwaccels
|
||||
```
|
||||
|
||||
In the app, prefer `CUDA`. Try `VDPAU` only as a fallback.
|
||||
|
||||
## GStreamer / WebKit Video Stack
|
||||
|
||||
These packages are useful for WebKit video support, especially if native
|
||||
`<video>` playback is used:
|
||||
|
||||
```bash
|
||||
sudo apt install \
|
||||
gstreamer1.0-libav \
|
||||
gstreamer1.0-plugins-base \
|
||||
gstreamer1.0-plugins-good \
|
||||
gstreamer1.0-plugins-bad \
|
||||
gstreamer1.0-plugins-ugly \
|
||||
gstreamer1.0-vaapi
|
||||
```
|
||||
|
||||
## Verification Commands
|
||||
|
||||
Check available acceleration support:
|
||||
|
||||
```bash
|
||||
ffmpeg -hide_banner -hwaccels
|
||||
vainfo
|
||||
vdpauinfo
|
||||
ls -l /dev/dri/renderD*
|
||||
```
|
||||
|
||||
Test VAAPI:
|
||||
|
||||
```bash
|
||||
ffmpeg -v verbose -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -i input.mp4 -f null -
|
||||
```
|
||||
|
||||
Test NVIDIA CUDA:
|
||||
|
||||
```bash
|
||||
ffmpeg -v verbose -hwaccel cuda -i input.mp4 -f null -
|
||||
```
|
||||
|
||||
## App Settings
|
||||
|
||||
Recommended hardware decode setting:
|
||||
|
||||
- Intel / AMD: `VAAPI`
|
||||
- NVIDIA: `CUDA`
|
||||
- Unknown or mixed systems: `auto`
|
||||
- If unstable: `software`
|
||||
|
||||
## WebKit Workaround
|
||||
|
||||
On machines where WebKit or the Intel iGPU compositor is the bottleneck, try:
|
||||
|
||||
```bash
|
||||
WEBKIT_DISABLE_COMPOSITING_MODE=1 ./sam-tool
|
||||
```
|
||||
|
||||
This disables WebKit accelerated compositing and can improve playback on some
|
||||
Linux systems.
|
||||
164
sam-tool-tauri/docs/POLYGON_SMOOTHING.md
Normal file
164
sam-tool-tauri/docs/POLYGON_SMOOTHING.md
Normal file
@@ -0,0 +1,164 @@
|
||||
# Tuning polygon output (point count + edge smoothness)
|
||||
|
||||
When SAM returns a mask, the backend converts it to a polygon before sending
|
||||
it back to the Tauri annotation tool. Two things you can tune:
|
||||
|
||||
1. **Point count** — how many vertices the polygon has. Each vertex is
|
||||
draggable in the UI, so too many makes editing painful.
|
||||
2. **Edge smoothness** — how the polygon traces curved edges (smooth curves
|
||||
vs. pixel staircases).
|
||||
|
||||
Both are controlled in one file:
|
||||
[`sam2-backend/adapters/utils.py`](../sam2-backend/adapters/utils.py).
|
||||
|
||||
---
|
||||
|
||||
## Where it's called
|
||||
|
||||
Both adapters do the same thing in their predict path:
|
||||
|
||||
```python
|
||||
# sam2-backend/adapters/sam2_adapter.py
|
||||
return PredictResult(
|
||||
polygon=_mask_to_polygon(masks_np[best]),
|
||||
score=best_score,
|
||||
)
|
||||
|
||||
# sam2-backend/adapters/sam3_adapter.py — same call site
|
||||
```
|
||||
|
||||
The defaults are `min_points=4`, `max_points=10`, `smooth=True`. You can
|
||||
override per-call:
|
||||
|
||||
```python
|
||||
polygon=_mask_to_polygon(masks_np[best], min_points=3, max_points=15)
|
||||
```
|
||||
|
||||
…or change the defaults in `utils.py::mask_to_polygon` and they'll apply
|
||||
to every model.
|
||||
|
||||
---
|
||||
|
||||
## Knobs
|
||||
|
||||
### 1. Point count band — `min_points` / `max_points`
|
||||
|
||||
The simplifier uses Douglas–Peucker (`cv2.approxPolyDP`) with a binary
|
||||
search over `epsilon` to land in `[min_points, max_points]`. Within that
|
||||
band it prefers **more** detail (i.e. closer to `max_points`).
|
||||
|
||||
| Band | Use it when… |
|
||||
|------------|-----------------------------------------------------------------------------|
|
||||
| `4..10` | Default. Comfortable to drag, captures common shapes. |
|
||||
| `4..15` | Curvy / organic objects (animals, foliage). 15 vertices follow a curve well. |
|
||||
| `3..6` | Very simple shapes (rectangles, traffic signs). Fewer drag handles. |
|
||||
| `6..20` | High-fidelity work, you don't mind dragging more. |
|
||||
|
||||
**Avoid `max_points > 25`** — past that, the binary search needs many
|
||||
iterations and the marginal benefit per vertex drops sharply. If you need
|
||||
that many points, the bottleneck is mask quality, not simplification.
|
||||
|
||||
### 2. Mask smoothing — `smooth=True | False`
|
||||
|
||||
Before extracting the contour, the helper optionally runs:
|
||||
|
||||
1. **Morphological close + open** (ellipse kernel) — fills single-pixel
|
||||
notches and removes pepper noise.
|
||||
2. **Light Gaussian blur + re-threshold** — sub-pixel edge smoothing so
|
||||
curves don't trace as staircases.
|
||||
|
||||
Pass `smooth=False` to skip both steps and trace the raw mask exactly. Use
|
||||
this when:
|
||||
- The mask has fine features that smoothing erases (thin antennae, sharp
|
||||
90° corners that get rounded).
|
||||
- You're A/B testing whether smoothing is helping or hurting.
|
||||
|
||||
```python
|
||||
polygon=_mask_to_polygon(masks_np[best], smooth=False)
|
||||
```
|
||||
|
||||
### 3. Smoothing kernel size — `_smooth_mask`
|
||||
|
||||
Inside `_smooth_mask` (called by `mask_to_polygon` when `smooth=True`),
|
||||
kernel size scales with the **blob's bounding box diagonal**:
|
||||
|
||||
```python
|
||||
k = int(round(blob_diag * 0.015)) # ← morphology kernel multiplier
|
||||
k = max(3, min(k | 1, 11)) # odd, clamped 3..11
|
||||
|
||||
blur_k = max(3, min(k | 1, 9)) # ← Gaussian blur kernel
|
||||
```
|
||||
|
||||
**The `0.015` multiplier is the main dial.**
|
||||
|
||||
| Multiplier | Effect |
|
||||
|------------|-----------------------------------------------------------------|
|
||||
| `0.008` | Very gentle. Preserves thin features. Use if smoothing dilates the polygon. |
|
||||
| `0.015` | Default. Good balance for typical objects. |
|
||||
| `0.025` | Aggressive. Use only on visibly noisy masks; will round corners. |
|
||||
|
||||
The 11-pixel hard cap means giant masks don't get insanely smoothed.
|
||||
|
||||
### 4. Drop the Gaussian, keep morphology
|
||||
|
||||
If hard 90° corners get rounded but you still want morphology to clean up
|
||||
notches, delete just the Gaussian step in `_smooth_mask`:
|
||||
|
||||
```python
|
||||
# remove these two lines
|
||||
m = cv2.GaussianBlur(m, (blur_k, blur_k), 0)
|
||||
_, m = cv2.threshold(m, 127, 255, cv2.THRESH_BINARY)
|
||||
```
|
||||
|
||||
This keeps close+open (which is mostly noise removal) and skips the
|
||||
sub-pixel curve smoothing.
|
||||
|
||||
---
|
||||
|
||||
## Diagnostic recipe — find the right knob in 3 minutes
|
||||
|
||||
1. **Draw a representative bbox in the Tauri app.** Screenshot the result.
|
||||
2. **Set `smooth=False`** in the adapter, restart, draw the same bbox.
|
||||
- If polygon is *better* (less bloat, hugs edges) → smoothing is over-aggressive. Lower the kernel multiplier or delete the Gaussian step.
|
||||
- If polygon is *worse* (jagged stairs) → smoothing was helping. Move on to step 3.
|
||||
3. **Bump `max_points` to 15 or 20**, restart.
|
||||
- If polygon is *better* → simplification was too aggressive. Land on `max_points=12..15` as your new default.
|
||||
- If polygon is the same → the mask itself is the bottleneck (see "If the raw mask is bad" below).
|
||||
4. **Restore `smooth=True`** and try a model swap (SAM 2.1 large ↔ SAM 3 tracker) — the underlying mask differs noticeably between models.
|
||||
|
||||
---
|
||||
|
||||
## If the raw mask is bad
|
||||
|
||||
If after all the above the polygon still looks wrong, the input to the
|
||||
simplifier is bad — fix that, not the polygon code. Common moves:
|
||||
|
||||
- **Pad the bbox before sending** to SAM (Meta's reference scripts use
|
||||
`pad=10` pixels). SAM gives crisper masks with a small box margin.
|
||||
- **Try a larger SAM model** (e.g. switch the dropdown from `SAM 2.1
|
||||
Hiera Small` to `SAM 2.1 Hiera Large`).
|
||||
- **Tighten the bbox** around the actual object — SAM tends to fill the
|
||||
box, so a generous box gets a bloated mask.
|
||||
|
||||
---
|
||||
|
||||
## Quick reference — every dial in one block
|
||||
|
||||
```python
|
||||
# sam2-backend/adapters/utils.py
|
||||
|
||||
mask_to_polygon(
|
||||
mask,
|
||||
min_points=4, # never below this
|
||||
max_points=10, # never above this; fewer = easier to drag
|
||||
smooth=True, # turn off to trace raw mask
|
||||
)
|
||||
|
||||
# Inside _smooth_mask:
|
||||
k = int(round(blob_diag * 0.015)) # ← MAIN DIAL
|
||||
# smaller = preserve detail
|
||||
# bigger = more rounded
|
||||
```
|
||||
|
||||
If you change defaults, restart the backend (`./run.sh`). No frontend
|
||||
restart needed — the polygon arrives over the existing `/predict` call.
|
||||
@@ -2,7 +2,7 @@ use crate::video::VideoInfo;
|
||||
use std::collections::{HashMap, VecDeque};
|
||||
use std::io::{BufRead, BufReader};
|
||||
use std::path::PathBuf;
|
||||
use std::process::{Child, ChildStdout, Command, Stdio};
|
||||
use std::process::{Child, ChildStderr, ChildStdout, Command, Stdio};
|
||||
use std::sync::{Arc, Condvar, Mutex};
|
||||
use std::thread::{self, JoinHandle};
|
||||
use std::time::{Duration, Instant};
|
||||
@@ -14,6 +14,10 @@ const READ_BUF_BYTES: usize = 256 * 1024;
|
||||
/// If the requested frame is this far ahead of the worker, kill + reseek.
|
||||
const SEEK_AHEAD_THRESHOLD: u64 = 60;
|
||||
const DECODE_TIMEOUT: Duration = Duration::from_secs(15);
|
||||
/// Keep the last N stderr lines from ffmpeg so error messages can include
|
||||
/// the real reason (e.g. "vaapi: failed to load driver") instead of just
|
||||
/// "pipe closed before frame completed".
|
||||
const STDERR_TAIL_LINES: usize = 32;
|
||||
|
||||
struct FrameCache {
|
||||
map: HashMap<u64, Vec<u8>>,
|
||||
@@ -89,20 +93,32 @@ struct SharedState {
|
||||
struct Shared {
|
||||
state: Mutex<SharedState>,
|
||||
cond: Condvar,
|
||||
/// Ring buffer of recent ffmpeg stderr lines, filled by a sibling drain
|
||||
/// thread. Cleared at each new seek.
|
||||
stderr_tail: Mutex<VecDeque<String>>,
|
||||
}
|
||||
|
||||
pub struct DecoderSession {
|
||||
path: PathBuf,
|
||||
info: VideoInfo,
|
||||
/// ffmpeg `-hwaccel` argument. None / empty = software decode. Cleared
|
||||
/// on any decode failure so the next attempt uses software — we'd
|
||||
/// rather degrade than keep retrying a flaky GPU path.
|
||||
hwaccel: Option<String>,
|
||||
shared: Arc<Shared>,
|
||||
worker: Option<JoinHandle<()>>,
|
||||
/// Sibling thread draining ffmpeg's stderr into `shared.stderr_tail`.
|
||||
/// Joined in `stop_worker` so the tail doesn't get clobbered by a
|
||||
/// straggling drain across seeks.
|
||||
stderr_drain: Option<JoinHandle<()>>,
|
||||
}
|
||||
|
||||
impl DecoderSession {
|
||||
pub fn open(path: PathBuf, info: VideoInfo) -> Self {
|
||||
pub fn open(path: PathBuf, info: VideoInfo, hwaccel: Option<String>) -> Self {
|
||||
Self {
|
||||
path,
|
||||
info,
|
||||
hwaccel: hwaccel.filter(|s| !s.is_empty()),
|
||||
shared: Arc::new(Shared {
|
||||
state: Mutex::new(SharedState {
|
||||
child: None,
|
||||
@@ -115,8 +131,10 @@ impl DecoderSession {
|
||||
shutdown: false,
|
||||
}),
|
||||
cond: Condvar::new(),
|
||||
stderr_tail: Mutex::new(VecDeque::new()),
|
||||
}),
|
||||
worker: None,
|
||||
stderr_drain: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -132,6 +150,28 @@ impl DecoderSession {
|
||||
}
|
||||
|
||||
pub fn decode_at(&mut self, idx: u64) -> Result<Vec<u8>, String> {
|
||||
match self.decode_at_inner(idx) {
|
||||
Ok(bytes) => Ok(bytes),
|
||||
Err(e) => {
|
||||
// Auto-fallback: any decode failure while hwaccel is on
|
||||
// retries once with software. Covers both first-frame init
|
||||
// failures and later seek/codec-edge failures (e.g. a GPU
|
||||
// path that worked on h264 but chokes on an HEVC segment).
|
||||
// Once cleared, subsequent failures propagate.
|
||||
if let Some(prev) = self.hwaccel.take() {
|
||||
eprintln!(
|
||||
"[decoder] hwaccel '{prev}' failed ({e}); falling back to software"
|
||||
);
|
||||
self.stop_worker();
|
||||
self.decode_at_inner(idx)
|
||||
} else {
|
||||
Err(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn decode_at_inner(&mut self, idx: u64) -> Result<Vec<u8>, String> {
|
||||
if idx >= self.info.total_frames {
|
||||
return Err(format!(
|
||||
"frame {} out of range (total {})",
|
||||
@@ -194,7 +234,8 @@ impl DecoderSession {
|
||||
fn seek_and_start_worker(&mut self, idx: u64) -> Result<(), String> {
|
||||
self.stop_worker();
|
||||
|
||||
let (child, stdout) = spawn_ffmpeg(&self.path, &self.info, idx)?;
|
||||
let (child, stdout, stderr) =
|
||||
spawn_ffmpeg(&self.path, &self.info, idx, self.hwaccel.as_deref())?;
|
||||
|
||||
{
|
||||
let mut st = self.shared.state.lock().unwrap();
|
||||
@@ -205,8 +246,24 @@ impl DecoderSession {
|
||||
st.error = None;
|
||||
st.shutdown = false;
|
||||
}
|
||||
self.shared.stderr_tail.lock().unwrap().clear();
|
||||
self.shared.cond.notify_all();
|
||||
|
||||
// Drain ffmpeg's stderr into a ring buffer so error messages can
|
||||
// include the real reason. We must drain even on success — leaving
|
||||
// stderr unread would let the pipe buffer fill and stall ffmpeg.
|
||||
let drain_shared = self.shared.clone();
|
||||
self.stderr_drain = Some(thread::spawn(move || {
|
||||
let reader = BufReader::new(stderr);
|
||||
for line in reader.lines().map_while(Result::ok) {
|
||||
let mut tail = drain_shared.stderr_tail.lock().unwrap();
|
||||
if tail.len() >= STDERR_TAIL_LINES {
|
||||
tail.pop_front();
|
||||
}
|
||||
tail.push_back(line);
|
||||
}
|
||||
}));
|
||||
|
||||
let shared = self.shared.clone();
|
||||
let handle = thread::spawn(move || {
|
||||
worker_loop(stdout, shared);
|
||||
@@ -232,6 +289,12 @@ impl DecoderSession {
|
||||
if let Some(handle) = self.worker.take() {
|
||||
let _ = handle.join();
|
||||
}
|
||||
// Killing the child closed stderr, so the drain thread sees EOF and
|
||||
// exits — joining it here prevents a straggling write into the new
|
||||
// session's stderr_tail after the next seek clears it.
|
||||
if let Some(handle) = self.stderr_drain.take() {
|
||||
let _ = handle.join();
|
||||
}
|
||||
|
||||
let mut st = self.shared.state.lock().unwrap();
|
||||
st.shutdown = false;
|
||||
@@ -246,15 +309,55 @@ impl Drop for DecoderSession {
|
||||
}
|
||||
}
|
||||
|
||||
/// Pick a VAAPI render node. `SAM_TOOL_VAAPI_DEVICE` overrides; otherwise we
|
||||
/// probe `/dev/dri/renderD128..199` for the first that exists. Multi-GPU
|
||||
/// systems can end up with the integrated card on D128 and a discrete card
|
||||
/// on D129+ — the env var is the escape hatch when the auto pick is wrong.
|
||||
fn vaapi_device() -> String {
|
||||
if let Ok(p) = std::env::var("SAM_TOOL_VAAPI_DEVICE") {
|
||||
if !p.is_empty() {
|
||||
return p;
|
||||
}
|
||||
}
|
||||
for n in 128..200u32 {
|
||||
let p = format!("/dev/dri/renderD{n}");
|
||||
if std::path::Path::new(&p).exists() {
|
||||
return p;
|
||||
}
|
||||
}
|
||||
"/dev/dri/renderD128".to_string()
|
||||
}
|
||||
|
||||
fn spawn_ffmpeg(
|
||||
path: &PathBuf,
|
||||
info: &VideoInfo,
|
||||
idx: u64,
|
||||
) -> Result<(Child, BufReader<ChildStdout>), String> {
|
||||
hwaccel: Option<&str>,
|
||||
) -> Result<(Child, BufReader<ChildStdout>, ChildStderr), String> {
|
||||
let path_str = path.to_str().ok_or_else(|| "non-utf8 path".to_string())?;
|
||||
let t = idx as f64 / info.fps;
|
||||
let mut cmd = Command::new("ffmpeg");
|
||||
cmd.args(["-v", "error", "-threads", "0"]);
|
||||
// -hwaccel must come BEFORE -i. Output goes to CPU memory by default
|
||||
// (no -hwaccel_output_format), so the downstream mjpeg encoder works
|
||||
// unchanged. If the GPU init fails, ffmpeg will exit nonzero and the
|
||||
// worker thread surfaces the error via shared.error.
|
||||
match hwaccel {
|
||||
Some("auto") => {
|
||||
cmd.args(["-hwaccel", "auto"]);
|
||||
}
|
||||
Some("vaapi") => {
|
||||
let dev = vaapi_device();
|
||||
cmd.args(["-hwaccel", "vaapi", "-vaapi_device", &dev]);
|
||||
}
|
||||
Some("cuda") => {
|
||||
cmd.args(["-hwaccel", "cuda"]);
|
||||
}
|
||||
Some("vdpau") => {
|
||||
cmd.args(["-hwaccel", "vdpau"]);
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
if t > 1.0 {
|
||||
cmd.args(["-ss", &format!("{:.3}", t - 1.0)]);
|
||||
cmd.args(["-i", path_str]);
|
||||
@@ -273,13 +376,21 @@ fn spawn_ffmpeg(
|
||||
"mjpeg",
|
||||
"pipe:1",
|
||||
]);
|
||||
cmd.stdout(Stdio::piped()).stderr(Stdio::null());
|
||||
cmd.stdout(Stdio::piped()).stderr(Stdio::piped());
|
||||
let mut child = cmd.spawn().map_err(|e| format!("ffmpeg spawn: {e}"))?;
|
||||
let stdout = child
|
||||
.stdout
|
||||
.take()
|
||||
.ok_or_else(|| "ffmpeg stdout unavailable".to_string())?;
|
||||
Ok((child, BufReader::with_capacity(READ_BUF_BYTES, stdout)))
|
||||
let stderr = child
|
||||
.stderr
|
||||
.take()
|
||||
.ok_or_else(|| "ffmpeg stderr unavailable".to_string())?;
|
||||
Ok((
|
||||
child,
|
||||
BufReader::with_capacity(READ_BUF_BYTES, stdout),
|
||||
stderr,
|
||||
))
|
||||
}
|
||||
|
||||
fn worker_loop(mut stdout: BufReader<ChildStdout>, shared: Arc<Shared>) {
|
||||
@@ -315,8 +426,19 @@ fn worker_loop(mut stdout: BufReader<ChildStdout>, shared: Arc<Shared>) {
|
||||
shared.cond.notify_all();
|
||||
}
|
||||
Err(e) => {
|
||||
// ffmpeg closes stderr right alongside stdout; on real
|
||||
// failures the drain thread may not have flushed yet, so
|
||||
// we wait briefly. On a clean exit there's nothing to
|
||||
// flush — collect_stderr_tail short-circuits via the
|
||||
// child's exit code so end-of-stream EOF isn't penalised.
|
||||
let tail = collect_stderr_tail(&shared, Duration::from_millis(150));
|
||||
let mut st = shared.state.lock().unwrap();
|
||||
if e.contains("pipe closed") || e.contains("EOF") {
|
||||
if !tail.is_empty() {
|
||||
// With `-v error`, anything on stderr means ffmpeg
|
||||
// actually failed (vs a clean exit) — surface it as a
|
||||
// real error so decode_at can fall back / report.
|
||||
st.error = Some(format!("{e}: ffmpeg: {tail}"));
|
||||
} else if e.contains("pipe closed") || e.contains("EOF") {
|
||||
st.eof = true;
|
||||
} else {
|
||||
st.error = Some(e);
|
||||
@@ -328,6 +450,50 @@ fn worker_loop(mut stdout: BufReader<ChildStdout>, shared: Arc<Shared>) {
|
||||
}
|
||||
}
|
||||
|
||||
fn read_tail_now(shared: &Arc<Shared>) -> Option<String> {
|
||||
let tail = shared.stderr_tail.lock().unwrap();
|
||||
if tail.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(tail.iter().cloned().collect::<Vec<_>>().join(" | "))
|
||||
}
|
||||
}
|
||||
|
||||
fn collect_stderr_tail(shared: &Arc<Shared>, max_wait: Duration) -> String {
|
||||
// Fast path: drain already flushed — common when ffmpeg writes its
|
||||
// error before exiting and stdout EOF arrives after the drain.
|
||||
if let Some(tail) = read_tail_now(shared) {
|
||||
return tail;
|
||||
}
|
||||
// Skip the poll on clean ffmpeg exit (exit code 0): natural end-of-
|
||||
// stream has nothing on stderr (we run with `-v error`), and 150ms ×
|
||||
// every EOF seek near the final frames adds up.
|
||||
let exited_cleanly = {
|
||||
let mut st = shared.state.lock().unwrap();
|
||||
st.child
|
||||
.as_mut()
|
||||
.and_then(|c| c.try_wait().ok().flatten())
|
||||
.map(|status| status.success())
|
||||
.unwrap_or(false)
|
||||
};
|
||||
if exited_cleanly {
|
||||
return String::new();
|
||||
}
|
||||
// Slow path: ffmpeg exited non-zero (or hasn't exited yet) and the
|
||||
// drain thread hasn't flushed. Poll briefly so the error message
|
||||
// carries the real reason.
|
||||
let deadline = Instant::now() + max_wait;
|
||||
loop {
|
||||
if let Some(tail) = read_tail_now(shared) {
|
||||
return tail;
|
||||
}
|
||||
if Instant::now() >= deadline {
|
||||
return String::new();
|
||||
}
|
||||
thread::sleep(Duration::from_millis(10));
|
||||
}
|
||||
}
|
||||
|
||||
fn read_jpeg_bytes(reader: &mut BufReader<ChildStdout>) -> Result<Vec<u8>, String> {
|
||||
let mut out = Vec::with_capacity(64 * 1024);
|
||||
let mut last = 0u8;
|
||||
|
||||
@@ -33,9 +33,14 @@ fn open_video(
|
||||
return Err(format!("file not found: {}", p.display()));
|
||||
}
|
||||
let info = video::probe(&p)?;
|
||||
let decoder = DecoderSession::open(p, info.clone());
|
||||
if let Ok(settings) = settings::load(&app) {
|
||||
decoder.set_cache_capacity(settings.cache_capacity);
|
||||
let settings = settings::load(&app).ok();
|
||||
let hwaccel = settings
|
||||
.as_ref()
|
||||
.map(|s| s.hwaccel.clone())
|
||||
.filter(|s| !s.is_empty());
|
||||
let decoder = DecoderSession::open(p, info.clone(), hwaccel);
|
||||
if let Some(s) = settings.as_ref() {
|
||||
decoder.set_cache_capacity(s.cache_capacity);
|
||||
}
|
||||
let mut guard = state.decoder.lock().map_err(|e| format!("lock: {e}"))?;
|
||||
*guard = Some(decoder);
|
||||
@@ -117,23 +122,27 @@ fn bulk_extract(
|
||||
out_dir: String,
|
||||
shapes_by_frame: HashMap<String, Vec<coco::Shape>>,
|
||||
author: String,
|
||||
frame_skip: Option<u64>,
|
||||
) -> Result<u64, String> {
|
||||
if end < start {
|
||||
return Err("end must be >= start".into());
|
||||
}
|
||||
let step = frame_skip.unwrap_or(1).max(1);
|
||||
let mut guard = state.decoder.lock().map_err(|e| format!("lock: {e}"))?;
|
||||
let decoder = guard.as_mut().ok_or("no video loaded")?;
|
||||
let dir = PathBuf::from(&out_dir);
|
||||
let total = end - start + 1;
|
||||
let total = ((end - start) / step) + 1;
|
||||
let empty: Vec<coco::Shape> = Vec::new();
|
||||
let mut done: u64 = 0;
|
||||
for idx in start..=end {
|
||||
let mut idx = start;
|
||||
while idx <= end {
|
||||
let shapes = shapes_by_frame.get(&idx.to_string()).unwrap_or(&empty);
|
||||
extract::extract_one(decoder, idx, &dir, shapes, &author)?;
|
||||
done += 1;
|
||||
if done == 1 || done % 5 == 0 || done == total {
|
||||
let _ = app.emit("bulk-extract-progress", BulkProgress { done, total });
|
||||
}
|
||||
idx += step;
|
||||
}
|
||||
Ok(done)
|
||||
}
|
||||
|
||||
@@ -15,6 +15,15 @@ pub struct Settings {
|
||||
pub sam_model: Option<String>,
|
||||
#[serde(default = "default_cache_capacity")]
|
||||
pub cache_capacity: usize,
|
||||
/// ffmpeg `-hwaccel` argument. Empty string = software decode (default).
|
||||
/// Recognized: "auto", "vaapi", "cuda", "vdpau". Applied on next
|
||||
/// `open_video` or worker respawn.
|
||||
#[serde(default)]
|
||||
pub hwaccel: String,
|
||||
/// Decimation factor used by playback (advance by N frames per tick) and
|
||||
/// bulk extract (take every Nth frame in range). 1 = no decimation.
|
||||
#[serde(default = "default_frame_skip")]
|
||||
pub frame_skip: u64,
|
||||
}
|
||||
|
||||
fn default_sam_url() -> String {
|
||||
@@ -25,11 +34,18 @@ fn default_cache_capacity() -> usize {
|
||||
crate::decoder::DEFAULT_CACHE_CAPACITY
|
||||
}
|
||||
|
||||
fn default_frame_skip() -> u64 {
|
||||
1
|
||||
}
|
||||
|
||||
fn normalize(mut s: Settings) -> Settings {
|
||||
s.cache_capacity = s.cache_capacity.clamp(
|
||||
crate::decoder::MIN_CACHE_CAPACITY,
|
||||
crate::decoder::MAX_CACHE_CAPACITY,
|
||||
);
|
||||
if s.frame_skip < 1 {
|
||||
s.frame_skip = 1;
|
||||
}
|
||||
s
|
||||
}
|
||||
|
||||
@@ -40,6 +56,8 @@ impl Default for Settings {
|
||||
sam_enabled: false,
|
||||
sam_model: None,
|
||||
cache_capacity: default_cache_capacity(),
|
||||
hwaccel: String::new(),
|
||||
frame_skip: default_frame_skip(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -75,7 +75,7 @@ export default function App() {
|
||||
<main>
|
||||
{mode === "extract" && (
|
||||
<ErrorBoundary label="Extract mode">
|
||||
<ExtractMode username={username} />
|
||||
<ExtractMode username={username} settings={settings} />
|
||||
</ErrorBoundary>
|
||||
)}
|
||||
{mode === "annotate" && (
|
||||
|
||||
@@ -16,7 +16,10 @@ export const FrameCanvas = forwardRef<HTMLCanvasElement, Props>(function FrameCa
|
||||
className="canvas-frame"
|
||||
style={{ aspectRatio: `${video.width} / ${video.height}` }}
|
||||
>
|
||||
<canvas ref={ref} width={video.width} height={video.height} />
|
||||
{/* Backing-store width/height are set imperatively in drawBitmap so we
|
||||
can cap it at the displayed (CSS) size × DPR — drawing 4K every frame
|
||||
into a 1080p window is the main playback cost on slow systems. */}
|
||||
<canvas ref={ref} />
|
||||
{extracted && (
|
||||
<div className="frame-extracted-banner">EXTRACTED</div>
|
||||
)}
|
||||
|
||||
@@ -257,6 +257,50 @@ export function SettingsModal({ open, onClose, onSaved }: Props) {
|
||||
</div>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div className="settings-row">
|
||||
<label>
|
||||
<div className="settings-label">
|
||||
ffmpeg hardware decode
|
||||
<span className="dim"> · applies on next video open</span>
|
||||
</div>
|
||||
<select
|
||||
value={settings.hwaccel}
|
||||
onChange={(e) =>
|
||||
setSettings({ ...settings, hwaccel: e.target.value })
|
||||
}
|
||||
>
|
||||
<option value="">software (safe default)</option>
|
||||
<option value="auto">auto (let ffmpeg pick)</option>
|
||||
<option value="vaapi">vaapi — Intel / AMD Linux</option>
|
||||
<option value="cuda">cuda — NVIDIA, modern path</option>
|
||||
<option value="vdpau">vdpau — NVIDIA / Linux, legacy</option>
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div className="settings-row">
|
||||
<label>
|
||||
<div className="settings-label">
|
||||
Frame skip
|
||||
<span className="dim">
|
||||
{" "}
|
||||
· 1 = no skip; N = every Nth frame in playback & bulk extract
|
||||
</span>
|
||||
</div>
|
||||
<input
|
||||
type="number"
|
||||
min={1}
|
||||
max={60}
|
||||
step={1}
|
||||
value={settings.frame_skip}
|
||||
onChange={(e) => {
|
||||
const n = Math.max(1, Math.floor(Number(e.target.value) || 1));
|
||||
setSettings({ ...settings, frame_skip: n });
|
||||
}}
|
||||
/>
|
||||
</label>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
{error && <div className="picker-error">{error}</div>}
|
||||
|
||||
@@ -37,7 +37,8 @@ export const api = {
|
||||
end: number,
|
||||
outDir: string,
|
||||
shapesByFrame: Record<string, CocoShape[]>,
|
||||
author: string
|
||||
author: string,
|
||||
frameSkip?: number
|
||||
) =>
|
||||
invoke<number>("bulk_extract", {
|
||||
start,
|
||||
@@ -45,6 +46,7 @@ export const api = {
|
||||
outDir,
|
||||
shapesByFrame,
|
||||
author,
|
||||
frameSkip,
|
||||
}),
|
||||
bulkUndo: (start: number, end: number, outDir: string) =>
|
||||
invoke<number>("bulk_undo", { start, end, outDir }),
|
||||
|
||||
@@ -2,7 +2,7 @@ import { useCallback, useEffect, useMemo, useRef, useState } from "react";
|
||||
import { open as openDialog } from "@tauri-apps/plugin-dialog";
|
||||
import { listen } from "@tauri-apps/api/event";
|
||||
import { api } from "../ipc";
|
||||
import type { CocoShape, ImportedCoco, VideoInfo } from "../types";
|
||||
import type { CocoShape, ImportedCoco, Settings, VideoInfo } from "../types";
|
||||
import { FrameCanvas } from "../components/extract/FrameCanvas";
|
||||
import { drawOverlay } from "../components/extract/overlay";
|
||||
|
||||
@@ -11,6 +11,7 @@ const FRAME_BITMAP_CACHE_LIMIT = 24;
|
||||
|
||||
interface Props {
|
||||
username: string;
|
||||
settings: Settings | null;
|
||||
}
|
||||
|
||||
interface BulkProgress {
|
||||
@@ -19,7 +20,14 @@ interface BulkProgress {
|
||||
phase: "extract" | "undo";
|
||||
}
|
||||
|
||||
export function ExtractMode({ username }: Props) {
|
||||
export function ExtractMode({ username, settings }: Props) {
|
||||
// frame_skip ≥ 1 means: during playback advance by N frames per tick, and
|
||||
// bulk-extract takes every Nth frame across [start, end]. Kept in a ref so
|
||||
// the long-lived playback loop sees changes without re-running its effect.
|
||||
const frameSkipRef = useRef<number>(Math.max(1, settings?.frame_skip ?? 1));
|
||||
useEffect(() => {
|
||||
frameSkipRef.current = Math.max(1, settings?.frame_skip ?? 1);
|
||||
}, [settings?.frame_skip]);
|
||||
const [video, setVideo] = useState<VideoInfo | null>(null);
|
||||
const [frameIdx, setFrameIdx] = useState(0);
|
||||
const [playing, setPlaying] = useState(false);
|
||||
@@ -53,9 +61,6 @@ export function ExtractMode({ username }: Props) {
|
||||
const frameMapRef = useRef<Map<number, CocoShape[]>>(new Map());
|
||||
const overlayVisibleRef = useRef(overlayVisible);
|
||||
const annotatedFrames = useRef<Set<number>>(new Set());
|
||||
// Playback clock anchor — set when playback starts, rebased on seeks so a
|
||||
// slider drag during play doesn't get overwritten by the tick loop.
|
||||
const playAnchorRef = useRef<{ time: number; idx: number } | null>(null);
|
||||
|
||||
useEffect(() => { frameIdxRef.current = frameIdx; }, [frameIdx]);
|
||||
useEffect(() => {
|
||||
@@ -69,6 +74,12 @@ export function ExtractMode({ username }: Props) {
|
||||
const renderedIdx = useRef(-1);
|
||||
const decodingInFlight = useRef(false);
|
||||
const bitmapCacheRef = useRef<Map<number, ImageBitmap>>(new Map());
|
||||
// Last decode/render error message. Set in `pump` on failure, cleared on
|
||||
// any successful render, and observed by the playback loop so a mid-play
|
||||
// failure stops playback with a visible banner instead of leaving the UI
|
||||
// stuck in "playing" while the canvas freezes.
|
||||
const decodeErrorRef = useRef<string | null>(null);
|
||||
const [playbackError, setPlaybackError] = useState<string | null>(null);
|
||||
|
||||
const clearBitmapCache = useCallback(() => {
|
||||
for (const bitmap of bitmapCacheRef.current.values()) {
|
||||
@@ -105,31 +116,82 @@ export function ExtractMode({ username }: Props) {
|
||||
const drawBitmap = useCallback(
|
||||
(bitmap: ImageBitmap, atFrameIdx: number) => {
|
||||
const canvas = canvasRef.current;
|
||||
if (!canvas) return;
|
||||
if (!canvas || !video) return;
|
||||
const ctx = canvas.getContext("2d");
|
||||
if (!ctx) return;
|
||||
ctx.clearRect(0, 0, canvas.width, canvas.height);
|
||||
ctx.drawImage(bitmap, 0, 0, canvas.width, canvas.height);
|
||||
|
||||
// Backing store = displayed CSS rect × DPR, capped at video native
|
||||
// resolution (anything above is upsampling for no benefit). Halves
|
||||
// drawImage cost on 4K-into-1080p, which is the common slow-system
|
||||
// case. Coords below are in image-pixel space — the transform handles
|
||||
// the mapping to backing pixels, so overlay.ts is unchanged.
|
||||
const rect = canvas.getBoundingClientRect();
|
||||
const dpr = window.devicePixelRatio || 1;
|
||||
const targetW = Math.max(
|
||||
1,
|
||||
Math.min(video.width, Math.round(rect.width * dpr))
|
||||
);
|
||||
const targetH = Math.max(
|
||||
1,
|
||||
Math.min(video.height, Math.round(rect.height * dpr))
|
||||
);
|
||||
if (canvas.width !== targetW) canvas.width = targetW;
|
||||
if (canvas.height !== targetH) canvas.height = targetH;
|
||||
|
||||
ctx.setTransform(
|
||||
targetW / video.width,
|
||||
0,
|
||||
0,
|
||||
targetH / video.height,
|
||||
0,
|
||||
0
|
||||
);
|
||||
ctx.clearRect(0, 0, video.width, video.height);
|
||||
ctx.drawImage(bitmap, 0, 0, video.width, video.height);
|
||||
|
||||
if (overlayVisibleRef.current) {
|
||||
const shapes = frameMapRef.current.get(atFrameIdx);
|
||||
if (shapes && shapes.length > 0) {
|
||||
drawOverlay(ctx, shapes, canvas.width, canvas.height);
|
||||
drawOverlay(ctx, shapes, video.width, video.height);
|
||||
}
|
||||
}
|
||||
ctx.setTransform(1, 0, 0, 1, 0, 0);
|
||||
},
|
||||
[]
|
||||
[video]
|
||||
);
|
||||
|
||||
// Returns true if we drew, false if we were preempted by a newer request,
|
||||
// and throws if bitmap creation failed — pump uses this to decide whether
|
||||
// to mark the frame rendered or surface the error to the playback loop.
|
||||
const renderFrame = useCallback(
|
||||
async (bytes: ArrayBuffer, atFrameIdx: number) => {
|
||||
async (bytes: ArrayBuffer, atFrameIdx: number): Promise<boolean> => {
|
||||
// Diagnostic timing log: set `window.SAM_FRAME_DEBUG = true` in devtools
|
||||
// on a slow machine to see per-frame bitmap and draw timings — confirms
|
||||
// where the bottleneck lives (decode vs createImageBitmap vs drawImage).
|
||||
const debug = (window as unknown as { SAM_FRAME_DEBUG?: boolean })
|
||||
.SAM_FRAME_DEBUG === true;
|
||||
const t0 = debug ? performance.now() : 0;
|
||||
const blob = new Blob([bytes], { type: "image/jpeg" });
|
||||
let bitmap: ImageBitmap;
|
||||
try { bitmap = await createImageBitmap(blob); }
|
||||
catch (e) { console.error("createImageBitmap failed", e); return; }
|
||||
try {
|
||||
bitmap = await createImageBitmap(blob);
|
||||
} catch (e) {
|
||||
throw new Error(
|
||||
`createImageBitmap failed: ${e instanceof Error ? e.message : String(e)}`
|
||||
);
|
||||
}
|
||||
const t1 = debug ? performance.now() : 0;
|
||||
rememberBitmap(atFrameIdx, bitmap);
|
||||
if (requestedIdx.current !== atFrameIdx) return;
|
||||
if (requestedIdx.current !== atFrameIdx) return false;
|
||||
drawBitmap(bitmap, atFrameIdx);
|
||||
if (debug) {
|
||||
const t2 = performance.now();
|
||||
// eslint-disable-next-line no-console
|
||||
console.debug(
|
||||
`[SAM] f${atFrameIdx} bytes=${bytes.byteLength} bitmap=${(t1 - t0).toFixed(0)}ms draw=${(t2 - t1).toFixed(0)}ms`
|
||||
);
|
||||
}
|
||||
return true;
|
||||
},
|
||||
[drawBitmap, rememberBitmap]
|
||||
);
|
||||
@@ -149,11 +211,17 @@ export function ExtractMode({ username }: Props) {
|
||||
try {
|
||||
const bytes = await api.decodeFrame(want);
|
||||
if (requestedIdx.current !== want) continue;
|
||||
await renderFrame(bytes, want);
|
||||
renderedIdx.current = want;
|
||||
const drew = await renderFrame(bytes, want);
|
||||
if (drew) {
|
||||
renderedIdx.current = want;
|
||||
decodeErrorRef.current = null;
|
||||
}
|
||||
// !drew means renderFrame was preempted by a newer request; loop
|
||||
// continues and the new target gets a fresh decode.
|
||||
} catch (e) {
|
||||
if (requestedIdx.current !== want) continue;
|
||||
console.error("decode_frame failed", e);
|
||||
console.error("decode/render failed", e);
|
||||
decodeErrorRef.current = e instanceof Error ? e.message : String(e);
|
||||
break;
|
||||
}
|
||||
}
|
||||
@@ -181,47 +249,114 @@ export function ExtractMode({ username }: Props) {
|
||||
|
||||
useEffect(() => clearBitmapCache, [clearBitmapCache]);
|
||||
|
||||
// Redraw on canvas resize: backing-store is sized from CSS rect × DPR, so
|
||||
// a window resize/maximize would otherwise leave the old backing dimensions
|
||||
// until the next decode landed. Cheap — just re-runs drawBitmap on the
|
||||
// already-cached frame.
|
||||
useEffect(() => {
|
||||
const canvas = canvasRef.current;
|
||||
if (!canvas || !video) return;
|
||||
let scheduled = false;
|
||||
const ro = new ResizeObserver(() => {
|
||||
if (scheduled) return;
|
||||
scheduled = true;
|
||||
requestAnimationFrame(() => {
|
||||
scheduled = false;
|
||||
const idx = frameIdxRef.current;
|
||||
const cached = getCachedBitmap(idx);
|
||||
if (cached) drawBitmap(cached, idx);
|
||||
else { renderedIdx.current = -1; pump(); }
|
||||
});
|
||||
});
|
||||
ro.observe(canvas);
|
||||
return () => ro.disconnect();
|
||||
}, [video, drawBitmap, getCachedBitmap, pump]);
|
||||
|
||||
// --- playback loop ---------------------------------------------------
|
||||
// Render-paced: advance ONLY after the previous frame actually drew. On a
|
||||
// fast box this hits target fps; on a slow box it plays smoothly at the
|
||||
// achievable decode/render rate instead of dropping 9-of-10 frames at the
|
||||
// canvas while the wall-clock UI counter races ahead.
|
||||
useEffect(() => {
|
||||
if (!playing || !video) return;
|
||||
playAnchorRef.current = {
|
||||
time: performance.now(),
|
||||
idx: frameIdxRef.current,
|
||||
};
|
||||
let cancelled = false;
|
||||
let nextDueAt = performance.now();
|
||||
|
||||
let raf = 0;
|
||||
const tick = () => {
|
||||
const anchor = playAnchorRef.current;
|
||||
if (!anchor) return;
|
||||
const elapsed = (performance.now() - anchor.time) / 1000;
|
||||
const want = Math.min(
|
||||
video.total_frames - 1,
|
||||
anchor.idx + Math.floor(elapsed * video.fps * speed)
|
||||
);
|
||||
// Only trigger the review-mode auto-pause once we've actually advanced
|
||||
// past the resume point. Otherwise pressing play while parked on an
|
||||
// annotated frame would re-pause on the very first tick, making the
|
||||
// play button look broken — the user would have to step a frame
|
||||
// forward manually before play could "stick".
|
||||
if (
|
||||
reviewMode &&
|
||||
want > anchor.idx &&
|
||||
annotatedFrames.current.has(want)
|
||||
) {
|
||||
setFrameIdx(want);
|
||||
setPlaying(false);
|
||||
return;
|
||||
const waitForFrame = (target: number) =>
|
||||
new Promise<void>((resolve) => {
|
||||
const check = () => {
|
||||
if (
|
||||
cancelled ||
|
||||
renderedIdx.current === target ||
|
||||
decodeErrorRef.current !== null
|
||||
) {
|
||||
resolve();
|
||||
return;
|
||||
}
|
||||
requestAnimationFrame(check);
|
||||
};
|
||||
check();
|
||||
});
|
||||
|
||||
const loop = async () => {
|
||||
while (!cancelled && playing) {
|
||||
const skip = Math.max(1, frameSkipRef.current | 0);
|
||||
// Real-time pacing regardless of skip: skip controls density,
|
||||
// `speed` controls speed. Without the skip factor here, skip=3 would
|
||||
// play at 3× wall-clock — which is what the speed slider is for.
|
||||
const interval = (skip * 1000) / (video.fps * speed);
|
||||
const current = frameIdxRef.current;
|
||||
let target = Math.min(video.total_frames - 1, current + skip);
|
||||
if (target <= current) {
|
||||
setPlaying(false);
|
||||
return;
|
||||
}
|
||||
// Review-mode scan: with skip > 1 a naive +skip jump can leap past
|
||||
// annotations between current+1 and current+skip. Walk that window
|
||||
// and clamp `target` down to the first annotated frame.
|
||||
if (reviewMode) {
|
||||
for (let i = current + 1; i <= target; i++) {
|
||||
if (annotatedFrames.current.has(i)) {
|
||||
target = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
requestedIdx.current = target;
|
||||
pump();
|
||||
await waitForFrame(target);
|
||||
if (cancelled) return;
|
||||
// Decode failed mid-play: stop and surface so the UI doesn't claim to
|
||||
// be playing while the canvas is frozen on the last good frame.
|
||||
if (decodeErrorRef.current) {
|
||||
setPlaybackError(decodeErrorRef.current);
|
||||
setPlaying(false);
|
||||
return;
|
||||
}
|
||||
setFrameIdx(target);
|
||||
// Pause AFTER rendering so the user actually sees the annotated frame.
|
||||
if (reviewMode && annotatedFrames.current.has(target)) {
|
||||
setPlaying(false);
|
||||
return;
|
||||
}
|
||||
|
||||
const now = performance.now();
|
||||
const due = nextDueAt + interval;
|
||||
if (due > now) {
|
||||
await new Promise((r) => setTimeout(r, due - now));
|
||||
nextDueAt = due;
|
||||
} else {
|
||||
// Behind real-time — don't accumulate debt and then "catch up" by
|
||||
// skipping frames; just reset the clock to now.
|
||||
nextDueAt = now;
|
||||
}
|
||||
}
|
||||
setFrameIdx(want);
|
||||
if (want >= video.total_frames - 1) { setPlaying(false); return; }
|
||||
raf = requestAnimationFrame(tick);
|
||||
};
|
||||
raf = requestAnimationFrame(tick);
|
||||
loop();
|
||||
return () => {
|
||||
cancelAnimationFrame(raf);
|
||||
playAnchorRef.current = null;
|
||||
cancelled = true;
|
||||
};
|
||||
}, [playing, video, speed, reviewMode]);
|
||||
}, [playing, video, speed, reviewMode, pump]);
|
||||
|
||||
// --- navigation helpers ----------------------------------------------
|
||||
const seekToFrame = useCallback((idx: number) => {
|
||||
@@ -237,11 +372,6 @@ export function ExtractMode({ username }: Props) {
|
||||
pump();
|
||||
}
|
||||
setFrameIdx((prev) => (prev === clamped ? prev : clamped));
|
||||
// If a playback tick is active, rebase its anchor so the loop continues
|
||||
// from the new position instead of snapping back.
|
||||
if (playAnchorRef.current) {
|
||||
playAnchorRef.current = { time: performance.now(), idx: clamped };
|
||||
}
|
||||
}, [drawBitmap, getCachedBitmap, pump, video]);
|
||||
|
||||
const stepFrames = useCallback((delta: number) => {
|
||||
@@ -252,7 +382,15 @@ export function ExtractMode({ username }: Props) {
|
||||
|
||||
const togglePlay = useCallback(() => {
|
||||
if (!video) return;
|
||||
setPlaying((p) => !p);
|
||||
setPlaying((p) => {
|
||||
if (!p) {
|
||||
// Fresh play attempt — clear any stale decode error so the loop can
|
||||
// re-arm. If it fails again, the banner will reappear.
|
||||
decodeErrorRef.current = null;
|
||||
setPlaybackError(null);
|
||||
}
|
||||
return !p;
|
||||
});
|
||||
}, [video]);
|
||||
|
||||
const bumpSpeed = useCallback((dir: 1 | -1) => {
|
||||
@@ -326,22 +464,31 @@ export function ExtractMode({ username }: Props) {
|
||||
};
|
||||
}, [bulkStart, bulkEnd]);
|
||||
|
||||
const bulkCount = useMemo(() => {
|
||||
if (!bulkRange) return 0;
|
||||
const skip = Math.max(1, settings?.frame_skip ?? 1);
|
||||
return Math.floor((bulkRange.end - bulkRange.start) / skip) + 1;
|
||||
}, [bulkRange, settings?.frame_skip]);
|
||||
|
||||
async function runBulkExtract() {
|
||||
if (!outputFolder || !bulkRange) return;
|
||||
setPlaying(false);
|
||||
setBulkError(null);
|
||||
setBulkProgress({ done: 0, total: bulkRange.end - bulkRange.start + 1, phase: "extract" });
|
||||
const skip = Math.max(1, settings?.frame_skip ?? 1);
|
||||
const total = Math.floor((bulkRange.end - bulkRange.start) / skip) + 1;
|
||||
setBulkProgress({ done: 0, total, phase: "extract" });
|
||||
const unlisten = await listen<{ done: number; total: number }>(
|
||||
"bulk-extract-progress",
|
||||
(e) => setBulkProgress({ ...e.payload, phase: "extract" })
|
||||
);
|
||||
// Only include annotations for frames we'll actually emit; backend will
|
||||
// step start..end by `skip` and shapesByFrame is keyed by absolute idx.
|
||||
const shapesByFrame: Record<string, CocoShape[]> = {};
|
||||
if (coco) {
|
||||
for (const [key, shapes] of Object.entries(coco.frames)) {
|
||||
const idx = Number(key);
|
||||
if (idx >= bulkRange.start && idx <= bulkRange.end) {
|
||||
shapesByFrame[key] = shapes;
|
||||
}
|
||||
for (let idx = bulkRange.start; idx <= bulkRange.end; idx += skip) {
|
||||
const key = String(idx);
|
||||
const shapes = coco.frames[key];
|
||||
if (shapes) shapesByFrame[key] = shapes;
|
||||
}
|
||||
}
|
||||
try {
|
||||
@@ -350,7 +497,8 @@ export function ExtractMode({ username }: Props) {
|
||||
bulkRange.end,
|
||||
outputFolder,
|
||||
shapesByFrame,
|
||||
username
|
||||
username,
|
||||
skip
|
||||
);
|
||||
const existing = await api.listExtractedFrames(outputFolder);
|
||||
setExtractedFrames(new Set(existing));
|
||||
@@ -401,6 +549,16 @@ export function ExtractMode({ username }: Props) {
|
||||
const tag = (e.target as HTMLElement | null)?.tagName;
|
||||
if (tag === "INPUT" || tag === "TEXTAREA") return;
|
||||
const step = e.ctrlKey || e.metaKey ? 10 : 1;
|
||||
// Adaptive throttle: drop OS key-repeats when the previous step is
|
||||
// still decoding. On fast systems this is a no-op; on slow systems it
|
||||
// prevents a held arrow from queueing 30 seeks/sec the pump can't keep
|
||||
// up with, which is what causes the "huge gap between keypress and
|
||||
// frame" feel.
|
||||
const isHeldRepeat =
|
||||
e.repeat &&
|
||||
(e.key === "ArrowRight" || e.key === "ArrowLeft") &&
|
||||
requestedIdx.current !== renderedIdx.current;
|
||||
if (isHeldRepeat) { e.preventDefault(); return; }
|
||||
switch (e.key) {
|
||||
case " ":
|
||||
e.preventDefault(); togglePlay(); break;
|
||||
@@ -463,6 +621,8 @@ export function ExtractMode({ username }: Props) {
|
||||
setPlaying(false);
|
||||
setSpeed(1);
|
||||
renderedIdx.current = -1;
|
||||
decodeErrorRef.current = null;
|
||||
setPlaybackError(null);
|
||||
// Reset per-video extraction context, then auto-adopt the per-video
|
||||
// default folder. If the user already extracted frames into it in a
|
||||
// prior session, listExtractedFrames rehydrates them immediately.
|
||||
@@ -590,7 +750,7 @@ export function ExtractMode({ username }: Props) {
|
||||
onClick={runBulkExtract}
|
||||
disabled={!outputFolder || !bulkRange || !!bulkProgress}
|
||||
>
|
||||
Bulk Extract{bulkRange && ` (${bulkRange.end - bulkRange.start + 1})`}
|
||||
Bulk Extract{bulkRange && ` (${bulkCount})`}
|
||||
</button>
|
||||
<button
|
||||
onClick={runBulkUndo}
|
||||
@@ -683,6 +843,22 @@ export function ExtractMode({ username }: Props) {
|
||||
</button>
|
||||
<span className="dim shortcuts">space · ← → · ctrl+← → · ↑ ↓ · H · E · U · [ ]</span>
|
||||
</div>
|
||||
|
||||
{playbackError && (
|
||||
<p className="error" role="alert">
|
||||
Playback stopped: {playbackError}
|
||||
{" "}
|
||||
<button
|
||||
className="link"
|
||||
onClick={() => {
|
||||
decodeErrorRef.current = null;
|
||||
setPlaybackError(null);
|
||||
}}
|
||||
>
|
||||
dismiss
|
||||
</button>
|
||||
</p>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -93,6 +93,10 @@ export interface Settings {
|
||||
sam_enabled: boolean;
|
||||
sam_model?: string | null;
|
||||
cache_capacity: number;
|
||||
/** ffmpeg `-hwaccel` argument. "" = software. Recognized: auto, vaapi, cuda, vdpau. */
|
||||
hwaccel: string;
|
||||
/** Decimation factor for playback advancement + bulk extract. 1 = no decimation. */
|
||||
frame_skip: number;
|
||||
}
|
||||
|
||||
export interface SamModelInfo {
|
||||
|
||||
Reference in New Issue
Block a user