165 lines
6.1 KiB
Markdown
165 lines
6.1 KiB
Markdown
# Tuning polygon output (point count + edge smoothness)
|
||
|
||
When SAM returns a mask, the backend converts it to a polygon before sending
|
||
it back to the Tauri annotation tool. Two things you can tune:
|
||
|
||
1. **Point count** — how many vertices the polygon has. Each vertex is
|
||
draggable in the UI, so too many makes editing painful.
|
||
2. **Edge smoothness** — how the polygon traces curved edges (smooth curves
|
||
vs. pixel staircases).
|
||
|
||
Both are controlled in one file:
|
||
[`sam2-backend/adapters/utils.py`](../sam2-backend/adapters/utils.py).
|
||
|
||
---
|
||
|
||
## Where it's called
|
||
|
||
Both adapters do the same thing in their predict path:
|
||
|
||
```python
|
||
# sam2-backend/adapters/sam2_adapter.py
|
||
return PredictResult(
|
||
polygon=_mask_to_polygon(masks_np[best]),
|
||
score=best_score,
|
||
)
|
||
|
||
# sam2-backend/adapters/sam3_adapter.py — same call site
|
||
```
|
||
|
||
The defaults are `min_points=4`, `max_points=10`, `smooth=True`. You can
|
||
override per-call:
|
||
|
||
```python
|
||
polygon=_mask_to_polygon(masks_np[best], min_points=3, max_points=15)
|
||
```
|
||
|
||
…or change the defaults in `utils.py::mask_to_polygon` and they'll apply
|
||
to every model.
|
||
|
||
---
|
||
|
||
## Knobs
|
||
|
||
### 1. Point count band — `min_points` / `max_points`
|
||
|
||
The simplifier uses Douglas–Peucker (`cv2.approxPolyDP`) with a binary
|
||
search over `epsilon` to land in `[min_points, max_points]`. Within that
|
||
band it prefers **more** detail (i.e. closer to `max_points`).
|
||
|
||
| Band | Use it when… |
|
||
|------------|-----------------------------------------------------------------------------|
|
||
| `4..10` | Default. Comfortable to drag, captures common shapes. |
|
||
| `4..15` | Curvy / organic objects (animals, foliage). 15 vertices follow a curve well. |
|
||
| `3..6` | Very simple shapes (rectangles, traffic signs). Fewer drag handles. |
|
||
| `6..20` | High-fidelity work, you don't mind dragging more. |
|
||
|
||
**Avoid `max_points > 25`** — past that, the binary search needs many
|
||
iterations and the marginal benefit per vertex drops sharply. If you need
|
||
that many points, the bottleneck is mask quality, not simplification.
|
||
|
||
### 2. Mask smoothing — `smooth=True | False`
|
||
|
||
Before extracting the contour, the helper optionally runs:
|
||
|
||
1. **Morphological close + open** (ellipse kernel) — fills single-pixel
|
||
notches and removes pepper noise.
|
||
2. **Light Gaussian blur + re-threshold** — sub-pixel edge smoothing so
|
||
curves don't trace as staircases.
|
||
|
||
Pass `smooth=False` to skip both steps and trace the raw mask exactly. Use
|
||
this when:
|
||
- The mask has fine features that smoothing erases (thin antennae, sharp
|
||
90° corners that get rounded).
|
||
- You're A/B testing whether smoothing is helping or hurting.
|
||
|
||
```python
|
||
polygon=_mask_to_polygon(masks_np[best], smooth=False)
|
||
```
|
||
|
||
### 3. Smoothing kernel size — `_smooth_mask`
|
||
|
||
Inside `_smooth_mask` (called by `mask_to_polygon` when `smooth=True`),
|
||
kernel size scales with the **blob's bounding box diagonal**:
|
||
|
||
```python
|
||
k = int(round(blob_diag * 0.015)) # ← morphology kernel multiplier
|
||
k = max(3, min(k | 1, 11)) # odd, clamped 3..11
|
||
|
||
blur_k = max(3, min(k | 1, 9)) # ← Gaussian blur kernel
|
||
```
|
||
|
||
**The `0.015` multiplier is the main dial.**
|
||
|
||
| Multiplier | Effect |
|
||
|------------|-----------------------------------------------------------------|
|
||
| `0.008` | Very gentle. Preserves thin features. Use if smoothing dilates the polygon. |
|
||
| `0.015` | Default. Good balance for typical objects. |
|
||
| `0.025` | Aggressive. Use only on visibly noisy masks; will round corners. |
|
||
|
||
The 11-pixel hard cap means giant masks don't get insanely smoothed.
|
||
|
||
### 4. Drop the Gaussian, keep morphology
|
||
|
||
If hard 90° corners get rounded but you still want morphology to clean up
|
||
notches, delete just the Gaussian step in `_smooth_mask`:
|
||
|
||
```python
|
||
# remove these two lines
|
||
m = cv2.GaussianBlur(m, (blur_k, blur_k), 0)
|
||
_, m = cv2.threshold(m, 127, 255, cv2.THRESH_BINARY)
|
||
```
|
||
|
||
This keeps close+open (which is mostly noise removal) and skips the
|
||
sub-pixel curve smoothing.
|
||
|
||
---
|
||
|
||
## Diagnostic recipe — find the right knob in 3 minutes
|
||
|
||
1. **Draw a representative bbox in the Tauri app.** Screenshot the result.
|
||
2. **Set `smooth=False`** in the adapter, restart, draw the same bbox.
|
||
- If polygon is *better* (less bloat, hugs edges) → smoothing is over-aggressive. Lower the kernel multiplier or delete the Gaussian step.
|
||
- If polygon is *worse* (jagged stairs) → smoothing was helping. Move on to step 3.
|
||
3. **Bump `max_points` to 15 or 20**, restart.
|
||
- If polygon is *better* → simplification was too aggressive. Land on `max_points=12..15` as your new default.
|
||
- If polygon is the same → the mask itself is the bottleneck (see "If the raw mask is bad" below).
|
||
4. **Restore `smooth=True`** and try a model swap (SAM 2.1 large ↔ SAM 3 tracker) — the underlying mask differs noticeably between models.
|
||
|
||
---
|
||
|
||
## If the raw mask is bad
|
||
|
||
If after all the above the polygon still looks wrong, the input to the
|
||
simplifier is bad — fix that, not the polygon code. Common moves:
|
||
|
||
- **Pad the bbox before sending** to SAM (Meta's reference scripts use
|
||
`pad=10` pixels). SAM gives crisper masks with a small box margin.
|
||
- **Try a larger SAM model** (e.g. switch the dropdown from `SAM 2.1
|
||
Hiera Small` to `SAM 2.1 Hiera Large`).
|
||
- **Tighten the bbox** around the actual object — SAM tends to fill the
|
||
box, so a generous box gets a bloated mask.
|
||
|
||
---
|
||
|
||
## Quick reference — every dial in one block
|
||
|
||
```python
|
||
# sam2-backend/adapters/utils.py
|
||
|
||
mask_to_polygon(
|
||
mask,
|
||
min_points=4, # never below this
|
||
max_points=10, # never above this; fewer = easier to drag
|
||
smooth=True, # turn off to trace raw mask
|
||
)
|
||
|
||
# Inside _smooth_mask:
|
||
k = int(round(blob_diag * 0.015)) # ← MAIN DIAL
|
||
# smaller = preserve detail
|
||
# bigger = more rounded
|
||
```
|
||
|
||
If you change defaults, restart the backend (`./run.sh`). No frontend
|
||
restart needed — the polygon arrives over the existing `/predict` call.
|