osm_bug fix

TEST_PLAN.txt

@@ -0,0 +1,326 @@
+KML MAP TOOL — DIAGNOSTIC TEST PLAN
+====================================
+
+Goal: isolate where pan/zoom lag actually originates (WebView vs app code vs
+data scale) before deciding which optimisation to invest in next.
+
+How to use: run sections in the order at the bottom of this file. Record FPS
+and console-counter numbers as you go. The numbers — not feel — decide what
+ships next.
+
+------------------------------------------------------------------------------
+A. WEBVIEW BASELINE TESTS (do first, cheapest)
+------------------------------------------------------------------------------
+
+A1. Confirm GPU acceleration
+
+    Command:  glxinfo | grep "OpenGL renderer"
+
+    Pass:     Output names a real GPU
+              (e.g. "Mesa Intel(R) HD Graphics 630",
+               "NVIDIA GeForce RTX 5070/PCIe/SSE2").
+    Fail:     Output says "llvmpipe" (software rendering).
+
+    If fail:  No code change will help. Fix drivers / Mesa / permissions
+              first. Stop here.
+
+
+A2. WebKitGTK DMA-BUF mitigation
+
+    Run two sessions and compare FPS at the same dataset/zoom:
+        Baseline:    npm run tauri dev
+        Mitigated:   WEBKIT_DISABLE_DMABUF_RENDERER=1 npm run tauri dev
+
+    Expected: FPS jumps >= 1.5x with the env var set on Linux + Mesa
+              setups suffering DMA-BUF compositor regressions.
+
+    If pass:  WebKitGTK compositor is the bottleneck. Document the env
+              var in INSTALLATION.md as a required workaround.
+
+
+A3. Tauri WKGTK vs Chromium A/B
+
+    Terminal 1 (Tauri / WebKitGTK):
+        npm run tauri dev
+        Pan/zoom over a fixed area at z=14 for 5 seconds, record FPS.
+
+    Terminal 2 (vanilla browser):
+        npm run dev
+        Open http://localhost:1420 in Firefox or Chromium.
+        Login as "browser-test" (synthetic ~5000 dots around Hyderabad).
+        Settings -> Performance mode ON (FPS counter top-centre).
+        Pan/zoom at z=14 for 5 seconds, record FPS.
+
+    Compare: same code, different WebView. Big delta = WebView is the
+             bottleneck. Small delta = the issue is below the WebView
+             (drivers/compositor/dataset).
+
+
+A4. Same build, cross-OS (if Windows machine available)
+
+    Build / run the Tauri app on Windows (uses WebView2 = Chromium-
+    based). Repeat A3 measurement with the same dataset.
+
+    Pass:  Windows holds steady FPS while Linux does not -> WebKitGTK
+           is conclusively the issue on the user's Linux box.
+
+
+------------------------------------------------------------------------------
+B. APP-LEVEL ISOLATION (uses perfMode + console counters)
+------------------------------------------------------------------------------
+
+B1. perfMode on/off A/B
+
+    1. Settings -> Performance mode OFF. Pan/zoom 5 seconds. Record FPS.
+    2. Settings -> Performance mode ON. Pan/zoom 5 seconds. Record FPS.
+
+    Big jump (>= +20 FPS):  Layer complexity matters. Move to test
+                            B3 to find the most expensive layer.
+    No change:              Bottleneck is below the layer effect
+                            (WebView / Mesa / compositor). Move to A2/A3.
+
+
+B2. Console-count check
+
+    1. Open DevTools console.
+    2. Pan continuously for 5 seconds.
+
+    Expected:
+        "MapView render"        increments by ~2 per gesture
+                                (movestart + moveend toggles).
+        "deck layer rebuild"    increments by <= 2 per gesture.
+
+    Fail mode: either counter increments by tens or hundreds during
+               a single pan -> a regression has snuck in. Investigate.
+
+
+B3. Layer-by-layer cost ladder (with perfMode OFF)
+
+    Toggle right-panel Layer checkboxes off one at a time.
+    Record FPS after each step:
+
+        All on (baseline)   FPS = ____
+        Scope off           FPS = ____
+        Metadata off        FPS = ____
+        OSM off             FPS = ____
+        Range off           FPS = ____
+        Fixed off           FPS = ____
+
+    The biggest single-step jump = the layer to simplify first.
+
+
+B4. Filter-down ladder
+
+    1. Filter to a single video. Record FPS.
+    2. Then filter to a single asset_name. Record FPS.
+    3. Then zoom in by 2 levels. Record FPS.
+
+    If FPS climbs sharply, dataset density is the load and the answer
+    is zoom-tier sampling, not layer code.
+
+
+------------------------------------------------------------------------------
+C. SCENARIO REGRESSION TESTS (verify recent fixes)
+------------------------------------------------------------------------------
+
+C1. OSM import auto-recentre
+
+    1. Open app fresh (default viewport = [51.4, 25.5], near Qatar).
+    2. Do NOT pan.
+    3. Import an OSM GeoJSON for a far-away region (e.g. Hyderabad).
+
+    Expected: map flies to imported region, roads visible immediately,
+              status bar shows "Loaded N road segment(s). Centred on
+              imported area."
+
+    Fail:     roads invisible after import. Was the user-reported bug
+              "imports 10-15 times before showing".
+
+
+C2. Pan/zoom doesn't churn React
+
+    1. Open DevTools -> React DevTools -> Profiler.
+    2. Click record. Pan continuously 5 seconds. Stop record.
+
+    Expected: <= 2 AppShell renders per gesture (movestart + moveend).
+    Fail:     ~60 renders/sec -> the move-sync regression is back.
+
+
+C3. Decorative overlays hide during gestures
+
+    1. Tap a metadata polyline to select it (arrows, S/E rings, name
+       badge appear).
+    2. Begin a pan or zoom.
+
+    Expected: arrows, name badge, S/E rings/labels, link halos disappear
+              during the gesture and reappear on moveend.
+
+
+C4. zoomBand quantisation
+
+    1. Open DevTools console (counters visible from B2).
+    2. Mouse-wheel zoom by tiny increments within one 0.5-zoom band
+       (e.g. 14.0 -> 14.1 -> 14.2 -> 14.3).
+
+    Expected: "deck layer rebuild" does NOT increment for sub-band moves.
+    It should only increment when crossing a 0.5 boundary.
+
+
+C5. Stale-fetch guard
+
+    1. Pan rapidly across many tiles for 10 seconds without stopping.
+
+    Expected: assets array updates only with the latest viewport's data.
+              No flicker of old viewports' points appearing late.
+
+
+C6. Compare mode default off
+
+    1. Open app fresh.
+    2. Inspect .map-area in DevTools.
+
+    Expected: exactly one <MapView> child rendered. Toggle the Compare
+              checkbox in the Basemap panel and a second canvas mounts;
+              uncheck it and the second canvas unmounts.
+
+
+C7. Browser-mode shim
+
+    1. npm run dev (no Tauri).
+    2. Open http://localhost:1420 in Firefox or Chromium.
+
+    Expected:
+      - Login screen lists one user "Browser Test", no console errors.
+      - After login, ~5000 dots visible around Hyderabad.
+      - Pan/zoom works.
+      - Filtering, search, perfMode toggle all work.
+      - Edit operations log "[browser-mode] invoke(...) is not stubbed"
+        warnings — expected, those mutations are no-ops in browser mode.
+
+
+------------------------------------------------------------------------------
+D. STRESS / SCALE TESTS
+------------------------------------------------------------------------------
+
+D1. 50k assets in viewport
+
+    1. Import the largest realistic dataset.
+    2. Zoom out so most points are in viewport.
+    3. Pan across — measure FPS and frame consistency.
+
+    Acceptable:  >= 30 FPS with perfMode OFF
+                 >= 50 FPS with perfMode ON
+
+
+D2. Dense OSM (no class filter)
+
+    1. Import a full-city OSM GeoJSON WITHOUT the major-only filter
+       (i.e. accept all road classes).
+    2. Zoom out to z=12.
+
+    If FPS tanks: zoom-tier simplification (Test 4 plan in chat) or
+                  class-prune workflow is required for that workload.
+
+
+D3. Many metadata polylines
+
+    1. Import the per-video metadata JSON with 40+ tracks.
+    2. Zoom out to fit all tracks. Pan.
+
+    With viewport-cull + zoom-decimation already in place, should still
+    be smooth. If not, lower the per-track point cap or the global
+    arrow cap.
+
+
+D4. Long edit session (memory leak check)
+
+    1. Drag 100 asset pins in succession.
+    2. Mix snap-to-road, side-toggle, rename, delete, restore, undo.
+    3. Watch memory in DevTools.
+
+    Expected: heap stays stable. No leaks from event listeners or
+              MapLibre Marker objects.
+
+
+------------------------------------------------------------------------------
+E. FEATURE SMOKE TESTS (run before any release)
+------------------------------------------------------------------------------
+
+For each, confirm the feature still works after recent edits.
+
+  1. Login + create user + switch user
+  2. Import fixed assets JSON         -> markers visible
+  3. Import range assets JSON         -> S/M/E lines visible
+  4. Import KML scope                 -> polygon visible,
+                                         out-of-scope filter works
+  5. Import per-video metadata JSON   -> coloured polylines visible
+  6. Import OSM GeoJSON               -> roads visible AND map
+                                         auto-centres (see C1)
+  7. Click asset                      -> image popup opens
+                                         (popupEnabled on)
+  8. Drag pin                         -> asset moves, action appears
+                                         in Recent actions
+  9. Lasso (circle / rect / polygon)  -> bulk delete / set-scope /
+                                         restore work
+ 10. Find duplicates within viewport
+ 11. Mark anchor -> drag -> Distribute correction
+ 12. Auto-link in polygon (L<->R, by-video, nearest)
+ 13. Right-click two assets          -> manual link, pink line
+     Right-click them again          -> unlink
+ 14. OSM edit mode                   -> drag vertex, shift-click + Del,
+                                        simplify slider all work
+ 15. Search by lat/lng               -> red goto pin appears
+ 16. Search by video name            -> flies to video, highlights
+                                        metadata track
+ 17. Export Source JSON / GeoJSON / KML / CSV
+     Round-trip: re-import the Source JSON cleanly
+ 18. Settings: toggle popup, aspect ratio, snap-to-centerline names,
+              perfMode
+ 19. Undo last action                -> state restored
+ 20. Reset DB                        -> app empty, but classes.txt
+                                        suggestions still load
+
+
+------------------------------------------------------------------------------
+RECOMMENDED RUN ORDER (fastest signal first)
+------------------------------------------------------------------------------
+
+   1. A1, A2          (5 min, no code touched)
+   2. B2              (1 min — confirms recent perf changes are working)
+   3. B1              (2 min — quantifies what perfMode buys)
+   4. A3              (10 min — Tauri vs Chromium)
+   5. B3              (10 min — which layer is most expensive)
+   6. C1              (1 min — verifies the OSM auto-recentre fix)
+   7. D1, D2, D3      (only if A2/A3 inconclusive)
+   8. E (full smoke)  (before any release)
+
+
+------------------------------------------------------------------------------
+DECISION TREE
+------------------------------------------------------------------------------
+
+   A2 alone fixes the user's lag                ->  ship the env var in
+                                                    INSTALLATION.md, done.
+
+   A3 shows Chromium is dramatically faster     ->  plan browser-mode
+                                                    deployment for Linux
+                                                    QA reviewers (keep
+                                                    Tauri for the import
+                                                    operator).
+
+   B3 isolates one heavy layer                  ->  zoom-tier
+                                                    simplification for
+                                                    that layer only.
+
+   B1 jump is large but A3 delta is small       ->  invest in zoom-tier
+                                                    simplification
+                                                    (Test 4 plan).
+
+   None of the above conclusive                 ->  re-run on the actual
+                                                    user hardware before
+                                                    investing further.
+
+
+==============================================================================
+End of test plan.
+==============================================================================

src-tauri/src/db.rs

@@ -187,7 +187,9 @@ async fn apply_schema(pool: &SqlitePool) -> Result<()> {
             min_lat  REAL NOT NULL,
             max_lat  REAL NOT NULL,
             min_lng  REAL NOT NULL,
-            max_lng  REAL NOT NULL
+            max_lng  REAL NOT NULL,
+            oneway   INTEGER NOT NULL DEFAULT 0,
+            highway  TEXT
         )",
     )
     .execute(pool)

src-tauri/src/lib.rs

@@ -3,7 +3,7 @@ mod db;
 mod scope;
 
 use commands::{
-    delete_asset, delete_assets_by_video, get_assets_in_bbox, get_osm_roads,
+    assets_bbox, delete_asset, delete_assets_by_video, get_assets_in_bbox, get_osm_roads,
     get_scope_polygons, import_fixed_assets, import_kml_scope, import_osm_geojson,
     import_range_assets, list_assets, read_video_polylines_json, list_recent_actions, merge_polylines, reset_assets_to_original,
     read_metadata_polyline, redo_action, reset_database, snap_assets_in_bbox,
@@ -11,7 +11,8 @@ use commands::{
     clear_osm_roads, clear_scope_polygons, data_source_counts, change_assets_side, clear_link, clear_manual_scope,
     create_osm_road, create_user, delete_road_vertices, set_link, simplify_road,
     delete_roads_by_highway, export_osm_roads, find_duplicate_clusters,
-    flip_road_direction, generate_osm_for_bbox, list_road_classes, merge_roads_by_name,
+    flip_road_direction, generate_osm_for_bbox, generate_osm_near_assets,
+    list_road_classes, merge_roads_by_name,
     delete_assets_bulk, delete_osm_road, delete_user, export_assets,
     list_asset_names, list_classes, list_users, list_video_names,
     rename_assets, restore_assets, set_assets_by_video_in_scope,
@@ -88,6 +89,8 @@ pub fn run() {
             merge_roads_by_name,
             export_osm_roads,
             generate_osm_for_bbox,
+            generate_osm_near_assets,
+            assets_bbox,
             flip_road_direction,
             change_assets_side,
             auto_link_in_polygon,

src/App.tsx

@@ -149,6 +149,39 @@ function FpsCounter() {
   );
 }
 
+// Lightweight indicator while the Overpass fetch is in flight. Single line,
+// no animation, no fixed positioning — just a small chip in the corner so
+// the user knows the click is doing something.
+function OsmFetchIndicator() {
+  const [elapsed, setElapsed] = useState(0);
+  useEffect(() => {
+    const start = performance.now();
+    const id = window.setInterval(() => {
+      setElapsed(Math.floor((performance.now() - start) / 1000));
+    }, 500);
+    return () => window.clearInterval(id);
+  }, []);
+  return (
+    <div
+      style={{
+        position: "fixed",
+        bottom: 12,
+        right: 12,
+        zIndex: 9500,
+        background: "rgba(20, 20, 20, 0.8)",
+        color: "#fff",
+        padding: "6px 10px",
+        borderRadius: 4,
+        fontFamily: "system-ui, sans-serif",
+        fontSize: 12,
+        pointerEvents: "none",
+      }}
+    >
+      Fetching OSM… {elapsed}s
+    </div>
+  );
+}
+
 function AppShell({
   currentUser,
   onLogout,
@@ -238,6 +271,7 @@ function AppShell({
     if (typeof window === "undefined") return false;
     return window.localStorage.getItem(PERF_MODE_KEY) === "1";
   });
+  const [osmFetchActive, setOsmFetchActive] = useState<boolean>(false);
   const [filters, setFilters] = useState<Filters>({
     videos: new Set(),
     sides: new Set(),
@@ -1155,7 +1189,11 @@ function AppShell({
     refreshScopePolygons().catch((e) =>
       setStatus(`Scope load failed: ${e}`),
     );
-    refreshOsmRoads().catch(() => {});
+    // Surface OSM load failures — silent catch made startup look fine
+    // even when the roads table couldn't be read.
+    refreshOsmRoads().catch((e) =>
+      setStatus(`OSM roads load failed: ${e}`),
+    );
     refreshActions().catch(() => {});
     refreshAssetNames().catch(() => {});
     refreshVideoNames().catch(() => {});
@@ -1234,6 +1272,7 @@ function AppShell({
         coords: drawRoadCoords,
       });
       await refreshOsmRoads();
+      await refreshDataCounts();
       setDrawRoadMode(false);
       setDrawRoadCoords([]);
       setEditingRoadId(newId);
@@ -1252,6 +1291,7 @@ function AppShell({
     try {
       await invoke("delete_osm_road", { id: editingRoadId });
       await refreshOsmRoads();
+      await refreshDataCounts();
       setEditingRoadId(null);
       setStatus(`Deleted road #${editingRoadId}.`);
     } catch (e) {
@@ -1304,13 +1344,26 @@ function AppShell({
           "tertiary_link",
         ];
       }
-      setStatus("Importing OSM roads…");
+      setStatus(`Importing OSM roads from ${picked.split(/[\\/]/).pop()}…`);
       const count = await invoke<number>("import_osm_geojson", args);
       const { bounds } = await refreshOsmRoads();
+      // Keep the sidebar Loaded-data card in sync. Without this the OSM
+      // count stayed at its last value, which made imports look like they
+      // hadn't done anything.
+      await refreshDataCounts();
       // Auto-recentre the map on the imported road bbox. Without this, if
       // the user's current viewport is far from the imported area, the
       // roads render off-screen and the import looks silently broken.
-      if (bounds) {
+      if (count === 0) {
+        // Explicit message — silent "Loaded 0" looked like the import
+        // did nothing. Common cause: the major-only filter removed every
+        // road class in the file, or the GeoJSON had no LineStrings.
+        setStatus(
+          majorOnly
+            ? `0 road segments matched the major-only filter. Re-import and click Cancel on the filter prompt to import all classes.`
+            : `0 road segments imported. Check that the GeoJSON has LineString features.`,
+        );
+      } else if (bounds) {
         const center: [number, number] = [
           (bounds.minLng + bounds.maxLng) / 2,
           (bounds.minLat + bounds.maxLat) / 2,
@@ -1356,33 +1409,54 @@ function AppShell({
 
   async function handleGenerateOsm() {
     if (busy) return;
-    if (!boundsRef.current) {
-      setStatus("Pan/zoom the map first.");
-      return;
-    }
     const folder = await ensureOsmFolder();
     if (!folder) return;
-    const b = boundsRef.current;
     const sep =
       folder.endsWith("/") || folder.endsWith("\\") ? "" : "/";
     const ts = new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19);
-    const path = `${folder}${sep}osm_${b.south.toFixed(4)}_${b.west.toFixed(4)}_${b.north.toFixed(4)}_${b.east.toFixed(4)}_${ts}.geojson`;
+    const hasAssets = dataCounts.fixed_assets + dataCounts.range_assets > 0;
     setBusy(true);
-    setStatus("Fetching OSM from Overpass…");
+    setOsmFetchActive(true);
     try {
-      const n = await invoke<number>("generate_osm_for_bbox", {
+      if (hasAssets) {
-        minLat: b.south,
+        // Tight per-asset filter: fetch a small candidate set with
-        minLng: b.west,
+        // around:60m, then keep only roads where ≥2 assets sit within 30 m
-        maxLat: b.north,
+        // of the centerline. That way the output is the road(s) the
-        maxLng: b.east,
+        // assets are actually on, not every street that happens to be
-        path,
+        // within bbox or radius.
-      });
+        setStatus("Fetching OSM near assets from Overpass…");
-      setStatus(
+        const path = `${folder}${sep}osm_on_asset_roads_${ts}.geojson`;
-        `Generated ${n} OSM way(s) → ${path}. Use Import → OSM roads to load.`,
+        const n = await invoke<number>("generate_osm_near_assets", {
-      );
+          radiusM: 60,
+          maxSamples: 800,
+          path,
+        });
+        setStatus(
+          `Generated ${n} road(s) carrying assets → ${path}. Use Import → OSM roads to load.`,
+        );
+      } else if (boundsRef.current) {
+        const b = boundsRef.current;
+        const path = `${folder}${sep}osm_${b.south.toFixed(4)}_${b.west.toFixed(4)}_${b.north.toFixed(4)}_${b.east.toFixed(4)}_${ts}.geojson`;
+        setStatus("Fetching OSM for visible viewport from Overpass…");
+        const n = await invoke<number>("generate_osm_for_bbox", {
+          minLat: b.south,
+          minLng: b.west,
+          maxLat: b.north,
+          maxLng: b.east,
+          path,
+        });
+        setStatus(
+          `Generated ${n} OSM way(s) from viewport → ${path}. Use Import → OSM roads to load.`,
+        );
+      } else {
+        setStatus(
+          "No assets imported and no map viewport set. Import assets or pan/zoom the map first.",
+        );
+      }
     } catch (e) {
       setStatus(`Generate OSM failed: ${e}`);
     } finally {
+      setOsmFetchActive(false);
       setBusy(false);
     }
   }
@@ -1422,6 +1496,8 @@ function AppShell({
         }
       }
       await refreshOsmRoads();
+      // Sidebar count would otherwise show pre-prune total.
+      await refreshDataCounts();
       setOsmToolsOpen(false);
       setStatus(
         summary.length > 0 ? `OSM tools: ${summary.join("; ")}.` : "Nothing selected.",
@@ -1688,7 +1764,7 @@ function AppShell({
       >;
       for (const e of merged) {
         if (e.kind !== "target") continue;
-        // Find bracketing anchors by rank.
+        // Find bracketing anchors using row_id ordering (rank in the merged list).
         let left: (typeof anchorByRank)[number] | null = null;
         let right: (typeof anchorByRank)[number] | null = null;
         for (const a of anchorByRank) {
@@ -1701,8 +1777,35 @@ function AppShell({
         let dlat: number;
         let dlng: number;
         if (left && right) {
-          const span = right.rank - left.rank;
+          // Interpolate by **geographic position** of the target projected
-          const tt = span > 0 ? (e.rank - left.rank) / span : 0;
+          // onto the line segment between the bracketing anchors' original
+          // positions — i.e. along the road direction. This replaces the
+          // previous rank-based tt, which over-corrected clustered targets
+          // toward the next anchor regardless of how close they actually
+          // were geographically. Latitude is scaled by cos(centre_lat) so
+          // the projection isn't distorted by lat/lng's non-isotropic
+          // metric at the dataset's latitude (Hyderabad ≈ 17°: 1° lng is
+          // ~95 % of 1° lat there).
+          const lx = left.anchor.origLng;
+          const ly = left.anchor.origLat;
+          const rx = right.anchor.origLng;
+          const ry = right.anchor.origLat;
+          const lat0 = ((ly + ry) / 2) * (Math.PI / 180);
+          const cosLat = Math.cos(lat0);
+          const tx = (e.asset.lng as number) - lx;
+          const ty = (e.asset.lat as number) - ly;
+          const sx = rx - lx;
+          const sy = ry - ly;
+          // Scale lng deltas by cosLat so the dot product is in metres²-equiv.
+          const seg2 = sx * sx * cosLat * cosLat + sy * sy;
+          let tt: number;
+          if (seg2 > 0) {
+            const dot = tx * sx * cosLat * cosLat + ty * sy;
+            tt = Math.max(0, Math.min(1, dot / seg2));
+          } else {
+            // Degenerate: anchors coincide. Fall back to left's offset.
+            tt = 0;
+          }
           dlat = left.anchor.dlat + (right.anchor.dlat - left.anchor.dlat) * tt;
           dlng = left.anchor.dlng + (right.anchor.dlng - left.anchor.dlng) * tt;
         } else {
@@ -2183,6 +2286,7 @@ function AppShell({
   return (
     <div className="app">
       {perfMode && <FpsCounter />}
+      {osmFetchActive && <OsmFetchIndicator />}
       <div className="map-area">
         <MapView
           basemapId={primaryId}
@@ -2886,10 +2990,10 @@ function AppShell({
                   setOsmToolsOpen(false);
                   handleGenerateOsm();
                 }}
-                disabled={busy || !boundsRef.current}
+                disabled={busy}
-                title="Fetch OSM ways for the visible map area from Overpass and save to your OSM folder"
+                title="Fetch OSM ways from Overpass for the imported assets' bbox (with a small buffer for nearby highways). Falls back to the visible map area when no assets are loaded."
               >
-                Generate (visible bbox) →
+                Generate roads KML →
               </button>
               <button
                 className="primary-btn"