Compare commits

...

11 Commits

Author SHA1 Message Date
cf4ae6e146 chainage panel update 2026-07-08 18:02:01 +05:30
8238bcf981 compose update 2026-07-08 17:16:56 +05:30
e7ab0f3368 chainage compute and map view 2026-07-08 17:03:24 +05:30
78b1666402 map updates 2026-07-08 15:06:57 +05:30
5c129bbdfe port changed 2026-07-07 22:45:34 +05:30
308d6bc129 map and chainage calculation 2026-07-07 22:42:47 +05:30
05ce1f2cdb video_name missing in export 2026-07-06 19:47:31 +05:30
4d71e08aa4 detailed data for verifications 2026-07-02 18:48:15 +05:30
bf019f86c3 minor bugs 2026-06-23 20:17:32 +05:30
c81e503af0 Merge remote-tracking branch 'origin/main' 2026-06-16 14:43:46 +05:30
7cb85d5ebc Central Verification Dashboard 2026-06-16 14:32:21 +05:30
41 changed files with 14998 additions and 1 deletions

4
.gitignore vendored Normal file
View File

@@ -0,0 +1,4 @@
target/
node_modules/
dist/
.env

245
README.md
View File

@@ -1,2 +1,245 @@
# Centralised_VideoVerification_Dashboard
# Central Verification Dashboard (Phase 1)
Dockerized pipeline that ingests a folder of videos + their sibling export-JSONs into Postgres and
serves a manager dashboard: completion overview, per-user leaderboard, throughput + ETA, and a
per-video table. Standalone — it does **not** touch the desktop video-annotator app.
> Phase 1 = the dashboard pipeline. Worker pull/push (claim a video, verify, push) + token auth are
> **Phase 3** and not built yet. So in this phase the dashboard shows the **annotation snapshot**
> ingested from the NAS JSONs: `pending` (no JSON) vs `annotated` (has a JSON) vs `verified`
> (a pushed pass — 0 until Phase 3).
## Run it
```bash
cd central
docker compose up --build
```
- Dashboard → http://localhost:8081
- API → http://localhost:8080 (e.g. `curl localhost:8080/api/projects`)
- Postgres → localhost:5433 (user/pass/db = `central`)
On startup the server creates a `default` project pointing at `INGEST_DIR` (`/data`, mounted from
`./sample-data`) and ingests it once. Click **Re-ingest** in the dashboard to re-scan after files
change. Re-ingest is idempotent and never downgrades a `verified` video.
## Point it at the NAS (on the VM)
The ingest folder is just a path/volume. On the VM, mount the Synology share and map it to `/data`:
```yaml
# docker-compose.override.yml on the VM
services:
server:
volumes:
- /mnt/synology/<client-folder>:/data:ro
```
Then `POST /api/projects/<id>/ingest` (or restart the server). No code changes.
## Database backups (→ NAS)
A `db-backup` sidecar (`prodrigestivill/postgres-backup-local`) runs `pg_dump` on a schedule
and writes gzipped dumps straight to the NAS, pruning old ones automatically.
- **Schedule:** daily at 02:00 IST (`SCHEDULE` + `TZ: Asia/Kolkata` in `docker-compose.yml`).
- **Destination:** the `nasbackups` NFS volume → `:/volume4/Saudi_Video_Sync/Verification_Dashboard_Backup`
on the NAS (`NFS_HOST`, default `192.168.1.199`). **Create that folder on the NAS** and allow
rw NFS from the Docker host; the host needs an NFS client (`nfs-common`).
- **Retention:** `BACKUP_KEEP_DAYS=30`, `BACKUP_KEEP_WEEKS=8`, `BACKUP_KEEP_MONTHS=6` (tune to taste).
Dumps land under `daily/`, `weekly/`, `monthly/` as `central-YYYYMMDD-HHMMSS.sql.gz`.
```bash
docker compose up -d db-backup # start it
docker compose exec db-backup /backup.sh # run an immediate backup (verify it writes to the NAS)
docker compose logs db-backup # check the schedule/last run
```
**Restore** a dump into the running DB:
```bash
gunzip -c /path/on/nas/daily/central-YYYYMMDD-HHMMSS.sql.gz \
| docker compose exec -T db psql -U central -d central
```
(For a clean restore, drop+recreate the DB first, or restore into a scratch DB and compare.)
Test a restore periodically — an untested backup isn't a backup.
## Layout
```
central/
db/init.sql # Postgres schema (projects, videos, annotations)
server/ # Rust/Axum API + folder ingest (embeds db/init.sql)
dashboard/ # React + Vite, served by nginx (proxies /api → server)
sample-data/ # demo: one annotated video JSON + two empty .mp4 placeholders
docker-compose.yml
```
## Ingest details
For each video file (`.mp4/.mkv/.avi/.mov/.m4v/.webm`) it looks for a sibling JSON named
`<file>_annotations.json`, `<file>.json`, or `<stem>.json` (video-annotator's export format —
`{ video, fixed_annotations[], range_annotations[] }`). From it the server derives per-video
annotation count, `annotated_by` (primary), `annotation_time_ms`, and `annotated_at` (latest
annotation timestamp, used for the throughput chart), and stores the full doc in `videos.raw_json`
for Phase 3 pull. Videos without a JSON are recorded as `pending`.
## API
### Clients → projects
A **client** (e.g. *IRB*) owns one or more **projects**; each project points at a folder via
`source_path` + `source_kind`. Project names are unique **per client**. Use the dashboard's
**Admin** tab: sign in (token) → create client (admin) → create project → **Sync**. A `default`
client/project is seeded **only on a fresh DB** (local dev); once real clients exist it never reappears.
**Sync from NFS (`source_kind='nfs'`).** A project's `source_path` may be a **bare export path**
(`/volume4/Saudi_Video_Sync` — the host comes from the `NFS_HOST` env, so the **IP stays server-side and
is never shown/returned**), or include the host explicitly as `host:/volume4/Share` or
`nfs://host/volume4/Share`. On sync the server mounts the share on demand (a fast TCP probe to port 2049
fails an unreachable NAS in ~3 s; the mount is bounded so a stuck NAS can't hang the request).
`source_kind='local'` keeps the old behaviour (a path inside the container, incl. a compose-managed NFS
volume). Point a project at a **subfolder** (e.g. `/volume4/Saudi_Video_Sync/IRB_Master_VIdeos`), not a
huge share root.
**Subfolders.** Ingest walks the folder **recursively**, so videos nested in subfolders are included;
each video's `rel_path` carries its subfolder, which the dashboard shows in a **Folder** column.
> In-app NFS mounting needs `nfs-common` (in the image) and the server container's `cap_add: SYS_ADMIN`
> + `security_opt: apparmor:unconfined` (in `docker-compose.yml`; use `privileged: true` if your host's
> `mount.nfs` still refuses). `local`-only deployments need none of this.
>
> The server service also sets `init: true` — it spawns exiftool/mount.nfs children as PID 1, and
> without an init their orphans become unreapable zombies that block `docker stop`
> ("PID … is zombie and can not be killed"). Keep it if you copy the service elsewhere.
### Roles
- **admin** (role `admin`, or the `ADMIN_TOKEN`): everything — create clients, manage collaborators, etc.
- **collaborator** (role `worker`): can **add a project** and **change a project's sync directory**, but
**not** create clients or manage users.
### Dashboard read APIs (open — the dashboard is manager-only on the VM)
- `GET /api/clients` — clients + project counts
- `GET /api/projects` — projects + counts (each carries `client_id`, `client_name`, `source_kind`)
- `GET /api/projects/:id/videos` — per-video rows (incl. claim/verify columns)
- `GET /api/projects/:id/stats` — overview, annotator leaderboard, `verifiers`, throughput, eta_days
- `GET /api/projects/:id/activity` — live claims + recent `video_events`
- `GET /api/audit?limit=N` — org audit log (client/project created, sync dir changed, synced, user created)
- `POST /api/projects/:id/ingest` — re-scan the project's folder ("Sync"); attributes the actor if a token is sent
## Phase 3 — worker pull/push + token auth
Workers claim an annotated video, verify it in the desktop app, and push the verified
pass back. All Phase 3 routes require `Authorization: Bearer <token>`.
- **Auth model:** tokens are random 256-bit hex strings; only their SHA-256 hash is
stored (`users.token_hash`). An admin provisions users and hands out the token (shown
**once**). A master `ADMIN_TOKEN` env var grants admin (bootstrap / break-glass).
- **Status vs. claim:** the annotation `status` (`pending`/`annotated`/`verified`) is
unchanged; claiming is a separate dimension (`claimed_by` + `lease_expires_at`). A video
is claimable when `annotated` and unclaimed-or-lease-expired. A push flips it to
`verified`, records `completed_by`/`verify_time_ms`, and clears the claim. Re-ingest then
**skips** verified videos (the pushed pass is authoritative).
- **Leases auto-release:** a 60 s background task returns claims whose lease lapsed
(`LEASE_SECS`, default 900) to the pool and logs an `auto_release` event.
### Admin (require admin / `ADMIN_TOKEN`)
- `POST /api/users` `{username, display_name?, role?}``{username, role, token}` (token shown once)
- `GET /api/users` — list users (no tokens)
- `POST /api/clients` `{name}` → the new client (unique name; 409 on dup)
### Create / manage projects (require any valid token — workers included)
- `POST /api/projects` `{client_id, name, source_path, source_kind}` — create/repoint a project
under a client (idempotent on `(client_id, name)`); `source_kind` = `local`|`nfs`
- `POST /api/projects/:id/source` `{source_path, source_kind}` — change a project's sync directory
### Worker (require any valid token)
- `GET /api/auth/whoami` — validate token → `{username, role}`
- `POST /api/videos/:id/claim` — atomic claim (annotated + unclaimed/expired); returns
metadata + `download_url` + preloaded `annotations`
- `GET /api/videos/:id/download` — stream the source bytes (claimant/admin only)
- `POST /api/videos/:id/heartbeat` — extend the lease while verifying
- `POST /api/videos/:id/release` — abandon the claim, back to the pool
- `POST /api/videos/:id/push` — body = export doc + `verify_time_ms`; replace annotations,
mark `verified`, record verifier + time, clear claim
### Map / GPS / road_type / chainage
The server image bundles exiftool; a serial background sweep extracts each video's GPS
track (`gps_status`: `pending → ok | none | error`). The raw track is immutable —
a kept "Viterbi" correction lives beside it and is preferred by the map + chainage.
`road_type` + the four chainage values (`chainage_start`, `chainage_end`,
`sr_chainage_start`, `sr_chainage_end`) are injected into every claim/export doc and
re-stamped on push (server-authoritative). **Every annotation** additionally gets its
**frame-exact** `lat`/`lon`/`chainage`/`sr_chainage` (range annotations also `end_*`):
frame 15 of a 30 fps video is t = 0.5 s — halfway between two 1 Hz GPS samples — so it
gets the halfway position/chainage, not a second/video boundary.
**Chainage unit convention (everywhere — JSON + UI): kilometres with 4 decimals**,
e.g. `22.4456` (`chainage` = client-calibrated, `sr_chainage` = GPS-measured).
Distances are WGS84-ellipsoidal (no spherical bias); calibration anchors persist in
`chainage_calibrations` so this works at any time after a recompute.
- `GET /api/projects/:id/map` — per-video map rows: track (≤400 pts), status,
road_type, chainage, `gps_status` (member/admin)
- `GET /api/projects/:id/tracks.kml?token=` — all tracks as KML LineStrings colored by
status + calibration points (Google Earth)
- `POST /api/videos/:id/gps/rescan` — re-queue a video for the exiftool sweep (admin)
- `POST /api/projects/:id/gps/rescan_all` — re-queue every video (admin; background)
- `POST /api/videos/:id/gps/correct` `{action: preview|keep|discard}` — spike smoothing +
missing-start stitch from the previous file (by GPS
timestamp, duration-guarded); the scan sweep applies
the same pipeline automatically as 'auto' (raw never
changes; discard blocks re-auto-correction) (admin)
- `GET/POST /api/projects/:id/road_types` — list / replace label set (`{names}`, admin)
- `POST /api/projects/:id/road_type_assign` `{video_ids, road_type}` — bulk-assign ('' clears, admin)
- `GET/POST /api/projects/:id/chainage/points` — client calibration points (POST admin:
`{lat, lon, chainage_m, note?, road_type?}`)
- `POST /api/projects/:id/chainage/points/bulk` `{points: […]}` — transactional bulk insert;
the dashboard's **KML import** uses this (placemark
names like `12+400` or `12.4` become chainage) (admin)
- `DELETE /api/chainage/points/:id` — remove a point (admin)
- `POST /api/chainage/points/:id` `{note}` — edit a point's note in place; position/chainage
stay immutable (admin). Points carry an `origin` field:
`''` = manual, `'quick'` = generated by the quick assign
(replaced wholesale on the next quick run — identity is
the column, so renaming the note is safe)
- `GET /api/projects/:id/chainage/calibrations` — the saved calibrations, one per road-type group
(`road_type, route_len_m, anchor_count, computed_by,
computed_at, has_summary`) — every recompute persists one,
so any user sees what's already calibrated (member/admin)
- `GET /api/projects/:id/chainage/summary?road_type=` — the SAVED full result of the group's last
recompute (route, spans, per-video chainage, warnings),
null if never computed; the dashboard auto-loads it so a
newly signed-in admin sees the existing compute
(member/admin)
- `POST /api/projects/:id/chainage/recompute` `{video_ids, road_type}` — order the videos into a
route, snap the points, distribute the client-vs-GPS error
piecewise-linearly; writes `chainage_*` (client) +
`sr_chainage_*` (measured, immutable) + `route_seq` (admin).
Summary reports `majority_flipped` + per-video
`against_flow` (opposite to the route's dominant direction —
the only direction flag the UI shows). Warnings include
mixed directions (≥25% against the majority → probably both
carriageways in one group) and ambiguous untagged points
(used by this route but also within 150 m of another
road-type group's videos — tag them)
- `POST /api/projects/:id/chainage/quick` `{video_ids, start_m, end_m, road_type}` — the
two-point case: points at the route's two geometric ends
(notes `quick: chain start/end`, tagged with the group) +
same recompute (admin)
### Env
| var | default | meaning |
|---------------|------------------------------------------|----------------------------------------------------|
| `ADMIN_TOKEN` | `dev-admin-token` | master admin token — **change in production** |
| `LEASE_SECS` | `900` | claim lease length; auto-released after this |
| `NFS_HOST` | _(empty)_ | NAS host/IP for `nfs`-kind projects (server-side only, never returned) |
| `NFS_OPTS` | `nfsvers=4,ro,soft,timeo=30,retrans=2,retry=0` | mount options for `nfs`-kind projects |

3
dashboard/.dockerignore Normal file
View File

@@ -0,0 +1,3 @@
node_modules
dist
.vite

11
dashboard/Dockerfile Normal file
View File

@@ -0,0 +1,11 @@
FROM node:20-alpine AS build
WORKDIR /app
COPY package.json package-lock.json* ./
RUN npm install
COPY . .
RUN npm run build
FROM nginx:alpine
COPY --from=build /app/dist /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80

12
dashboard/index.html Normal file
View File

@@ -0,0 +1,12 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Verification Dashboard</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>

18
dashboard/nginx.conf Normal file
View File

@@ -0,0 +1,18 @@
server {
listen 80;
server_name _;
root /usr/share/nginx/html;
index index.html;
# SPA fallback
location / {
try_files $uri $uri/ /index.html;
}
# Proxy API calls to the server container (same Docker network).
location /api/ {
proxy_pass http://server:8080/api/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}

1920
dashboard/package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

25
dashboard/package.json Normal file
View File

@@ -0,0 +1,25 @@
{
"name": "central-dashboard",
"private": true,
"version": "0.1.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc -b && vite build",
"preview": "vite preview"
},
"dependencies": {
"fflate": "^0.8.3",
"leaflet": "^1.9.4",
"react": "^18.3.1",
"react-dom": "^18.3.1"
},
"devDependencies": {
"@types/leaflet": "^1.9.21",
"@types/react": "^18.3.12",
"@types/react-dom": "^18.3.1",
"@vitejs/plugin-react": "^4.3.4",
"typescript": "^5.6.3",
"vite": "^6.0.5"
}
}

View File

@@ -0,0 +1,460 @@
import { useCallback, useEffect, useState } from "react";
import { api, fmtBytes, type AuditRow, type Client, type Member, type ProjectSummary, type StorageInfo, type UserRow } from "./api";
/** Admin/collaborator console. The signed-in token (from the app gate) is used for
* everything — there's no separate login here. Role-aware: ml_support can add
* projects + change a project's sync directory; admins additionally create/delete
* clients, delete projects, and manage (create / reset-token / delete) collaborators. */
export function AdminPanel({ role, onChanged }: { token: string; role: string; onChanged: () => void }) {
const isAdmin = role === "admin";
const [clients, setClients] = useState<Client[]>([]);
const [projects, setProjects] = useState<ProjectSummary[]>([]);
const [users, setUsers] = useState<UserRow[]>([]);
const [audit, setAudit] = useState<AuditRow[]>([]);
const [storage, setStorage] = useState<StorageInfo | null>(null);
const [notice, setNotice] = useState<{ kind: "ok" | "err"; msg: string } | null>(null);
const [busy, setBusy] = useState(false);
// form state
const [clientName, setClientName] = useState("");
const [projClientId, setProjClientId] = useState<number | "">("");
const [projName, setProjName] = useState("");
const [projKind, setProjKind] = useState<"nfs" | "local">("nfs");
const [projPath, setProjPath] = useState("");
const [manageId, setManageId] = useState<number | "">("");
const [manageKind, setManageKind] = useState<"nfs" | "local">("nfs");
const [managePath, setManagePath] = useState("");
const [members, setMembers] = useState<Member[]>([]);
const [memberPick, setMemberPick] = useState("");
const [newUsername, setNewUsername] = useState("");
const [newDisplay, setNewDisplay] = useState("");
const [newRole, setNewRole] = useState<"ml_support" | "admin">("ml_support");
const [createdToken, setCreatedToken] = useState<{ username: string; token: string } | null>(null);
const [editing, setEditing] = useState<{ id: number; display_name: string; role: "ml_support" | "admin" } | null>(null);
const refreshUsers = useCallback(async () => {
try { setUsers(await api.listUsers()); } catch { setUsers([]); }
try { setStorage(await api.storage()); } catch { setStorage(null); }
}, []);
const refresh = useCallback(async () => {
try {
const [c, p, a] = await Promise.all([api.clients(), api.projects(), api.audit(50)]);
setClients(c);
setProjects(p);
setAudit(a);
} catch (e) {
setNotice({ kind: "err", msg: String(e) });
}
}, []);
useEffect(() => { void refresh(); }, [refresh]);
useEffect(() => { if (isAdmin) void refreshUsers(); }, [isAdmin, refreshUsers]);
// When a project is picked in "Manage", load its current sync dir + members.
const loadMembers = useCallback(async (id: number) => {
try { setMembers(await api.members(id)); } catch { setMembers([]); }
}, []);
useEffect(() => {
const p = projects.find((x) => x.id === manageId);
if (p) {
setManagePath(p.source_path);
setManageKind(p.source_kind === "nfs" ? "nfs" : "local");
}
if (manageId !== "" && isAdmin) void loadMembers(Number(manageId));
else setMembers([]);
}, [manageId, projects, isAdmin, loadMembers]);
const run = async (fn: () => Promise<string>, needAdmin = false) => {
if (needAdmin && !isAdmin) { setNotice({ kind: "err", msg: "admin only" }); return; }
setBusy(true);
setNotice(null);
try {
const msg = await fn();
setNotice({ kind: "ok", msg });
await refresh();
if (isAdmin) void refreshUsers();
onChanged();
} catch (e) {
setNotice({ kind: "err", msg: String(e) });
} finally {
setBusy(false);
}
};
const createClient = () => run(async () => {
const c = await api.createClient(clientName.trim());
setClientName("");
return `Created client '${c.name}'`;
}, true);
const deleteClient = (c: Client) => {
// Re-affirmation: typing the exact client name confirms a destructive cascade.
const typed = window.prompt(
`Delete client "${c.name}" and ALL its ${c.project_count} project(s), videos and annotations?\n\nThis cannot be undone. Type the client name to confirm:`,
);
if (typed == null) return;
if (typed.trim() !== c.name) { setNotice({ kind: "err", msg: "name didn't match — deletion cancelled" }); return; }
void run(async () => {
const r = await api.deleteClient(c.id);
return `Deleted client '${c.name}' (${r.projects_removed} project(s) removed)`;
}, true);
};
const createProject = () => run(async () => {
if (projClientId === "") throw new Error("pick a client");
await api.createProject(
{ client_id: Number(projClientId), name: projName.trim(), source_path: projPath.trim(), source_kind: projKind },
);
const msg = `Created project '${projName.trim()}'`;
setProjName(""); setProjPath("");
return msg;
});
const saveSyncDir = () => run(async () => {
if (manageId === "") throw new Error("pick a project");
await api.updateProjectSource(Number(manageId), { source_path: managePath.trim(), source_kind: manageKind });
return `Sync directory updated`;
});
const syncProject = () => run(async () => {
if (manageId === "") throw new Error("pick a project");
const r = await api.ingest(Number(manageId));
return `Synced — ingested ${r.ingested} video(s)`;
});
const deleteProject = () => {
if (manageId === "") { setNotice({ kind: "err", msg: "pick a project" }); return; }
const p = projects.find((x) => x.id === manageId);
if (!p) return;
if (!window.confirm(`Delete project "${p.name}" and all its videos + annotations? This cannot be undone.`)) return;
void run(async () => {
await api.deleteProject(p.id);
setManageId("");
return `Deleted project '${p.name}'`;
}, true);
};
const addMember = () => {
if (manageId === "" || !memberPick) return;
void run(async () => {
await api.addMember(Number(manageId), memberPick);
const u = memberPick;
setMemberPick("");
await loadMembers(Number(manageId));
return `Added '${u}' to the project`;
}, true);
};
const removeMember = (username: string) => {
if (manageId === "") return;
if (!confirm(`Remove "${username}" from this project? Their assignments here are cleared.`)) return;
void run(async () => {
await api.removeMember(Number(manageId), username);
await loadMembers(Number(manageId));
return `Removed '${username}' from the project`;
}, true);
};
const addUser = () => run(async () => {
const u = await api.createUser({ username: newUsername.trim(), display_name: newDisplay.trim(), role: newRole });
setCreatedToken({ username: u.username, token: u.token });
setNewUsername(""); setNewDisplay("");
return `Added ${u.role} '${u.username}'`;
}, true);
const resetToken = (u: UserRow) => {
if (!confirm(`Generate a new access token for "${u.username}"? Their current token stops working immediately.`)) return;
void run(async () => {
const r = await api.resetUserToken(u.id);
setCreatedToken({ username: r.username, token: r.token });
return `New token generated for '${r.username}'`;
}, true);
};
const saveEdit = () => {
if (!editing) return;
void run(async () => {
const r = await api.updateUser(editing.id, { display_name: editing.display_name.trim(), role: editing.role });
setEditing(null);
return `Updated user → ${r.display_name} (${r.role})`;
}, true);
};
const deleteUser = (u: UserRow) => {
if (!confirm(`Delete ${u.role} "${u.username}"? Their token stops working immediately. Past work keeps their name.`)) return;
void run(async () => {
await api.deleteUser(u.id);
return `Deleted user '${u.username}'`;
}, true);
};
const pathHint = projKind === "nfs"
? "/volume4/Saudi_Video_Sync (or nfs://host/volume4/Share)"
: "Container path (e.g. /data/Microsoft/India)";
const manageHint = manageKind === "nfs"
? "/volume4/Saudi_Video_Sync (or nfs://host/volume4/Share)"
: "Container path (e.g. /data/...)";
return (
<div className="admin">
{!isAdmin && (
<p className="dim" style={{ margin: "0 0 12px" }}>
ML support access: you can add projects and change a project's sync directory.
Creating/deleting clients, deleting projects, and managing collaborators is admin-only.
</p>
)}
{notice && (
<div className={notice.kind === "ok" ? "notice-ok" : "error"}>
{notice.kind === "ok" ? "✓ " : "⚠ "}{notice.msg}
</div>
)}
<div className="grid2">
{/* Create client — admin only */}
{isAdmin && (
<section className="panel">
<h3>Create client</h3>
<div className="form-col">
<label>Client name</label>
<input placeholder="e.g. Microsoft" value={clientName} onChange={(e) => setClientName(e.target.value)} />
<button className="btn primary" disabled={busy || !clientName.trim()} onClick={createClient}>
Create client
</button>
</div>
</section>
)}
{/* Create project — all */}
<section className="panel">
<h3>Create project</h3>
<div className="form-col">
<label>Client</label>
<select value={projClientId} onChange={(e) => setProjClientId(e.target.value === "" ? "" : Number(e.target.value))}>
<option value="">— pick a client —</option>
{clients.map((c) => <option key={c.id} value={c.id}>{c.name}</option>)}
</select>
<label>Project name</label>
<input placeholder="e.g. India" value={projName} onChange={(e) => setProjName(e.target.value)} />
<label>Storage</label>
<select value={projKind} onChange={(e) => setProjKind(e.target.value as "nfs" | "local")}>
<option value="nfs">NFS share</option>
<option value="local">Local / mounted path</option>
</select>
<label>{projKind === "nfs" ? "NAS export path" : "Container path"}</label>
<input className="mono" placeholder={pathHint} value={projPath} onChange={(e) => setProjPath(e.target.value)} />
<button
className="btn primary"
disabled={busy || projClientId === "" || !projName.trim() || !projPath.trim()}
onClick={createProject}
>
Create project
</button>
</div>
</section>
</div>
{/* Clients list with delete — admin only */}
{isAdmin && (
<section className="panel">
<h3>Clients</h3>
<table>
<thead><tr><th>Client</th><th>Projects</th><th>Created</th><th></th></tr></thead>
<tbody>
{clients.map((c) => (
<tr key={c.id}>
<td>{c.name}</td>
<td>{c.project_count}</td>
<td className="dim">{c.created_at.slice(0, 10)}</td>
<td><button className="btn danger" disabled={busy} onClick={() => deleteClient(c)}>Delete</button></td>
</tr>
))}
{clients.length === 0 && <tr><td colSpan={4} className="dim">no clients yet</td></tr>}
</tbody>
</table>
</section>
)}
{/* Manage / sync a project — all (delete is admin only) */}
<section className="panel">
<h3>Manage project — change sync directory &amp; sync</h3>
<div className="form-col" style={{ maxWidth: 560 }}>
<label>Project</label>
<select value={manageId} onChange={(e) => setManageId(e.target.value === "" ? "" : Number(e.target.value))}>
<option value="">— pick a project —</option>
{projects.map((p) => (
<option key={p.id} value={p.id}>{p.client_name ? `${p.client_name} / ${p.name}` : p.name}</option>
))}
</select>
{manageId !== "" && (
<>
<label>Storage</label>
<select value={manageKind} onChange={(e) => setManageKind(e.target.value as "nfs" | "local")}>
<option value="nfs">NFS share</option>
<option value="local">Local / mounted path</option>
</select>
<label>{manageKind === "nfs" ? "NAS export path" : "Container path"}</label>
<input className="mono" placeholder={manageHint} value={managePath} onChange={(e) => setManagePath(e.target.value)} />
<div className="form-row">
<button className="btn" disabled={busy || !managePath.trim()} onClick={saveSyncDir}>Save sync directory</button>
<button className="btn primary" disabled={busy} onClick={syncProject}>{busy ? "Syncing…" : "Sync now"}</button>
{isAdmin && <button className="btn danger" disabled={busy} onClick={deleteProject}>Delete project</button>}
</div>
</>
)}
</div>
{/* Project members — admin picks who can see / be assigned this project */}
{manageId !== "" && isAdmin && (
<div className="members-box">
<h4>Members <span className="dim">— who can see &amp; be assigned this project</span></h4>
<div className="form-row" style={{ marginBottom: 8 }}>
<select value={memberPick} onChange={(e) => setMemberPick(e.target.value)}>
<option value="">— pick a user to add —</option>
{users
.filter((u) => !members.some((m) => m.username === u.username))
.map((u) => <option key={u.id} value={u.username}>{u.username} ({u.role})</option>)}
</select>
<button className="btn" disabled={busy || !memberPick} onClick={addMember}>Add member</button>
</div>
<div className="member-chips">
{members.map((m) => (
<span className="member-chip" key={m.username}>
{m.username} <span className="dim">{m.role}</span>
<button className="chip-x" title="Remove from project" onClick={() => removeMember(m.username)}>×</button>
</span>
))}
{members.length === 0 && <span className="dim">No members yet — this project is hidden from all ml_support users.</span>}
</div>
</div>
)}
<p className="dim" style={{ marginTop: 8 }}>
Sync re-scans the folder (incl. subfolders) for new/changed videos + JSONs. For NFS, enter the
export path (e.g. <code>/volume4/Saudi_Video_Sync/IRB_Master_VIdeos</code>) — a bare path uses
the NAS address configured on the server; a full <code>nfs://host/path</code> also works.
</p>
</section>
{/* Collaborators — admin only */}
{isAdmin && (
<section className="panel">
<h3>Collaborators</h3>
{createdToken && (
<div className="token-box">
<div>
New token for <b>{createdToken.username}</b> — copy it now, it won't be shown again:
</div>
<div className="form-row" style={{ marginTop: 6 }}>
<code className="token-val">{createdToken.token}</code>
<button className="btn" onClick={() => navigator.clipboard?.writeText(createdToken.token)}>Copy</button>
<button className="btn" onClick={() => setCreatedToken(null)}>Dismiss</button>
</div>
</div>
)}
<div className="form-row" style={{ marginBottom: 10 }}>
<input placeholder="username" value={newUsername} onChange={(e) => setNewUsername(e.target.value)} />
<input placeholder="display name (optional)" value={newDisplay} onChange={(e) => setNewDisplay(e.target.value)} />
<select value={newRole} onChange={(e) => setNewRole(e.target.value as "ml_support" | "admin")}>
<option value="ml_support">ml_support</option>
<option value="admin">admin</option>
</select>
<button className="btn primary" disabled={busy || !newUsername.trim()} onClick={addUser}>Add collaborator</button>
</div>
<p className="dim" style={{ margin: "0 0 8px", fontSize: 12 }}>
Tokens are stored hashed and can't be displayed. Use <b>Reset token</b> to issue a new one
(shown once, here).
</p>
<table>
<thead><tr><th>User</th><th>Role</th><th>Active</th><th>Since</th><th></th></tr></thead>
<tbody>
{users.map((u) => editing && editing.id === u.id ? (
<tr key={u.id} className="row-editing">
<td>
<b>{u.username}</b>{" "}
<input value={editing.display_name} placeholder="display name"
onChange={(e) => setEditing({ ...editing, display_name: e.target.value })}
style={{ width: 140 }} />
</td>
<td>
<select value={editing.role} onChange={(e) => setEditing({ ...editing, role: e.target.value as "ml_support" | "admin" })}>
<option value="ml_support">ml_support</option>
<option value="admin">admin</option>
</select>
</td>
<td>{u.active ? "yes" : "no"}</td>
<td className="dim">{u.created_at.slice(0, 10)}</td>
<td className="row-actions">
<button className="btn primary" disabled={busy} onClick={saveEdit}>Save</button>
<button className="btn" disabled={busy} onClick={() => setEditing(null)}>Cancel</button>
</td>
</tr>
) : (
<tr key={u.id}>
<td>{u.username}{u.display_name && u.display_name !== u.username ? ` (${u.display_name})` : ""}</td>
<td><span className={`badge ${u.role === "admin" ? "verified" : "annotated"}`}>{u.role}</span></td>
<td>{u.active ? "yes" : "no"}</td>
<td className="dim">{u.created_at.slice(0, 10)}</td>
<td className="row-actions">
<button className="btn" disabled={busy} title="Edit display name + role"
onClick={() => setEditing({ id: u.id, display_name: u.display_name === u.username ? "" : u.display_name, role: u.role === "admin" ? "admin" : "ml_support" })}>Edit</button>
<button className="btn" disabled={busy} title="Generate a new token (current one stops working)" onClick={() => resetToken(u)}>Reset token</button>
<button className="btn danger" disabled={busy} title="Delete this user" onClick={() => deleteUser(u)}>Delete</button>
</td>
</tr>
))}
{users.length === 0 && <tr><td colSpan={5} className="dim">no collaborators yet</td></tr>}
</tbody>
</table>
</section>
)}
{/* Storage — admin only */}
{isAdmin && storage && (
<section className="panel">
<h3>Storage</h3>
<div className="storage-grid">
<div><span className="dim">Database size</span><b>{fmtBytes(storage.db_bytes)}</b></div>
<div><span className="dim">Videos</span><b>{storage.videos}</b></div>
<div><span className="dim">Annotations</span><b>{storage.annotations}</b></div>
<div><span className="dim">Event log rows</span><b>{storage.video_events}</b></div>
<div><span className="dim">Audit log rows</span><b>{storage.audit_events}</b></div>
</div>
<p className="dim" style={{ marginTop: 8 }}>
The activity + audit logs are trimmed to the most recent {storage.max_log_rows} rows nightly
(~00:00 UTC) so the database stays bounded. Verification (push) history is kept.
</p>
</section>
)}
{/* Activity log — all */}
<section className="panel">
<h3>Activity log</h3>
<table>
<thead><tr><th>Time</th><th>User</th><th>Action</th><th>Details</th></tr></thead>
<tbody>
{audit.map((a, i) => (
<tr key={i}>
<td className="dim mono">{a.at.slice(0, 19).replace("T", " ")}</td>
<td>{a.username || "—"}</td>
<td><span className={`badge ${a.action === "signed_in" ? "verified" : "annotated"}`}>{a.action}</span></td>
<td className="mono dim">{auditDetail(a)}</td>
</tr>
))}
{audit.length === 0 && <tr><td colSpan={4} className="dim">no activity yet</td></tr>}
</tbody>
</table>
</section>
</div>
);
}
function auditDetail(a: AuditRow): string {
const d = a.detail || {};
const parts: string[] = [];
for (const [k, v] of Object.entries(d)) parts.push(`${k}=${v}`);
return parts.join(" ");
}

709
dashboard/src/App.tsx Normal file
View File

@@ -0,0 +1,709 @@
import { useCallback, useEffect, useState } from "react";
import {
api, exportProjectUrl, exportVideoUrl, fmtChainageRange, fmtDuration, fmtWhen, passDeltas,
setAuthToken, statusLabel,
type Activity, type Member, type ProjectSummary, type RoadType, type Stats, type UserRow, type VideoRow,
} from "./api";
import { RemarkCell } from "./RemarkCell";
import { AdminPanel } from "./AdminPanel";
import { MembersPicker } from "./MembersPicker";
import { FoldersPicker } from "./FoldersPicker";
import { SignIn } from "./SignIn";
import { MapView } from "./MapView";
import { ChainagePanel } from "./ChainagePanel";
const TOKEN_KEY = "central-token";
const POLL_MS = 10_000;
const IDLE_MS = 10 * 60_000; // auto-logout after 10 minutes of inactivity
type Auth = { username: string; role: string };
export function App() {
// ---- auth gate ----
const [token, setToken] = useState<string>(() => localStorage.getItem(TOKEN_KEY) ?? "");
const [auth, setAuth] = useState<Auth | null>(null);
const [authChecked, setAuthChecked] = useState(false);
const [expiredReason, setExpiredReason] = useState<string | null>(null);
const isAdmin = auth?.role === "admin";
const signOut = useCallback((reason?: string) => {
setAuth(null);
setToken("");
setAuthToken("");
localStorage.removeItem(TOKEN_KEY);
setExpiredReason(reason ?? null);
}, []);
const onSignedIn = useCallback((t: string, me: Auth) => {
setAuthToken(t);
localStorage.setItem(TOKEN_KEY, t);
setToken(t);
setAuth(me);
setExpiredReason(null);
}, []);
// Restore a persisted session on load (silent whoami — no audit spam).
useEffect(() => {
let cancel = false;
(async () => {
if (token) {
setAuthToken(token);
try {
const me = await api.whoami(token);
if (!cancel) setAuth(me);
} catch {
if (!cancel) { setAuthToken(""); localStorage.removeItem(TOKEN_KEY); }
}
}
if (!cancel) setAuthChecked(true);
})();
return () => { cancel = true; };
// eslint-disable-next-line react-hooks/exhaustive-deps
}, []);
// Idle auto-logout: any activity resets a 10-minute timer.
useEffect(() => {
if (!auth) return;
let timer = 0;
const reset = () => {
clearTimeout(timer);
timer = window.setTimeout(
() => signOut("Your session expired because of inactivity. Please sign in again."),
IDLE_MS,
);
};
const events = ["mousemove", "mousedown", "keydown", "scroll", "touchstart", "click"];
events.forEach((e) => window.addEventListener(e, reset, { passive: true }));
reset();
return () => { clearTimeout(timer); events.forEach((e) => window.removeEventListener(e, reset)); };
}, [auth, signOut]);
// ---- dashboard data ----
const [view, setView] = useState<"dashboard" | "admin">("dashboard");
const [projects, setProjects] = useState<ProjectSummary[]>([]);
const [projectId, setProjectId] = useState<number | null>(null);
const [stats, setStats] = useState<Stats | null>(null);
const [activity, setActivity] = useState<Activity | null>(null);
const [videos, setVideos] = useState<VideoRow[]>([]);
const [error, setError] = useState<string | null>(null);
const [busy, setBusy] = useState(false);
// Assignment (admin): selected video ids; the dropdown lists the project's members.
const [selected, setSelected] = useState<Set<number>>(new Set());
const [members, setMembers] = useState<Member[]>([]);
const [assignee, setAssignee] = useState("");
// Videos panel: table ↔ map toggle + road types (labels for assignment/calibration).
const [videoView, setVideoView] = useState<"table" | "map">("table");
const [roadTypes, setRoadTypes] = useState<RoadType[]>([]);
const [rtAssign, setRtAssign] = useState("");
// All ml_support users (admin only) — the pool the members picker chooses from.
const [users, setUsers] = useState<UserRow[]>([]);
const mlUsers = users.filter((u) => u.role === "ml_support");
useEffect(() => {
if (!isAdmin) { setUsers([]); return; }
api.listUsers().then(setUsers).catch(() => setUsers([]));
}, [isAdmin]);
const toggleSel = (id: number) =>
setSelected((s) => { const n = new Set(s); n.has(id) ? n.delete(id) : n.add(id); return n; });
const loadProject = useCallback(async (id: number) => {
try {
const [s, v, a, m, rt] = await Promise.all([
api.stats(id), api.videos(id), api.activity(id),
api.members(id).catch(() => [] as Member[]),
api.roadTypes(id).catch(() => [] as RoadType[]),
]);
setStats(s);
setVideos(v);
setActivity(a);
setMembers(m);
setRoadTypes(rt);
setError(null);
} catch (e) {
setError(String(e));
}
}, []);
const loadProjects = useCallback(async () => {
try {
const p = await api.projects();
setProjects(p);
setProjectId((cur) => cur ?? (p[0]?.id ?? null));
setError(null);
} catch (e) {
setError(String(e));
}
}, []);
const assignSelected = useCallback(async () => {
if (!isAdmin || selected.size === 0 || projectId == null) return;
setBusy(true);
try {
await api.assign(projectId, [...selected], assignee);
setSelected(new Set());
await loadProject(projectId);
setError(null);
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
}, [isAdmin, selected, assignee, projectId, loadProject]);
// Bulk-assign the picked road type to the selected videos (admin; "" clears).
const applyRoadType = useCallback(async () => {
if (!isAdmin || selected.size === 0 || projectId == null) return;
setBusy(true);
try {
await api.assignRoadType(projectId, [...selected], rtAssign);
setSelected(new Set());
await loadProject(projectId);
setError(null);
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
}, [isAdmin, selected, rtAssign, projectId, loadProject]);
// A deleted road-type label can leave a stale dropdown selection: the select
// LOOKS like "(clear road type)" but still sends the dead label, which the server
// rejects — making un-assign impossible. Snap back to '' whenever that happens.
useEffect(() => {
if (rtAssign !== "" && !roadTypes.some((r) => r.name === rtAssign)) setRtAssign("");
}, [roadTypes, rtAssign]);
useEffect(() => { if (auth) void loadProjects(); }, [auth, loadProjects]);
useEffect(() => {
if (!auth || projectId == null) return;
setSelected(new Set()); // clear selection when switching projects
void loadProject(projectId);
const t = setInterval(() => { void loadProject(projectId); void loadProjects(); }, POLL_MS);
return () => clearInterval(t);
}, [auth, projectId, loadProject, loadProjects]);
const timer = useCallback(async (action: "start" | "stop" | "reset") => {
if (projectId == null) return;
if (action === "reset" && !window.confirm("Reset the project timer back to zero?")) return;
try {
await api.setTimer(projectId, action);
await loadProject(projectId);
} catch (e) {
setError(String(e));
}
}, [projectId, loadProject]);
const toggleIgnore = useCallback(async (v: VideoRow) => {
if (!isAdmin || projectId == null) return;
try {
await api.setIgnore(v.id, !v.ignored);
await loadProject(projectId);
} catch (e) {
setError(String(e));
}
}, [isAdmin, projectId, loadProject]);
const reingest = useCallback(async () => {
if (projectId == null) return;
setBusy(true);
try {
await api.ingest(projectId);
await loadProject(projectId);
await loadProjects();
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
}, [projectId, loadProject, loadProjects]);
if (!authChecked) {
return <div className="signin-screen"><div className="dim">Loading</div></div>;
}
if (!auth) {
return <SignIn onSignedIn={onSignedIn} reason={expiredReason} />;
}
const project = projects.find((p) => p.id === projectId) ?? null;
const ov = stats?.overview;
const groups = groupByClient(projects);
return (
<div className="page">
<header className="topbar">
<div className="brand">Verification Dashboard</div>
<div className="viewtabs">
<button className={view === "dashboard" ? "active" : ""} onClick={() => setView("dashboard")}>Dashboard</button>
<button className={view === "admin" ? "active" : ""} onClick={() => setView("admin")}>Admin</button>
</div>
{view === "dashboard" && (
<>
<select
value={projectId ?? ""}
onChange={(e) => setProjectId(Number(e.target.value))}
disabled={projects.length === 0}
>
{groups.map(([client, ps]) => (
<optgroup key={client} label={client}>
{ps.map((p) => (
<option key={p.id} value={p.id}>{p.name}</option>
))}
</optgroup>
))}
</select>
<button onClick={reingest} disabled={busy || projectId == null}>
{busy ? "Ingesting…" : "Re-ingest"}
</button>
</>
)}
<span className="spacer" />
{view === "dashboard" && project && (
<span className="dim path" title={project.source_path}>
{project.client_name ? `${project.client_name} · ` : ""}{project.source_path}
</span>
)}
<span className="whoami" title={`role: ${auth.role}`}>{auth.username} · {auth.role}</span>
<button className="signout-btn" onClick={() => signOut()}>Sign out</button>
</header>
{error && <div className="error"> {error}</div>}
{view === "admin" && <AdminPanel token={token} role={auth.role} onChanged={loadProjects} />}
{view === "dashboard" && (
<>
{/* Overview */}
{ov && (
<section className="cards">
<Card label="Total videos" value={ov.total} />
<Card label="Pending" value={ov.pending} tone="warn" />
<Card label="Annotated" value={ov.annotated} tone="info" />
<Card label="Verified" value={ov.verified} tone="ok" />
<div className="card progress-card">
<div className="card-label">Verified progress</div>
<div className="bar">
<div className="bar-fill" style={{ width: `${ov.pct_done}%` }} />
</div>
<div className="card-value sm">{ov.pct_done.toFixed(1)}%
{stats?.timeline.eta_ms != null && (
<span className="dim"> · ETA ~{fmtDuration(stats.timeline.eta_ms)}</span>
)}
</div>
</div>
</section>
)}
{/* Project time & estimate (driven by the assigned users) */}
{stats && ov && (
<section className="panel time-panel">
<div className="panel-head">
<h3>Project time &amp; estimate</h3>
{isAdmin && projectId != null && (
<div className="timer-controls">
<span className={`timer-state ${stats.timeline.time_running ? "on" : "off"}`}>
{stats.timeline.time_running ? "● running" : "⏸ stopped"}
</span>
{stats.timeline.time_running
? <button className="btn" onClick={() => timer("stop")}>Stop</button>
: <button className="btn" onClick={() => timer("start")}>Start</button>}
<button className="btn danger" onClick={() => timer("reset")}>Reset</button>
</div>
)}
</div>
<div className="time-grid">
<div className="tcell big">
<span className="dim">Total project time</span>
<b>{fmtDuration(stats.timeline.elapsed_ms)} {stats.timeline.all_verified ? "✓" : ""}</b>
<span className="dim sub">inception {stats.timeline.all_verified ? "all verified" : "now (ongoing)"}</span>
</div>
<div className="tcell">
<span className="dim">Total hours all users</span>
<b>{fmtDuration(stats.timeline.total_spent_ms)}</b>
<span className="dim sub">annotation + verify, summed</span>
</div>
<div className="tcell">
<span className="dim">Assigned users</span>
<b>{stats.timeline.active_workers}</b>
<span className="dim sub">{stats.timeline.active_workers === 0 ? "assign users to estimate" : "drives the estimate"}</span>
</div>
<div className="tcell">
<span className="dim">Avg per video</span>
<b>{fmtDuration(stats.timeline.avg_video_ms)}</b>
<span className="dim sub">over verified videos</span>
</div>
<div className="tcell big">
<span className="dim">Estimated time to finish</span>
<b>{stats.timeline.eta_ms != null ? fmtDuration(stats.timeline.eta_ms) : "—"}</b>
<span className="dim sub">
{ov.total - ov.verified} left ÷ {Math.max(1, stats.timeline.active_workers)} user(s) @ {fmtDuration(stats.timeline.avg_video_ms)}/video
</span>
</div>
</div>
</section>
)}
<div className="grid2">
{/* Leaderboard + project members picker */}
<section className="panel">
<div className="panel-head">
<h3>Per-user leaderboard</h3>
{isAdmin && projectId != null && (
<MembersPicker
projectId={projectId}
users={mlUsers}
members={members}
onChange={() => loadProject(projectId)}
/>
)}
</div>
{isAdmin && members.length === 0 && (
<div className="members-cta">No users assigned to this project yet use <b>+ Assign users</b> to pick ml_support, then assign them videos.</div>
)}
<table>
<thead>
<tr><th>User</th><th>Videos</th><th>Annotations</th><th>Total time</th><th>Avg / video</th></tr>
</thead>
<tbody>
{stats?.leaderboard.map((r) => (
<tr key={r.user}>
<td>{r.user}</td>
<td>{r.videos}</td>
<td>{r.annotations}</td>
<td>{fmtDuration(r.total_time_ms)}</td>
<td>{fmtDuration(r.avg_time_ms)}</td>
</tr>
))}
{(!stats || stats.leaderboard.length === 0) && (
<tr><td colSpan={5} className="dim">no annotators yet</td></tr>
)}
</tbody>
</table>
</section>
{/* Verification pace (rate-based, replaces the old per-day bars) */}
<section className="panel">
<h3>Verification pace</h3>
{stats && ov ? (() => {
const t = stats.timeline;
const perDay = t.recent_window_days > 0 ? t.recent_verified / t.recent_window_days : 0;
const remaining = ov.total - ov.verified;
const paceDays = perDay > 0 ? Math.ceil(remaining / perDay) : null;
return (
<div className="pace-grid">
<div className="pcell">
<span className="dim">Verified / day</span>
<b>{perDay.toFixed(1)}</b>
<span className="dim sub">last {t.recent_window_days} days ({t.recent_verified} done)</span>
</div>
<div className="pcell">
<span className="dim">Avg per video</span>
<b>{fmtDuration(t.avg_video_ms)}</b>
<span className="dim sub">over verified videos</span>
</div>
<div className="pcell big">
<span className="dim">Finish at recent pace</span>
<b>{paceDays != null ? `~${paceDays} day${paceDays === 1 ? "" : "s"}` : "—"}</b>
<span className="dim sub">{perDay > 0 ? `${remaining} left ÷ ${perDay.toFixed(1)}/day (all users' actual output)` : "no recent activity"}</span>
</div>
<div className="pcell big">
<span className="dim">Est. to finish · {Math.max(1, t.active_workers)} user{t.active_workers === 1 ? "" : "s"}</span>
<b>{t.eta_ms != null ? fmtDuration(t.eta_ms) : "—"}</b>
<span className="dim sub">{remaining} left × {fmtDuration(t.avg_video_ms)} ÷ {Math.max(1, t.active_workers)} user{t.active_workers === 1 ? "" : "s"}</span>
</div>
</div>
);
})() : <div className="dim">no data yet</div>}
</section>
</div>
<div className="grid2">
{/* Verification leaderboard — assigned users ranked by annotation throughput
(most annotations in the least time), not by video count. */}
<section className="panel">
<h3>Verification leaderboard <span className="dim">· most annotations / min</span></h3>
<table>
<thead>
<tr><th>Rank</th><th>User</th><th>Annotations</th><th>Anno/min</th><th>Verified</th><th>Total time</th></tr>
</thead>
<tbody>
{stats?.verifiers.map((r) => (
<tr key={r.user} className={r.rank === 1 ? "rank-top" : undefined}>
<td>{r.rank > 0 ? (r.rank === 1 ? "🥇 1" : `#${r.rank}`) : <span className="dim"></span>}</td>
<td>{r.user}</td>
<td>{r.annotations > 0 ? r.annotations : <span className="dim">0</span>}</td>
<td>{r.annotations_per_min > 0 ? r.annotations_per_min.toFixed(1) : <span className="dim"></span>}</td>
<td>{r.videos > 0 ? r.videos : <span className="dim">0</span>}</td>
<td>{r.total_time_ms > 0 ? fmtDuration(r.total_time_ms) : <span className="dim"></span>}</td>
</tr>
))}
{(!stats || stats.verifiers.length === 0) && (
<tr><td colSpan={6} className="dim">no assigned users yet</td></tr>
)}
</tbody>
</table>
</section>
{/* Recent activity (live claims + event feed) */}
<section className="panel">
<h3>Recent activity</h3>
{activity && activity.active_claims.length > 0 && (
<div className="active-claims">
{activity.active_claims.map((c) => (
<div className="active-claim" key={c.video_id} title={`lease expires ${fmtWhen(c.lease_expires_at)}`}>
🔒 <b>{c.claimed_by}</b> has <span className="mono">{c.file_name}</span> <span className="dim">· since {fmtWhen(c.claimed_at)}</span>
</div>
))}
</div>
)}
<div className="activity-feed">
{activity?.recent_events.map((e, i) => (
<div className="activity-row" key={i}>
<span className={`evt evt-${e.event}`}>{e.event}</span>
<span className="mono">{e.file_name}</span>
<span className="dim">{e.username} · {fmtWhen(e.at)}</span>
</div>
))}
{(!activity || activity.recent_events.length === 0) && <div className="dim">no activity yet</div>}
</div>
</section>
</div>
{/* Per-video table / map */}
<section className="panel">
<div className="panel-head">
<h3>
Videos ({videos.length})
<span className="viewtabs videos-toggle">
<button className={videoView === "table" ? "active" : ""} onClick={() => setVideoView("table")}>Table</button>
<button className={videoView === "map" ? "active" : ""} onClick={() => setVideoView("map")}>Map</button>
</span>
</h3>
<div className="panel-actions">
{isAdmin && projectId != null && (
<FoldersPicker projectId={projectId} onChange={() => loadProject(projectId)} />
)}
{isAdmin && videoView === "table" && (
<span className="assign-bar">
<span className="dim">{selected.size} selected </span>
<select value={assignee} onChange={(e) => setAssignee(e.target.value)} title="Assign to a project member (or unassign)">
<option value="">(unassign)</option>
{members.map((m) => <option key={m.username} value={m.username}>{m.username}</option>)}
</select>
<button className="btn" disabled={busy || selected.size === 0 || (assignee !== "" && members.length === 0)} onClick={assignSelected}>Assign</button>
<select value={rtAssign} onChange={(e) => setRtAssign(e.target.value)} title="Set the road type of the selected videos (or clear it)">
<option value="">(clear road type)</option>
{roadTypes.map((r) => <option key={r.name} value={r.name}>{r.name}</option>)}
</select>
<button className="btn" disabled={busy || selected.size === 0} onClick={applyRoadType}
title={rtAssign ? "Apply the picked road type to the selected videos" : "Remove the road type from the selected videos"}>
{rtAssign ? "Set road" : "Clear road"}
</button>
</span>
)}
{projectId != null && videos.some((v) => v.annotation_count > 0) && (
<a className="btn" href={exportProjectUrl(projectId)} download title="Download all annotations in this project as one JSON">
Export all
</a>
)}
</div>
</div>
{videoView === "map" && projectId != null && (
<MapView projectId={projectId} isAdmin={isAdmin} onChanged={() => loadProject(projectId)} />
)}
{videoView === "table" && (
<div className="table-scroll">
<table>
<thead>
<tr>
{isAdmin && <th><input type="checkbox" title="select all"
checked={selected.size > 0 && selected.size === videos.length}
onChange={(e) => setSelected(e.target.checked ? new Set(videos.map((v) => v.id)) : new Set())} /></th>}
<th>#</th><th>Folder</th><th>File</th><th>Status</th><th>Road · Chainage</th><th>Annotator</th><th>Annotations</th>
<th>Review</th><th>Verify time</th><th>Resolution</th><th>Annotated</th><th>Remarks</th>
</tr>
</thead>
<tbody>
{videos.map((v, idx) => (
<tr key={v.id} className={`${selected.has(v.id) ? "row-sel " : ""}${v.ignored ? "row-ignored" : ""}`}>
{isAdmin && <td><input type="checkbox" checked={selected.has(v.id)} onChange={() => toggleSel(v.id)} /></td>}
<td className="dim">{idx + 1}</td>
<td className="dim mono" title={v.rel_path}>{subfolder(v.rel_path) || "—"}</td>
<td className="mono">{v.file_name}</td>
<td>
<span className={`badge ${v.workflow_status}`}>{statusLabel(v.workflow_status)}</span>
{v.ignored && <span className="badge ignored-tag" title={v.ignored_by ? `ignored by ${v.ignored_by}` : "ignored"}>ignored</span>}
{isAdmin && (
<button className="ignore-btn" onClick={() => toggleIgnore(v)}
title={v.ignored ? "Restore this video" : "Ignore this video (cross it out)"}>
{v.ignored ? "undo ignore" : "ignore"}
</button>
)}
{v.claimed_by ? (
<div className="claim-line" title={v.lease_expires_at ? `lease expires ${fmtWhen(v.lease_expires_at)}` : undefined}>
🔒 claimed by <b>{v.claimed_by}</b>{v.claimed_at ? ` · ${fmtWhen(v.claimed_at)}` : ""}
</div>
) : v.workflow_status === "assigned" && v.assigned_to ? (
<div className="claim-line"> assigned to <b>{v.assigned_to}</b></div>
) : v.completed_at ? (
<div className="claim-line" title={`last push ${fmtWhen(v.completed_at)}`}> <b>{v.verifiers || v.completed_by}</b> · {fmtWhen(v.completed_at)}</div>
) : null}
</td>
<td>
{v.road_type ? <span className="badge road-tag">{v.road_type}</span> : <span className="dim"></span>}
<div className="claim-line" title={
v.sr_chainage_start_m != null
? `sr (GPS-measured): ${fmtChainageRange(v.sr_chainage_start_m, v.sr_chainage_end_m)}`
: v.gps_status === "ok" ? "GPS scanned — chainage not calibrated yet"
: v.gps_status === "error" ? `GPS scan failed: ${v.gps_error ?? ""}`
: v.gps_status === "none" ? "no GPS metadata in this video"
: "GPS scan pending"
}>
{v.chainage_start_m != null
? fmtChainageRange(v.chainage_start_m, v.chainage_end_m)
: <span className="dim">{v.gps_status === "ok" ? "gps ✓" : v.gps_status === "pending" ? "gps …" : `gps ${v.gps_status}`}</span>}
</div>
<GpsPoints v={v} />
</td>
<td>{v.verifiers || v.primary_annotator || (v.assigned_to ? <span className="dim">assigned: {v.assigned_to}</span> : "—")}</td>
<td>
{v.annotation_count > 0
? <a href={exportVideoUrl(v.id)} download title="Download this video's annotations (JSON)">{v.annotation_count} </a>
: 0}
{(v.imported_count > 0 || (v.passes?.length ?? 0) > 0) && (() => {
const deltas = passDeltas(v.imported_count, v.passes ?? []);
return (
<div className="imported-line" title="imported baseline → per-pass changes (verify / re-verify)">
<span className="dim">imported {v.imported_count}</span>
{deltas.length > 0
? deltas.map((d, i) => (
<span key={i} className={d.delta >= 0 ? "ann-add" : "ann-del"}
title={`${d.label} by ${d.username}${d.total} total`}>
{" · "}{d.label} {d.delta >= 0 ? "+" : ""}{d.delta}
</span>
))
: <>
{v.annotation_count < v.imported_count &&
<span className="ann-del" title={`${v.imported_count - v.annotation_count} deleted by the user`}> · {v.imported_count - v.annotation_count}</span>}
{v.annotation_count > v.imported_count &&
<span className="ann-add" title={`${v.annotation_count - v.imported_count} added by the user`}> · +{v.annotation_count - v.imported_count}</span>}
</>}
</div>
);
})()}
</td>
<td>{v.review_count > 0
? <span className="review-flag" title={`${v.review_count} annotation(s) flagged for review`}>🚩 {v.review_count}</span>
: <span className="dim"></span>}</td>
<td title="total across all passes">{fmtDuration(v.verify_time_ms)}</td>
<td className="dim">{v.width && v.height ? `${v.width}×${v.height}` : "—"}</td>
<td className="dim">{v.annotated_at ? v.annotated_at.slice(0, 10) : "—"}</td>
<td className="remark-col">
<RemarkCell
videoId={v.id}
initial={v.remarks}
handRaised={v.hand_raised}
handBy={v.hand_raised_by}
/>
</td>
</tr>
))}
{videos.length === 0 && <tr><td colSpan={isAdmin ? 13 : 12} className="dim">no videos ingested</td></tr>}
</tbody>
</table>
</div>
)}
</section>
{/* Chainage calibration + road-type labels (admin) */}
{isAdmin && projectId != null && (
<ChainagePanel
projectId={projectId}
videos={videos}
roadTypes={roadTypes}
onChanged={() => loadProject(projectId)}
/>
)}
</>
)}
</div>
);
}
/** The subfolder portion of a rel_path (everything before the last '/'), or "" at the root. */
function subfolder(rel: string): string {
const i = rel.lastIndexOf("/");
return i >= 0 ? rel.slice(0, i) : "";
}
/** Group projects under their client name, preserving the server's ordering. */
function groupByClient(projects: ProjectSummary[]): [string, ProjectSummary[]][] {
const order: string[] = [];
const map = new Map<string, ProjectSummary[]>();
for (const p of projects) {
const key = p.client_name || "—";
if (!map.has(key)) { map.set(key, []); order.push(key); }
map.get(key)!.push(p);
}
return order.map((k) => [k, map.get(k)!]);
}
/** GPS fix count vs the ~1 fix/s expected for the video's duration. A mismatch gets
* a ⚠ that expands (on click) into a plain-language explanation of how the missing/
* extra data is handled — interpolation, start-stitching, end clamping. */
function GpsPoints({ v }: { v: VideoRow }) {
const [open, setOpen] = useState(false);
if (v.gps_status !== "ok" || v.gps_sample_count == null) return null;
const durS = v.duration_s ?? (v.frame_count && v.fps ? v.frame_count / v.fps : null);
const expected = durS ? Math.round(durS) : null;
const diff = expected != null ? v.gps_sample_count - expected : 0;
const off = expected != null && Math.abs(diff) > Math.max(3, expected * 0.05);
const bigGap = (v.gps_max_gap_s ?? 0) > 5;
return (
<div className="claim-line gps-pts">
{v.gps_sample_count} gps pts
{expected != null && <span className="dim"> / ~{expected}</span>}
{(off || bigGap) && (
<button
className="gps-warn"
onClick={() => setOpen((o) => !o)}
title="Fix count doesn't match the video duration — click to see how the gaps are handled"
>
</button>
)}
{open && (off || bigGap) && (
<div className="gps-explain" onClick={() => setOpen(false)}>
{diff < 0 || (!off && bigGap) ? (
<>
Expected ~{expected ?? "?"} fixes (1 per second of video), found {v.gps_sample_count}
{bigGap && <> largest dropout <b>{v.gps_max_gap_s!.toFixed(0)} s</b></>}.
Positions inside a dropout are <b>time-interpolated</b>: each frame is placed
proportionally on a straight line between the two nearest real fixes (constant
speed assumed), and its chainage follows the same interpolation. A missing{" "}
<b>start</b> is stitched from the previous video via GPS timestamps when possible;
a missing <b>end</b> clamps to the last fix. The longer the dropout, the rougher
the approximation.
</>
) : diff > 0 ? (
<>
Found {v.gps_sample_count} fixes but only ~{expected} expected the camera wrote
more than one fix per second (or duplicated samples). Extra fixes are used as-is;
nothing is approximated.
</>
) : null}
</div>
)}
</div>
);
}
function Card({ label, value, tone }: { label: string; value: number; tone?: string }) {
return (
<div className={`card ${tone ?? ""}`}>
<div className="card-label">{label}</div>
<div className="card-value">{value}</div>
</div>
);
}

View File

@@ -0,0 +1,697 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { unzipSync } from "fflate";
import {
api, fmtChainage, fmtChainageRange, fmtWhen,
type CalibrationRow, type CalibrationSummary, type ChainagePoint, type RoadType, type VideoRow,
} from "./api";
/** The import format clients should follow: one Point placemark per survey point,
* chainage in the name ("12+400" = 12 km + 400 m, or decimal km). */
const SAMPLE_KML = `<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2"><Document>
<name>chainage points — sample</name>
<!-- One Placemark per survey point.
name = the chainage at that point: "12+400" (12 km + 400 m) or "12.4" (km)
coordinates = lon,lat[,alt] (KML order: longitude first!)
The dashboard reads the name (or description) and the Point coordinates.
LineStrings/paths are ignored — only Point placemarks import. -->
<Placemark><name>0+000</name><Point><coordinates>78.185718,21.105570,0</coordinates></Point></Placemark>
<Placemark><name>5+000</name><Point><coordinates>78.158000,21.092045,0</coordinates></Point></Placemark>
<Placemark><name>10+000</name><Point><coordinates>78.123885,21.091447,0</coordinates></Point></Placemark>
</Document></kml>
`;
function downloadSampleKml() {
const url = URL.createObjectURL(new Blob([SAMPLE_KML], { type: "application/vnd.google-earth.kml+xml" }));
const a = document.createElement("a");
a.href = url;
a.download = "chainage_points_sample.kml";
a.click();
URL.revokeObjectURL(url);
}
/** One placemark parsed out of an uploaded KML, editable before import. */
interface KmlRow {
name: string;
lat: number;
lon: number;
/** Editable chainage in km (prefilled from the placemark name/description). */
km: string;
include: boolean;
}
/** Chainage from free text: road notation "12+400" (km+m) always wins; otherwise the
* first number, interpreted in `unit`. Returns meters, or null when nothing parses. */
function parseChainageText(text: string, unit: "km" | "m"): number | null {
const plus = text.match(/(\d+)\s*\+\s*(\d+(?:\.\d+)?)/);
if (plus) return parseInt(plus[1], 10) * 1000 + parseFloat(plus[2]);
const num = text.match(/-?\d+(?:[.,]\d+)?/);
if (!num) return null;
const v = parseFloat(num[0].replace(",", "."));
if (isNaN(v)) return null;
return unit === "km" ? v * 1000 : v;
}
/** Extract Point placemarks (name + lat/lon) from a KML document. LineStrings and
* multi-coordinate placemarks are skipped (counted) — only survey points import. */
function parseKmlPoints(text: string): { rows: { name: string; lat: number; lon: number }[]; skipped: number } {
const doc = new DOMParser().parseFromString(text, "application/xml");
if (doc.getElementsByTagName("parsererror").length > 0) {
throw new Error("not a valid KML/XML file (KMZ must be unzipped first)");
}
const rows: { name: string; lat: number; lon: number }[] = [];
let skipped = 0;
for (const pm of Array.from(doc.getElementsByTagNameNS("*", "Placemark"))) {
const coords = pm.getElementsByTagNameNS("*", "coordinates")[0]?.textContent?.trim() ?? "";
const tuples = coords.split(/\s+/).filter(Boolean);
if (tuples.length !== 1) {
skipped += 1; // a LineString/Polygon (or empty) — not a survey point
continue;
}
const [lon, lat] = tuples[0].split(",").map(parseFloat);
if (isNaN(lat) || isNaN(lon)) {
skipped += 1;
continue;
}
const name = pm.getElementsByTagNameNS("*", "name")[0]?.textContent?.trim()
|| pm.getElementsByTagNameNS("*", "description")[0]?.textContent?.trim()
|| "";
rows.push({ name, lat, lon });
}
return { rows, skipped };
}
/**
* Admin chainage panel: manage road-type labels, enter the client's calibration
* points (lat/lon ↔ chainage), run the two-point quick assign ("first video starts
* at X, last ends at Y") or a full recompute over the current group, and read the
* resulting per-span scales + per-video chainage. `sr` (GPS-measured) chainage is
* shown but never editable — only the client-calibrated values come from points.
*/
export function ChainagePanel({
projectId,
videos,
roadTypes,
onChanged,
}: {
projectId: number;
videos: VideoRow[];
roadTypes: RoadType[];
onChanged: () => void;
}) {
const [open, setOpen] = useState(false);
const [group, setGroup] = useState(""); // road_type calibration group ('' = all videos)
const [points, setPoints] = useState<ChainagePoint[]>([]);
const [calibs, setCalibs] = useState<CalibrationRow[]>([]);
const [calibsErr, setCalibsErr] = useState<string | null>(null);
const [summary, setSummary] = useState<CalibrationSummary | null>(null);
// true = loaded from the server (someone's earlier compute); false = just run here.
const [summaryIsSaved, setSummaryIsSaved] = useState(false);
const [busy, setBusy] = useState(false);
const [error, setError] = useState<string | null>(null);
// add-point form
const [latlon, setLatlon] = useState("");
const [pointKm, setPointKm] = useState("");
const [pointNote, setPointNote] = useState("");
// inline note edit on an existing point
const [noteEdit, setNoteEdit] = useState<{ id: number; text: string } | null>(null);
// quick two-point form
const [startKm, setStartKm] = useState("0");
const [endKm, setEndKm] = useState("");
// road-type editor
const [newType, setNewType] = useState("");
// KML import preview
const [kmlRows, setKmlRows] = useState<KmlRow[] | null>(null);
const [kmlUnit, setKmlUnit] = useState<"km" | "m">("km");
const [kmlSkipped, setKmlSkipped] = useState(0);
const kmlFileRef = useRef<HTMLInputElement>(null);
const loadPoints = useCallback(async () => {
try {
setPoints(await api.chainagePoints(projectId));
} catch (e) {
setError(String(e));
}
// Calibration list failing must not hide the points editor — but it must not
// fail silently either (a broken endpoint/auth issue should be visible).
try {
setCalibs(await api.chainageCalibrations(projectId));
setCalibsErr(null);
} catch (e) {
setCalibs([]);
setCalibsErr(String(e));
}
}, [projectId]);
useEffect(() => {
if (open) void loadPoints();
}, [open, loadPoints]);
// On open / group change, show the SAVED result of the last recompute (persisted
// server-side) so a freshly signed-in admin sees the existing compute for LHS/RHS
// instead of an empty panel asking to recompute.
useEffect(() => {
setSummary(null);
setSummaryIsSaved(false);
if (!open) return;
let alive = true;
api.chainageSummary(projectId, group)
.then((s) => {
if (alive && s && typeof s === "object") {
setSummary(s);
setSummaryIsSaved(true);
}
})
.catch(() => {}); // calibrations block surfaces endpoint failures already
return () => {
alive = false;
};
}, [projectId, group, open]);
/** The route's videos = current group, ignored ones excluded. */
const groupVideos = useMemo(
() => videos.filter((v) => !v.ignored && (group === "" || v.road_type === group)),
[videos, group],
);
const groupPoints = useMemo(
() => points.filter((p) => p.road_type === group || p.road_type === ""),
[points, group],
);
// Suggest quick-assign values from the group's own videos: an existing
// calibration's range when one exists, else 0 → the summed GPS length rounded to
// the nearest whole km. Re-suggested when the group changes (still editable).
useEffect(() => {
if (!open) return;
const ch = groupVideos
.flatMap((v) => [v.chainage_start_m, v.chainage_end_m])
.filter((x): x is number => x != null);
if (ch.length > 0) {
setStartKm((Math.min(...ch) / 1000).toFixed(4));
setEndKm((Math.max(...ch) / 1000).toFixed(4));
} else {
const len = groupVideos.reduce((s, v) => s + (v.gps_len_m ?? 0), 0);
setStartKm("0");
setEndKm(len > 0 ? String(Math.max(1, Math.round(len / 1000))) : "");
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [open, group]);
const run = async (fn: () => Promise<CalibrationSummary>) => {
setBusy(true);
setError(null);
try {
setSummary(await fn());
setSummaryIsSaved(false);
await loadPoints();
onChanged();
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
};
const addPoint = async () => {
const m = latlon.trim().match(/^\s*(-?\d+(?:\.\d+)?)\s*[,;\s]\s*(-?\d+(?:\.\d+)?)\s*$/);
const km = parseFloat(pointKm);
if (!m || isNaN(km)) {
setError('Point needs "lat, lon" and a chainage in km.');
return;
}
setBusy(true);
setError(null);
try {
await api.addChainagePoint(projectId, {
lat: parseFloat(m[1]),
lon: parseFloat(m[2]),
chainage_m: km * 1000,
note: pointNote.trim(),
road_type: group,
});
setLatlon("");
setPointKm("");
setPointNote("");
await loadPoints();
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
};
const deletePoint = async (id: number) => {
setBusy(true);
try {
await api.deleteChainagePoint(id);
await loadPoints();
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
};
/** Persist an inline note edit (Enter or blur). */
const saveNote = async () => {
if (!noteEdit) return;
const { id, text } = noteEdit;
setNoteEdit(null);
try {
await api.updateChainagePointNote(id, text);
await loadPoints();
} catch (e) {
setError(String(e));
}
};
const quick = () => {
const s = parseFloat(startKm);
const e = parseFloat(endKm);
if (isNaN(s) || isNaN(e)) {
setError("Quick assign needs start and end chainage in km.");
return;
}
void run(() =>
api.chainageQuick(projectId, {
video_ids: groupVideos.map((v) => v.id),
start_m: s * 1000,
end_m: e * 1000,
road_type: group,
}),
);
};
const recompute = () =>
run(() => api.chainageRecompute(projectId, groupVideos.map((v) => v.id), group));
const addRoadType = async () => {
const name = newType.trim();
if (!name) return;
setBusy(true);
setError(null);
try {
await api.setRoadTypes(projectId, [...roadTypes.map((r) => r.name), name]);
setNewType("");
onChanged();
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
};
const onKmlFile = async (file: File) => {
setError(null);
try {
let text: string;
if (/\.kmz$/i.test(file.name)) {
// KMZ = zipped KML; take the first .kml entry (usually doc.kml).
const entries = unzipSync(new Uint8Array(await file.arrayBuffer()));
const key = Object.keys(entries).find((k) => k.toLowerCase() === "doc.kml")
?? Object.keys(entries).find((k) => /\.kml$/i.test(k));
if (!key) {
setError(`${file.name} contains no .kml document.`);
return;
}
text = new TextDecoder().decode(entries[key]);
} else {
text = await file.text();
}
const { rows, skipped } = parseKmlPoints(text);
if (rows.length === 0) {
setError(`No point placemarks found in ${file.name}${skipped ? ` (${skipped} non-point entries skipped)` : ""}.`);
return;
}
setKmlSkipped(skipped);
setKmlRows(rows.map((r) => {
const m = parseChainageText(r.name, kmlUnit);
return { ...r, km: m != null ? (m / 1000).toFixed(4) : "", include: m != null };
}));
} catch (e) {
setError(String(e));
} finally {
if (kmlFileRef.current) kmlFileRef.current.value = ""; // allow re-picking the same file
}
};
/** Re-read every row's chainage from its placemark name under the new unit. */
const setKmlUnitAndReparse = (u: "km" | "m") => {
setKmlUnit(u);
setKmlRows((rows) => rows?.map((r) => {
const m = parseChainageText(r.name, u);
return { ...r, km: m != null ? (m / 1000).toFixed(4) : "", include: m != null };
}) ?? null);
};
const importKml = async () => {
if (!kmlRows) return;
const points = kmlRows
.filter((r) => r.include && r.km.trim() !== "" && !isNaN(parseFloat(r.km)))
.map((r) => ({
lat: r.lat,
lon: r.lon,
chainage_m: parseFloat(r.km) * 1000,
note: r.name,
road_type: group,
}));
if (points.length === 0) {
setError("No rows selected with a valid chainage.");
return;
}
setBusy(true);
setError(null);
try {
await api.addChainagePointsBulk(projectId, points);
setKmlRows(null);
await loadPoints();
onChanged();
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
};
const removeRoadType = async (name: string) => {
setBusy(true);
setError(null);
try {
await api.setRoadTypes(projectId, roadTypes.map((r) => r.name).filter((n) => n !== name));
if (group === name) setGroup("");
onChanged();
} catch (e) {
setError(String(e));
} finally {
setBusy(false);
}
};
if (!open) {
return (
<section className="panel">
<div className="panel-head">
<h3>Chainage &amp; road types</h3>
<button className="btn" onClick={() => setOpen(true)}>Open </button>
</div>
</section>
);
}
return (
<section className="panel">
<div className="panel-head">
<h3>Chainage &amp; road types</h3>
<button className="btn" onClick={() => setOpen(false)}>Close </button>
</div>
{error && <div className="error"> {error}</div>}
{/* Road-type labels (project-wide) */}
<div className="chainage-block">
<div className="dim block-label">Road types (assign them to selected videos in the table below)</div>
<div className="member-chips">
{roadTypes.map((r) => (
<span key={r.name} className="member-chip">
{r.name} <span className="dim">({r.video_count})</span>
<button className="chip-x" disabled={busy} onClick={() => removeRoadType(r.name)} title="Remove label">×</button>
</span>
))}
<input
className="chip-input"
value={newType}
onChange={(e) => setNewType(e.target.value)}
onKeyDown={(e) => { if (e.key === "Enter") void addRoadType(); }}
placeholder="Add: LHS / RHS / MCW…"
/>
<button className="btn" disabled={busy || !newType.trim()} onClick={addRoadType}>Add</button>
</div>
</div>
{/* Calibration group + the videos it covers */}
<div className="chainage-block form-row">
<label className="dim">Calibration group</label>
<select value={group} onChange={(e) => setGroup(e.target.value)}>
<option value="">(all videos)</option>
{roadTypes.map((r) => (
<option key={r.name} value={r.name}>{r.name}</option>
))}
</select>
<span className="dim">
{groupVideos.length} video(s) in the route · {groupPoints.length} calibration point(s)
</span>
</div>
{/* Saved calibrations — persisted server-side, so every user (dashboard AND
desktop exports) sees the same computed chainage without re-running anything. */}
{(calibs.length > 0 || calibsErr) && (
<div className="chainage-block">
<div className="dim block-label">Saved calibrations (shared all users see these chainages)</div>
{calibsErr && <div className="err-tag" style={{ fontSize: 12 }}> calibrations unavailable: {calibsErr}</div>}
{calibs.map((c) => (
<div key={c.road_type || "(any)"} className="dim" style={{ fontSize: 12 }}>
<b>{c.road_type || "(ungrouped)"}</b> · route {fmtChainage(c.route_len_m)} km
· {c.anchor_count} anchor(s) · computed by {c.computed_by} · {fmtWhen(c.computed_at)}
{!c.has_summary && (
<span className="err-tag" title="This calibration predates saved results — select its group and hit 'Recompute from points' once (same points ⇒ same values); after that every admin sees the full result here.">
{" "}· result not saved yet recompute once
</span>
)}
</div>
))}
</div>
)}
<div className="grid2">
{/* Control points */}
<div className="chainage-block">
<div className="dim block-label">Client calibration points (lat/lon chainage)</div>
<table>
<thead>
<tr><th>Chainage</th><th>Lat, Lon</th><th>Group</th><th>Note</th><th></th></tr>
</thead>
<tbody>
{groupPoints.map((p) => (
<tr key={p.id}>
<td>
{fmtChainage(p.chainage_m)}
{p.origin === "quick" && (
<span className="dim" title="Generated by the two-point quick assign — replaced automatically on the next quick run"> · auto</span>
)}
</td>
<td className="mono">{p.lat.toFixed(6)}, {p.lon.toFixed(6)}</td>
<td className="dim">{p.road_type || "any"}</td>
<td
className="dim"
style={{ cursor: "pointer" }}
title="Click to edit — e.g. 'start point', 'toll plaza', a survey remark"
onClick={() => { if (noteEdit?.id !== p.id) setNoteEdit({ id: p.id, text: p.note }); }}
>
{noteEdit?.id === p.id ? (
<input
autoFocus
value={noteEdit.text}
style={{ width: 140 }}
onChange={(e) => setNoteEdit({ id: p.id, text: e.target.value })}
onKeyDown={(e) => {
if (e.key === "Enter") void saveNote();
if (e.key === "Escape") setNoteEdit(null);
}}
onBlur={() => void saveNote()}
/>
) : (
p.note || "— (click to add)"
)}
</td>
<td><button className="chip-x" disabled={busy} onClick={() => deletePoint(p.id)} title="Delete point">×</button></td>
</tr>
))}
{groupPoints.length === 0 && (
<tr><td colSpan={5} className="dim">no points yet add them, or use the quick assign</td></tr>
)}
</tbody>
</table>
<div className="form-row">
<input value={latlon} onChange={(e) => setLatlon(e.target.value)} placeholder="lat, lon" style={{ width: 190 }} />
<input value={pointKm} onChange={(e) => setPointKm(e.target.value)} placeholder="chainage (km)" style={{ width: 110 }} />
<input value={pointNote} onChange={(e) => setPointNote(e.target.value)} placeholder="note (optional)" style={{ width: 140 }} />
<button className="btn" disabled={busy} onClick={addPoint}>Add point</button>
<input
ref={kmlFileRef}
type="file"
accept=".kml,.kmz,.xml"
style={{ display: "none" }}
onChange={(e) => { const f = e.target.files?.[0]; if (f) void onKmlFile(f); }}
/>
<button className="btn" disabled={busy} onClick={() => kmlFileRef.current?.click()}
title="Upload a KML/KMZ of survey points — chainage read from each placemark's name (12+400 or a number)">
Import KML/KMZ
</button>
<button className="btn" onClick={downloadSampleKml}
title="Download the template your client's points file should follow">
Sample KML
</button>
</div>
{kmlRows && (
<div className="kml-preview">
<div className="form-row">
<b>{kmlRows.filter((r) => r.include).length}</b>
<span className="dim">of {kmlRows.length} points will import into group "{group || "any"}"
{kmlSkipped > 0 && ` · ${kmlSkipped} non-point placemark(s) skipped`}</span>
<span className="dim">· values in names are</span>
<select value={kmlUnit} onChange={(e) => setKmlUnitAndReparse(e.target.value as "km" | "m")}>
<option value="km">km</option>
<option value="m">meters</option>
</select>
<span className="dim">("12+400" always = 12 km + 400 m)</span>
</div>
<div className="kml-preview-list">
<table>
<thead>
<tr><th></th><th>Placemark</th><th>Lat, Lon</th><th>Chainage (km)</th></tr>
</thead>
<tbody>
{kmlRows.map((r, i) => (
<tr key={i} className={r.include ? undefined : "row-ignored"}>
<td><input type="checkbox" checked={r.include}
onChange={() => setKmlRows((rows) => rows!.map((x, k) => k === i ? { ...x, include: !x.include } : x))} /></td>
<td className="mono">{r.name || <span className="dim">(unnamed)</span>}</td>
<td className="mono dim">{r.lat.toFixed(6)}, {r.lon.toFixed(6)}</td>
<td><input value={r.km} style={{ width: 90 }} placeholder="?"
onChange={(e) => setKmlRows((rows) => rows!.map((x, k) => k === i ? { ...x, km: e.target.value, include: true } : x))} /></td>
</tr>
))}
</tbody>
</table>
</div>
<div className="form-row">
<button className="btn primary" disabled={busy} onClick={importKml}>
Import {kmlRows.filter((r) => r.include).length} point(s)
</button>
<button className="btn" disabled={busy} onClick={() => setKmlRows(null)}>Cancel</button>
</div>
</div>
)}
</div>
{/* Quick assign + recompute */}
<div className="chainage-block">
<div className="dim block-label">
Two-point quick assign client gives just the chainage at the route's two ends; the
error is distributed over the whole route. Values are pre-filled from the {group ? `"${group}"` : "selected"} videos.
</div>
<div className="form-row">
<label className="dim">One end</label>
<input value={startKm} onChange={(e) => setStartKm(e.target.value)} style={{ width: 90 }} /> km
<label className="dim">other end</label>
<input value={endKm} onChange={(e) => setEndKm(e.target.value)} style={{ width: 90 }} /> km
<button className="btn" disabled={busy}
title="The 0 landed on the wrong physical end? Swap the two values and re-compute."
onClick={() => { const s = startKm; setStartKm(endKm); setEndKm(s); }}>
⇄ swap
</button>
<button className="btn primary" disabled={busy || groupVideos.length === 0} onClick={quick}>
Assign &amp; compute
</button>
</div>
<div className="dim" style={{ marginTop: 6, fontSize: 12 }}>
The two auto points ("quick: chain start/end") are saved into group <b>{group || "any"}</b>.
After computing, check the km stones on the map — if 0 sits at the wrong end of the road,
hit ⇄ swap and compute again. Footage direction doesn't matter (handled automatically).
</div>
<div className="form-row" style={{ marginTop: 10 }}>
<button className="btn" disabled={busy || groupVideos.length === 0} onClick={recompute}
title="Re-run the calibration with the points above">
Recompute from points
</button>
<span className="dim">orders the {groupVideos.length} video(s) into a route, snaps the points, distributes the error</span>
</div>
</div>
</div>
{/* Result summary — freshly run, or the saved result of the last recompute */}
{summary && (
<div className="chainage-block">
{summaryIsSaved && (() => {
const c = calibs.find((x) => x.road_type === group);
return (
<div className="dim" style={{ fontSize: 12, marginBottom: 4 }}>
Saved result of the last compute
{c ? <> by <b>{c.computed_by}</b> · {fmtWhen(c.computed_at)}</> : null}
{" "}(recompute only if points or videos changed)
</div>
);
})()}
<div className="block-label">
Route <b>{(summary.route_len_m / 1000).toFixed(3)} km</b> (GPS-measured)
· {summary.points_used} point(s) used
{summary.points_excluded > 0 && <span className="err-tag"> · {summary.points_excluded} excluded</span>}
{summary.reversed && <span className="dim"> · direction auto-reversed</span>}
{summary.majority_flipped && (
<span className="dim" title="Most videos here were filmed driving toward decreasing chainage — that's this route's normal pattern, so only videos running AGAINST it are flagged below.">
{" "}· footage mostly runs high low chainage
</span>
)}
{summary.videos.length > 1 && (() => {
const ag = summary.videos.filter((v) => v.against_flow).length;
return (
<span className={ag > 0 ? "err-tag" : "dim"}
title={ag > 0 ? "Videos running against the route's dominant direction — on a divided road this often means the video belongs to the other carriageway (wrong group)." : "Every video runs in the route's dominant direction."}>
{" "}· {summary.videos.length - ag}/{summary.videos.length} follow the dominant direction{ag > 0 ? `, ${ag} against` : ""}
</span>
);
})()}
</div>
{summary.warnings.length > 0 && (
<ul className="chainage-warnings">
{summary.warnings.map((w, i) => <li key={i}> {w}</li>)}
</ul>
)}
{summary.spans.length > 0 && (
<table>
<thead>
<tr><th>Span (client)</th><th>GPS length</th><th>Client length</th><th>Scale</th></tr>
</thead>
<tbody>
{summary.spans.map((sp, i) => (
<tr key={i} className={Math.abs(sp.scale - 1) > 0.05 ? "span-warn" : undefined}>
<td>{fmtChainage(sp.from_chainage_m)} {fmtChainage(sp.to_chainage_m)}</td>
<td>{(sp.gps_len_m / 1000).toFixed(3)} km</td>
<td>{(sp.client_len_m / 1000).toFixed(3)} km</td>
<td>×{sp.scale.toFixed(4)}</td>
</tr>
))}
</tbody>
</table>
)}
<table style={{ marginTop: 8 }}>
<thead>
<tr><th>#</th><th>Video</th><th>Chainage (client)</th><th>SR chainage (GPS)</th><th></th></tr>
</thead>
<tbody>
{summary.videos.map((v) => (
<tr key={v.id}>
<td className="dim">{v.seq + 1}</td>
<td className="mono">{v.file_name}</td>
<td>{fmtChainageRange(v.chainage_start_m, v.chainage_end_m)}</td>
<td className="dim">{fmtChainageRange(v.sr_start_m, v.sr_end_m)}</td>
<td className="dim">{v.against_flow && (
<span className="err-tag" title="This video's travel direction is OPPOSITE to most of the route. Chainage is still computed correctly — but if it's unexpected, the video may belong to the other carriageway (wrong group).">
against flow
</span>
)}</td>
</tr>
))}
</tbody>
</table>
{summary.skipped.length > 0 && (
<div className="dim" style={{ marginTop: 6 }}>
Skipped (no GPS): {summary.skipped.join(", ")}
</div>
)}
</div>
)}
</section>
);
}

View File

@@ -0,0 +1,78 @@
import { useCallback, useEffect, useRef, useState } from "react";
import { api, type Folder } from "./api";
/**
* Dropdown checklist of the project's folders. Ticking folders restricts the videos
* shown/counted to those folders (admins verify only some folders; the rest are
* backup/misc). None ticked = consider all folders. Admin only.
*/
export function FoldersPicker({ projectId, onChange }: { projectId: number; onChange: () => void }) {
const [open, setOpen] = useState(false);
const [folders, setFolders] = useState<Folder[]>([]);
const [busy, setBusy] = useState(false);
const [err, setErr] = useState<string | null>(null);
const ref = useRef<HTMLDivElement>(null);
const load = useCallback(async () => {
try { setFolders(await api.folders(projectId)); } catch (e) { setErr(String(e)); }
}, [projectId]);
useEffect(() => { if (open) void load(); }, [open, load]);
useEffect(() => {
if (!open) return;
const h = (e: MouseEvent) => { if (ref.current && !ref.current.contains(e.target as Node)) setOpen(false); };
document.addEventListener("mousedown", h);
return () => document.removeEventListener("mousedown", h);
}, [open]);
const includedCount = folders.filter((f) => f.included).length;
const commit = async (next: Folder[]) => {
setFolders(next);
setBusy(true);
setErr(null);
try {
await api.setFolders(projectId, next.filter((f) => f.included).map((f) => f.folder));
onChange();
} catch (e) {
setErr(String(e));
await load();
} finally {
setBusy(false);
}
};
const toggle = (folder: string) =>
commit(folders.map((f) => (f.folder === folder ? { ...f, included: !f.included } : f)));
const considerAll = () => commit(folders.map((f) => ({ ...f, included: false })));
const label = includedCount === 0 ? "Folders: all ▾" : `Folders: ${includedCount}/${folders.length}`;
return (
<div className="members-picker" ref={ref}>
<button className="btn" onClick={() => setOpen((v) => !v)} title="Pick which folders to verify">{label}</button>
{open && (
<div className="members-dropdown">
<div className="folders-hint dim">Tick folders to verify. None ticked = all folders shown.</div>
<div className="members-list">
{folders.map((f) => (
<label key={f.folder} className={`members-item${f.included ? " on" : ""}`}>
<input type="checkbox" checked={f.included} disabled={busy} onChange={() => toggle(f.folder)} />
<span className="mi-name">{f.folder || "(root)"}</span>
<span className="dim mi-disp">{f.video_count}</span>
</label>
))}
{folders.length === 0 && <div className="dim mi-empty">no folders sync the project first</div>}
</div>
{err && <div className="mi-err"> {err}</div>}
<div className="members-foot">
<button className="btn" disabled={busy || includedCount === 0} onClick={considerAll}>Consider all</button>
<span className="dim" style={{ marginLeft: 8 }}>
{includedCount === 0 ? "all folders shown" : `${includedCount} selected`}
</span>
</div>
</div>
)}
</div>
);
}

1133
dashboard/src/MapView.tsx Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,88 @@
import { useEffect, useRef, useState } from "react";
import { api, type Member, type UserRow } from "./api";
/**
* Dropdown checklist to pick which ml_support users belong to a project. Searchable
* + scrollable (built for ~30 users). Ticking a user adds them as a project member;
* un-ticking removes them. Members are who can see the project and be assigned videos.
* Admin only. By default no one is selected.
*/
export function MembersPicker({
projectId,
users,
members,
onChange,
}: {
projectId: number;
users: UserRow[]; // all ml_support users
members: Member[];
onChange: () => void; // reload the project's members after a change
}) {
const [open, setOpen] = useState(false);
const [q, setQ] = useState("");
const [busy, setBusy] = useState<string | null>(null);
const [err, setErr] = useState<string | null>(null);
const ref = useRef<HTMLDivElement>(null);
const memberSet = new Set(members.map((m) => m.username));
useEffect(() => {
if (!open) return;
const h = (e: MouseEvent) => { if (ref.current && !ref.current.contains(e.target as Node)) setOpen(false); };
document.addEventListener("mousedown", h);
return () => document.removeEventListener("mousedown", h);
}, [open]);
const toggle = async (username: string) => {
setBusy(username);
setErr(null);
try {
if (memberSet.has(username)) await api.removeMember(projectId, username);
else await api.addMember(projectId, username);
onChange();
} catch (e) {
setErr(String(e));
} finally {
setBusy(null);
}
};
const filtered = users.filter((u) => u.username.toLowerCase().includes(q.trim().toLowerCase()));
return (
<div className="members-picker" ref={ref}>
<button className="btn" onClick={() => setOpen((v) => !v)}>
{members.length === 0 ? "+ Assign users" : `Members (${members.length}) ▾`}
</button>
{open && (
<div className="members-dropdown">
<input
className="members-search"
placeholder="search ml_support…"
value={q}
onChange={(e) => setQ(e.target.value)}
autoFocus
/>
<div className="members-list">
{filtered.map((u) => (
<label key={u.id} className={`members-item${memberSet.has(u.username) ? " on" : ""}`}>
<input
type="checkbox"
checked={memberSet.has(u.username)}
disabled={busy === u.username}
onChange={() => toggle(u.username)}
/>
<span className="mi-name">{u.username}</span>
{u.display_name && u.display_name !== u.username && <span className="dim mi-disp">{u.display_name}</span>}
</label>
))}
{users.length === 0 && <div className="dim mi-empty">no ml_support users yet create some in Admin</div>}
{users.length > 0 && filtered.length === 0 && <div className="dim mi-empty">no matches</div>}
</div>
{err && <div className="mi-err"> {err}</div>}
<div className="members-foot dim">{members.length} selected · ticked users can be assigned videos</div>
</div>
)}
</div>
);
}

View File

@@ -0,0 +1,110 @@
import { useState } from "react";
import { api, type Remark } from "./api";
/**
* Remark + raise-hand cell for the dashboard video table. Collapsed shows just a
* count (💬 N) so it never overflows the column; expand for the full thread + an add
* box. The ✋ raise-hand toggle lets anyone start a conversation on a video; anyone
* (user or admin) can lower it. Both use the signed-in token automatically.
*/
export function RemarkCell({
videoId,
initial,
handRaised,
handBy,
}: {
videoId: number;
initial: Remark[];
handRaised: boolean;
handBy: string | null;
}) {
const [thread, setThread] = useState<Remark[]>(initial ?? []);
const [raised, setRaised] = useState(handRaised);
const [by, setBy] = useState(handBy);
const [open, setOpen] = useState(false);
const [draft, setDraft] = useState("");
const [busy, setBusy] = useState(false);
const [err, setErr] = useState<string | null>(null);
const add = async () => {
const body = draft.trim();
if (!body) return;
setBusy(true);
setErr(null);
try {
setThread(await api.addRemark(videoId, body));
setDraft("");
} catch (e) {
setErr(String(e));
} finally {
setBusy(false);
}
};
const toggleHand = async () => {
const next = !raised;
setBusy(true);
setErr(null);
try {
const r = await api.setHand(videoId, next);
setRaised(r.hand_raised);
setBy(r.hand_raised ? r.by : null);
} catch (e) {
setErr(String(e));
} finally {
setBusy(false);
}
};
const handBtn = (
<button
className={`hand-btn${raised ? " raised" : ""}`}
disabled={busy}
onClick={toggleHand}
title={raised ? `✋ raised by ${by ?? "?"} — click to lower` : "Raise hand to start a conversation"}
>
</button>
);
if (!open) {
return (
<div className="remark-cell">
{handBtn}
<button className="remark-collapsed" onClick={() => setOpen(true)} title="View / add remarks">
{thread.length === 0
? <span className="dim"> add remark</span>
: <span className="remark-badge">💬 {thread.length}</span>}
</button>
</div>
);
}
return (
<div className="remark-open">
<div className="remark-open-head">
{handBtn}
{raised && <span className="hand-note"> {by}</span>}
<span style={{ flex: 1 }} />
<button className="btn" onClick={() => setOpen(false)}>Close</button>
</div>
<div className="remark-thread">
{thread.length === 0 && <div className="dim">No remarks yet.</div>}
{thread.map((r, i) => (
<div key={i} className="remark-line"><b>{r.username}:</b> {r.body}</div>
))}
</div>
<div className="remark-add">
<input
value={draft}
onChange={(e) => setDraft(e.target.value)}
onKeyDown={(e) => { if (e.key === "Enter") add(); }}
placeholder="Add a remark…"
autoFocus
/>
<button className="btn primary" disabled={busy || !draft.trim()} onClick={add}>{busy ? "…" : "Add"}</button>
</div>
{err && <div className="dim" style={{ color: "#ff8b8b" }}> {err}</div>}
</div>
);
}

56
dashboard/src/SignIn.tsx Normal file
View File

@@ -0,0 +1,56 @@
import { useState } from "react";
import { api } from "./api";
/**
* Full-screen sign-in gate. No data is shown until a valid token is entered — this
* applies to both admins and ml_support. `reason` shows why a previous session ended
* (e.g. expired due to inactivity).
*/
export function SignIn({
onSignedIn,
reason,
}: {
onSignedIn: (token: string, auth: { username: string; role: string }) => void;
reason?: string | null;
}) {
const [token, setToken] = useState("");
const [busy, setBusy] = useState(false);
const [err, setErr] = useState<string | null>(null);
const signIn = async () => {
const t = token.trim();
if (!t) return;
setBusy(true);
setErr(null);
try {
const auth = await api.login(t); // validates + records a signed_in audit event
onSignedIn(t, auth);
} catch (e) {
setErr(String(e));
} finally {
setBusy(false);
}
};
return (
<div className="signin-screen">
<div className="signin-card">
<h1>Verification Dashboard</h1>
<p className="dim">Sign in with your access token to continue.</p>
{reason && <div className="signin-reason"> {reason}</div>}
<input
type="password"
placeholder="paste your access token"
value={token}
onChange={(e) => setToken(e.target.value)}
onKeyDown={(e) => { if (e.key === "Enter") signIn(); }}
autoFocus
/>
<button className="btn primary" disabled={busy || !token.trim()} onClick={signIn}>
{busy ? "Signing in…" : "Sign in"}
</button>
{err && <div className="error"> {err}</div>}
</div>
</div>
);
}

545
dashboard/src/api.ts Normal file
View File

@@ -0,0 +1,545 @@
const BASE = (import.meta.env.VITE_API_BASE as string) || "/api";
export interface ProjectSummary {
id: number;
name: string;
source_path: string;
source_kind: string; // 'local' | 'nfs'
client_id: number | null;
client_name: string;
total: number;
pending: number;
annotated: number;
verified: number;
}
export interface UserRow {
id: number;
username: string;
display_name: string;
role: string;
active: boolean;
created_at: string;
}
export interface AuditRow {
at: string;
username: string;
action: string;
detail: Record<string, unknown>;
}
export interface Client {
id: number;
name: string;
created_at: string;
project_count: number;
}
export interface Remark {
username: string;
body: string;
created_at: string;
}
/** One verification pass (a push). `count` is the annotation total after that pass;
* null for legacy pushes logged before we recorded it. */
export interface Pass {
username: string;
at: string;
count: number | null;
}
export interface VideoRow {
id: number;
file_name: string;
rel_path: string;
has_json: boolean;
width: number | null;
height: number | null;
fps: number | null;
frame_count: number | null;
annotation_count: number;
imported_count: number; // baseline at claim/ingest — vs annotation_count shows deletions
annotation_time_ms: number;
primary_annotator: string;
status: string;
annotated_at: string | null;
// Phase 3 claim/verify dimension (orthogonal to status).
claimed_by: string | null;
claimed_at: string | null;
lease_expires_at: string | null;
completed_by: string | null;
completed_at: string | null;
verify_time_ms: number; // total across all passes
verify_count: number; // 0 none, 1 verified, ≥2 re-verified
verifiers: string; // distinct verifiers, e.g. "ravi, john"
review_count: number; // annotations flagged for review still open
assigned_to: string | null;
workflow_status: string; // pending|annotated|assigned|in-progress|verified|re-verified
hand_raised: boolean;
hand_raised_by: string | null;
hand_raised_at: string | null;
ignored: boolean;
ignored_by: string | null;
remarks: Remark[];
passes: Pass[]; // per-pass annotation-count history (push order)
road_type: string; // '' = unassigned
gps_status: string; // pending | ok | none | error
gps_len_m: number | null;
gps_error: string | null;
gps_sample_count: number | null; // raw fixes found (≈1/s expected)
gps_max_gap_s: number | null; // biggest dropout between fixes
duration_s: number | null; // real container duration from the scan
chainage_start_m: number | null; // client-calibrated
chainage_end_m: number | null;
sr_chainage_start_m: number | null; // GPS-measured (immutable)
sr_chainage_end_m: number | null;
route_seq: number | null;
}
/** One video as the map sees it. `track` = [[lat, lon, t_s], …] (corrected when a
* correction was kept, else raw); null until the GPS sweep has scanned it. */
export interface MapVideo {
id: number;
file_name: string;
rel_path: string;
status: string;
workflow_status: string;
ignored: boolean;
hand_raised: boolean;
assigned_to: string | null;
claimed_by: string | null;
annotation_count: number;
road_type: string;
gps_status: string;
gps_len_m: number | null;
gps_error: string | null;
chainage_start_m: number | null;
chainage_end_m: number | null;
sr_chainage_start_m: number | null;
sr_chainage_end_m: number | null;
route_seq: number | null;
remark_count: number;
track: [number, number, number][] | null;
corrected: boolean;
gps_corrected_by: string | null; // 'auto' or a username
gps_corrected_note: string | null;
gps_sample_count: number | null; // raw fixes found by the scan
gps_max_gap_s: number | null; // biggest dropout between fixes
duration_s: number | null;
gps_start_epoch: number | null; // recording window (UTC epoch of first/last fix)
gps_end_epoch: number | null;
speed_avg_mps: number | null; // over the full-resolution track
speed_max_mps: number | null;
}
export interface RoadType {
name: string;
video_count: number;
}
export interface ChainagePoint {
id: number;
lat: number;
lon: number;
chainage_m: number;
note: string;
road_type: string;
/** '' = manual; 'quick' = generated by quick assign (replaced on the next run). */
origin: string;
created_by: string;
created_at: string;
}
export interface SpanRow {
from_chainage_m: number;
to_chainage_m: number;
gps_len_m: number;
client_len_m: number;
scale: number;
}
export interface VideoChainageRow {
id: number;
file_name: string;
seq: number;
flipped: boolean;
/** Runs OPPOSITE to the majority of the route — the only direction flag shown. */
against_flow: boolean;
sr_start_m: number;
sr_end_m: number;
chainage_start_m: number;
chainage_end_m: number;
}
export interface CalibrationSummary {
route_len_m: number;
reversed: boolean;
/** Most footage runs toward decreasing chainage — the route's normal pattern. */
majority_flipped: boolean;
points_used: number;
points_excluded: number;
spans: SpanRow[];
warnings: string[];
videos: VideoChainageRow[];
skipped: string[];
}
/** A saved calibration (persisted per road-type group by every recompute) — any
* member who opens the project sees what's already computed, by whom and when. */
export interface CalibrationRow {
road_type: string;
route_len_m: number;
anchor_count: number;
computed_by: string;
computed_at: string;
/** False only for pre-summary calibrations — one recompute seeds the saved result. */
has_summary: boolean;
}
/** Result of the GPS correction preview/keep — original + corrected tracks + stats.
* The pipeline = spike removal + (when a temporal predecessor exists) a
* missing-start stitch that re-bases the clock to the true video start. */
export interface CorrectResult {
action: string;
points: number;
replaced: number;
stitched_from: string | null; // predecessor file the start was borrowed from
shift_s: number; // seconds of GPS missing at the start (clock re-based by this)
note: string;
len_before_m: number;
len_after_m: number;
original: [number, number, number][];
corrected: [number, number, number][];
}
export interface LeaderRow {
user: string;
videos: number;
annotations: number;
total_time_ms: number;
avg_time_ms: number;
}
export interface VerifierRow {
user: string;
videos: number;
total_time_ms: number;
avg_time_ms: number;
annotations: number; // total annotations this user authored
annotations_per_min: number; // throughput: annotations added per minute (ranking metric)
rank: number; // 1 = highest throughput; 0 = not yet ranked
}
export interface ProjectTime {
active_workers: number;
avg_video_ms: number;
total_spent_ms: number;
elapsed_ms: number;
all_verified: boolean;
eta_ms: number | null;
time_running: boolean;
last_activity_at: string | null;
recent_verified: number;
recent_window_days: number;
}
export interface Folder {
folder: string; // '' = project root
video_count: number;
included: boolean;
}
export interface Stats {
overview: {
total: number;
pending: number;
annotated: number;
verified: number;
pct_done: number;
};
leaderboard: LeaderRow[];
verifiers: VerifierRow[];
timeline: ProjectTime;
}
export interface Member {
username: string;
display_name: string;
role: string;
added_at: string;
}
export interface StorageInfo {
db_bytes: number;
videos: number;
annotations: number;
video_events: number;
audit_events: number;
max_log_rows: number;
}
export interface ActiveClaim {
video_id: number;
file_name: string;
claimed_by: string;
claimed_at: string;
lease_expires_at: string;
}
export interface ActivityEvent {
video_id: number;
file_name: string;
username: string;
event: string; // claim | release | push | auto_release | reopen
at: string;
}
export interface Activity {
active_claims: ActiveClaim[];
recent_events: ActivityEvent[];
}
// The signed-in bearer token. Set once after login (App gate); every request
// attaches it automatically so the whole dashboard is behind auth.
let authToken = "";
export function setAuthToken(t: string) {
authToken = t;
}
/** Download URLs for annotation export. Served as <a download> attachments, which
* can't set a header — so the token rides as a query param (validated server-side). */
export const exportVideoUrl = (id: number) =>
`${BASE}/videos/${id}/export?token=${encodeURIComponent(authToken)}`;
export const exportProjectUrl = (id: number) =>
`${BASE}/projects/${id}/export?token=${encodeURIComponent(authToken)}`;
/** All video tracks (colored by status) + calibration points as a Google-Earth KML. */
export const tracksKmlUrl = (id: number) =>
`${BASE}/projects/${id}/tracks.kml?token=${encodeURIComponent(authToken)}`;
function authHeaders(token?: string): Record<string, string> {
const t = token ?? authToken;
return t ? { Authorization: `Bearer ${t}` } : {};
}
async function get<T>(path: string, token?: string): Promise<T> {
const r = await fetch(`${BASE}${path}`, { headers: authHeaders(token) });
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
}
/** POST with optional JSON body; uses the explicit token or the signed-in one. */
async function post<T>(path: string, body?: unknown, token?: string): Promise<T> {
const headers: Record<string, string> = { ...authHeaders(token) };
if (body !== undefined) headers["Content-Type"] = "application/json";
const r = await fetch(`${BASE}${path}`, {
method: "POST",
headers,
body: body !== undefined ? JSON.stringify(body) : undefined,
});
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
}
/** DELETE with the signed-in (or explicit) token. */
async function del<T>(path: string, token?: string): Promise<T> {
const r = await fetch(`${BASE}${path}`, { method: "DELETE", headers: authHeaders(token) });
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
}
export const api = {
projects: () => get<ProjectSummary[]>("/projects"),
clients: () => get<Client[]>("/clients"),
videos: (id: number) => get<VideoRow[]>(`/projects/${id}/videos`),
stats: (id: number) => get<Stats>(`/projects/${id}/stats`),
activity: (id: number) => get<Activity>(`/projects/${id}/activity`),
audit: (limit = 100) => get<AuditRow[]>(`/audit?limit=${limit}`),
// Sync re-scans the project's NAS folder; the token (if any) attributes the action.
ingest: (id: number, token?: string) =>
post<{ ingested: number; source_path: string }>(`/projects/${id}/ingest`, undefined, token),
whoami: async (token: string): Promise<{ username: string; role: string }> => {
const r = await fetch(`${BASE}/auth/whoami`, { headers: { Authorization: `Bearer ${token}` } });
if (!r.ok) throw new Error(`${r.status} ${await r.text()}`);
return r.json();
},
// Explicit sign-in: validates the token AND records a `signed_in` audit event.
login: (token: string) => post<{ username: string; role: string }>("/auth/login", undefined, token),
// Any authenticated user: add a project, change its sync directory.
createProject: (
p: { client_id: number; name: string; source_path: string; source_kind: string },
token?: string,
) => post<{ id: number }>("/projects", p, token),
updateProjectSource: (
id: number,
p: { source_path: string; source_kind: string },
token?: string,
) => post<{ id: number }>(`/projects/${id}/source`, p, token),
// Admin only.
createClient: (name: string, token?: string) => post<Client>("/clients", { name }, token),
deleteClient: (id: number, token?: string) => del<{ deleted: boolean; projects_removed: number }>(`/clients/${id}`, token),
deleteProject: (id: number, token?: string) => del<{ deleted: boolean }>(`/projects/${id}`, token),
deleteUser: (id: number, token?: string) => del<{ deleted: boolean; username: string }>(`/users/${id}`, token),
listUsers: (token?: string) => get<UserRow[]>("/users", token),
createUser: (
u: { username: string; display_name?: string; role: string },
token?: string,
) => post<{ username: string; role: string; token: string }>("/users", u, token),
// Admin only: regenerate a user's token (when they forget it). Returns the new token once.
resetUserToken: (id: number, token?: string) =>
post<{ username: string; role: string; token: string }>(`/users/${id}/token`, undefined, token),
// Admin only: edit a user's display name + role.
updateUser: (id: number, body: { display_name: string; role: string }, token?: string) =>
post<{ updated: boolean; display_name: string; role: string }>(`/users/${id}/update`, body, token),
// Append a remark (any authed user). Returns the updated thread.
addRemark: (videoId: number, body: string, token?: string) =>
post<Remark[]>(`/videos/${videoId}/remarks`, { body }, token),
// Raise/lower the hand on a video (any authed user). Anyone can lower it.
setHand: (videoId: number, raised: boolean, token?: string) =>
post<{ hand_raised: boolean; by: string }>(`/videos/${videoId}/hand`, { raised }, token),
// Ignore / un-ignore a video (admin only).
setIgnore: (videoId: number, ignored: boolean, token?: string) =>
post<{ ignored: boolean }>(`/videos/${videoId}/ignore`, { ignored }, token),
// Assign (or unassign with assignee="") videos to a member. Any authed user.
assign: (projectId: number, video_ids: number[], assignee: string, token?: string) =>
post<{ assigned: number; assignee: string }>(`/projects/${projectId}/assign`, { video_ids, assignee }, token),
// Project membership (admin picks who can see/be-assigned a project).
members: (projectId: number) => get<Member[]>(`/projects/${projectId}/members`),
addMember: (projectId: number, username: string, token?: string) =>
post<{ added: boolean }>(`/projects/${projectId}/members`, { username }, token),
removeMember: (projectId: number, username: string, token?: string) =>
del<{ removed: boolean }>(`/projects/${projectId}/members/${encodeURIComponent(username)}`, token),
// Admin: DB size + row counts (how much space the app uses).
storage: () => get<StorageInfo>("/admin/storage"),
// Project timer (admin): start | stop | reset.
setTimer: (projectId: number, action: "start" | "stop" | "reset", token?: string) =>
post<{ action: string }>(`/projects/${projectId}/timer`, { action }, token),
// Folders to consider for verification.
folders: (projectId: number) => get<Folder[]>(`/projects/${projectId}/folders`),
setFolders: (projectId: number, folders: string[], token?: string) =>
post<{ folders: number }>(`/projects/${projectId}/folders`, { folders }, token),
// ---- Map view + GPS pipeline + road_type + chainage ----
map: (projectId: number) => get<MapVideo[]>(`/projects/${projectId}/map`),
remarks: (videoId: number) => get<Remark[]>(`/videos/${videoId}/remarks`),
// Queue a fresh exiftool scan for a video / for every scanned video (admin).
gpsRescan: (videoId: number) => post<{ queued: boolean }>(`/videos/${videoId}/gps/rescan`),
gpsRescanAll: (projectId: number) => post<{ queued: number }>(`/projects/${projectId}/gps/rescan_all`),
// "Viterbi" correction: preview (nothing saved) / keep (persist) / discard (revert).
gpsCorrect: (videoId: number, action: "preview" | "keep" | "discard") =>
post<CorrectResult>(`/videos/${videoId}/gps/correct`, { action }),
// Road types: admin-defined labels + bulk assignment to selected videos.
roadTypes: (projectId: number) => get<RoadType[]>(`/projects/${projectId}/road_types`),
setRoadTypes: (projectId: number, names: string[]) =>
post<{ road_types: number }>(`/projects/${projectId}/road_types`, { names }),
assignRoadType: (projectId: number, video_ids: number[], road_type: string) =>
post<{ assigned: number }>(`/projects/${projectId}/road_type_assign`, { video_ids, road_type }),
// Chainage: client calibration points + the recompute/quick-assign actions.
chainagePoints: (projectId: number) => get<ChainagePoint[]>(`/projects/${projectId}/chainage/points`),
addChainagePoint: (
projectId: number,
p: { lat: number; lon: number; chainage_m: number; note?: string; road_type?: string },
) => post<ChainagePoint>(`/projects/${projectId}/chainage/points`, p),
// Bulk insert (KML import): one transaction, all-or-nothing.
addChainagePointsBulk: (
projectId: number,
points: { lat: number; lon: number; chainage_m: number; note?: string; road_type?: string }[],
) => post<{ added: number }>(`/projects/${projectId}/chainage/points/bulk`, { points }),
deleteChainagePoint: (pointId: number) => del<{ deleted: boolean }>(`/chainage/points/${pointId}`),
// Edit a point's note in place (admin); position/chainage stay immutable.
updateChainagePointNote: (pointId: number, note: string) =>
post<{ updated: boolean }>(`/chainage/points/${pointId}`, { note }),
// Saved calibrations per group (member or admin) — proof the chainage is computed.
chainageCalibrations: (projectId: number) =>
get<CalibrationRow[]>(`/projects/${projectId}/chainage/calibrations`),
// The saved result of the last recompute for a group (null if never computed) —
// any admin who signs in later sees the existing compute without re-running it.
chainageSummary: (projectId: number, road_type: string) =>
get<CalibrationSummary | null>(
`/projects/${projectId}/chainage/summary?road_type=${encodeURIComponent(road_type)}`,
),
chainageRecompute: (projectId: number, video_ids: number[], road_type: string) =>
post<CalibrationSummary>(`/projects/${projectId}/chainage/recompute`, { video_ids, road_type }),
chainageQuick: (
projectId: number,
body: { video_ids: number[]; start_m: number; end_m: number; road_type: string },
) => post<CalibrationSummary>(`/projects/${projectId}/chainage/quick`, body),
};
/** Display label for a workflow/status value. `annotated` really means "has
* annotations, nobody verified them yet"; `pending` means the video has no
* annotations at all — admins need to see that plainly. Display-only: the DB/API
* keep the raw values. */
export function statusLabel(s: string): string {
if (s === "annotated") return "awaiting verification";
if (s === "pending") return "no annotations";
return s;
}
/** Chainage convention (user-fixed): km with FOUR decimals — "22.4456".
* Input is meters (internal unit); em-dash when not computed yet. */
export function fmtChainage(m: number | null | undefined): string {
if (m == null || isNaN(m)) return "—";
return (m / 1000).toFixed(4);
}
/** "12.4000 → 13.7500" for a start/end pair (direction preserved — a flipped video
* legitimately runs high→low); em-dash when not computed. */
export function fmtChainageRange(a: number | null | undefined, b: number | null | undefined): string {
if (a == null || b == null) return "—";
return `${(a / 1000).toFixed(4)}${(b / 1000).toFixed(4)}`;
}
/** Per-pass annotation deltas from the imported baseline: verify +Δ, re-verify +Δ, …
* Skips legacy passes that have no recorded count. The first pass is "verify",
* subsequent ones "re-verify". `total` is the running annotation count after the pass. */
export function passDeltas(
importedCount: number,
passes: Pass[],
): { label: string; delta: number; username: string; total: number }[] {
const out: { label: string; delta: number; username: string; total: number }[] = [];
let prev = importedCount;
let n = 0;
for (const p of passes ?? []) {
if (p.count == null) continue;
n += 1;
out.push({ label: n === 1 ? "verify" : "re-verify", delta: p.count - prev, username: p.username, total: p.count });
prev = p.count;
}
return out;
}
/** Human-readable byte size. */
export function fmtBytes(n: number): string {
if (!n || n <= 0) return "0 B";
const u = ["B", "KB", "MB", "GB", "TB"];
const i = Math.min(u.length - 1, Math.floor(Math.log(n) / Math.log(1024)));
return `${(n / 1024 ** i).toFixed(i ? 1 : 0)} ${u[i]}`;
}
/** Readable date+time for a server ISO timestamp, shown in IST (Asia/Kolkata)
* regardless of the viewer's/VM's timezone; "" when missing/invalid. Server
* timestamps are UTC (TIMESTAMPTZ), so the instant is converted, not relabelled. */
export function fmtWhen(iso: string | null): string {
if (!iso) return "";
const d = new Date(iso);
if (isNaN(d.getTime())) return "";
return d.toLocaleString("en-IN", {
timeZone: "Asia/Kolkata",
month: "short",
day: "numeric",
hour: "2-digit",
minute: "2-digit",
}) + " IST";
}
export function fmtDuration(ms: number): string {
if (!ms || ms <= 0) return "—";
const s = Math.round(ms / 1000);
const h = Math.floor(s / 3600);
const m = Math.floor((s % 3600) / 60);
const sec = s % 60;
if (h > 0) return `${h}h ${m}m`;
if (m > 0) return `${m}m ${sec}s`;
return `${sec}s`;
}

10
dashboard/src/main.tsx Normal file
View File

@@ -0,0 +1,10 @@
import React from "react";
import ReactDOM from "react-dom/client";
import { App } from "./App";
import "./styles.css";
ReactDOM.createRoot(document.getElementById("root")!).render(
<React.StrictMode>
<App />
</React.StrictMode>
);

357
dashboard/src/styles.css Normal file
View File

@@ -0,0 +1,357 @@
:root {
--bg0: #0e1222; --bg1: #151a30; --bg2: #1c223d; --border: #283154;
--text: #e8e9f1; --dim: #8f95ae; --accent: #f5c838;
--ok: #5ccf75; --info: #6ea8ff; --warn: #f5a623; --danger: #ff7a7a;
}
* { box-sizing: border-box; }
body { margin: 0; background: var(--bg0); color: var(--text);
font: 14px/1.5 system-ui, -apple-system, Segoe UI, Roboto, sans-serif; }
.page { max-width: 1200px; margin: 0 auto; padding: 16px; }
.dim { color: var(--dim); }
.mono, .path { font-family: ui-monospace, Menlo, monospace; font-size: 12px; }
.topbar { display: flex; align-items: center; gap: 12px; padding: 10px 0 16px; }
.brand { font-size: 18px; font-weight: 700; }
.spacer { flex: 1; }
.topbar select, .topbar button {
background: var(--bg2); color: var(--text); border: 1px solid var(--border);
border-radius: 6px; padding: 6px 12px; cursor: pointer; font-size: 13px;
}
.topbar button:hover:not(:disabled) { background: var(--border); }
.topbar button:disabled { opacity: 0.5; cursor: default; }
.path { max-width: 360px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.error { background: rgba(255,122,122,0.12); border: 1px solid var(--danger);
color: #ffb3a8; padding: 8px 12px; border-radius: 6px; margin-bottom: 12px; }
.cards { display: grid; grid-template-columns: repeat(4, 1fr) 1.6fr; gap: 12px; margin-bottom: 16px; }
.card { background: var(--bg1); border: 1px solid var(--border); border-radius: 10px; padding: 14px; }
.card-label { color: var(--dim); font-size: 12px; text-transform: uppercase; letter-spacing: 0.04em; }
.card-value { font-size: 30px; font-weight: 700; margin-top: 4px; }
.card-value.sm { font-size: 16px; }
.card.ok .card-value { color: var(--ok); }
.card.info .card-value { color: var(--info); }
.card.warn .card-value { color: var(--warn); }
.progress-card { display: flex; flex-direction: column; justify-content: center; }
.bar { height: 10px; background: var(--bg2); border-radius: 6px; overflow: hidden; margin: 8px 0; }
.bar-fill { height: 100%; background: var(--ok); transition: width 0.4s; }
.grid2 { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin-bottom: 16px; }
@media (max-width: 860px) { .grid2 { grid-template-columns: 1fr; } .cards { grid-template-columns: repeat(2, 1fr); } }
.panel { background: var(--bg1); border: 1px solid var(--border); border-radius: 10px; padding: 14px; }
.panel h3 { margin: 0 0 10px; font-size: 14px; }
table { width: 100%; border-collapse: collapse; }
th, td { text-align: left; padding: 7px 8px; border-bottom: 1px solid var(--border); font-size: 13px; }
th { color: var(--dim); font-weight: 600; font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em; }
tbody tr:hover { background: rgba(255,255,255,0.02); }
.badge { padding: 2px 8px; border-radius: 10px; font-size: 11px; font-weight: 600; }
.badge.pending { background: rgba(245,166,35,0.18); color: var(--warn); }
.badge.annotated { background: rgba(110,168,255,0.18); color: var(--info); }
.badge.verified { background: rgba(92,207,117,0.18); color: var(--ok); }
.throughput { display: flex; flex-direction: column; gap: 6px; }
.tp-row { display: grid; grid-template-columns: 90px 1fr 34px; align-items: center; gap: 8px; }
.tp-date { font-size: 11px; }
.tp-bar { height: 12px; background: var(--bg2); border-radius: 6px; overflow: hidden; }
.tp-fill { height: 100%; background: var(--accent); }
.tp-count { text-align: right; font-variant-numeric: tabular-nums; }
/* View toggle (Dashboard | Admin) */
.viewtabs { display: inline-flex; border: 1px solid var(--border); border-radius: 6px; overflow: hidden; }
.viewtabs button {
background: var(--bg1); color: var(--dim); border: none; padding: 6px 14px;
cursor: pointer; font-size: 13px;
}
.viewtabs button.active { background: var(--bg2); color: var(--text); font-weight: 600; }
.viewtabs button:not(.active):hover { color: var(--text); }
/* Admin forms */
.admin { display: flex; flex-direction: column; gap: 16px; }
.admin .grid2 { margin-bottom: 0; }
.form-row { display: flex; align-items: center; gap: 8px; flex-wrap: wrap; }
.form-col { display: flex; flex-direction: column; gap: 6px; max-width: 480px; }
.form-col label { color: var(--dim); font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em; }
.admin input, .admin select {
background: var(--bg2); color: var(--text); border: 1px solid var(--border);
border-radius: 6px; padding: 7px 10px; font-size: 13px; font-family: inherit;
}
.admin input:focus, .admin select:focus { outline: none; border-color: var(--info); }
.btn {
background: var(--bg2); color: var(--text); border: 1px solid var(--border);
border-radius: 6px; padding: 7px 14px; cursor: pointer; font-size: 13px;
}
.btn:hover:not(:disabled) { background: var(--border); }
.btn:disabled { opacity: 0.5; cursor: default; }
.btn.primary { background: var(--info); border-color: var(--info); color: #0a1020; font-weight: 600; align-self: flex-start; }
.btn.primary:hover:not(:disabled) { filter: brightness(1.08); background: var(--info); }
.ok-tag { color: var(--ok); font-size: 12px; }
.err-tag { color: var(--danger); font-size: 12px; }
.notice-ok { background: rgba(92,207,117,0.12); border: 1px solid var(--ok);
color: #b6f0c3; padding: 8px 12px; border-radius: 6px; }
/* One-time token reveal */
.token-box { background: rgba(245,200,56,0.08); border: 1px solid var(--accent);
border-radius: 8px; padding: 10px 12px; margin-bottom: 10px; font-size: 13px; }
.token-val { background: var(--bg0); border: 1px solid var(--border); border-radius: 6px;
padding: 6px 8px; font-family: ui-monospace, Menlo, monospace; font-size: 12px;
word-break: break-all; flex: 1; }
/* Remark thread cell (video table) — needs a guaranteed floor so the crowded table
can't crush it; the table itself scrolls horizontally instead (.table-scroll). */
.remark-col { max-width: 360px; min-width: 150px; }
.remark-collapsed { background: none; border: none; color: var(--fg, #dce3f5); cursor: pointer;
text-align: left; padding: 2px 0; font-size: 12px; display: inline-flex; gap: 6px; align-items: center;
max-width: 340px; white-space: nowrap; }
/* Wide tables scroll inside their panel instead of squeezing columns off the edge. */
.table-scroll { overflow-x: auto; }
.remark-collapsed:hover { filter: brightness(1.2); }
.remark-badge { background: var(--bg0); border: 1px solid var(--border); border-radius: 10px;
padding: 1px 7px; font-size: 11px; white-space: nowrap; }
.remark-snip { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.remark-open { display: flex; flex-direction: column; gap: 6px; min-width: 280px; }
.remark-thread { max-height: 120px; overflow: auto; display: flex; flex-direction: column; gap: 3px;
background: var(--bg0); border: 1px solid var(--border); border-radius: 6px; padding: 6px 8px; }
.remark-line { font-size: 12px; line-height: 1.35; }
.remark-line b { color: var(--info); }
.remark-add { display: flex; gap: 6px; align-items: center; }
.remark-add input { flex: 1; background: var(--bg0); color: inherit; border: 1px solid var(--border);
border-radius: 6px; padding: 6px 8px; font-size: 12px; }
/* Claim / completion subtitle under the status badge (video table) */
.claim-line { margin-top: 3px; font-size: 11px; color: var(--dim); white-space: nowrap; }
.claim-line b { color: var(--fg, #dce3f5); font-weight: 600; }
/* Flagged-for-review count (video table) */
.review-flag { background: rgba(245,166,35,0.18); color: var(--warn); border-radius: 10px; padding: 2px 8px; font-size: 11px; font-weight: 600; white-space: nowrap; }
/* Panel header row (title + actions, e.g. Export all) */
.panel-head { display: flex; align-items: center; justify-content: space-between; gap: 12px; }
.panel-head h3 { margin: 0; }
/* Recent activity feed + verification panel */
.active-claims { display: flex; flex-direction: column; gap: 4px; margin-bottom: 10px; }
.active-claim { font-size: 12px; background: rgba(110,168,255,0.08); border: 1px solid var(--border); border-radius: 6px; padding: 5px 8px; }
.activity-feed { display: flex; flex-direction: column; gap: 3px; max-height: 260px; overflow: auto; }
.activity-row { display: flex; align-items: center; gap: 8px; font-size: 12px; padding: 2px 0; }
.evt { font-size: 10px; font-weight: 700; text-transform: uppercase; border-radius: 4px; padding: 1px 6px; min-width: 56px; text-align: center; }
.evt-push { background: rgba(92,207,117,0.18); color: var(--ok); }
.evt-claim { background: rgba(110,168,255,0.18); color: var(--info); }
.evt-release { background: rgba(245,166,35,0.18); color: var(--warn); }
.evt-auto_release { background: rgba(245,166,35,0.12); color: var(--warn); }
.evt-reopen { background: rgba(160,160,160,0.18); color: var(--dim); }
/* Workflow-status badges (assigned / in-progress / re-verified) + assignment UI */
.badge.assigned { background: rgba(180,140,255,0.18); color: #b48cff; }
.badge.in-progress { background: rgba(110,168,255,0.20); color: var(--info); }
.badge.re-verified { background: rgba(92,207,117,0.22); color: var(--ok); }
.panel-actions { display: flex; align-items: center; gap: 12px; }
.assign-bar { display: flex; align-items: center; gap: 6px; font-size: 12px; }
.assign-bar select { background: var(--bg0); color: inherit; border: 1px solid var(--border); border-radius: 6px; padding: 4px 6px; font-size: 12px; }
tr.row-sel { background: rgba(110,168,255,0.08); }
/* ---- Sign-in gate ---- */
.signin-screen { display: flex; align-items: center; justify-content: center; min-height: 100vh; padding: 16px; }
.signin-card { background: var(--bg1); border: 1px solid var(--border); border-radius: 12px;
padding: 28px; width: 100%; max-width: 380px; box-shadow: 0 8px 32px rgba(0,0,0,0.4); }
.signin-card h1 { margin: 0 0 4px; font-size: 20px; }
.signin-card p { margin: 0 0 16px; }
.signin-card input { width: 100%; background: var(--bg2); color: var(--text); border: 1px solid var(--border);
border-radius: 6px; padding: 9px 11px; font-size: 14px; margin-bottom: 12px; }
.signin-card input:focus { outline: none; border-color: var(--info); }
.signin-card .btn.primary { width: 100%; justify-content: center; align-self: stretch; text-align: center; }
.signin-card .error { margin-top: 12px; margin-bottom: 0; }
.signin-reason { background: rgba(245,166,35,0.12); border: 1px solid var(--warn); color: #f3d3a0;
border-radius: 6px; padding: 8px 10px; font-size: 13px; margin-bottom: 14px; }
/* Signed-in identity + sign out in the top bar */
.whoami { color: var(--dim); font-size: 12px; white-space: nowrap; }
.signout-btn { background: var(--bg2); color: var(--text); border: 1px solid var(--border);
border-radius: 6px; padding: 6px 12px; cursor: pointer; font-size: 13px; }
.signout-btn:hover { background: var(--border); }
/* Danger / destructive buttons + row action group */
.btn.danger { border-color: var(--danger); color: #ffb3a8; }
.btn.danger:hover:not(:disabled) { background: rgba(255,122,122,0.15); }
.row-actions { display: flex; gap: 6px; }
/* Imported-vs-current annotation count (deletions/additions) */
.imported-line { font-size: 11px; margin-top: 2px; white-space: nowrap; }
.ann-del { color: var(--danger); font-weight: 600; }
.ann-add { color: var(--ok); font-weight: 600; }
/* Raise-hand toggle in the remark cell */
.remark-cell { display: flex; align-items: center; gap: 6px; }
.hand-btn { background: var(--bg2); border: 1px solid var(--border); border-radius: 6px;
cursor: pointer; font-size: 13px; line-height: 1; padding: 3px 6px; filter: grayscale(1) opacity(0.55); }
.hand-btn:hover:not(:disabled) { filter: none; }
.hand-btn.raised { filter: none; background: rgba(245,166,35,0.18); border-color: var(--warn); }
.hand-note { color: var(--warn); font-size: 11px; white-space: nowrap; }
.remark-open-head { display: flex; align-items: center; gap: 6px; }
/* Ignored video row (admin "ignore") — crossed out + dimmed */
tr.row-ignored td { text-decoration: line-through; opacity: 0.5; }
tr.row-ignored .badge, tr.row-ignored .ignore-btn { text-decoration: none; }
.badge.ignored-tag { background: rgba(160,160,160,0.18); color: var(--dim); margin-left: 6px; text-decoration: none; }
.ignore-btn { background: none; border: none; color: var(--dim); cursor: pointer; font-size: 11px;
margin-left: 8px; text-decoration: underline; padding: 0; }
.ignore-btn:hover { color: var(--text); }
/* Project time & estimate panel */
.time-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(150px, 1fr)); gap: 10px; }
.tcell { display: flex; flex-direction: column; gap: 2px; background: var(--bg2);
border: 1px solid var(--border); border-radius: 8px; padding: 10px 12px; }
.tcell b { font-size: 18px; }
.tcell .sub { font-size: 11px; }
.tcell.big { background: rgba(110,168,255,0.08); border-color: rgba(110,168,255,0.35); }
.tcell.big b { font-size: 20px; }
/* Members picker (dropdown checklist on the leaderboard panel) */
.members-picker { position: relative; }
.members-dropdown { position: absolute; right: 0; top: calc(100% + 6px); z-index: 30; width: 280px;
background: var(--bg1); border: 1px solid var(--border); border-radius: 8px;
box-shadow: 0 10px 30px rgba(0,0,0,0.45); padding: 8px; }
.members-search { width: 100%; background: var(--bg2); color: var(--text); border: 1px solid var(--border);
border-radius: 6px; padding: 6px 8px; font-size: 13px; margin-bottom: 6px; }
.members-list { max-height: 260px; overflow: auto; display: flex; flex-direction: column; gap: 1px; }
.members-item { display: flex; align-items: center; gap: 8px; padding: 5px 6px; border-radius: 6px; cursor: pointer; font-size: 13px; }
.members-item:hover { background: var(--bg2); }
.members-item.on { background: rgba(92,207,117,0.10); }
.members-item .mi-disp { font-size: 11px; margin-left: auto; }
.mi-empty { padding: 8px; font-size: 12px; }
.mi-err { color: var(--danger); font-size: 12px; padding: 4px 6px; }
.members-foot { font-size: 11px; border-top: 1px solid var(--border); margin-top: 6px; padding-top: 6px; }
.members-cta { background: rgba(245,166,35,0.10); border: 1px solid rgba(245,166,35,0.4);
color: #f3d3a0; border-radius: 6px; padding: 7px 10px; font-size: 12px; margin-bottom: 10px; }
.folders-hint { font-size: 11px; padding: 2px 4px 6px; }
/* Verification pace panel */
.pace-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(120px, 1fr)); gap: 10px; }
.pcell { display: flex; flex-direction: column; gap: 2px; background: var(--bg2);
border: 1px solid var(--border); border-radius: 8px; padding: 10px 12px; }
.pcell b { font-size: 18px; }
.pcell .sub { font-size: 11px; }
.pcell.big { background: rgba(110,168,255,0.08); border-color: rgba(110,168,255,0.35); }
tr.rank-top { background: rgba(245,200,56,0.08); }
/* Project timer controls */
.timer-controls { display: flex; align-items: center; gap: 8px; }
.timer-state { font-size: 12px; font-weight: 600; }
.timer-state.on { color: var(--ok); }
.timer-state.off { color: var(--dim); }
/* Project members manager (Admin) */
.members-box { margin-top: 14px; border-top: 1px solid var(--border); padding-top: 12px; }
.members-box h4 { margin: 0 0 8px; font-size: 13px; }
.member-chips { display: flex; flex-wrap: wrap; gap: 6px; }
.member-chip { display: inline-flex; align-items: center; gap: 6px; background: var(--bg2);
border: 1px solid var(--border); border-radius: 14px; padding: 3px 6px 3px 10px; font-size: 12px; }
.chip-x { background: none; border: none; color: var(--danger); cursor: pointer; font-size: 14px;
line-height: 1; padding: 0 2px; }
.chip-x:hover { filter: brightness(1.2); }
/* ---- Map view ---- */
.videos-toggle { margin-left: 12px; vertical-align: middle; }
.mapview { display: flex; flex-direction: column; gap: 10px; }
.map-toolbar { display: flex; align-items: center; gap: 12px; flex-wrap: wrap; }
.map-search { position: relative; }
.map-search input { background: var(--bg2); color: var(--text); border: 1px solid var(--border);
border-radius: 6px; padding: 7px 10px; font-size: 13px; width: 260px; }
.map-search input:focus { outline: none; border-color: var(--info); }
.map-search-results { position: absolute; top: calc(100% + 4px); left: 0; z-index: 1000; width: 340px;
background: var(--bg1); border: 1px solid var(--border); border-radius: 8px;
box-shadow: 0 10px 30px rgba(0,0,0,0.45); overflow: hidden; }
.map-search-results button { display: block; width: 100%; text-align: left; background: none; border: none;
color: var(--text); padding: 7px 10px; cursor: pointer; font-size: 12px; }
.map-search-results button:hover { background: var(--bg2); }
.map-colorby { display: inline-flex; border: 1px solid var(--border); border-radius: 6px; overflow: hidden; }
.map-colorby button { background: var(--bg1); color: var(--dim); border: none; padding: 6px 12px;
cursor: pointer; font-size: 12px; }
.map-colorby button.active { background: var(--bg2); color: var(--text); font-weight: 600; }
.map-chips { display: flex; gap: 6px; flex-wrap: wrap; }
.map-chip { background: var(--bg2); border: 1px solid var(--border); border-radius: 12px;
padding: 3px 10px; font-size: 11px; font-weight: 600; cursor: pointer; color: var(--text); }
.map-chip.off { opacity: 0.35; text-decoration: line-through; }
.map-body { display: grid; grid-template-columns: 1fr 320px; gap: 12px; }
@media (max-width: 900px) { .map-body { grid-template-columns: 1fr; } }
.map-canvas { height: 560px; border: 1px solid var(--border); border-radius: 10px;
background: var(--bg2); z-index: 0; }
.map-side { display: flex; flex-direction: column; gap: 10px; max-height: 560px; overflow: auto; }
.map-hint { background: var(--bg2); border: 1px solid var(--border); border-radius: 8px;
padding: 12px; font-size: 12px; }
.map-details { background: var(--bg2); border: 1px solid var(--border); border-radius: 8px;
padding: 10px 12px; display: flex; flex-direction: column; gap: 8px; }
.map-details-head { display: flex; align-items: center; justify-content: space-between; gap: 8px; }
.map-details-head b { word-break: break-all; font-size: 12px; }
.map-kv { display: grid; grid-template-columns: 92px 1fr; gap: 3px 8px; font-size: 12px; }
.map-actions { display: flex; align-items: center; gap: 6px; flex-wrap: wrap; }
.map-correction { background: rgba(92,207,117,0.08); border: 1px solid rgba(92,207,117,0.4);
border-radius: 8px; padding: 8px 10px; font-size: 12px; display: flex; flex-direction: column; gap: 6px; }
.map-remarks { display: flex; flex-direction: column; gap: 6px; font-size: 12px; }
.map-nometa { background: var(--bg2); border: 1px solid var(--border); border-radius: 8px; padding: 10px 12px; }
.map-nometa-head { font-size: 12px; margin-bottom: 6px; }
.map-nometa-head b { color: var(--warn); }
.map-nometa-list { display: flex; flex-direction: column; gap: 3px; max-height: 200px; overflow: auto; }
.map-nometa-row { display: flex; align-items: center; gap: 8px; font-size: 12px; }
.map-nometa-name { background: none; border: none; color: var(--text); cursor: pointer; padding: 0;
font-size: 11px; text-align: left; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; flex: 1; }
.map-nometa-name:hover { color: var(--info); }
.btn.mini { padding: 1px 7px; font-size: 11px; }
/* Travel-direction arrows along every track (bigger on the selected one) */
.dir-arrow-wrap { background: none; border: none; }
.dir-arrow { color: #ffffff; font-size: 13px; line-height: 16px; text-align: center;
text-shadow: 0 0 3px #000, 0 0 5px #000; pointer-events: none; }
.dir-arrow.sm { font-size: 10px; line-height: 12px; opacity: 0.85; }
/* Whole-km chainage milestones pinned on calibrated tracks */
.km-mark-wrap { background: none; border: none; }
.km-mark { min-width: 22px; padding: 0 4px; text-align: center; font-size: 10px; font-weight: 700;
color: #0a1020; background: #f5c838; border: 1px solid #0a1020; border-radius: 8px;
line-height: 13px; box-shadow: 0 0 3px rgba(0,0,0,.6); }
/* Idle / dropout tags in the data-quality panel */
.idle-tag { color: var(--warn); font-size: 11px; white-space: nowrap; }
/* Per-video GPS coverage bar (details card): green real fixes, amber interpolated
gaps, red missing start/end, blue tick = stitched start */
.gps-bar { position: relative; height: 8px; min-width: 120px; background: #2a3350;
border-radius: 4px; overflow: hidden; margin-top: 4px; }
.gps-bar-seg { position: absolute; top: 0; bottom: 0; }
.gps-bar-seg.ok { background: #5ccf75; }
.gps-bar-seg.gap { background: #f5a623; }
.gps-bar-seg.miss { background: #ff5252; }
.gps-bar-stitch { position: absolute; top: 0; bottom: 0; width: 3px; background: #6ea8ff; }
/* Leaflet dark-theme nudges */
.leaflet-container { font: inherit; }
.leaflet-container a.leaflet-attribution-flag { display: none !important; }
/* ---- Road type + chainage ---- */
.badge.road-tag { background: rgba(245,200,56,0.16); color: var(--accent); }
.chainage-block { margin-top: 12px; }
.block-label { font-size: 12px; margin-bottom: 6px; }
.chainage-block input, .chainage-block select { background: var(--bg2); color: var(--text);
border: 1px solid var(--border); border-radius: 6px; padding: 6px 8px; font-size: 12px; }
.chainage-block input:focus, .chainage-block select:focus { outline: none; border-color: var(--info); }
.chip-input { width: 150px; }
.chainage-warnings { margin: 6px 0; padding-left: 18px; color: #f3d3a0; font-size: 12px; }
tr.span-warn { background: rgba(245,166,35,0.10); }
.kml-preview { margin-top: 10px; border: 1px solid var(--border); border-radius: 8px;
padding: 10px 12px; background: var(--bg2); display: flex; flex-direction: column; gap: 8px; }
.kml-preview-list { max-height: 240px; overflow: auto; }
/* GPS fix-count line + the tap-to-explain data-quality warning (video table) */
.gps-pts { position: relative; }
.gps-warn { background: rgba(245,166,35,0.18); border: 1px solid var(--warn); color: var(--warn);
border-radius: 8px; cursor: pointer; font-size: 11px; line-height: 1; padding: 1px 5px; margin-left: 5px; }
.gps-warn:hover { filter: brightness(1.2); }
.gps-explain { position: absolute; z-index: 40; margin-top: 4px; width: 340px; white-space: normal;
background: var(--bg1); border: 1px solid var(--warn); border-radius: 8px; padding: 9px 11px;
font-size: 12px; line-height: 1.45; box-shadow: 0 10px 30px rgba(0,0,0,0.45); cursor: pointer; }
.gps-explain b { color: var(--warn); }
/* Map details: correction attribution line */
.corrected-note { font-size: 11px; margin-top: 2px; white-space: normal; }
/* Storage panel */
.storage-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(140px, 1fr)); gap: 12px; }
.storage-grid > div { display: flex; flex-direction: column; gap: 2px; background: var(--bg2);
border: 1px solid var(--border); border-radius: 8px; padding: 10px 12px; }
.storage-grid b { font-size: 18px; }

1
dashboard/src/vite-env.d.ts vendored Normal file
View File

@@ -0,0 +1 @@
/// <reference types="vite/client" />

20
dashboard/tsconfig.json Normal file
View File

@@ -0,0 +1,20 @@
{
"compilerOptions": {
"target": "ES2020",
"useDefineForClassFields": true,
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"module": "ESNext",
"skipLibCheck": true,
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"jsx": "react-jsx",
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true
},
"include": ["src"]
}

View File

@@ -0,0 +1 @@
{"root":["./src/AdminPanel.tsx","./src/App.tsx","./src/ChainagePanel.tsx","./src/FoldersPicker.tsx","./src/MapView.tsx","./src/MembersPicker.tsx","./src/RemarkCell.tsx","./src/SignIn.tsx","./src/api.ts","./src/main.tsx","./src/vite-env.d.ts"],"version":"5.9.3"}

12
dashboard/vite.config.ts Normal file
View File

@@ -0,0 +1,12 @@
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";
export default defineConfig({
plugins: [react()],
server: {
// `npm run dev` proxies /api to the server so the app is same-origin in dev.
proxy: {
"/api": { target: "http://localhost:8080", changeOrigin: true },
},
},
});

318
db/init.sql Normal file
View File

@@ -0,0 +1,318 @@
-- Central verification DB (Phase 1: dashboard pipeline).
-- Ingest reads a folder of videos + sibling export-JSONs and fills these tables.
CREATE TABLE IF NOT EXISTS projects (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
source_path TEXT NOT NULL, -- local folder now; mounted NAS share on the VM
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS videos (
id SERIAL PRIMARY KEY,
project_id INTEGER NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
file_name TEXT NOT NULL,
rel_path TEXT NOT NULL,
has_json BOOLEAN NOT NULL DEFAULT FALSE,
width INTEGER,
height INTEGER,
fps DOUBLE PRECISION,
frame_count INTEGER,
annotation_count INTEGER NOT NULL DEFAULT 0,
annotation_time_ms BIGINT NOT NULL DEFAULT 0,
primary_annotator TEXT NOT NULL DEFAULT '',
-- 'pending' = no JSON yet (not annotated)
-- 'annotated'= has annotations from the NAS JSON (awaiting verification)
-- 'verified' = a worker pushed a verified pass (Phase 3)
status TEXT NOT NULL DEFAULT 'pending',
annotated_at TIMESTAMPTZ, -- max(annotation.created_at), used for throughput
raw_json JSONB, -- full imported export doc (for Phase 3 pull)
ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (project_id, rel_path)
);
CREATE INDEX IF NOT EXISTS idx_videos_project ON videos(project_id);
CREATE INDEX IF NOT EXISTS idx_videos_status ON videos(project_id, status);
CREATE TABLE IF NOT EXISTS annotations (
id SERIAL PRIMARY KEY,
video_id INTEGER NOT NULL REFERENCES videos(id) ON DELETE CASCADE,
frame_number INTEGER NOT NULL,
label TEXT NOT NULL,
side TEXT NOT NULL DEFAULT '',
shape_type TEXT NOT NULL DEFAULT 'bbox',
vertices JSONB,
annotated_by TEXT NOT NULL DEFAULT '',
review_status TEXT NOT NULL DEFAULT 'none',
remark TEXT NOT NULL DEFAULT '',
subclass TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL DEFAULT ''
);
CREATE INDEX IF NOT EXISTS idx_annotations_video ON annotations(video_id);
CREATE INDEX IF NOT EXISTS idx_annotations_user ON annotations(annotated_by);
-- ---------------------------------------------------------------------------
-- Phase 3: worker pull/push (claim → verify → push) + token auth.
-- These run on every startup; ALTER ... IF NOT EXISTS keeps them idempotent and
-- migrates existing Postgres volumes without a separate migration step.
-- ---------------------------------------------------------------------------
-- Provisioned workers/admins. Tokens are never stored in clear: token_hash is
-- the lowercase hex SHA-256 of the issued token. Admin provisions these.
CREATE TABLE IF NOT EXISTS users (
id SERIAL PRIMARY KEY,
username TEXT NOT NULL UNIQUE,
display_name TEXT NOT NULL DEFAULT '',
token_hash TEXT NOT NULL,
role TEXT NOT NULL DEFAULT 'ml_support', -- 'ml_support' | 'admin'
active BOOLEAN NOT NULL DEFAULT TRUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_users_token ON users(token_hash);
-- Roles were renamed worker → ml_support (only two roles now: ml_support | admin).
-- Idempotent: migrate existing rows + the column default on already-provisioned DBs.
UPDATE users SET role='ml_support' WHERE role='worker';
ALTER TABLE users ALTER COLUMN role SET DEFAULT 'ml_support';
-- Claiming is orthogonal to the annotation `status` above: a worker claims an
-- 'annotated' video (exclusive lease), verifies it, then pushes → 'verified'.
ALTER TABLE videos ADD COLUMN IF NOT EXISTS claimed_by TEXT;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS claimed_at TIMESTAMPTZ;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS lease_expires_at TIMESTAMPTZ;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS completed_by TEXT;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS completed_at TIMESTAMPTZ;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS verify_time_ms BIGINT NOT NULL DEFAULT 0;
-- A video is claimable when annotated and either unclaimed or its lease lapsed.
CREATE INDEX IF NOT EXISTS idx_videos_claim ON videos(project_id, claimed_by, lease_expires_at);
-- Imported baseline: how many annotations the video started a pass with (NAS-ingested
-- count, or the worker's claim-time count on push). vs annotation_count it reveals how
-- many the worker added/deleted during verification.
ALTER TABLE videos ADD COLUMN IF NOT EXISTS imported_count INTEGER NOT NULL DEFAULT 0;
-- "Raise hand" signal: anyone (ml_support or admin) can raise a hand on a video to
-- start a conversation; anyone can lower it. Orthogonal to the remark thread.
ALTER TABLE videos ADD COLUMN IF NOT EXISTS hand_raised BOOLEAN NOT NULL DEFAULT FALSE;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS hand_raised_by TEXT;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS hand_raised_at TIMESTAMPTZ;
-- "Ignore" flag: admins can mark a video as ignored (excluded from the workflow);
-- the dashboard/desktop cross out the row. Admin-only to set.
ALTER TABLE videos ADD COLUMN IF NOT EXISTS ignored BOOLEAN NOT NULL DEFAULT FALSE;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS ignored_by TEXT;
-- Admin assignment: which user a video is assigned to (advisory owner). Orthogonal to
-- claim/status; drives the 'assigned' workflow state until someone claims it.
ALTER TABLE videos ADD COLUMN IF NOT EXISTS assigned_to TEXT;
CREATE INDEX IF NOT EXISTS idx_videos_assigned ON videos(project_id, assigned_to);
-- Audit + analytics: every claim/release/push/auto_release/ingest.
CREATE TABLE IF NOT EXISTS video_events (
id SERIAL PRIMARY KEY,
video_id INTEGER NOT NULL REFERENCES videos(id) ON DELETE CASCADE,
username TEXT NOT NULL DEFAULT '',
event TEXT NOT NULL, -- 'claim'|'release'|'push'|'auto_release'|'reopen'
at TIMESTAMPTZ NOT NULL DEFAULT now(),
meta JSONB NOT NULL DEFAULT '{}'::jsonb
);
CREATE INDEX IF NOT EXISTS idx_events_video ON video_events(video_id);
CREATE INDEX IF NOT EXISTS idx_events_at ON video_events(at);
-- ---------------------------------------------------------------------------
-- Clients own projects (one client → many projects). Admins create these from
-- the dashboard; each project points at a NAS folder (`source_path`).
-- Idempotent: safe to re-run on every startup.
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS clients (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
-- projects gains a client owner; name becomes unique PER client
-- (Microsoft/India and Google/India can both exist).
ALTER TABLE projects ADD COLUMN IF NOT EXISTS client_id INTEGER REFERENCES clients(id) ON DELETE CASCADE;
-- One-time backfill: only if a client-less project exists, park it under a 'default' client.
-- Guarded so a deleted 'default' is NOT resurrected on every boot.
DO $$ BEGIN
IF EXISTS (SELECT 1 FROM projects WHERE client_id IS NULL) THEN
INSERT INTO clients (name) VALUES ('default') ON CONFLICT (name) DO NOTHING;
UPDATE projects SET client_id = (SELECT id FROM clients WHERE name='default') WHERE client_id IS NULL;
END IF;
END $$;
-- Swap the old global UNIQUE(name) for UNIQUE(client_id, name). DROP IF EXISTS is idempotent;
-- ADD CONSTRAINT is guarded by a DO-block (Postgres has no ADD CONSTRAINT IF NOT EXISTS).
ALTER TABLE projects DROP CONSTRAINT IF EXISTS projects_name_key;
DO $$ BEGIN
IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname='projects_client_name_key') THEN
ALTER TABLE projects ADD CONSTRAINT projects_client_name_key UNIQUE (client_id, name);
END IF;
END $$;
CREATE INDEX IF NOT EXISTS idx_projects_client ON projects(client_id);
-- How to reach a project's source folder. 'local' = scan source_path directly (a path inside the
-- container, incl. compose-managed NFS volumes); 'nfs' = mount NFS_HOST:source_path on demand
-- (the NAS host/IP lives only in the NFS_HOST env, never here).
ALTER TABLE projects ADD COLUMN IF NOT EXISTS source_kind TEXT NOT NULL DEFAULT 'local';
-- ---------------------------------------------------------------------------
-- Audit log of org-level actions (client/project created, sync dir changed,
-- synced, user created) with timestamp + actor.
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS audit_events (
id SERIAL PRIMARY KEY,
at TIMESTAMPTZ NOT NULL DEFAULT now(),
username TEXT NOT NULL DEFAULT '',
action TEXT NOT NULL, -- 'client_created'|'project_created'|'project_source_changed'|'synced'|'user_created'
detail JSONB NOT NULL DEFAULT '{}'::jsonb
);
CREATE INDEX IF NOT EXISTS idx_audit_at ON audit_events(at);
-- ---------------------------------------------------------------------------
-- Per-video remark thread. Admins and workers both append; everyone sees the
-- same thread (synced). Each line is attributed to its author + timestamp.
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS video_remarks (
id SERIAL PRIMARY KEY,
video_id INTEGER NOT NULL REFERENCES videos(id) ON DELETE CASCADE,
username TEXT NOT NULL,
body TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_video_remarks_video ON video_remarks(video_id, created_at);
-- ---------------------------------------------------------------------------
-- Project membership: which users are picked for a project. Admins manage this.
-- Non-admins only see projects they're a member of; videos can only be assigned
-- to members.
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS project_members (
project_id INTEGER NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
username TEXT NOT NULL,
added_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (project_id, username)
);
CREATE INDEX IF NOT EXISTS idx_project_members_user ON project_members(username);
-- ---------------------------------------------------------------------------
-- Project timer: a stopwatch for "active project time". Admins start/stop/reset it;
-- a push (verify/re-verify) auto-resumes it; 72h of inactivity auto-stops it (idle
-- time isn't counted). elapsed = accumulated + (running ? now - started_at : 0).
-- ---------------------------------------------------------------------------
ALTER TABLE projects ADD COLUMN IF NOT EXISTS time_running BOOLEAN NOT NULL DEFAULT TRUE;
ALTER TABLE projects ADD COLUMN IF NOT EXISTS time_accumulated_ms BIGINT NOT NULL DEFAULT 0;
ALTER TABLE projects ADD COLUMN IF NOT EXISTS time_started_at TIMESTAMPTZ;
ALTER TABLE projects ADD COLUMN IF NOT EXISTS last_activity_at TIMESTAMPTZ;
-- Backfill: running projects count from their creation; seed last activity.
UPDATE projects SET time_started_at = created_at WHERE time_running AND time_started_at IS NULL;
UPDATE projects SET last_activity_at = created_at WHERE last_activity_at IS NULL;
-- ---------------------------------------------------------------------------
-- Folders to consider for verification. When a project has ANY rows here, only
-- videos in those folders are shown/counted; empty = consider all folders.
-- Admins add/modify/delete the set.
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS project_folders (
project_id INTEGER NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
folder TEXT NOT NULL, -- directory part of rel_path ('' = project root)
PRIMARY KEY (project_id, folder)
);
-- ---------------------------------------------------------------------------
-- GPS pipeline (map view + chainage). A background sweep extracts each video's
-- GPS track with exiftool (dashcam Doc{N} samples), one video at a time:
-- gps_status: 'pending' → 'ok' (track stored) | 'none' (no GPS metadata)
-- | 'error' (exiftool/mount failure, see gps_error)
-- The RAW track is immutable once scanned; a kept "Viterbi" correction goes to
-- gps_track_corrected (map/chainage prefer it, raw is never modified).
-- Track format: [[lat, lon, t_s], ...] (t_s = seconds from first sample).
-- ---------------------------------------------------------------------------
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_status TEXT NOT NULL DEFAULT 'pending';
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_track JSONB;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_len_m DOUBLE PRECISION;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_error TEXT;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_scanned_at TIMESTAMPTZ;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_track_corrected JSONB;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_corrected_by TEXT;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_corrected_at TIMESTAMPTZ;
-- Human-readable summary of what the correction did ("2 spikes · start stitched …").
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_corrected_note TEXT;
-- Data-quality + continuity metadata from the raw scan: absolute epoch of the
-- first/last GPS fix (lets consecutive files stitch across a missing start),
-- raw sample count and the largest inter-fix gap (dropout visibility).
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_start_epoch DOUBLE PRECISION;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_end_epoch DOUBLE PRECISION;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_sample_count INTEGER;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS gps_max_gap_s DOUBLE PRECISION;
-- Real media duration read from the container by the exiftool scan. Load-bearing for
-- the stitch guard: an inter-file gap is only "missing start" when the track covers
-- LESS than the file's duration — otherwise the camera simply wasn't recording.
ALTER TABLE videos ADD COLUMN IF NOT EXISTS duration_s DOUBLE PRECISION;
CREATE INDEX IF NOT EXISTS idx_videos_gps ON videos(gps_status) WHERE gps_status = 'pending';
-- ---------------------------------------------------------------------------
-- Road types: admin-defined per-project labels (LHS/RHS/MCW/Servicelane, …)
-- assigned to videos. Injected into the export/claim JSON as `road_type` and
-- re-stamped on push (server-authoritative).
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS project_road_types (
project_id INTEGER NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
name TEXT NOT NULL,
PRIMARY KEY (project_id, name)
);
ALTER TABLE videos ADD COLUMN IF NOT EXISTS road_type TEXT NOT NULL DEFAULT '';
-- ---------------------------------------------------------------------------
-- Chainage (linear referencing with piecewise-linear calibration).
-- chainage_points = admin-entered client calibration pairs (lat/lon ↔ chainage).
-- Optional road_type tag scopes a point to one calibration group (LHS vs RHS are
-- separate routes); '' = usable by any group.
-- Per video:
-- sr_chainage_* = ACTUAL measured position along the GPS route (immutable,
-- regenerated only from GPS — never admin-editable)
-- chainage_* = client-calibrated values (error distributed between
-- consecutive control points)
-- route_seq = position of the video along the calibrated route
-- ---------------------------------------------------------------------------
CREATE TABLE IF NOT EXISTS chainage_points (
id SERIAL PRIMARY KEY,
project_id INTEGER NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
lat DOUBLE PRECISION NOT NULL,
lon DOUBLE PRECISION NOT NULL,
chainage_m DOUBLE PRECISION NOT NULL,
note TEXT NOT NULL DEFAULT '',
road_type TEXT NOT NULL DEFAULT '',
created_by TEXT NOT NULL DEFAULT '',
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_chainage_points_project ON chainage_points(project_id);
-- origin marks generated points ('quick' = the two-point quick assign) so they can
-- be replaced on the next quick run regardless of the (freely editable) note text.
ALTER TABLE chainage_points ADD COLUMN IF NOT EXISTS origin TEXT NOT NULL DEFAULT '';
UPDATE chainage_points SET origin='quick' WHERE origin='' AND note LIKE 'quick:%';
-- The full result of the last recompute (route/spans/per-video/warnings JSON) so
-- any admin who signs in later sees the existing computed data, not a recompute ask.
ALTER TABLE chainage_calibrations ADD COLUMN IF NOT EXISTS summary JSONB;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS chainage_start_m DOUBLE PRECISION;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS chainage_end_m DOUBLE PRECISION;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS sr_chainage_start_m DOUBLE PRECISION;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS sr_chainage_end_m DOUBLE PRECISION;
ALTER TABLE videos ADD COLUMN IF NOT EXISTS route_seq INTEGER;
-- The piecewise-linear calibration itself (per project + group), persisted at
-- recompute time as sorted [s_m, chainage_m] anchors. Needed to evaluate chainage at
-- ANY route position later — e.g. frame-exact per-annotation chainage in exports.
CREATE TABLE IF NOT EXISTS chainage_calibrations (
project_id INTEGER NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
road_type TEXT NOT NULL DEFAULT '',
anchors JSONB NOT NULL,
route_len_m DOUBLE PRECISION NOT NULL DEFAULT 0,
computed_by TEXT NOT NULL DEFAULT '',
computed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (project_id, road_type)
);

105
docker-compose.yml Normal file
View File

@@ -0,0 +1,105 @@
services:
db:
restart: unless-stopped
image: postgres:16-alpine
environment:
POSTGRES_USER: central
POSTGRES_PASSWORD: central
POSTGRES_DB: central
volumes:
- pgdata:/var/lib/postgresql/data
- ./db/init.sql:/docker-entrypoint-initdb.d/init.sql:ro
ports:
- "5433:5432" # host 5433 → container 5432 (avoid clashing local pg)
healthcheck:
test: ["CMD-SHELL", "pg_isready -U central -d central"]
interval: 5s
timeout: 3s
retries: 10
server:
restart: unless-stopped
# The server spawns exiftool + mount.nfs children; without an init as PID 1
# their orphans become unreapable zombies and docker can't stop the container
# ("PID … is zombie and can not be killed"). tini forwards signals + reaps.
init: true
build:
context: .
dockerfile: server/Dockerfile
environment:
DATABASE_URL: postgres://central:central@db:5432/central
BIND_ADDR: 0.0.0.0:8080
# Folder the ingest scans. Locally this is ./sample-data (mounted below);
# on the VM, mount the Synology share here instead and point INGEST_DIR at it.
INGEST_DIR: /data
# Phase 3: master token that grants admin (provision users, create projects).
# CHANGE THIS in production — it is a break-glass credential.
ADMIN_TOKEN: 198eb78d0687cca88e22f76759f0154bf85c3f926f225c6c7c84538a57c25ef7
# Claim lease length in seconds (auto-released after this if not heartbeated).
LEASE_SECS: ${LEASE_SECS:-900}
# NAS host for `nfs`-kind projects. The IP stays here (server-side); users only ever enter the
# export path (e.g. /volume4/Saudi_Video_Sync). Leave empty to disable in-app NFS mounting.
NFS_HOST: ${NFS_HOST:-192.168.1.199}
NFS_OPTS: ${NFS_OPTS:-nfsvers=4,ro,soft,timeo=30,retrans=2,retry=0}
volumes:
- ./sample-data:/data:ro
# Needed for the server to mount `nfs`-kind shares on demand. Not required if you only use
# `local`-kind projects (plain paths or compose-managed NFS volumes). If mount.nfs still fails
# on your host, replace these two with `privileged: true`.
cap_add:
- SYS_ADMIN
security_opt:
- apparmor:unconfined
depends_on:
db:
condition: service_healthy
ports:
- "8091:8080"
dashboard:
restart: unless-stopped
build: ./dashboard
environment:
# The dashboard calls the API through nginx's /api proxy, so no host needed.
VITE_API_BASE: /api
depends_on:
- server
ports:
- "8090:80"
# Nightly pg_dump → NAS, with automatic daily/weekly/monthly pruning.
# Writes gzipped SQL dumps to /backups (an NFS mount of the NAS folder below).
db-backup:
image: prodrigestivill/postgres-backup-local:16
restart: unless-stopped
environment:
POSTGRES_HOST: db
POSTGRES_PORT: "5432"
POSTGRES_DB: central
POSTGRES_USER: central
POSTGRES_PASSWORD: central
# Schedule is evaluated in this timezone (IST). 02:00 IST daily (off-peak).
TZ: Asia/Kolkata
SCHEDULE: "0 2 * * *"
# Tiered retention — tune to taste (DB is tiny, so generous is cheap).
BACKUP_KEEP_DAYS: "30"
BACKUP_KEEP_WEEKS: "8"
BACKUP_KEEP_MONTHS: "6"
# Healthcheck window so a missed run is visible (`docker inspect` health).
HEALTHCHECK_PORT: "8080"
volumes:
- nasbackups:/backups
depends_on:
db:
condition: service_healthy
volumes:
pgdata:
# NFS mount of the NAS backup folder. The folder must already exist on the NAS and
# allow rw NFS from this Docker host. Host needs an NFS client (nfs-common).
nasbackups:
driver: local
driver_opts:
type: nfs
o: "addr=${NFS_HOST:-192.168.1.199},nfsvers=4,rw,soft,timeo=30,retrans=2"
device: ":/volume4/Saudi_Video_Sync/Verification_Dashboard_Backup"

169
docs/er-diagram.md Normal file
View File

@@ -0,0 +1,169 @@
# Central DB — ER diagram
Schema for the `central/` verification dashboard (Postgres). Source of truth:
[`central/db/init.sql`](../db/init.sql) (re-run idempotently on every server boot).
This diagram was generated from the **live** database and verified against it.
> Render: GitHub renders the Mermaid block below directly; in VS Code use the
> *Markdown Preview Mermaid* extension (or any Mermaid viewer).
## Hierarchy at a glance
```
clients ──1:N──▶ projects ──1:N──▶ videos ──1:N──▶ annotations
├──1:N──▶ video_remarks (remark thread)
└──1:N──▶ video_events (claim/push log)
users — provisioned workers/admins (token auth)
audit_events — org-level action log
(both link to the hierarchy only "softly", by username string — no FK)
```
## Entityrelationship diagram
```mermaid
erDiagram
clients ||--o{ projects : "owns"
projects ||--o{ videos : "contains"
videos ||--o{ annotations : "has"
videos ||--o{ video_remarks : "thread"
videos ||--o{ video_events : "logs"
clients {
int id PK
text name UK "unique client name"
timestamptz created_at
}
projects {
int id PK
int client_id FK "→ clients.id (CASCADE); nullable"
text name "UK(client_id, name)"
text source_path "NAS export path only — host lives in NFS_HOST env"
text source_kind "'local' | 'nfs'"
timestamptz created_at
}
videos {
int id PK "= serverVideoId on the desktop"
int project_id FK "→ projects.id (CASCADE)"
text file_name
text rel_path "UK(project_id, rel_path); keeps subfolders"
bool has_json "sibling export-JSON exists"
int width
int height
float8 fps
int frame_count
int annotation_count
bigint annotation_time_ms
text primary_annotator
text status "pending | annotated | verified"
timestamptz annotated_at "max(annotation.created_at)"
jsonb raw_json "full imported export doc (Phase-3 pull)"
timestamptz ingested_at
text claimed_by "claim dimension — orthogonal to status"
timestamptz claimed_at
timestamptz lease_expires_at
text completed_by "set on push (verify pass)"
timestamptz completed_at
bigint verify_time_ms
}
annotations {
int id PK
int video_id FK "→ videos.id (CASCADE)"
int frame_number "0-indexed"
text label
text side
text shape_type "bbox | polygon"
jsonb vertices
text annotated_by "username (soft link to users)"
text review_status "default 'none'"
text remark "per-annotation note (NOT the thread)"
text subclass
text created_at
}
video_remarks {
int id PK
int video_id FK "→ videos.id (CASCADE)"
text username "author (soft link to users)"
text body
timestamptz created_at
}
video_events {
int id PK
int video_id FK "→ videos.id (CASCADE)"
text username
text event "claim | release | push | auto_release | reopen"
timestamptz at
jsonb meta
}
users {
int id PK
text username UK
text display_name
text token_hash "SHA-256 of issued token (never clear)"
text role "worker | admin"
bool active
timestamptz created_at
}
audit_events {
int id PK
timestamptz at
text username "actor"
text action "client_created | project_created | project_source_changed | synced | user_created | user_token_reset"
jsonb detail
}
```
## Relationships
| Parent | Child | Cardinality | On delete | Constraint |
|---|---|---|---|---|
| `clients.id` | `projects.client_id` | 1 : N | CASCADE | FK |
| `projects.id` | `videos.project_id` | 1 : N | CASCADE | FK |
| `videos.id` | `annotations.video_id` | 1 : N | CASCADE | FK |
| `videos.id` | `video_remarks.video_id` | 1 : N | CASCADE | FK |
| `videos.id` | `video_events.video_id` | 1 : N | CASCADE | FK |
**Soft links (by `username` string — no FK):** `users.username` is referenced as text by
`annotations.annotated_by`, `videos.claimed_by` / `completed_by` / `primary_annotator`,
`video_remarks.username`, `video_events.username`, and `audit_events.username`. These are
intentionally not foreign keys so historical attribution survives a user being removed.
## Unique constraints & key indexes
- **Unique:** `clients(name)`, `projects(client_id, name)`, `videos(project_id, rel_path)`, `users(username)`.
- **Indexes:** `idx_videos_project`, `idx_videos_status(project_id,status)`, `idx_videos_claim(project_id,claimed_by,lease_expires_at)`,
`idx_annotations_video`, `idx_annotations_user`, `idx_users_token`, `idx_video_remarks_video(video_id,created_at)`,
`idx_events_video`, `idx_events_at`, `idx_audit_at`, `idx_projects_client`.
## Notes
- **Two orthogonal dimensions on `videos`:** annotation `status` (`pending`/`annotated`/`verified`)
vs. the claim/lease columns (`claimed_by` + `lease_expires_at`). Claimable = `status IN ('pending','annotated')`
and unclaimed-or-lease-expired; push → `verified` + `completed_by`/`verify_time_ms`.
- **A range annotation** in the export doc expands to **12 rows** here (start frame, and end frame if present).
- The NAS host/IP is never persisted — `projects.source_path` holds only the export path; the host lives in the `NFS_HOST` server env.
- Everything cascades from `clients`: deleting a client removes its projects → videos → annotations/remarks/events.
---
### Regenerate / verify against the live DB
```bash
# columns
docker exec central-db-1 psql -U central -d central -c "\d+ videos"
# foreign keys
docker exec central-db-1 psql -U central -d central -Atc "
SELECT tc.table_name, kcu.column_name, ccu.table_name, rc.delete_rule
FROM information_schema.table_constraints tc
JOIN information_schema.key_column_usage kcu USING (constraint_name)
JOIN information_schema.constraint_column_usage ccu USING (constraint_name)
JOIN information_schema.referential_constraints rc USING (constraint_name)
WHERE tc.constraint_type='FOREIGN KEY';"
```

View File

View File

@@ -0,0 +1,33 @@
{
"video": {
"file_name": "2026_0508_092422_F.MP4",
"width": 1920,
"height": 1080,
"fps": 30.0,
"frame_count": 18000,
"annotation_time_ms": 1804807
},
"fixed_annotations": [
{
"label": "CCTV", "side": "Right", "frame_number": 114, "type": "bbox",
"vertices": [{"x":0.530,"y":0.306},{"x":0.768,"y":0.306},{"x":0.768,"y":0.549},{"x":0.530,"y":0.549}],
"annotated_by": "shubham", "created_at": "2026-06-10 10:22:40"
},
{
"label": "Street_Light", "side": "Right", "frame_number": 128, "type": "bbox",
"vertices": [{"x":0.541,"y":0.117},{"x":0.668,"y":0.117},{"x":0.668,"y":0.487},{"x":0.541,"y":0.487}],
"annotated_by": "shubham", "created_at": "2026-06-10 10:22:49"
},
{
"label": "Mandatory_Board", "side": "Right", "frame_number": 136, "type": "bbox",
"vertices": [{"x":0.670,"y":0.563},{"x":0.729,"y":0.563},{"x":0.729,"y":0.675},{"x":0.670,"y":0.675}],
"annotated_by": "shubham", "created_at": "2026-06-10 10:23:08"
},
{
"label": "Delineator", "side": "Right", "frame_number": 12238, "type": "bbox",
"vertices": [{"x":0.795,"y":0.644},{"x":0.880,"y":0.644},{"x":0.880,"y":0.770},{"x":0.795,"y":0.770}],
"annotated_by": "shubham", "created_at": "2026-06-09 10:28:47"
}
],
"range_annotations": []
}

View File

2377
server/Cargo.lock generated Normal file

File diff suppressed because it is too large Load Diff

22
server/Cargo.toml Normal file
View File

@@ -0,0 +1,22 @@
[package]
name = "central-server"
version = "0.1.0"
edition = "2021"
[dependencies]
axum = "0.7"
tokio = { version = "1", features = ["full"] }
tower-http = { version = "0.5", features = ["cors", "trace"] }
sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "postgres", "chrono", "json"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
chrono = { version = "0.4", features = ["serde"] }
walkdir = "2"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
anyhow = "1"
# Phase 3: token hashing (sha256) + random token generation + file-stream download.
sha2 = "0.10"
rand = "0.8"
hex = "0.4"
tokio-util = { version = "0.7", features = ["io"] }

20
server/Dockerfile Normal file
View File

@@ -0,0 +1,20 @@
# Build context is the `central/` dir (see docker-compose) so we can copy both
# server/ and db/ — the server embeds db/init.sql via include_str!("../../db/init.sql").
FROM rust:1-bookworm AS build
WORKDIR /build
COPY server/Cargo.toml server/Cargo.toml
COPY server/src server/src
COPY db db
WORKDIR /build/server
RUN cargo build --release
FROM debian:bookworm-slim
# nfs-common provides mount.nfs for `nfs`-kind projects (mounted on demand at sync/download time).
# libimage-exiftool-perl provides exiftool for the GPS-track extraction sweep (map/chainage).
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates nfs-common \
libimage-exiftool-perl \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /build/server/target/release/central-server /usr/local/bin/central-server
ENV BIND_ADDR=0.0.0.0:8080
EXPOSE 8080
CMD ["central-server"]

435
server/src/admin.rs Normal file
View File

@@ -0,0 +1,435 @@
//! Phase 3 admin API — provisioning. Gated by `AuthUser::require_admin()`, which is
//! satisfied by a user with role 'admin' or by the master `ADMIN_TOKEN`.
use crate::auth::{gen_token, sha256_hex, AuthUser};
use crate::models::*;
use crate::{err, AppState, ApiResult};
use axum::{
extract::{Path as AxPath, State},
http::StatusCode,
Json,
};
use serde_json::{json, Value};
fn is_unique_violation(e: &sqlx::Error) -> bool {
e.as_database_error().map(|d| d.is_unique_violation()).unwrap_or(false)
}
/// Record an org-level audit event (best-effort: log on failure, never block the action).
pub async fn audit(db: &sqlx::PgPool, username: &str, action: &str, detail: Value) {
if let Err(e) =
sqlx::query("INSERT INTO audit_events (username, action, detail) VALUES ($1,$2,$3)")
.bind(username)
.bind(action)
.bind(detail)
.execute(db)
.await
{
tracing::warn!("failed to record audit '{action}' by '{username}': {e}");
}
}
/// `POST /api/users` — provision a worker/admin. Generates the token, stores only its
/// hash, and returns the clear token **once** (it can never be retrieved again).
pub async fn create_user(
State(s): State<AppState>,
user: AuthUser,
Json(req): Json<NewUserRequest>,
) -> ApiResult<NewUserResponse> {
user.require_admin()?;
if req.username.trim().is_empty() {
return Err((StatusCode::BAD_REQUEST, "username is required".into()));
}
// Only two roles exist: 'admin' and 'ml_support' (anything else normalizes to ml_support).
let role = if req.role == "admin" { "admin" } else { "ml_support" };
let display_name = if req.display_name.trim().is_empty() {
req.username.clone()
} else {
req.display_name.clone()
};
let token = gen_token();
let token_hash = sha256_hex(&token);
let res = sqlx::query(
"INSERT INTO users (username, display_name, token_hash, role) VALUES ($1,$2,$3,$4)",
)
.bind(&req.username)
.bind(&display_name)
.bind(&token_hash)
.bind(role)
.execute(&s.db)
.await;
match res {
Ok(_) => {
audit(&s.db, &user.username, "user_created",
json!({ "username": req.username, "role": role })).await;
Ok(Json(NewUserResponse {
username: req.username,
display_name,
role: role.into(),
token,
}))
}
Err(e) if is_unique_violation(&e) => {
Err((StatusCode::CONFLICT, "username already exists".into()))
}
Err(e) => Err(err(e)),
}
}
/// `POST /api/users/:id/token` — regenerate a user's access token (admin only), e.g.
/// when they forget it. The old token stops working immediately; the new clear token
/// is returned **once**. The master `ADMIN_TOKEN` env is unaffected (it's not a row).
pub async fn rotate_token(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
) -> ApiResult<NewUserResponse> {
user.require_admin()?;
let token = gen_token();
let token_hash = sha256_hex(&token);
let row: Option<(String, String, String)> = sqlx::query_as(
"UPDATE users SET token_hash=$2 WHERE id=$1 RETURNING username, display_name, role",
)
.bind(id)
.bind(&token_hash)
.fetch_optional(&s.db)
.await
.map_err(err)?;
let Some((username, display_name, role)) = row else {
return Err((StatusCode::NOT_FOUND, "user not found".into()));
};
audit(&s.db, &user.username, "user_token_reset",
json!({ "user_id": id, "username": username })).await;
Ok(Json(NewUserResponse { username, display_name, role, token }))
}
/// `POST /api/users/:id/update {display_name, role}` — edit a user's display name +
/// role (admin only). Empty display_name falls back to the username. Tokens are not
/// affected (they're hashed; use rotate_token to issue a new one).
pub async fn update_user(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
Json(req): Json<UpdateUserRequest>,
) -> ApiResult<Value> {
user.require_admin()?;
let username: Option<String> = sqlx::query_scalar("SELECT username FROM users WHERE id=$1")
.bind(id)
.fetch_optional(&s.db)
.await
.map_err(err)?;
let Some(username) = username else {
return Err((StatusCode::NOT_FOUND, "user not found".into()));
};
let role = if req.role == "admin" { "admin" } else { "ml_support" };
let display_name = if req.display_name.trim().is_empty() {
username.clone()
} else {
req.display_name.trim().to_string()
};
sqlx::query("UPDATE users SET display_name=$2, role=$3 WHERE id=$1")
.bind(id)
.bind(&display_name)
.bind(role)
.execute(&s.db)
.await
.map_err(err)?;
audit(&s.db, &user.username, "user_updated",
json!({ "user_id": id, "username": username, "display_name": display_name, "role": role })).await;
Ok(Json(json!({ "updated": true, "display_name": display_name, "role": role })))
}
/// `GET /api/users` — list provisioned users (never exposes tokens).
pub async fn list_users(State(s): State<AppState>, user: AuthUser) -> ApiResult<Vec<UserRow>> {
user.require_admin()?;
let rows = sqlx::query_as::<_, UserRow>(
"SELECT id, username, display_name, role, active, created_at FROM users ORDER BY id",
)
.fetch_all(&s.db)
.await
.map_err(err)?;
Ok(Json(rows))
}
/// `POST /api/clients` — create a client. Unique on name.
pub async fn create_client(
State(s): State<AppState>,
user: AuthUser,
Json(req): Json<NewClientRequest>,
) -> ApiResult<ClientRow> {
user.require_admin()?;
if req.name.trim().is_empty() {
return Err((StatusCode::BAD_REQUEST, "client name is required".into()));
}
let res: Result<(i32, chrono::DateTime<chrono::Utc>), sqlx::Error> = sqlx::query_as(
"INSERT INTO clients (name) VALUES ($1) RETURNING id, created_at",
)
.bind(&req.name)
.fetch_one(&s.db)
.await;
match res {
Ok((id, created_at)) => {
audit(&s.db, &user.username, "client_created", json!({ "client_id": id, "name": req.name })).await;
Ok(Json(ClientRow {
id,
name: req.name,
created_at,
project_count: 0,
}))
}
Err(e) if is_unique_violation(&e) => {
Err((StatusCode::CONFLICT, "client name already exists".into()))
}
Err(e) => Err(err(e)),
}
}
fn norm_kind(k: &str) -> &'static str {
if k == "nfs" { "nfs" } else { "local" }
}
/// `POST /api/projects` — create (or repoint) a project under a client. Allowed for **any
/// authenticated user** (collaborators add projects); idempotent on `(client_id, name)`.
pub async fn create_project(
State(s): State<AppState>,
user: AuthUser,
Json(req): Json<NewProjectRequest>,
) -> ApiResult<Value> {
if req.name.trim().is_empty() {
return Err((StatusCode::BAD_REQUEST, "project name is required".into()));
}
if req.source_path.trim().is_empty() {
return Err((StatusCode::BAD_REQUEST, "source path is required".into()));
}
let kind = norm_kind(&req.source_kind);
let client_exists: Option<i32> = sqlx::query_scalar("SELECT id FROM clients WHERE id=$1")
.bind(req.client_id)
.fetch_optional(&s.db)
.await
.map_err(err)?;
if client_exists.is_none() {
return Err((StatusCode::NOT_FOUND, "client not found".into()));
}
let id: i32 = sqlx::query_scalar(
r#"INSERT INTO projects (name, source_path, source_kind, client_id) VALUES ($1,$2,$3,$4)
ON CONFLICT (client_id, name)
DO UPDATE SET source_path=EXCLUDED.source_path, source_kind=EXCLUDED.source_kind
RETURNING id"#,
)
.bind(&req.name)
.bind(&req.source_path)
.bind(kind)
.bind(req.client_id)
.fetch_one(&s.db)
.await
.map_err(err)?;
audit(&s.db, &user.username, "project_created",
json!({ "project_id": id, "client_id": req.client_id, "name": req.name,
"source_path": req.source_path, "source_kind": kind })).await;
Ok(Json(json!({
"id": id, "client_id": req.client_id, "name": req.name,
"source_path": req.source_path, "source_kind": kind
})))
}
/// `POST /api/projects/:id/source` — change a project's sync directory (any authenticated user).
pub async fn update_project_source(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
Json(req): Json<UpdateSourceRequest>,
) -> ApiResult<Value> {
if req.source_path.trim().is_empty() {
return Err((StatusCode::BAD_REQUEST, "source path is required".into()));
}
let kind = norm_kind(&req.source_kind);
let updated: Option<i32> = sqlx::query_scalar(
"UPDATE projects SET source_path=$2, source_kind=$3 WHERE id=$1 RETURNING id",
)
.bind(id)
.bind(&req.source_path)
.bind(kind)
.fetch_optional(&s.db)
.await
.map_err(err)?;
if updated.is_none() {
return Err((StatusCode::NOT_FOUND, "project not found".into()));
}
audit(&s.db, &user.username, "project_source_changed",
json!({ "project_id": id, "source_path": req.source_path, "source_kind": kind })).await;
Ok(Json(json!({ "id": id, "source_path": req.source_path, "source_kind": kind })))
}
/// `DELETE /api/projects/:id` — delete a project and everything under it (videos,
/// annotations, events, remarks all cascade). Admin only.
pub async fn delete_project(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
) -> ApiResult<Value> {
user.require_admin()?;
let name: Option<String> = sqlx::query_scalar("SELECT name FROM projects WHERE id=$1")
.bind(id)
.fetch_optional(&s.db)
.await
.map_err(err)?;
let Some(name) = name else {
return Err((StatusCode::NOT_FOUND, "project not found".into()));
};
sqlx::query("DELETE FROM projects WHERE id=$1")
.bind(id)
.execute(&s.db)
.await
.map_err(err)?;
audit(&s.db, &user.username, "project_deleted", json!({ "project_id": id, "name": name })).await;
Ok(Json(json!({ "deleted": true, "id": id })))
}
/// `DELETE /api/clients/:id` — delete a client and ALL its projects + their data
/// (cascades through projects → videos → annotations). Admin only. The dashboard
/// re-affirms (type-the-name) before calling this.
pub async fn delete_client(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
) -> ApiResult<Value> {
user.require_admin()?;
let name: Option<String> = sqlx::query_scalar("SELECT name FROM clients WHERE id=$1")
.bind(id)
.fetch_optional(&s.db)
.await
.map_err(err)?;
let Some(name) = name else {
return Err((StatusCode::NOT_FOUND, "client not found".into()));
};
let project_count: i64 =
sqlx::query_scalar("SELECT COUNT(*) FROM projects WHERE client_id=$1")
.bind(id)
.fetch_one(&s.db)
.await
.map_err(err)?;
sqlx::query("DELETE FROM clients WHERE id=$1")
.bind(id)
.execute(&s.db)
.await
.map_err(err)?;
audit(&s.db, &user.username, "client_deleted",
json!({ "client_id": id, "name": name, "projects_removed": project_count })).await;
Ok(Json(json!({ "deleted": true, "id": id, "projects_removed": project_count })))
}
/// `GET /api/projects/:id/members` — users picked for a project. Admin or member.
pub async fn list_members(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
) -> ApiResult<Vec<MemberRow>> {
crate::require_project_access(&s, &user, id).await?;
let rows = sqlx::query_as::<_, MemberRow>(
r#"SELECT m.username,
COALESCE(u.display_name, m.username) AS display_name,
COALESCE(u.role, '') AS role,
m.added_at
FROM project_members m
LEFT JOIN users u ON u.username = m.username
WHERE m.project_id=$1
ORDER BY m.username"#,
)
.bind(id)
.fetch_all(&s.db)
.await
.map_err(err)?;
Ok(Json(rows))
}
/// `POST /api/projects/:id/members {username}` — pick a user for a project. Admin only.
pub async fn add_member(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
Json(req): Json<AddMemberRequest>,
) -> ApiResult<Value> {
user.require_admin()?;
let username = req.username.trim().to_string();
if username.is_empty() {
return Err((StatusCode::BAD_REQUEST, "username is required".into()));
}
let exists: Option<i32> = sqlx::query_scalar("SELECT id FROM users WHERE username=$1 AND active")
.bind(&username)
.fetch_optional(&s.db)
.await
.map_err(err)?;
if exists.is_none() {
return Err((StatusCode::BAD_REQUEST, format!("unknown user '{username}'")));
}
sqlx::query(
"INSERT INTO project_members (project_id, username) VALUES ($1,$2) ON CONFLICT DO NOTHING",
)
.bind(id)
.bind(&username)
.execute(&s.db)
.await
.map_err(err)?;
audit(&s.db, &user.username, "project_member_added",
json!({ "project_id": id, "username": username })).await;
Ok(Json(json!({ "added": true, "username": username })))
}
/// `DELETE /api/projects/:id/members/:username` — remove a user from a project (admin
/// only). Also clears any of that user's video assignments within the project.
pub async fn remove_member(
State(s): State<AppState>,
user: AuthUser,
AxPath((id, username)): AxPath<(i32, String)>,
) -> ApiResult<Value> {
user.require_admin()?;
sqlx::query("DELETE FROM project_members WHERE project_id=$1 AND username=$2")
.bind(id)
.bind(&username)
.execute(&s.db)
.await
.map_err(err)?;
// Unassign their videos in this project so nothing stays assigned to a non-member.
sqlx::query("UPDATE videos SET assigned_to=NULL WHERE project_id=$1 AND assigned_to=$2")
.bind(id)
.bind(&username)
.execute(&s.db)
.await
.map_err(err)?;
audit(&s.db, &user.username, "project_member_removed",
json!({ "project_id": id, "username": username })).await;
Ok(Json(json!({ "removed": true, "username": username })))
}
/// `DELETE /api/users/:id` — remove a collaborator (admin or ml_support). Admin only.
/// You cannot delete the account you're signed in as (use a different admin / the
/// master token). Annotations/events keep their text `annotated_by`/`username`
/// attribution (those are not FKs), so history is preserved.
pub async fn delete_user(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
) -> ApiResult<Value> {
user.require_admin()?;
let target: Option<String> = sqlx::query_scalar("SELECT username FROM users WHERE id=$1")
.bind(id)
.fetch_optional(&s.db)
.await
.map_err(err)?;
let Some(username) = target else {
return Err((StatusCode::NOT_FOUND, "user not found".into()));
};
if username == user.username {
return Err((StatusCode::BAD_REQUEST, "you cannot delete your own account".into()));
}
sqlx::query("DELETE FROM users WHERE id=$1")
.bind(id)
.execute(&s.db)
.await
.map_err(err)?;
audit(&s.db, &user.username, "user_deleted", json!({ "user_id": id, "username": username })).await;
Ok(Json(json!({ "deleted": true, "id": id, "username": username })))
}

93
server/src/auth.rs Normal file
View File

@@ -0,0 +1,93 @@
//! Phase 3 token auth. Workers/admins authenticate with `Authorization: Bearer <token>`.
//! Tokens are random 32-byte hex strings; only their SHA-256 hash is stored
//! (`users.token_hash`). A master `ADMIN_TOKEN` env var maps to a synthetic admin
//! identity so the first real user can be provisioned (bootstrap).
use crate::AppState;
use axum::{
async_trait,
extract::FromRequestParts,
http::{header::AUTHORIZATION, request::Parts, StatusCode},
};
use rand::RngCore;
use sha2::{Digest, Sha256};
/// Lowercase hex SHA-256 of `s` — used to hash tokens before storage/lookup.
pub fn sha256_hex(s: &str) -> String {
let mut h = Sha256::new();
h.update(s.as_bytes());
hex::encode(h.finalize())
}
/// Generate a fresh 256-bit token as a 64-char hex string.
pub fn gen_token() -> String {
let mut bytes = [0u8; 32];
rand::thread_rng().fill_bytes(&mut bytes);
hex::encode(bytes)
}
/// The authenticated caller. Use as a handler argument to require a valid token.
pub struct AuthUser {
pub username: String,
pub role: String,
}
impl AuthUser {
pub fn is_admin(&self) -> bool {
self.role == "admin"
}
/// 403 unless this caller is an admin.
pub fn require_admin(&self) -> Result<(), (StatusCode, String)> {
if self.is_admin() {
Ok(())
} else {
Err((StatusCode::FORBIDDEN, "admin only".into()))
}
}
}
fn bearer(parts: &Parts) -> Option<String> {
let raw = parts.headers.get(AUTHORIZATION)?.to_str().ok()?;
raw.strip_prefix("Bearer ")
.or_else(|| raw.strip_prefix("bearer "))
.map(|t| t.trim().to_string())
.filter(|t| !t.is_empty())
}
/// Resolve a raw token to an `AuthUser`, or `None` if it's empty/unknown/inactive.
/// Used both by the `FromRequestParts` extractor (header auth) and by endpoints that
/// must accept a `?token=` query param (file downloads via `<a download>`, which can't
/// set an Authorization header).
pub async fn validate_token(state: &AppState, token: &str) -> Option<AuthUser> {
let token = token.trim();
if token.is_empty() {
return None;
}
// Master admin token (bootstrap / break-glass).
if !state.admin_token.is_empty() && token == state.admin_token {
return Some(AuthUser { username: "admin".into(), role: "admin".into() });
}
let hash = sha256_hex(token);
sqlx::query_as::<_, (String, String)>(
"SELECT username, role FROM users WHERE token_hash=$1 AND active",
)
.bind(&hash)
.fetch_optional(&state.db)
.await
.ok()
.flatten()
.map(|(username, role)| AuthUser { username, role })
}
#[async_trait]
impl FromRequestParts<AppState> for AuthUser {
type Rejection = (StatusCode, String);
async fn from_request_parts(parts: &mut Parts, state: &AppState) -> Result<Self, Self::Rejection> {
let token = bearer(parts)
.ok_or((StatusCode::UNAUTHORIZED, "missing bearer token".to_string()))?;
validate_token(state, &token)
.await
.ok_or((StatusCode::UNAUTHORIZED, "invalid or inactive token".into()))
}
}

1259
server/src/chainage.rs Normal file

File diff suppressed because it is too large Load Diff

999
server/src/gps.rs Normal file
View File

@@ -0,0 +1,999 @@
//! GPS pipeline: exiftool track extraction (background sweep), shared track math,
//! and the Stage-1 "Viterbi" smoother with its preview/keep/discard endpoint.
//!
//! Track representation everywhere: `[[lat, lon, t_s], …]` (t_s = seconds since the
//! first GPS sample). The RAW extracted track (`videos.gps_track`) is immutable once
//! scanned; a kept correction lives in `gps_track_corrected` and is preferred by the
//! map + chainage. Raw is never modified — discarding a correction restores raw.
//!
//! Lifecycle (`videos.gps_status`): 'pending' → 'ok' | 'none' (no GPS metadata) |
//! 'error' (exiftool/mount failure, message in `gps_error`). The sweep processes ONE
//! video at a time (exiftool -ee reads the whole file over NFS — never hammer the NAS).
use crate::auth::AuthUser;
use crate::{err, ApiResult, AppState};
use axum::{
extract::{Path as AxPath, State},
http::StatusCode,
Json,
};
use serde::Deserialize;
use serde_json::{json, Value};
use std::collections::BTreeMap;
use std::time::Duration;
/// One GPS sample: [lat, lon, t_s].
pub type Pt = [f64; 3];
/// Max samples stored per track (1 Hz ⇒ ~83 min of video at full fidelity).
const MAX_TRACK_PTS: usize = 5000;
/// Hard deadline for one exiftool scan (-ee reads the whole file over NFS).
const EXIFTOOL_TIMEOUT: Duration = Duration::from_secs(600);
/// Sweep pacing: idle poll when nothing is pending / pause between videos / back-off
/// after a failure (so an unreachable NAS isn't probed in a hot loop).
const SWEEP_IDLE: Duration = Duration::from_secs(60);
const SWEEP_BETWEEN: Duration = Duration::from_millis(500);
const SWEEP_AFTER_ERROR: Duration = Duration::from_secs(5);
// ---- track math (shared with chainage) ----
/// WGS84 ellipsoid: semi-major axis + first eccentricity squared.
const WGS84_A: f64 = 6_378_137.0;
const WGS84_E2: f64 = 0.006_694_379_990_14;
/// Meters per degree of latitude / longitude at a given latitude, from the WGS84
/// meridian (M) and prime-vertical (N) radii. Distance and snapping both use this,
/// so every meter in the system is measured the same way.
pub fn wgs84_m_per_deg(lat_deg: f64) -> (f64, f64) {
let phi = lat_deg.to_radians();
let w = (1.0 - WGS84_E2 * phi.sin().powi(2)).sqrt();
let m = WGS84_A * (1.0 - WGS84_E2) / (w * w * w);
let n = WGS84_A / w;
let rad = std::f64::consts::PI / 180.0;
(m * rad, n * phi.cos() * rad)
}
/// WGS84 local-ellipsoid distance between two nearby points. Essentially exact at
/// GPS sample spacing (meterstens of meters) and <2e-5 relative for spans up to
/// ~100 km — unlike spherical haversine, whose 0.10.3% systematic bias would land
/// directly in sr_chainage. (Not valid across the ±180° meridian — irrelevant here.)
pub fn dist_m(lat1: f64, lon1: f64, lat2: f64, lon2: f64) -> f64 {
let (mlat, mlon) = wgs84_m_per_deg((lat1 + lat2) * 0.5);
let dy = (lat2 - lat1) * mlat;
let dx = (lon2 - lon1) * mlon;
(dx * dx + dy * dy).sqrt()
}
pub fn polyline_len_m(track: &[Pt]) -> f64 {
track
.windows(2)
.map(|w| dist_m(w[0][0], w[0][1], w[1][0], w[1][1]))
.sum()
}
/// Parse a stored JSONB track (`[[lat,lon,t_s],…]`) back into points. Tolerant:
/// skips malformed entries.
pub fn parse_track(v: &Value) -> Vec<Pt> {
v.as_array()
.map(|a| {
a.iter()
.filter_map(|p| {
let p = p.as_array()?;
Some([p.first()?.as_f64()?, p.get(1)?.as_f64()?, p.get(2).and_then(|t| t.as_f64()).unwrap_or(0.0)])
})
.collect()
})
.unwrap_or_default()
}
/// Serialize a track for storage/transport, rounding to keep the JSON small
/// (1e-6 deg ≈ 0.1 m; 0.1 s time resolution).
pub fn track_value(track: &[Pt]) -> Value {
Value::Array(
track
.iter()
.map(|p| {
json!([
(p[0] * 1e6).round() / 1e6,
(p[1] * 1e6).round() / 1e6,
(p[2] * 10.0).round() / 10.0
])
})
.collect(),
)
}
/// Average + maximum speed over a track (m/s), from the full-resolution samples.
/// Max is measured over single hops (dt clamped ≥ 0.5 s so duplicate timestamps
/// can't fabricate teleports). None when the track has no elapsed time.
pub fn speed_stats(track: &[Pt]) -> (Option<f64>, Option<f64>) {
if track.len() < 2 {
return (None, None);
}
let elapsed = track.last().unwrap()[2] - track[0][2];
let avg = (elapsed > 0.0).then(|| polyline_len_m(track) / elapsed);
let max = track
.windows(2)
.map(|w| dist_m(w[0][0], w[0][1], w[1][0], w[1][1]) / (w[1][2] - w[0][2]).max(0.5))
.fold(None::<f64>, |acc, v| Some(acc.map_or(v, |a| a.max(v))));
(avg, max)
}
/// Cap a track at `max` points (uniform stride, always keeping the last point).
pub fn downsample(track: Vec<Pt>, max: usize) -> Vec<Pt> {
if track.len() <= max || max < 2 {
return track;
}
let stride = track.len().div_ceil(max);
let last = *track.last().unwrap();
let mut out: Vec<Pt> = track.into_iter().step_by(stride).collect();
if out.last() != Some(&last) {
out.push(last);
}
out
}
// ---- exiftool extraction ----
/// What one exiftool scan yields: the GPS samples (t_s relative to the first fix),
/// the first fix's absolute epoch (None when the camera wrote no GPSDateTime), the
/// container's media duration in seconds, and the video frame rate. Empty samples =
/// no GPS metadata. Duration + fps let us do frame→position math on raw NAS videos
/// that were never annotated (no sibling JSON ⇒ no fps in our DB otherwise).
struct ScanResult {
samples: Vec<Pt>,
t0_abs: Option<f64>,
duration_s: Option<f64>,
fps: Option<f64>,
}
/// Run exiftool over a video and collect its embedded GPS time-series (Doc{N}
/// groups, the format dashcams/DJI write at ~1 Hz). Same flags as the proven
/// desktop extractor, plus container Duration + VideoFrameRate.
async fn run_exiftool(path: &std::path::Path) -> Result<ScanResult, String> {
let child = tokio::process::Command::new("exiftool")
.args([
"-ee", "-api", "LargeFileSupport=1", "-G3", "-json", "-n",
"-GPSLatitude", "-GPSLongitude", "-GPSDateTime", "-Duration", "-VideoFrameRate",
])
.arg(path)
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.kill_on_drop(true)
.spawn()
.map_err(|e| format!("failed to run exiftool (is it installed?): {e}"))?;
let output = match tokio::time::timeout(EXIFTOOL_TIMEOUT, child.wait_with_output()).await {
Ok(r) => r.map_err(|e| format!("exiftool failed: {e}"))?,
// kill_on_drop terminates the scan when the timeout drops the child.
Err(_) => return Err("exiftool timed out (file too slow to read?)".into()),
};
let stdout = String::from_utf8_lossy(&output.stdout);
let parsed: Value = serde_json::from_str(stdout.trim()).map_err(|_| {
let se = String::from_utf8_lossy(&output.stderr);
let msg: String = se.trim().chars().take(300).collect();
if msg.is_empty() {
"exiftool produced no output".to_string()
} else {
format!("exiftool: {msg}")
}
})?;
let obj = parsed
.as_array()
.and_then(|a| a.first())
.and_then(|v| v.as_object())
.ok_or_else(|| "exiftool output was not a non-empty array".to_string())?;
// exiftool reports per-file problems inline (e.g. "Error: File is empty").
if let Some((_, e)) = obj.iter().find(|(k, _)| *k == "Error" || k.ends_with(":Error")) {
if let Some(msg) = e.as_str() {
return Err(format!("exiftool: {msg}"));
}
}
// Numeric field across any -G3 group (container + per-track may each report one);
// `reduce` picks the max, which is the sensible choice for both Duration and fps.
let numeric = |name: &str| -> Option<f64> {
obj.iter()
.filter(|(k, _)| k.as_str() == name || k.ends_with(&format!(":{name}")))
.filter_map(|(_, v)| v.as_f64().or_else(|| v.as_str().and_then(|s| s.parse().ok())))
.reduce(f64::max)
};
let duration = numeric("Duration");
let fps = numeric("VideoFrameRate").filter(|f| *f > 0.0);
let (samples, t0_abs) = collect_samples(obj);
Ok(ScanResult { samples, t0_abs, duration_s: duration, fps })
}
/// Group `Doc{N}:GPSLatitude/GPSLongitude/GPSDateTime` keys into ordered samples
/// with relative timestamps (falls back to 1 Hz when GPSDateTime is missing).
/// Also returns the first fix's ABSOLUTE epoch — GPS time is UTC on every camera,
/// so consecutive files can be matched by timestamp (start-of-video stitching).
fn collect_samples(obj: &serde_json::Map<String, Value>) -> (Vec<Pt>, Option<f64>) {
#[derive(Default)]
struct Partial {
lat: Option<f64>,
lon: Option<f64>,
t: Option<f64>,
}
let mut by_doc: BTreeMap<u32, Partial> = BTreeMap::new();
for (k, v) in obj {
let Some((doc_idx, field)) = split_doc_key(k) else { continue };
let entry = by_doc.entry(doc_idx).or_default();
match field {
"GPSLatitude" => entry.lat = v.as_f64(),
"GPSLongitude" => entry.lon = v.as_f64(),
// Plausibility gate: cameras write garbage dates (1970/zero) on some
// fixes — treat anything outside [2000, ~2096] as unstamped.
"GPSDateTime" => {
entry.t = v
.as_str()
.and_then(parse_gps_datetime)
.filter(|&t| (946_684_800.0..4_100_000_000.0).contains(&t))
}
_ => {}
}
}
let raw: Vec<(u32, f64, f64, Option<f64>)> = by_doc
.into_iter()
.filter_map(|(idx, p)| Some((idx, p.lat?, p.lon?, p.t)))
.collect();
let t0_abs = raw.first().and_then(|r| r.3); // real epoch only when stamped
// ONE timebase for the whole track. Mixing them (epoch for stamped samples,
// doc index for unstamped ones) once produced billion-second "dropouts" when a
// camera skipped GPSDateTime on some fixes — so unless EVERY sample is stamped,
// fall back to index-based 1 Hz for all of them.
let index_based = |raw: &[(u32, f64, f64, Option<f64>)]| -> Vec<Pt> {
let idx0 = raw.first().map(|r| r.0).unwrap_or(0);
raw.iter().map(|&(idx, lat, lon, _)| [lat, lon, (idx - idx0) as f64]).collect()
};
let all_stamped = !raw.is_empty() && raw.iter().all(|r| r.3.is_some());
let mut samples: Vec<Pt> = if all_stamped {
let t0 = raw[0].3.unwrap();
raw.iter().map(|&(_, lat, lon, t)| [lat, lon, t.unwrap() - t0]).collect()
} else {
index_based(&raw)
};
// Clock-glitch guard: a negative or >1 h step between consecutive fixes means
// the stamps themselves are unusable (camera clock jumped) — rebuild the whole
// track on the index timebase rather than trust any of them.
if samples
.windows(2)
.any(|w| !(0.0..=3600.0).contains(&(w[1][2] - w[0][2])))
{
samples = index_based(&raw);
}
(samples, t0_abs)
}
fn split_doc_key(k: &str) -> Option<(u32, &str)> {
let (group, field) = k.split_once(':')?;
let idx: u32 = group.strip_prefix("Doc")?.parse().ok()?;
Some((idx, field))
}
/// "2025:08:02 09:33:31Z" → seconds since epoch (approx; only relative deltas are
/// used). Civil-to-days via Howard Hinnant's algorithm — no chrono parse needed for
/// exiftool's colon-separated dates.
fn parse_gps_datetime(s: &str) -> Option<f64> {
let s = s.trim_end_matches('Z').trim();
let (date, time) = s.split_once(' ')?;
let mut dp = date.split(':');
let y: i64 = dp.next()?.parse().ok()?;
let mo: i64 = dp.next()?.parse().ok()?;
let d: i64 = dp.next()?.parse().ok()?;
let mut tp = time.split(':');
let h: i64 = tp.next()?.parse().ok()?;
let mi: i64 = tp.next()?.parse().ok()?;
let sec: f64 = tp.next()?.parse().ok()?;
let y_adj = if mo <= 2 { y - 1 } else { y };
let era = y_adj.div_euclid(400);
let yoe = y_adj - era * 400;
let m = if mo > 2 { mo - 3 } else { mo + 9 };
let doy = (153 * m + 2) / 5 + d - 1;
let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
let days = era * 146097 + doe - 719468;
Some(days as f64 * 86400.0 + (h * 3600 + mi * 60) as f64 + sec)
}
// ---- background sweep ----
/// Forever-loop: pick ONE pending video, scan it, store the result, repeat.
/// Serial by design (requirement: throttle — never scan the NAS in parallel).
pub async fn sweep(state: AppState) {
loop {
let next: Option<(i32, String, String, String)> = sqlx::query_as(
r#"SELECT v.id, v.rel_path, p.source_path, p.source_kind
FROM videos v JOIN projects p ON v.project_id = p.id
WHERE v.gps_status = 'pending'
ORDER BY v.id LIMIT 1"#,
)
.fetch_optional(&state.db)
.await
.unwrap_or(None);
let Some((id, rel_path, source_path, source_kind)) = next else {
tokio::time::sleep(SWEEP_IDLE).await;
continue;
};
let ok = scan_one(&state, id, &rel_path, &source_path, &source_kind).await;
tokio::time::sleep(if ok { SWEEP_BETWEEN } else { SWEEP_AFTER_ERROR }).await;
}
}
/// Scan a single video and persist the outcome. Returns false on error (the video
/// is marked 'error' either way, so the sweep never re-picks it until a re-scan).
async fn scan_one(state: &AppState, id: i32, rel_path: &str, source_path: &str, source_kind: &str) -> bool {
let (sp, kind, host, opts) =
(source_path.to_string(), source_kind.to_string(), state.nfs_host.clone(), state.nfs_opts.clone());
let dir = match tokio::task::spawn_blocking(move || crate::nfs::resolve_dir(&sp, &kind, &host, &opts)).await {
Ok(Ok(d)) => d,
Ok(Err(e)) => return mark_error(&state.db, id, &e).await,
Err(e) => return mark_error(&state.db, id, &format!("mount task failed: {e}")).await,
};
match run_exiftool(&dir.join(rel_path)).await {
Ok(ScanResult { samples, t0_abs, duration_s, fps }) if samples.len() >= 2 => {
// Data-quality stats are measured BEFORE the storage cap: the raw fix
// count and the largest inter-fix gap (dropouts) must reflect reality.
let sample_count = samples.len() as i32;
let max_gap = samples.windows(2).map(|w| w[1][2] - w[0][2]).fold(0.0_f64, f64::max);
let end_epoch = t0_abs.map(|t0| t0 + samples.last().unwrap()[2]);
let samples = downsample(samples, MAX_TRACK_PTS);
let len = polyline_len_m(&samples);
let res = sqlx::query(
r#"UPDATE videos SET gps_status='ok', gps_track=$2, gps_len_m=$3, gps_error=NULL,
gps_scanned_at=now(),
gps_start_epoch=$4, gps_end_epoch=$5, gps_sample_count=$6, gps_max_gap_s=$7,
duration_s=$8,
-- fill fps for raw NAS videos (no sibling JSON) so frame→position
-- math works; never clobber a JSON/desktop-derived fps.
fps=COALESCE(fps, $9),
-- a fresh raw track invalidates any correction derived from the old one
gps_track_corrected=NULL, gps_corrected_by=NULL, gps_corrected_at=NULL,
gps_corrected_note=NULL
WHERE id=$1"#,
)
.bind(id)
.bind(track_value(&samples))
.bind(len)
.bind(t0_abs)
.bind(end_epoch)
.bind(sample_count)
.bind(max_gap)
.bind(duration_s)
.bind(fps)
.execute(&state.db)
.await;
match res {
Ok(_) => {
tracing::info!("gps: video {id} ok — {sample_count} samples, {len:.0} m, max gap {max_gap:.0}s");
// Auto-correct this video (spikes + start stitch), then its
// temporal successor — its start stitch may only now be possible.
auto_correct(&state.db, id).await;
if let Some(end) = end_epoch {
let succ: Option<i32> = sqlx::query_scalar(
r#"SELECT v.id FROM videos v, videos me
WHERE me.id=$1 AND v.id<>$1 AND v.project_id=me.project_id
AND v.gps_status='ok' AND v.gps_start_epoch IS NOT NULL
AND v.gps_start_epoch > $2 AND v.gps_start_epoch - $2 <= $3
AND (CASE WHEN v.rel_path LIKE '%/%' THEN regexp_replace(v.rel_path,'/[^/]*$','') ELSE '' END)
= (CASE WHEN me.rel_path LIKE '%/%' THEN regexp_replace(me.rel_path,'/[^/]*$','') ELSE '' END)
ORDER BY v.gps_start_epoch LIMIT 1"#,
)
.bind(id)
.bind(end)
.bind(STITCH_MAX_GAP_S)
.fetch_optional(&state.db)
.await
.unwrap_or(None);
if let Some(succ) = succ {
auto_correct(&state.db, succ).await;
}
}
true
}
Err(e) => mark_error(&state.db, id, &format!("db write failed: {e}")).await,
}
}
Ok(_) => {
let _ = sqlx::query(
"UPDATE videos SET gps_status='none', gps_error=NULL, gps_scanned_at=now() WHERE id=$1",
)
.bind(id)
.execute(&state.db)
.await;
tracing::info!("gps: video {id} has no GPS metadata");
true
}
Err(e) => mark_error(&state.db, id, &e).await,
}
}
async fn mark_error(db: &sqlx::PgPool, id: i32, msg: &str) -> bool {
tracing::warn!("gps: video {id} scan failed: {msg}");
let _ = sqlx::query(
"UPDATE videos SET gps_status='error', gps_error=$2, gps_scanned_at=now() WHERE id=$1",
)
.bind(id)
.bind(msg)
.execute(db)
.await;
false
}
// ---- Stage-1 "Viterbi" smoother ----
/// Speed gate: hops implying more than this are implausible (≈160 km/h).
const MAX_SPEED_MPS: f64 = 45.0;
/// Cost of dropping (re-interpolating) one sample. An outlier is dropped when
/// keeping it costs more than this.
const DROP_PENALTY: f64 = 30.0;
/// Max consecutive dropped samples + 1 (DP transition window).
const VITERBI_WINDOW: usize = 12;
/// Min-cost keep/drop path over the samples (Viterbi over a keep-lattice):
/// transition cost between consecutive KEPT samples grows quadratically with the
/// implied speed above `MAX_SPEED_MPS`; every dropped sample costs `DROP_PENALTY`.
/// Dropped samples are re-interpolated (by time) between their kept neighbors —
/// principled outlier rejection with no road network. Returns the corrected track
/// (same length + timestamps as the input) and how many samples were replaced.
pub fn viterbi_smooth(track: &[Pt]) -> (Vec<Pt>, usize) {
let n = track.len();
if n < 3 {
return (track.to_vec(), 0);
}
let trans_cost = |a: &Pt, b: &Pt| -> f64 {
let dt = (b[2] - a[2]).max(0.5);
let v = dist_m(a[0], a[1], b[0], b[1]) / dt;
let excess = (v - MAX_SPEED_MPS).max(0.0);
(excess / 5.0).powi(2)
};
// cost[i] = min cost of a path whose last kept sample is i.
let mut cost = vec![f64::INFINITY; n];
let mut prev = vec![usize::MAX; n];
for i in 0..n {
if i <= VITERBI_WINDOW {
// Start the path at i, paying for the dropped leading samples.
cost[i] = DROP_PENALTY * i as f64;
}
for j in i.saturating_sub(VITERBI_WINDOW)..i {
if cost[j].is_finite() {
let c = cost[j] + DROP_PENALTY * (i - j - 1) as f64 + trans_cost(&track[j], &track[i]);
if c < cost[i] {
cost[i] = c;
prev[i] = j;
}
}
}
}
// End the path at the sample minimizing total cost incl. dropped trailing ones.
let mut end = n - 1;
let mut best = f64::INFINITY;
for i in n.saturating_sub(VITERBI_WINDOW + 1)..n {
let c = cost[i] + DROP_PENALTY * (n - 1 - i) as f64;
if c < best {
best = c;
end = i;
}
}
let mut kept = vec![false; n];
let mut i = end;
loop {
kept[i] = true;
if prev[i] == usize::MAX {
break;
}
i = prev[i];
}
// Rebuild: kept samples stay; dropped ones are re-interpolated between the
// nearest kept neighbors (clamped to the first/last kept at the edges).
let kept_idx: Vec<usize> = (0..n).filter(|&k| kept[k]).collect();
let mut out = track.to_vec();
let mut replaced = 0usize;
for k in 0..n {
if kept[k] {
continue;
}
let after = kept_idx.partition_point(|&ki| ki < k);
let (lo, hi) = match (after.checked_sub(1).map(|a| kept_idx[a]), kept_idx.get(after)) {
(Some(lo), Some(&hi)) => (lo, hi),
(None, Some(&hi)) => (hi, hi),
(Some(lo), None) => (lo, lo),
(None, None) => continue, // unreachable: at least one sample is kept
};
let (a, b) = (&track[lo], &track[hi]);
let f = if b[2] > a[2] { ((track[k][2] - a[2]) / (b[2] - a[2])).clamp(0.0, 1.0) } else { 0.0 };
out[k] = [a[0] + (b[0] - a[0]) * f, a[1] + (b[1] - a[1]) * f, track[k][2]];
replaced += 1;
}
(out, replaced)
}
// ---- auto-correction: spikes + start-of-video stitching ----
/// Two files are one continuous recording only when the GPS gap between them is at
/// most this many seconds.
const STITCH_MAX_GAP_S: f64 = 180.0;
/// Gaps at or below this are the normal ~1 Hz cadence across a file boundary —
/// nothing is missing.
const STITCH_MIN_GAP_S: f64 = 2.0;
/// Decide the time-shift for a missing-start stitch. `gap_s` = seconds between the
/// predecessor's last fix and this video's first fix; `span_s` = this track's own
/// first→last duration; `duration_s` = the video's real length when known
/// (frame_count / fps); `jump_m` = distance from the predecessor's last fix to this
/// video's first fix. Returns the seconds this track starts late (its samples are
/// shifted by this), or None when nothing should be stitched.
fn plan_stitch(gap_s: f64, span_s: f64, duration_s: Option<f64>, jump_m: f64) -> Option<f64> {
if gap_s <= STITCH_MIN_GAP_S || gap_s > STITCH_MAX_GAP_S {
return None; // healthy rollover, or not a continuous recording
}
if jump_m / gap_s > MAX_SPEED_MPS {
return None; // the vehicle can't have covered the jump — different session
}
let mut shift = gap_s - 1.0; // one normal sample period isn't missing data
if let Some(dur) = duration_s {
// The track already covers the whole file ⇒ the gap was the PREDECESSOR's
// problem (missing end), not ours — shifting would misalign every frame.
let missing = dur - span_s;
if missing < STITCH_MIN_GAP_S {
return None;
}
shift = shift.min(missing);
}
(shift > 0.5).then_some(shift)
}
/// The unified correction result: Viterbi spike removal + (when a temporal
/// predecessor exists in the same folder) a missing-start stitch that re-bases the
/// clock so t = 0 is the actual video start. Raw is never modified.
pub struct CorrectionOutcome {
pub raw: Vec<Pt>,
pub corrected: Vec<Pt>,
pub replaced: usize,
pub stitched_from: Option<String>,
pub shift_s: f64,
pub note: String,
}
impl CorrectionOutcome {
pub fn changed(&self) -> bool {
self.replaced > 0 || self.stitched_from.is_some()
}
}
/// Compute (but do not store) the correction for a video. Errors when the video has
/// no usable raw track.
pub async fn compute_correction(db: &sqlx::PgPool, video_id: i32) -> Result<CorrectionOutcome, String> {
#[allow(clippy::type_complexity)]
let row: Option<(Option<Value>, Option<f64>, Option<f64>, Option<i32>, Option<f64>, i32, String)> = sqlx::query_as(
r#"SELECT gps_track, gps_start_epoch, fps, frame_count, duration_s, project_id, rel_path
FROM videos WHERE id=$1 AND gps_status='ok'"#,
)
.bind(video_id)
.fetch_optional(db)
.await
.map_err(|e| e.to_string())?;
let Some((track, start_epoch, fps, frame_count, duration_s, project_id, rel_path)) = row else {
return Err("video has no GPS track".into());
};
let raw = track.as_ref().map(parse_track).unwrap_or_default();
if raw.len() < 2 {
return Err("video has no GPS track".into());
}
let (mut corrected, replaced) = viterbi_smooth(&raw);
let mut notes: Vec<String> = Vec::new();
if replaced > 0 {
notes.push(format!("{replaced} spike(s) re-interpolated"));
}
// Missing-start stitch: the nearest earlier-ending video in the same folder,
// matched by absolute GPS time (UTC on every camera — no timezone games).
let mut stitched_from = None;
let mut shift_s = 0.0;
if let Some(start_epoch) = start_epoch {
let folder = match rel_path.rfind('/') {
Some(i) => rel_path[..i].to_string(),
None => String::new(),
};
let prev: Option<(String, f64, Value)> = sqlx::query_as(
r#"SELECT file_name, gps_end_epoch, gps_track FROM videos
WHERE project_id=$1 AND id<>$2 AND gps_track IS NOT NULL AND gps_end_epoch IS NOT NULL
AND gps_end_epoch < $3 AND $3 - gps_end_epoch <= $4
AND (CASE WHEN rel_path LIKE '%/%' THEN regexp_replace(rel_path,'/[^/]*$','') ELSE '' END) = $5
ORDER BY gps_end_epoch DESC LIMIT 1"#,
)
.bind(project_id)
.bind(video_id)
.bind(start_epoch)
.bind(STITCH_MAX_GAP_S)
.bind(&folder)
.fetch_optional(db)
.await
.map_err(|e| e.to_string())?;
if let Some((prev_name, prev_end, prev_track)) = prev {
// Borrow from the predecessor's RAW track (clean provenance — its own
// correction may already contain stitched points).
let prev_pts = parse_track(&prev_track);
if let Some(anchor) = prev_pts.last() {
let gap = start_epoch - prev_end;
let span = corrected.last().unwrap()[2] - corrected[0][2];
// Container duration (from the scan) first; the annotation-doc
// fps/frame_count only as a fallback for pre-upgrade rows.
let duration = duration_s.or(match (fps, frame_count) {
(Some(f), Some(n)) if f > 0.0 => Some(n as f64 / f),
_ => None,
});
let jump = dist_m(anchor[0], anchor[1], corrected[0][0], corrected[0][1]);
if let Some(shift) = plan_stitch(gap, span, duration, jump) {
for p in corrected.iter_mut() {
p[2] += shift;
}
corrected.insert(0, [anchor[0], anchor[1], 0.0]);
shift_s = shift;
notes.push(format!(
"start stitched from {prev_name} (first {shift:.1}s of GPS were missing — clock re-based)"
));
stitched_from = Some(prev_name);
}
}
}
}
Ok(CorrectionOutcome {
raw,
corrected,
replaced,
stitched_from,
shift_s,
note: notes.join(" · "),
})
}
/// Compute + store the correction as 'auto' — but never over a human decision:
/// only when the video is untouched or its current correction is itself 'auto'.
/// A no-op correction clears a stale auto one. Best-effort (sweep path).
pub async fn auto_correct(db: &sqlx::PgPool, video_id: i32) {
let eligible: Option<bool> = sqlx::query_scalar(
"SELECT (gps_corrected_by IS NULL OR gps_corrected_by='auto') FROM videos WHERE id=$1 AND gps_status='ok'",
)
.bind(video_id)
.fetch_optional(db)
.await
.ok()
.flatten();
if eligible != Some(true) {
return;
}
match compute_correction(db, video_id).await {
Ok(o) if o.changed() => {
let len = polyline_len_m(&o.corrected);
let res = sqlx::query(
r#"UPDATE videos SET gps_track_corrected=$2, gps_corrected_by='auto',
gps_corrected_at=now(), gps_corrected_note=$3, gps_len_m=$4 WHERE id=$1"#,
)
.bind(video_id)
.bind(track_value(&o.corrected))
.bind(&o.note)
.bind(len)
.execute(db)
.await;
match res {
Ok(_) => tracing::info!("gps: video {video_id} auto-corrected — {}", o.note),
Err(e) => tracing::warn!("gps: video {video_id} auto-correct store failed: {e}"),
}
}
Ok(o) => {
// Nothing to fix (anymore) — drop a stale auto correction if present.
let _ = sqlx::query(
r#"UPDATE videos SET gps_track_corrected=NULL, gps_corrected_by=NULL,
gps_corrected_at=NULL, gps_corrected_note=NULL, gps_len_m=$2
WHERE id=$1 AND gps_corrected_by='auto'"#,
)
.bind(video_id)
.bind(polyline_len_m(&o.raw))
.execute(db)
.await;
}
Err(e) => tracing::debug!("gps: video {video_id} auto-correct skipped: {e}"),
}
}
// ---- endpoints ----
/// `POST /api/videos/:id/gps/rescan` — queue a video for a fresh metadata scan
/// (admin). The sweep picks it up; any previous error is cleared.
pub async fn rescan(State(s): State<AppState>, user: AuthUser, AxPath(id): AxPath<i32>) -> ApiResult<Value> {
user.require_admin()?;
let updated: Option<i32> =
sqlx::query_scalar("UPDATE videos SET gps_status='pending', gps_error=NULL WHERE id=$1 RETURNING id")
.bind(id)
.fetch_optional(&s.db)
.await
.map_err(err)?;
if updated.is_none() {
return Err((StatusCode::NOT_FOUND, "video not found".into()));
}
Ok(Json(json!({ "queued": true })))
}
#[derive(Deserialize)]
pub struct CorrectRequest {
/// 'preview' (compute, don't save) | 'keep' (save corrected) | 'discard' (revert to raw).
pub action: String,
}
/// `POST /api/videos/:id/gps/correct {action}` — the keep-or-discard correction flow
/// (admin), running the SAME pipeline the scan sweep auto-applies (spike removal +
/// missing-start stitch). Preview returns original + corrected tracks so the map can
/// overlay them; keep persists (attributed to the admin); discard reverts to raw AND
/// tombstones the video so the sweep won't re-auto-correct it (a re-scan resets that).
pub async fn correct(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
Json(req): Json<CorrectRequest>,
) -> ApiResult<Value> {
user.require_admin()?;
let exists: Option<String> = sqlx::query_scalar("SELECT gps_status FROM videos WHERE id=$1")
.bind(id)
.fetch_optional(&s.db)
.await
.map_err(err)?;
let Some(gps_status) = exists else {
return Err((StatusCode::NOT_FOUND, "video not found".into()));
};
if gps_status != "ok" {
return Err((StatusCode::BAD_REQUEST, "video has no GPS track".into()));
}
match req.action.as_str() {
"discard" => {
let raw_track: Option<Value> = sqlx::query_scalar("SELECT gps_track FROM videos WHERE id=$1")
.bind(id)
.fetch_one(&s.db)
.await
.map_err(err)?;
let len = raw_track.as_ref().map(|t| polyline_len_m(&parse_track(t))).unwrap_or_default();
// Tombstone: gps_corrected_by keeps the admin's name with a NULL track,
// so auto_correct (which only touches NULL/'auto') leaves this video raw.
sqlx::query(
r#"UPDATE videos SET gps_track_corrected=NULL,
gps_corrected_by=$2, gps_corrected_at=now(),
gps_corrected_note='reverted to raw — auto-correction off (re-scan to reset)',
gps_len_m=$3
WHERE id=$1"#,
)
.bind(id)
.bind(&user.username)
.bind(len)
.execute(&s.db)
.await
.map_err(err)?;
Ok(Json(json!({ "action": "discard", "len_m": len })))
}
"preview" | "keep" => {
let o = compute_correction(&s.db, id).await.map_err(|e| (StatusCode::BAD_REQUEST, e))?;
let len_before = polyline_len_m(&o.raw);
let len_after = polyline_len_m(&o.corrected);
if req.action == "keep" {
sqlx::query(
r#"UPDATE videos SET gps_track_corrected=$2, gps_corrected_by=$3,
gps_corrected_at=now(), gps_corrected_note=$4, gps_len_m=$5 WHERE id=$1"#,
)
.bind(id)
.bind(track_value(&o.corrected))
.bind(&user.username)
.bind(&o.note)
.bind(len_after)
.execute(&s.db)
.await
.map_err(err)?;
crate::worker::log_event(&s.db, id, &user.username, "gps_corrected",
json!({ "replaced": o.replaced, "stitched": o.stitched_from.is_some() })).await;
}
Ok(Json(json!({
"action": req.action,
"points": o.raw.len(),
"replaced": o.replaced,
"stitched_from": o.stitched_from,
"shift_s": o.shift_s,
"note": o.note,
"len_before_m": len_before,
"len_after_m": len_after,
"original": track_value(&o.raw),
"corrected": track_value(&o.corrected),
})))
}
_ => Err((StatusCode::BAD_REQUEST, "action must be preview, keep or discard".into())),
}
}
/// `POST /api/projects/:id/gps/rescan_all` — queue EVERY already-scanned video in the
/// project for a fresh scan (admin). Needed once after upgrading: pre-upgrade scans
/// have no absolute timestamps, so stitching/data-quality stats only exist after a
/// re-scan. The serial sweep works through them in the background.
pub async fn rescan_all(State(s): State<AppState>, user: AuthUser, AxPath(id): AxPath<i32>) -> ApiResult<Value> {
user.require_admin()?;
let n = sqlx::query(
"UPDATE videos SET gps_status='pending', gps_error=NULL WHERE project_id=$1 AND gps_status <> 'pending'",
)
.bind(id)
.execute(&s.db)
.await
.map_err(err)?
.rows_affected();
crate::admin::audit(&s.db, &user.username, "gps_rescan_all", json!({ "project_id": id, "queued": n })).await;
Ok(Json(json!({ "queued": n })))
}
#[cfg(test)]
mod tests {
use super::*;
/// A straight east-west track at the equator: 1 pt/s, ~11.1 m apart (40 km/h).
fn straight(n: usize) -> Vec<Pt> {
(0..n).map(|i| [0.0, i as f64 * 0.0001, i as f64]).collect()
}
#[test]
fn smoother_removes_a_jump() {
let mut t = straight(60);
// one wild point 0.01° (~1.1 km) off the line
t[30][0] = 0.01;
let (fixed, replaced) = viterbi_smooth(&t);
assert_eq!(replaced, 1);
assert!(fixed[30][0].abs() < 1e-6, "outlier not re-interpolated: {}", fixed[30][0]);
// everything else untouched
assert_eq!(fixed[29], t[29]);
assert_eq!(fixed[31], t[31]);
}
#[test]
fn smoother_keeps_a_clean_track() {
let t = straight(60);
let (fixed, replaced) = viterbi_smooth(&t);
assert_eq!(replaced, 0);
assert_eq!(fixed, t);
}
#[test]
fn mixed_gps_datetime_never_mixes_timebases() {
// Doc1 has NO GPSDateTime, Doc2/Doc3 do (real epoch ~1.69e9). The old code
// seeded t0 from the index and then subtracted it from the epoch — a
// 1,693,958,401-second "dropout". Every sample must share one timebase.
let mut obj = serde_json::Map::new();
let ins = |o: &mut serde_json::Map<String, Value>, d: u32, lat: f64, t: Option<&str>| {
o.insert(format!("Doc{d}:GPSLatitude"), serde_json::json!(lat));
o.insert(format!("Doc{d}:GPSLongitude"), serde_json::json!(78.0));
if let Some(t) = t {
o.insert(format!("Doc{d}:GPSDateTime"), serde_json::json!(t));
}
};
ins(&mut obj, 1, 21.0, None);
ins(&mut obj, 2, 21.0001, Some("2023:09:06 00:00:01Z"));
ins(&mut obj, 3, 21.0002, Some("2023:09:06 00:00:02Z"));
let (samples, t0_abs) = collect_samples(&obj);
assert_eq!(samples.len(), 3);
let max_gap = samples.windows(2).map(|w| w[1][2] - w[0][2]).fold(0.0_f64, f64::max);
assert!(max_gap <= 1.0 + 1e-9, "timebases mixed: max gap {max_gap}");
assert_eq!(samples[0][2], 0.0);
assert!(t0_abs.is_none(), "first fix is unstamped — no reliable epoch");
// Fully-stamped tracks still use real GPS time (t0_abs = first fix's epoch).
let mut obj2 = serde_json::Map::new();
ins(&mut obj2, 1, 21.0, Some("2023:09:06 00:00:00Z"));
ins(&mut obj2, 2, 21.0001, Some("2023:09:06 00:00:03Z"));
let (s2, t0b) = collect_samples(&obj2);
assert_eq!(s2[1][2], 3.0);
assert!(t0b.is_some());
// The live-data case: a GARBAGE first stamp (1970) followed by real ones —
// "all stamped", but the epoch jump would be the whole epoch. The
// plausibility gate (>2000) turns the 1970 stamp into "unstamped" → index
// timebase for the whole track, and no epoch is trusted for stitching.
let mut obj3 = serde_json::Map::new();
ins(&mut obj3, 1, 21.0, Some("1970:01:01 00:00:00Z"));
ins(&mut obj3, 2, 21.0001, Some("2023:09:06 00:00:01Z"));
ins(&mut obj3, 3, 21.0002, Some("2023:09:06 00:00:02Z"));
let (s3, t0c) = collect_samples(&obj3);
let g3 = s3.windows(2).map(|w| w[1][2] - w[0][2]).fold(0.0_f64, f64::max);
assert!(g3 <= 1.0 + 1e-9, "garbage stamp leaked: max gap {g3}");
assert!(t0c.is_none());
// Clock glitch WITHIN plausible dates (2023 → 2026 jump): the >1 h step
// guard rebuilds on the index timebase too.
let mut obj4 = serde_json::Map::new();
ins(&mut obj4, 1, 21.0, Some("2023:09:06 00:00:00Z"));
ins(&mut obj4, 2, 21.0001, Some("2026:05:28 12:00:00Z"));
ins(&mut obj4, 3, 21.0002, Some("2026:05:28 12:00:01Z"));
let (s4, _) = collect_samples(&obj4);
let g4 = s4.windows(2).map(|w| w[1][2] - w[0][2]).fold(0.0_f64, f64::max);
assert!(g4 <= 1.0 + 1e-9, "clock glitch leaked: max gap {g4}");
}
#[test]
fn smoother_removes_a_burst() {
let mut t = straight(80);
for k in 40..44 {
t[k][0] = 0.02; // 2.2 km off for 4 consecutive samples
}
let (fixed, replaced) = viterbi_smooth(&t);
assert_eq!(replaced, 4);
for k in 40..44 {
assert!(fixed[k][0].abs() < 1e-6);
}
}
#[test]
fn downsample_caps_and_keeps_last() {
let t = straight(12000);
let d = downsample(t.clone(), 5000);
assert!(d.len() <= 5001);
assert_eq!(d.last(), t.last());
}
#[test]
fn speed_stats_measures_avg_and_max() {
// ~11.13 m per second for 60 s → avg ≈ max ≈ 11.1 m/s (40 km/h).
let t = straight(60);
let (avg, max) = speed_stats(&t);
assert!((avg.unwrap() - 11.13).abs() < 0.1, "{avg:?}");
assert!((max.unwrap() - 11.13).abs() < 0.1, "{max:?}");
// One fast hop dominates max but barely moves avg.
let mut t2 = straight(60);
t2[30][1] += 0.0002; // extra ~22 m in that second
let (avg2, max2) = speed_stats(&t2);
assert!(max2.unwrap() > 25.0, "{max2:?}");
assert!(avg2.unwrap() < 13.0, "{avg2:?}");
}
#[test]
fn stitch_planning_rules() {
// Healthy 1 Hz rollover (1 s gap) → nothing missing.
assert_eq!(plan_stitch(1.0, 299.0, Some(300.0), 20.0), None);
// 9 s gap, duration unknown → shift 8 s.
assert_eq!(plan_stitch(9.0, 291.0, None, 100.0), Some(8.0));
// 9 s gap but the track already covers the whole file → the PREDECESSOR was
// missing its end; shifting us would misalign every frame.
assert_eq!(plan_stitch(9.0, 299.0, Some(300.0), 100.0), None);
// Duration caps the shift: 20 s gap but only 5 s missing from this file.
assert_eq!(plan_stitch(20.0, 295.0, Some(300.0), 100.0), Some(5.0));
// Teleport guard: 5 km jump in 9 s can't be the same vehicle.
assert_eq!(plan_stitch(9.0, 291.0, None, 5000.0), None);
// Different session: 10 minutes apart.
assert_eq!(plan_stitch(600.0, 291.0, None, 100.0), None);
}
#[test]
fn wgs84_distance_matches_reference_arcs() {
// Textbook WGS84 arc lengths: 1° latitude ≈ 110.574 km (equator) /
// 111.132 km (45°); 1° longitude at the equator ≈ 111.320 km.
assert!((dist_m(0.0, 0.0, 1.0, 0.0) - 110_574.0).abs() < 200.0);
assert!((dist_m(0.0, 0.0, 0.0, 1.0) - 111_320.0).abs() < 30.0);
assert!((dist_m(44.5, 0.0, 45.5, 0.0) - 111_132.0).abs() < 200.0);
}
#[test]
fn track_roundtrip() {
let t = straight(10);
let v = track_value(&t);
let back = parse_track(&v);
assert_eq!(back.len(), 10);
assert!((back[9][1] - t[9][1]).abs() < 1e-5);
}
}

277
server/src/ingest.rs Normal file
View File

@@ -0,0 +1,277 @@
use crate::models::ExportDoc;
use chrono::{DateTime, NaiveDateTime, Utc};
use sqlx::PgPool;
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use walkdir::WalkDir;
const VIDEO_EXTS: &[&str] = &["mp4", "mkv", "avi", "mov", "m4v", "webm"];
fn is_video(p: &Path) -> bool {
p.extension()
.and_then(|e| e.to_str())
.map(|e| VIDEO_EXTS.contains(&e.to_lowercase().as_str()))
.unwrap_or(false)
}
/// Find the sibling export-JSON for a video. Supports `<file>_annotations.json`
/// (video-annotator's default export name), `<file>.json`, and `<stem>.json`.
fn sibling_json(video: &Path) -> Option<PathBuf> {
let dir = video.parent()?;
let file_name = video.file_name()?.to_str()?;
let stem = video.file_stem()?.to_str()?;
let candidates = [
format!("{file_name}_annotations.json"),
format!("{file_name}.json"),
format!("{stem}.json"),
format!("{stem}_annotations.json"),
];
for c in candidates {
let p = dir.join(c);
if p.is_file() {
return Some(p);
}
}
None
}
fn parse_ts(s: &str) -> Option<DateTime<Utc>> {
if s.is_empty() {
return None;
}
if let Ok(dt) = DateTime::parse_from_rfc3339(s) {
return Some(dt.with_timezone(&Utc));
}
NaiveDateTime::parse_from_str(s, "%Y-%m-%d %H:%M:%S")
.ok()
.map(|n| n.and_utc())
}
/// Scan `dir`, upsert every video (and its annotations) into the project.
/// Idempotent: re-running updates rows; never downgrades a 'verified' video.
pub async fn ingest_project(db: &PgPool, project_id: i32, dir: &Path) -> anyhow::Result<usize> {
if !dir.is_dir() {
anyhow::bail!("ingest dir not found: {}", dir.display());
}
let mut count = 0usize;
for entry in WalkDir::new(dir).into_iter().filter_map(|e| e.ok()) {
let path = entry.path();
if !entry.file_type().is_file() || !is_video(path) {
continue;
}
let rel_path = path
.strip_prefix(dir)
.unwrap_or(path)
.to_string_lossy()
.to_string();
let file_name = path
.file_name()
.map(|n| n.to_string_lossy().to_string())
.unwrap_or_default();
// Never clobber a verified push (Phase 3): a worker's pushed annotations +
// completer/verify-time are authoritative, so skip re-ingesting this video.
// Also never clobber a video that is *actively claimed* (a worker is mid-edit)
// — a stale NAS JSON must not overwrite in-progress work. Both checks read the
// current row's status + claim lease.
let existing: Option<(String, Option<String>, Option<DateTime<Utc>>)> = sqlx::query_as(
"SELECT status, claimed_by, lease_expires_at FROM videos WHERE project_id=$1 AND rel_path=$2",
)
.bind(project_id)
.bind(&rel_path)
.fetch_optional(db)
.await?;
if let Some((status, claimed_by, lease)) = existing.as_ref() {
if status == "verified" {
tracing::debug!("skip re-ingest of verified video: {rel_path}");
continue;
}
let active_claim = claimed_by.is_some()
&& lease.map(|l| l > Utc::now()).unwrap_or(false);
if active_claim {
tracing::debug!("skip re-ingest of actively-claimed video: {rel_path}");
continue;
}
}
let json_path = sibling_json(path);
let has_json = json_path.is_some();
// Parse the sibling JSON if present.
let doc: Option<ExportDoc> = json_path.as_ref().and_then(|jp| {
std::fs::read_to_string(jp)
.ok()
.and_then(|t| serde_json::from_str::<ExportDoc>(&t).ok())
});
let (width, height, fps, frame_count, time_ms) = match doc.as_ref().and_then(|d| d.video.as_ref()) {
Some(v) => (
v.width.map(|x| x as i32),
v.height.map(|x| x as i32),
v.fps,
v.frame_count.map(|x| x as i32),
v.annotation_time_ms,
),
None => (None, None, None, None, 0i64),
};
// Aggregate annotation_count, primary annotator, annotated_at.
let mut ann_count: i32 = 0;
let mut by_freq: HashMap<String, i32> = HashMap::new();
let mut latest: Option<DateTime<Utc>> = None;
if let Some(d) = doc.as_ref() {
for a in &d.fixed_annotations {
ann_count += 1;
if !a.annotated_by.is_empty() {
*by_freq.entry(a.annotated_by.clone()).or_default() += 1;
}
if let Some(t) = parse_ts(&a.created_at) {
latest = Some(latest.map_or(t, |l| l.max(t)));
}
}
for a in &d.range_annotations {
ann_count += 1;
if !a.annotated_by.is_empty() {
*by_freq.entry(a.annotated_by.clone()).or_default() += 1;
}
if let Some(t) = parse_ts(&a.created_at) {
latest = Some(latest.map_or(t, |l| l.max(t)));
}
}
}
let primary_annotator = by_freq
.into_iter()
.max_by_key(|(_, c)| *c)
.map(|(u, _)| u)
.unwrap_or_default();
let status = if has_json && ann_count > 0 { "annotated" } else { "pending" };
let raw_json: Option<serde_json::Value> = json_path.as_ref().and_then(|jp| {
std::fs::read_to_string(jp).ok().and_then(|t| serde_json::from_str(&t).ok())
});
// Upsert the video + replace its annotation rows atomically: either the new
// metadata AND the new annotation set both land, or neither does. Prevents a
// failed insert mid-loop from leaving the video with partial/zero annotations.
let mut tx = db.begin().await?;
// Upsert the video. Keep 'verified' status sticky (Phase 3 pushes).
let video_id: i32 = sqlx::query_scalar(
r#"
INSERT INTO videos
(project_id, file_name, rel_path, has_json, width, height, fps, frame_count,
annotation_count, annotation_time_ms, primary_annotator, status, annotated_at, raw_json,
imported_count)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15)
ON CONFLICT (project_id, rel_path) DO UPDATE SET
file_name=EXCLUDED.file_name,
has_json=EXCLUDED.has_json,
width=EXCLUDED.width, height=EXCLUDED.height, fps=EXCLUDED.fps,
frame_count=EXCLUDED.frame_count,
annotation_count=EXCLUDED.annotation_count,
annotation_time_ms=EXCLUDED.annotation_time_ms,
primary_annotator=EXCLUDED.primary_annotator,
status = CASE WHEN videos.status='verified' THEN videos.status ELSE EXCLUDED.status END,
annotated_at=EXCLUDED.annotated_at,
raw_json=EXCLUDED.raw_json,
-- imported baseline = the NAS-ingested count (skipped for verified videos
-- via the early continue above, so a pushed pass's baseline is preserved).
imported_count=EXCLUDED.imported_count,
ingested_at=now()
RETURNING id
"#,
)
.bind(project_id)
.bind(&file_name)
.bind(&rel_path)
.bind(has_json)
.bind(width)
.bind(height)
.bind(fps)
.bind(frame_count)
.bind(ann_count)
.bind(time_ms)
.bind(&primary_annotator)
.bind(status)
.bind(latest)
.bind(raw_json)
.bind(ann_count)
.fetch_one(&mut *tx)
.await?;
// Replace detailed annotation rows for this video (same transaction).
if let Some(d) = doc.as_ref() {
replace_annotations(&mut tx, video_id, d).await?;
} else {
sqlx::query("DELETE FROM annotations WHERE video_id=$1")
.bind(video_id)
.execute(&mut *tx)
.await?;
}
tx.commit().await?;
count += 1;
}
Ok(count)
}
/// Replace all annotation rows for a video with the contents of an export doc.
/// Shared by ingest (NAS JSON) and the Phase 3 worker push. Returns the row count
/// written (each range annotation contributes 12 rows: start, and end if present).
///
/// Runs on a caller-supplied transaction connection so the delete + all inserts are
/// **atomic** — either every annotation row lands or none do. Callers (`ingest`, worker
/// `push`) own the transaction so the surrounding metadata update commits together with
/// the annotation set. A failed insert mid-loop rolls back the delete, so annotations
/// can never be partially lost.
pub async fn replace_annotations(
conn: &mut sqlx::PgConnection,
video_id: i32,
d: &ExportDoc,
) -> anyhow::Result<i32> {
sqlx::query("DELETE FROM annotations WHERE video_id=$1")
.bind(video_id)
.execute(&mut *conn)
.await?;
let mut written = 0i32;
for a in &d.fixed_annotations {
insert_ann(&mut *conn, video_id, a.frame_number, &a.label, &a.side, &a.shape_type,
&a.vertices, &a.annotated_by, &a.review_status, &a.remark, &a.subclass, &a.created_at).await?;
written += 1;
}
for a in &d.range_annotations {
insert_ann(&mut *conn, video_id, a.start_frame, &a.label, &a.side, &a.shape_type,
&a.start_vertices, &a.annotated_by, &a.review_status, &a.remark, &a.subclass, &a.created_at).await?;
written += 1;
if let (Some(ef), Some(ev)) = (a.end_frame, a.end_vertices.as_ref()) {
insert_ann(&mut *conn, video_id, ef, &a.label, &a.side, &a.shape_type,
ev, &a.annotated_by, &a.review_status, &a.remark, &a.subclass, &a.created_at).await?;
written += 1;
}
}
Ok(written)
}
#[allow(clippy::too_many_arguments)]
async fn insert_ann(
conn: &mut sqlx::PgConnection, video_id: i32, frame: i64, label: &str, side: &str, shape: &str,
vertices: &serde_json::Value, by: &str, review: &str, remark: &str, subclass: &str, created_at: &str,
) -> anyhow::Result<()> {
sqlx::query(
r#"INSERT INTO annotations
(video_id, frame_number, label, side, shape_type, vertices, annotated_by, review_status, remark, subclass, created_at)
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11)"#,
)
.bind(video_id)
.bind(frame as i32)
.bind(label)
.bind(side)
.bind(if shape.is_empty() { "bbox" } else { shape })
.bind(vertices)
.bind(by)
.bind(if review.is_empty() { "none" } else { review })
.bind(remark)
.bind(subclass)
.bind(created_at)
.execute(&mut *conn)
.await?;
Ok(())
}

1336
server/src/main.rs Normal file

File diff suppressed because it is too large Load Diff

516
server/src/models.rs Normal file
View File

@@ -0,0 +1,516 @@
use serde::{Deserialize, Serialize};
use serde_json::Value;
// ---- Parsed from the sibling export-JSON (video-annotator export format) ----
#[derive(Deserialize, Default)]
pub struct ExportDoc {
#[serde(default)]
pub video: Option<ExportVideo>,
#[serde(default)]
pub fixed_annotations: Vec<FixedAnn>,
#[serde(default)]
pub range_annotations: Vec<RangeAnn>,
}
#[derive(Deserialize, Default)]
pub struct ExportVideo {
#[serde(default)]
pub file_name: String,
#[serde(default)]
pub width: Option<i64>,
#[serde(default)]
pub height: Option<i64>,
#[serde(default)]
pub fps: Option<f64>,
#[serde(default)]
pub frame_count: Option<i64>,
#[serde(default)]
pub annotation_time_ms: i64,
}
#[derive(Deserialize)]
pub struct FixedAnn {
#[serde(default)]
pub label: String,
#[serde(default)]
pub side: String,
#[serde(default)]
pub frame_number: i64,
#[serde(default, rename = "type")]
pub shape_type: String,
#[serde(default)]
pub vertices: Value,
#[serde(default)]
pub annotated_by: String,
#[serde(default)]
pub remark: String,
#[serde(default)]
pub review_status: String,
#[serde(default)]
pub subclass: String,
#[serde(default)]
pub created_at: String,
}
#[derive(Deserialize)]
pub struct RangeAnn {
#[serde(default)]
pub label: String,
#[serde(default)]
pub side: String,
#[serde(default)]
pub start_frame: i64,
#[serde(default)]
pub start_vertices: Value,
#[serde(default)]
pub end_frame: Option<i64>,
#[serde(default)]
pub end_vertices: Option<Value>,
#[serde(default, rename = "type")]
pub shape_type: String,
#[serde(default)]
pub annotated_by: String,
#[serde(default)]
pub remark: String,
#[serde(default)]
pub review_status: String,
#[serde(default)]
pub subclass: String,
#[serde(default)]
pub created_at: String,
}
// ---- API response shapes ----
#[derive(Serialize)]
pub struct ProjectSummary {
pub id: i32,
pub name: String,
pub source_path: String,
pub source_kind: String,
pub client_id: Option<i32>,
pub client_name: String,
pub total: i64,
pub pending: i64,
pub annotated: i64,
pub verified: i64,
}
/// A client owning one or more projects (admin-created from the dashboard).
#[derive(Serialize)]
pub struct ClientRow {
pub id: i32,
pub name: String,
pub created_at: chrono::DateTime<chrono::Utc>,
pub project_count: i64,
}
/// Admin: `POST /api/clients` — create a client.
#[derive(Deserialize)]
pub struct NewClientRequest {
pub name: String,
}
#[derive(Serialize, sqlx::FromRow)]
pub struct VideoRow {
pub id: i32,
pub file_name: String,
pub rel_path: String,
pub has_json: bool,
pub width: Option<i32>,
pub height: Option<i32>,
pub fps: Option<f64>,
pub frame_count: Option<i32>,
pub annotation_count: i32,
// Baseline count this video started a verification pass with: the NAS-ingested
// count, or (after a push) the count the worker had at claim time. Compared with
// annotation_count it shows how many the worker added/deleted.
pub imported_count: i32,
pub annotation_time_ms: i64,
pub primary_annotator: String,
pub status: String,
pub annotated_at: Option<chrono::DateTime<chrono::Utc>>,
// Phase 3: claim/verify dimension (orthogonal to `status`).
pub claimed_by: Option<String>,
pub claimed_at: Option<chrono::DateTime<chrono::Utc>>,
pub lease_expires_at: Option<chrono::DateTime<chrono::Utc>>,
pub completed_by: Option<String>,
pub completed_at: Option<chrono::DateTime<chrono::Utc>>,
pub verify_time_ms: i64,
// Annotations still flagged for review (review_status='flagged') on this video.
pub review_count: i64,
// How many times pushed (1 = verified, ≥2 = re-verified) + the distinct verifiers
// ("ravi, john"), both derived from the push event log.
pub verify_count: i64,
pub verifiers: String,
// Admin assignment (advisory owner) + the derived lifecycle label for the UI badge:
// pending | annotated | assigned | in-progress | verified | re-verified.
pub assigned_to: Option<String>,
pub workflow_status: String,
// Road type label ('' = unassigned) — injected into the export/claim JSON.
pub road_type: String,
// GPS pipeline: pending | ok | none | error (+ measured track length / last error).
pub gps_status: String,
pub gps_len_m: Option<f64>,
pub gps_error: Option<String>,
// Data quality: raw fix count + biggest dropout — the table compares the count
// against the video duration (≈1 fix/s) and flags mismatches.
pub gps_sample_count: Option<i32>,
pub gps_max_gap_s: Option<f64>,
// Real container duration from the scan (raw NAS videos have no fps/frame_count).
pub duration_s: Option<f64>,
// Client-calibrated chainage + the immutable GPS-measured (sr) chainage, meters.
pub chainage_start_m: Option<f64>,
pub chainage_end_m: Option<f64>,
pub sr_chainage_start_m: Option<f64>,
pub sr_chainage_end_m: Option<f64>,
pub route_seq: Option<i32>,
// "Raise hand" signal — anyone may raise (to start a conversation) or lower it.
pub hand_raised: bool,
pub hand_raised_by: Option<String>,
pub hand_raised_at: Option<chrono::DateTime<chrono::Utc>>,
// "Ignore" flag — admin-only; the UI crosses out ignored rows.
pub ignored: bool,
pub ignored_by: Option<String>,
// Remark thread (json array of {username, body, created_at}); '[]' when none.
pub remarks: serde_json::Value,
// Per-pass annotation-count history (json array of {username, at, count}, one per
// push in order); '[]' when never pushed. Drives the imported→verify→re-verify
// count breakdown in the UI.
pub passes: serde_json::Value,
}
/// One row of `GET /api/projects/:id/map` — a video as the map sees it. `track` is
/// the (transport-downsampled) effective polyline: corrected when kept, else raw;
/// None until the GPS sweep has scanned the video (or when it has no metadata).
#[derive(Serialize, sqlx::FromRow)]
pub struct MapVideo {
pub id: i32,
pub file_name: String,
pub rel_path: String,
pub status: String,
pub workflow_status: String,
pub ignored: bool,
pub hand_raised: bool,
pub assigned_to: Option<String>,
pub claimed_by: Option<String>,
pub annotation_count: i32,
pub road_type: String,
pub gps_status: String,
pub gps_len_m: Option<f64>,
pub gps_error: Option<String>,
pub chainage_start_m: Option<f64>,
pub chainage_end_m: Option<f64>,
pub sr_chainage_start_m: Option<f64>,
pub sr_chainage_end_m: Option<f64>,
pub route_seq: Option<i32>,
pub remark_count: i64,
pub track: Option<Value>,
pub corrected: bool,
// Who/what produced the correction ('auto' or a username) + its summary note.
pub gps_corrected_by: Option<String>,
pub gps_corrected_note: Option<String>,
// Data quality from the raw scan: fix count + biggest dropout between fixes.
pub gps_sample_count: Option<i32>,
pub gps_max_gap_s: Option<f64>,
pub duration_s: Option<f64>,
// Recording window (absolute UTC epoch of first/last fix) — temporal ordering.
pub gps_start_epoch: Option<f64>,
pub gps_end_epoch: Option<f64>,
// Speed over the full-resolution track (m/s) — the map's speed mode/stats.
#[sqlx(default)]
pub speed_avg_mps: Option<f64>,
#[sqlx(default)]
pub speed_max_mps: Option<f64>,
}
/// `GET /api/projects/:id/road_types` — a label + how many videos carry it.
#[derive(Serialize, sqlx::FromRow)]
pub struct RoadTypeRow {
pub name: String,
pub video_count: i64,
}
/// `POST /api/projects/:id/road_types {names}` — replace the label set (admin).
#[derive(Deserialize)]
pub struct RoadTypesRequest {
pub names: Vec<String>,
}
/// `POST /api/projects/:id/road_type_assign {video_ids, road_type}` — bulk-assign.
#[derive(Deserialize)]
pub struct RoadTypeAssignRequest {
pub video_ids: Vec<i32>,
pub road_type: String, // '' = clear
}
/// `POST /api/videos/:id/hand {raised}` — raise/lower the hand on a video.
#[derive(Deserialize)]
pub struct HandRequest {
pub raised: bool,
}
/// `POST /api/videos/:id/ignore {ignored}` — admin-only ignore toggle.
#[derive(Deserialize)]
pub struct IgnoreRequest {
pub ignored: bool,
}
#[derive(Deserialize)]
pub struct AssignRequest {
pub video_ids: Vec<i32>,
pub assignee: String, // empty = unassign
}
/// A user picked for a project (project_members joined with users).
#[derive(Serialize, sqlx::FromRow)]
pub struct MemberRow {
pub username: String,
pub display_name: String,
pub role: String,
pub added_at: chrono::DateTime<chrono::Utc>,
}
#[derive(Deserialize)]
pub struct AddMemberRequest {
pub username: String,
}
/// One line in a video's remark thread.
#[derive(Serialize, sqlx::FromRow)]
pub struct Remark {
pub username: String,
pub body: String,
pub created_at: chrono::DateTime<chrono::Utc>,
}
#[derive(Deserialize)]
pub struct NewRemarkRequest {
pub body: String,
}
#[derive(Serialize)]
pub struct Overview {
pub total: i64,
pub pending: i64,
pub annotated: i64,
pub verified: i64,
pub pct_done: f64,
}
#[derive(Serialize)]
pub struct LeaderRow {
pub user: String,
pub videos: i64,
pub annotations: i64,
pub total_time_ms: i64,
pub avg_time_ms: i64,
}
#[derive(Serialize)]
pub struct Stats {
pub overview: Overview,
pub leaderboard: Vec<LeaderRow>,
// Phase 3: verification leaderboard (by `completed_by`).
pub verifiers: Vec<VerifierRow>,
// Project time, verification pace + worker-based ETA.
pub timeline: ProjectTime,
}
/// Aggregate project time metrics + a worker-parallelised ETA for the remaining work.
#[derive(Serialize)]
pub struct ProjectTime {
/// Users picked for the project (the parallelism factor for the ETA).
pub active_workers: i64,
/// Average effort per verified video (annotation + verify time).
pub avg_video_ms: i64,
/// Total time spent by all users across all videos (annotation + verify).
pub total_spent_ms: i64,
/// Wall-clock from project inception to last completion (or now if ongoing).
pub elapsed_ms: i64,
/// Whether every video is verified.
pub all_verified: bool,
/// Estimated remaining work time = remaining × avg_video ÷ active_workers.
pub eta_ms: Option<i64>,
/// Whether the project timer is currently running (accumulating).
pub time_running: bool,
/// Last verify/re-verify activity (drives the 72h auto-stop).
pub last_activity_at: Option<chrono::DateTime<chrono::Utc>>,
/// Verification pace: push (verify) events in the recent window, and the window
/// length in days. videos/day = recent_verified ÷ recent_window_days.
pub recent_verified: i64,
pub recent_window_days: i64,
}
/// `POST /api/projects/:id/timer {action}` — start | stop | reset (admin only).
#[derive(Deserialize)]
pub struct TimerRequest {
pub action: String,
}
/// `POST /api/projects/:id/folders {folders}` — set the folders to consider (admin).
#[derive(Deserialize)]
pub struct FoldersRequest {
pub folders: Vec<String>,
}
/// `GET /api/projects/:id/folders` — a folder in the project + its video count +
/// whether it's currently included in verification.
#[derive(Serialize, sqlx::FromRow)]
pub struct FolderRow {
pub folder: String,
pub video_count: i64,
pub included: bool,
}
#[derive(Serialize)]
pub struct VerifierRow {
pub user: String,
pub videos: i64,
pub total_time_ms: i64,
pub avg_time_ms: i64,
/// Total annotations this user authored (annotated_by) in the project.
pub annotations: i64,
/// Annotations added per minute of work (annotations ÷ total_time). This is the
/// ranking metric: "most annotations in the least time". 0 when no time/annotations.
pub annotations_per_min: f64,
/// 1-based rank by annotations-per-minute (highest first); 0 = not yet ranked
/// (no annotations or no recorded time).
pub rank: i64,
}
// ---- Phase 3: auth + worker pull/push shapes ----
/// `GET /api/auth/whoami` — identifies the bearer.
#[derive(Serialize)]
pub struct WhoAmI {
pub username: String,
pub role: String,
}
/// `POST /api/videos/:id/claim` response — everything the desktop needs to verify.
#[derive(Serialize)]
pub struct ClaimResponse {
pub video_id: i32,
pub file_name: String,
pub rel_path: String,
pub width: Option<i32>,
pub height: Option<i32>,
pub fps: Option<f64>,
pub frame_count: Option<i32>,
pub lease_expires_at: chrono::DateTime<chrono::Utc>,
pub download_url: String,
/// Full preloaded export doc (`{video, fixed_annotations[], range_annotations[]}`)
/// for the desktop to import via its existing import path.
pub annotations: Value,
}
/// Admin: `POST /api/users` — provision a worker/admin.
#[derive(Deserialize)]
pub struct NewUserRequest {
pub username: String,
#[serde(default)]
pub display_name: String,
#[serde(default = "default_role")]
pub role: String,
}
fn default_role() -> String {
"ml_support".into()
}
/// Admin: `POST /api/users/:id/update` — edit a user's display name + role.
#[derive(Deserialize)]
pub struct UpdateUserRequest {
#[serde(default)]
pub display_name: String,
#[serde(default)]
pub role: String,
}
/// Admin user-provision response — the **only** time the clear token is shown.
#[derive(Serialize)]
pub struct NewUserResponse {
pub username: String,
pub display_name: String,
pub role: String,
pub token: String,
}
#[derive(Serialize, sqlx::FromRow)]
pub struct UserRow {
pub id: i32,
pub username: String,
pub display_name: String,
pub role: String,
pub active: bool,
pub created_at: chrono::DateTime<chrono::Utc>,
}
/// `POST /api/projects` — create a project under a client (any authenticated user).
#[derive(Deserialize)]
pub struct NewProjectRequest {
pub client_id: i32,
pub name: String,
pub source_path: String,
/// 'local' (default) or 'nfs'.
#[serde(default = "default_source_kind")]
pub source_kind: String,
}
fn default_source_kind() -> String {
"local".into()
}
/// `POST /api/projects/:id/source` — change a project's sync directory (any authenticated user).
#[derive(Deserialize)]
pub struct UpdateSourceRequest {
pub source_path: String,
#[serde(default = "default_source_kind")]
pub source_kind: String,
}
/// `GET /api/admin/storage` — DB size + row counts (admin only).
#[derive(Serialize)]
pub struct StorageInfo {
pub db_bytes: i64,
pub videos: i64,
pub annotations: i64,
pub video_events: i64,
pub audit_events: i64,
/// Retention cap applied nightly to the event/audit logs.
pub max_log_rows: i64,
}
/// `GET /api/audit` — an org-level audit row.
#[derive(Serialize, sqlx::FromRow)]
pub struct AuditRow {
pub at: chrono::DateTime<chrono::Utc>,
pub username: String,
pub action: String,
pub detail: Value,
}
/// `GET /api/projects/:id/activity` — live claims + recent events.
#[derive(Serialize)]
pub struct Activity {
pub active_claims: Vec<ActiveClaim>,
pub recent_events: Vec<EventRow>,
}
#[derive(Serialize, sqlx::FromRow)]
pub struct ActiveClaim {
pub video_id: i32,
pub file_name: String,
pub claimed_by: String,
pub claimed_at: chrono::DateTime<chrono::Utc>,
pub lease_expires_at: chrono::DateTime<chrono::Utc>,
}
#[derive(Serialize, sqlx::FromRow)]
pub struct EventRow {
pub video_id: i32,
pub file_name: String,
pub username: String,
pub event: String,
pub at: chrono::DateTime<chrono::Utc>,
}

146
server/src/nfs.rs Normal file
View File

@@ -0,0 +1,146 @@
//! Resolve a project's source folder to a real local directory the server can scan.
//!
//! - `local` projects: `source_path` is already a path inside the container (a plain folder, or a
//! compose-managed NFS volume) — returned as-is.
//! - `nfs` projects: the share is mounted on demand at `/mnt/nfs/<sanitized>`. `source_path` may be:
//! * a bare export path `/volume4/Share` — the host comes from the `NFS_HOST` env (IP hidden), or
//! * `host:/volume4/Share`, or `nfs://host/volume4/Share` — host taken from the path itself.
use std::io::Read;
use std::net::{TcpStream, ToSocketAddrs};
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
const NFS_MOUNT_ROOT: &str = "/mnt/nfs";
/// Fast reachability probe: can we open a TCP connection to the NAS's NFS port?
/// `mount.nfs` can hang for minutes on a filtered host (the connect is uninterruptible),
/// so we bound the unreachable case here to a few seconds and fail with a clean message.
fn probe_reachable(nfs_host: &str) -> Result<(), String> {
let addr = format!("{nfs_host}:2049");
let sock = addr
.to_socket_addrs()
.map_err(|_| "could not resolve the configured NAS host".to_string())?
.next()
.ok_or_else(|| "could not resolve the configured NAS host".to_string())?;
TcpStream::connect_timeout(&sock, Duration::from_secs(3))
.map(|_| ())
.map_err(|_| "NAS not reachable (no NFS service on the configured host)".to_string())
}
/// Turn a string into a stable, filesystem-safe mountpoint dir name.
fn sanitize(s: &str) -> String {
s.trim_matches('/')
.chars()
.map(|c| if c.is_ascii_alphanumeric() || c == '-' || c == '.' { c } else { '_' })
.collect()
}
/// Parse an nfs `source_path` into `(host, export)`. Accepts:
/// `nfs://host/volume4/Share`, `host:/volume4/Share`, or bare `/volume4/Share`
/// (bare → host falls back to the server's `NFS_HOST`). The export is always absolute.
fn parse_nfs(source_path: &str, default_host: &str) -> Result<(String, String), String> {
let s = source_path.trim();
// nfs://host/export
let scheme = s.strip_prefix("nfs://").or_else(|| s.strip_prefix("NFS://"));
if let Some(rest) = scheme {
let (host, export) = rest
.split_once('/')
.ok_or_else(|| "nfs:// path must include an export, e.g. nfs://host/volume4/Share".to_string())?;
if host.is_empty() {
return Err("nfs:// path is missing the host".into());
}
return Ok((host.to_string(), format!("/{}", export.trim_start_matches('/'))));
}
// host:/export (standard NFS device notation)
if let Some((host, export)) = s.split_once(':') {
if !host.is_empty() && export.starts_with('/') {
return Ok((host.to_string(), export.to_string()));
}
}
// bare /export → host from server env
if s.starts_with('/') {
if default_host.is_empty() {
return Err("NFS host is not configured on the server (set NFS_HOST)".into());
}
return Ok((default_host.to_string(), s.to_string()));
}
Err(format!(
"invalid NFS path '{s}' — use /export, host:/export, or nfs://host/export"
))
}
/// Is `mp` already an active mountpoint? (Reads /proc/mounts — no extra binaries needed.)
fn is_mounted(mp: &Path) -> bool {
let target = mp.to_string_lossy();
std::fs::read_to_string("/proc/mounts")
.map(|s| s.lines().any(|l| l.split(' ').nth(1) == Some(target.as_ref())))
.unwrap_or(false)
}
/// Resolve `(source_path, source_kind)` to a local directory, mounting an NFS share if needed.
/// `nfs_host`/`nfs_opts` come from server env. Blocking (runs a `mount` for nfs) — call inside
/// `spawn_blocking`.
pub fn resolve_dir(
source_path: &str,
source_kind: &str,
nfs_host: &str,
nfs_opts: &str,
) -> Result<PathBuf, String> {
if source_kind != "nfs" {
return Ok(PathBuf::from(source_path));
}
let (host, export) = parse_nfs(source_path, nfs_host)?;
let mp = PathBuf::from(NFS_MOUNT_ROOT).join(sanitize(&format!("{host}_{export}")));
if is_mounted(&mp) {
return Ok(mp);
}
// Fail fast if the NAS isn't reachable, before the (potentially long-hanging) mount.
probe_reachable(&host)?;
std::fs::create_dir_all(&mp).map_err(|e| format!("mkdir {}: {e}", mp.display()))?;
let device = format!("{host}:{export}");
// Spawn the mount and enforce the deadline ourselves: a `mount.nfs` stuck mid-handshake goes
// uninterruptible (D state) and ignores SIGTERM/SIGKILL, so `timeout` can't bound it. We poll
// and return a clean error at the deadline; an orphaned mount that later succeeds will be picked
// up by `is_mounted` on the next sync.
let mut child = Command::new("mount")
.args(["-t", "nfs", "-o", nfs_opts, &device, &mp.to_string_lossy()])
.stdout(Stdio::null())
.stderr(Stdio::piped())
.spawn()
.map_err(|e| format!("failed to run mount (is nfs-common installed?): {e}"))?;
let deadline = Instant::now() + Duration::from_secs(15);
loop {
match child.try_wait() {
Ok(Some(status)) if status.success() => return Ok(mp),
Ok(Some(_)) => {
let mut err = String::new();
if let Some(mut e) = child.stderr.take() {
let _ = e.read_to_string(&mut err);
}
let safe = err.replace(&host, "<nas>");
return Err(format!("mount of '{export}' failed: {}", safe.trim()));
}
Ok(None) => {
if Instant::now() >= deadline {
let _ = child.kill();
return Err(format!(
"mount of '{export}' timed out — check the NAS exports this path to the server"
));
}
std::thread::sleep(Duration::from_millis(300));
}
Err(e) => return Err(format!("mount wait failed: {e}")),
}
}
}

379
server/src/worker.rs Normal file
View File

@@ -0,0 +1,379 @@
//! Phase 3 worker pull/push API. All routes require a valid bearer token (`AuthUser`).
//! Flow: claim (atomic, leased) → download local copy → verify in the desktop → push.
use crate::auth::AuthUser;
use crate::models::*;
use crate::{err, AppState, ApiResult};
use axum::{
body::Body,
extract::{Path as AxPath, State},
http::{header, StatusCode},
response::Response,
Json,
};
use serde_json::{json, Value};
use tokio_util::io::ReaderStream;
/// Record an audit/analytics event. Best-effort: errors are logged, not surfaced.
pub async fn log_event(db: &sqlx::PgPool, video_id: i32, username: &str, event: &str, meta: Value) {
if let Err(e) = sqlx::query(
"INSERT INTO video_events (video_id, username, event, meta) VALUES ($1,$2,$3,$4)",
)
.bind(video_id)
.bind(username)
.bind(event)
.bind(meta)
.execute(db)
.await
{
tracing::warn!("failed to log {event} event for video {video_id}: {e}");
}
}
/// `GET /api/auth/whoami` — validate the token and identify the caller.
pub async fn whoami(user: AuthUser) -> ApiResult<WhoAmI> {
Ok(Json(WhoAmI { username: user.username, role: user.role }))
}
/// `POST /api/auth/login` — like whoami, but records a `signed_in` audit event so
/// admins can see who logged in (Activity log). Called once on an explicit sign-in,
/// not on every token check, so the log isn't spammed.
pub async fn login(State(s): State<AppState>, user: AuthUser) -> ApiResult<WhoAmI> {
crate::admin::audit(&s.db, &user.username, "signed_in", json!({ "role": user.role })).await;
Ok(Json(WhoAmI { username: user.username, role: user.role }))
}
/// `POST /api/videos/:id/claim` — atomically claim a video. **Any** status is claimable:
/// `pending` (annotate from scratch), `annotated` (verify), or `verified` (re-verify —
/// anyone may re-pull a completed video and push again). Unclaimed or lease-expired only;
/// same-user re-claim is allowed (resume). Returns metadata + any preloaded annotations.
pub async fn claim(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
) -> ApiResult<ClaimResponse> {
#[allow(clippy::type_complexity)]
let row: Option<(i32, String, String, Option<i32>, Option<i32>, Option<f64>, Option<i32>, chrono::DateTime<chrono::Utc>, Option<Value>, String, Option<f64>, Option<f64>, Option<f64>, Option<f64>)> =
sqlx::query_as(
r#"
UPDATE videos
SET claimed_by=$2, claimed_at=now(), lease_expires_at=now() + ($3::int * interval '1 second')
WHERE id=$1
AND (claimed_by IS NULL OR claimed_by=$2 OR lease_expires_at < now())
RETURNING id, file_name, rel_path, width, height, fps, frame_count, lease_expires_at, raw_json,
road_type, chainage_start_m, chainage_end_m, sr_chainage_start_m, sr_chainage_end_m
"#,
)
.bind(id)
.bind(&user.username)
.bind(s.lease_secs)
.fetch_optional(&s.db)
.await
.map_err(err)?;
let Some((video_id, file_name, rel_path, width, height, fps, frame_count, lease_expires_at, raw_json,
road_type, ch_s, ch_e, sr_s, sr_e)) = row
else {
// Distinguish "doesn't exist" from "not claimable".
let exists: Option<i32> = sqlx::query_scalar("SELECT id FROM videos WHERE id=$1")
.bind(id)
.fetch_optional(&s.db)
.await
.map_err(err)?;
return Err(match exists {
None => (StatusCode::NOT_FOUND, "video not found".into()),
Some(_) => (
StatusCode::CONFLICT,
"video is held by another worker".into(),
),
});
};
log_event(&s.db, video_id, &user.username, "claim", json!({})).await;
// Inject the server-authoritative road_type + chainage into the doc the worker
// pulls (only when a doc exists — a pending video legitimately has none; push
// re-stamps regardless, so the metadata always persists). Each annotation also
// gets its frame-exact lat/lon + chainage.
let annotations = match raw_json {
Some(mut doc) => {
crate::stamp_road_meta(&mut doc, &road_type, (ch_s, ch_e, sr_s, sr_e));
if let Some(geo) = crate::chainage::video_geo(&s.db, video_id).await {
crate::chainage::stamp_annotation_geo(&mut doc, &geo);
}
doc
}
None => Value::Null,
};
Ok(Json(ClaimResponse {
video_id,
file_name,
rel_path,
width,
height,
fps,
frame_count,
lease_expires_at,
download_url: format!("/api/videos/{video_id}/download"),
annotations,
}))
}
/// `GET /api/videos/:id/download` — stream the video bytes from the source folder
/// (the NAS mount on the VM). Only the current claimant (or an admin) may download,
/// so NAS credentials never leave the server.
pub async fn download(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
) -> Result<Response, (StatusCode, String)> {
let row: Option<(String, String, Option<String>)> = sqlx::query_as(
r#"SELECT v.rel_path, v.file_name, v.claimed_by
FROM videos v WHERE v.id=$1"#,
)
.bind(id)
.fetch_optional(&s.db)
.await
.map_err(err)?;
let (rel_path, file_name, claimed_by) =
row.ok_or((StatusCode::NOT_FOUND, "video not found".to_string()))?;
if !user.is_admin() && claimed_by.as_deref() != Some(user.username.as_str()) {
return Err((StatusCode::FORBIDDEN, "you do not hold this claim".into()));
}
// Resolve against the project's source folder (mount NFS if `source_kind='nfs'`).
let (source_path, source_kind): (String, String) = sqlx::query_as(
"SELECT p.source_path, p.source_kind FROM projects p JOIN videos v ON v.project_id=p.id WHERE v.id=$1",
)
.bind(id)
.fetch_one(&s.db)
.await
.map_err(err)?;
let (sp, nfs_host, nfs_opts) = (source_path, s.nfs_host.clone(), s.nfs_opts.clone());
let dir = tokio::task::spawn_blocking(move || crate::nfs::resolve_dir(&sp, &source_kind, &nfs_host, &nfs_opts))
.await
.map_err(err)?
.map_err(|e| (StatusCode::BAD_REQUEST, e))?;
let full = dir.join(&rel_path);
let file = tokio::fs::File::open(&full).await.map_err(|e| {
(StatusCode::NOT_FOUND, format!("source file unavailable ({}): {e}", full.display()))
})?;
let len = file.metadata().await.map_err(err)?.len();
let body = Body::from_stream(ReaderStream::new(file));
Response::builder()
.header(header::CONTENT_TYPE, "application/octet-stream")
.header(header::CONTENT_LENGTH, len)
.header(
header::CONTENT_DISPOSITION,
format!("attachment; filename=\"{file_name}\""),
)
.body(body)
.map_err(err)
}
/// `POST /api/videos/:id/heartbeat` — extend the lease while verifying.
pub async fn heartbeat(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
) -> ApiResult<Value> {
let new_lease: Option<chrono::DateTime<chrono::Utc>> = sqlx::query_scalar(
r#"UPDATE videos
SET lease_expires_at = now() + ($3::int * interval '1 second')
WHERE id=$1 AND claimed_by=$2
RETURNING lease_expires_at"#,
)
.bind(id)
.bind(&user.username)
.bind(s.lease_secs)
.fetch_optional(&s.db)
.await
.map_err(err)?;
match new_lease {
Some(lease_expires_at) => Ok(Json(json!({ "lease_expires_at": lease_expires_at }))),
None => Err((StatusCode::CONFLICT, "you do not hold this claim".into())),
}
}
/// `POST /api/videos/:id/release` — abandon a claim, returning the video to the pool.
pub async fn release(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
) -> ApiResult<Value> {
let released: Option<i32> = sqlx::query_scalar(
r#"UPDATE videos
SET claimed_by=NULL, claimed_at=NULL, lease_expires_at=NULL
WHERE id=$1 AND claimed_by=$2
RETURNING id"#,
)
.bind(id)
.bind(&user.username)
.fetch_optional(&s.db)
.await
.map_err(err)?;
match released {
Some(_) => {
log_event(&s.db, id, &user.username, "release", json!({})).await;
Ok(Json(json!({ "released": true })))
}
None => Err((StatusCode::CONFLICT, "you do not hold this claim".into())),
}
}
/// `POST /api/videos/:id/push` — submit the verified export doc + verify time.
/// Replaces the annotations, marks the video `verified`, records the verifier and
/// time, and clears the claim. Reuses the same export wire format as ingest.
pub async fn push(
State(s): State<AppState>,
user: AuthUser,
AxPath(id): AxPath<i32>,
Json(body): Json<Value>,
) -> ApiResult<Value> {
// A non-admin pusher must currently hold an ACTIVE claim on this video. This blocks
// a stale/expired claim from overwriting newer authoritative server state: if the
// lease lapsed (auto-released, or re-claimed by someone else) the worker must
// re-claim before pushing — their local work is never lost, it just can't clobber a
// claim they no longer hold. Admins may always push.
#[allow(clippy::type_complexity)]
let row: Option<(Option<String>, Option<chrono::DateTime<chrono::Utc>>, String, Option<f64>, Option<f64>, Option<f64>, Option<f64>)> =
sqlx::query_as(
r#"SELECT claimed_by, lease_expires_at, road_type,
chainage_start_m, chainage_end_m, sr_chainage_start_m, sr_chainage_end_m
FROM videos WHERE id=$1"#,
)
.bind(id)
.fetch_optional(&s.db)
.await
.map_err(err)?;
let (claimed_by, lease_expires_at, road_type, ch_s, ch_e, sr_s, sr_e) =
row.ok_or((StatusCode::NOT_FOUND, "video not found".to_string()))?;
if !user.is_admin() {
let holds_active_claim = claimed_by.as_deref() == Some(user.username.as_str())
&& lease_expires_at.map(|l| l > chrono::Utc::now()).unwrap_or(false);
if !holds_active_claim {
return Err((
StatusCode::CONFLICT,
"your claim has expired or is held by someone else — re-claim the video before pushing".into(),
));
}
}
let verify_time_ms = body.get("verify_time_ms").and_then(|v| v.as_i64()).unwrap_or(0);
// The desktop reports how many annotations it had at claim time (the preloaded
// baseline). 0/absent → keep the existing baseline (set at ingest), so we never
// wipe it with an old client that doesn't send the field.
let imported_count = body.get("imported_count").and_then(|v| v.as_i64()).unwrap_or(0) as i32;
let doc: ExportDoc = serde_json::from_value(body.clone())
.map_err(|e| (StatusCode::BAD_REQUEST, format!("invalid export doc: {e}")))?;
// Adopt the doc's fps/frame_count/resolution BEFORE stamping. A raw (from-scratch)
// NAS video has no fps in our DB — but the desktop decoded the real file, so its
// fps is authoritative. Without this, per-annotation lat/lon/chainage can't be
// computed (frame→time needs fps) and the geo silently wouldn't be written.
// COALESCE(doc, existing) so a doc that omits a field never wipes it.
if let Some(v) = doc.video.as_ref() {
let _ = sqlx::query(
r#"UPDATE videos SET fps=COALESCE($2, fps), frame_count=COALESCE($3, frame_count),
width=COALESCE($4, width), height=COALESCE($5, height) WHERE id=$1"#,
)
.bind(id)
.bind(v.fps)
.bind(v.frame_count.map(|x| x as i32))
.bind(v.width.map(|x| x as i32))
.bind(v.height.map(|x| x as i32))
.execute(&s.db)
.await;
}
// Re-stamp the server-authoritative road_type + chainage onto the stored doc —
// whatever the worker's client sent (or dropped), the server's values win. Every
// annotation gets its frame-exact lat/lon + chainage too (from the fps just set).
let mut body = body;
crate::stamp_road_meta(&mut body, &road_type, (ch_s, ch_e, sr_s, sr_e));
if let Some(geo) = crate::chainage::video_geo(&s.db, id).await {
crate::chainage::stamp_annotation_geo(&mut body, &geo);
}
let ann_count = (doc.fixed_annotations.len() + doc.range_annotations.len()) as i32;
// Recompute the video-level annotator fields from the pushed doc (same idea as
// ingest) so the annotation leaderboard + throughput populate for pushed videos:
// primary_annotator = the most-frequent drawer; annotation_time_ms from the doc.
let mut freq: std::collections::HashMap<String, i32> = std::collections::HashMap::new();
for a in &doc.fixed_annotations {
if !a.annotated_by.is_empty() { *freq.entry(a.annotated_by.clone()).or_default() += 1; }
}
for a in &doc.range_annotations {
if !a.annotated_by.is_empty() { *freq.entry(a.annotated_by.clone()).or_default() += 1; }
}
let primary_annotator = freq.into_iter().max_by_key(|(_, c)| *c).map(|(u, _)| u).unwrap_or_default();
let ann_time_ms = doc.video.as_ref().map(|v| v.annotation_time_ms).unwrap_or(0);
// Replace the annotation set AND flip the video to verified in ONE transaction:
// either the pushed annotations + the verified status both commit, or neither does.
// This makes a push all-or-nothing — a verifier can never end up "verified" with a
// half-written annotation set, and a failed push leaves the prior pass intact.
let mut tx = s.db.begin().await.map_err(err)?;
crate::ingest::replace_annotations(&mut tx, id, &doc)
.await
.map_err(err)?;
// Accumulate verify_time_ms across passes (re-verification adds up). The list
// of verifiers + the pass count are derived from video_events (each push logs one).
sqlx::query(
r#"UPDATE videos
SET status='verified', completed_by=$2, completed_at=now(),
verify_time_ms = COALESCE(videos.verify_time_ms, 0) + $3,
annotation_count=$4, raw_json=$5,
primary_annotator=$6, annotation_time_ms=$7, annotated_at=now(),
-- imported_count is the ORIGINAL auto-imported baseline (set at ingest).
-- Keep it fixed once set: verification/re-verification must NOT fold added
-- annotations into it (otherwise "imported 200" would drift to 210). Only
-- establish it from the claim-time count if it was never set.
imported_count = COALESCE(NULLIF(videos.imported_count, 0), $8),
claimed_by=NULL, claimed_at=NULL, lease_expires_at=NULL
WHERE id=$1"#,
)
.bind(id)
.bind(&user.username)
.bind(verify_time_ms)
.bind(ann_count)
.bind(&body)
.bind(&primary_annotator)
.bind(ann_time_ms)
.bind(imported_count)
.execute(&mut *tx)
.await
.map_err(err)?;
tx.commit().await.map_err(err)?;
// Record this pass's resulting annotation count so the dashboard can show a
// per-pass breakdown (imported baseline → verify Δ → re-verify Δ …), not just a net.
log_event(
&s.db,
id,
&user.username,
"push",
json!({ "verify_time_ms": verify_time_ms, "annotation_count": ann_count }),
)
.await;
// Mark project activity + auto-resume its timer (a verify/re-verify restarts it).
let _ = sqlx::query(
r#"UPDATE projects SET last_activity_at=now(), time_running=TRUE,
time_started_at = CASE WHEN time_running THEN time_started_at ELSE now() END
WHERE id = (SELECT project_id FROM videos WHERE id=$1)"#,
)
.bind(id)
.execute(&s.db)
.await;
Ok(Json(json!({ "verified": true, "annotation_count": ann_count })))
}