246 lines
15 KiB
Markdown
246 lines
15 KiB
Markdown
# Central Verification Dashboard (Phase 1)
|
|
|
|
Dockerized pipeline that ingests a folder of videos + their sibling export-JSONs into Postgres and
|
|
serves a manager dashboard: completion overview, per-user leaderboard, throughput + ETA, and a
|
|
per-video table. Standalone — it does **not** touch the desktop video-annotator app.
|
|
|
|
> Phase 1 = the dashboard pipeline. Worker pull/push (claim a video, verify, push) + token auth are
|
|
> **Phase 3** and not built yet. So in this phase the dashboard shows the **annotation snapshot**
|
|
> ingested from the NAS JSONs: `pending` (no JSON) vs `annotated` (has a JSON) vs `verified`
|
|
> (a pushed pass — 0 until Phase 3).
|
|
|
|
## Run it
|
|
|
|
```bash
|
|
cd central
|
|
docker compose up --build
|
|
```
|
|
|
|
- Dashboard → http://localhost:8081
|
|
- API → http://localhost:8080 (e.g. `curl localhost:8080/api/projects`)
|
|
- Postgres → localhost:5433 (user/pass/db = `central`)
|
|
|
|
On startup the server creates a `default` project pointing at `INGEST_DIR` (`/data`, mounted from
|
|
`./sample-data`) and ingests it once. Click **Re-ingest** in the dashboard to re-scan after files
|
|
change. Re-ingest is idempotent and never downgrades a `verified` video.
|
|
|
|
## Point it at the NAS (on the VM)
|
|
|
|
The ingest folder is just a path/volume. On the VM, mount the Synology share and map it to `/data`:
|
|
|
|
```yaml
|
|
# docker-compose.override.yml on the VM
|
|
services:
|
|
server:
|
|
volumes:
|
|
- /mnt/synology/<client-folder>:/data:ro
|
|
```
|
|
|
|
Then `POST /api/projects/<id>/ingest` (or restart the server). No code changes.
|
|
|
|
## Database backups (→ NAS)
|
|
|
|
A `db-backup` sidecar (`prodrigestivill/postgres-backup-local`) runs `pg_dump` on a schedule
|
|
and writes gzipped dumps straight to the NAS, pruning old ones automatically.
|
|
|
|
- **Schedule:** daily at 02:00 IST (`SCHEDULE` + `TZ: Asia/Kolkata` in `docker-compose.yml`).
|
|
- **Destination:** the `nasbackups` NFS volume → `:/volume4/Saudi_Video_Sync/Verification_Dashboard_Backup`
|
|
on the NAS (`NFS_HOST`, default `192.168.1.199`). **Create that folder on the NAS** and allow
|
|
rw NFS from the Docker host; the host needs an NFS client (`nfs-common`).
|
|
- **Retention:** `BACKUP_KEEP_DAYS=30`, `BACKUP_KEEP_WEEKS=8`, `BACKUP_KEEP_MONTHS=6` (tune to taste).
|
|
Dumps land under `daily/`, `weekly/`, `monthly/` as `central-YYYYMMDD-HHMMSS.sql.gz`.
|
|
|
|
```bash
|
|
docker compose up -d db-backup # start it
|
|
docker compose exec db-backup /backup.sh # run an immediate backup (verify it writes to the NAS)
|
|
docker compose logs db-backup # check the schedule/last run
|
|
```
|
|
|
|
**Restore** a dump into the running DB:
|
|
|
|
```bash
|
|
gunzip -c /path/on/nas/daily/central-YYYYMMDD-HHMMSS.sql.gz \
|
|
| docker compose exec -T db psql -U central -d central
|
|
```
|
|
|
|
(For a clean restore, drop+recreate the DB first, or restore into a scratch DB and compare.)
|
|
Test a restore periodically — an untested backup isn't a backup.
|
|
|
|
## Layout
|
|
|
|
```
|
|
central/
|
|
db/init.sql # Postgres schema (projects, videos, annotations)
|
|
server/ # Rust/Axum API + folder ingest (embeds db/init.sql)
|
|
dashboard/ # React + Vite, served by nginx (proxies /api → server)
|
|
sample-data/ # demo: one annotated video JSON + two empty .mp4 placeholders
|
|
docker-compose.yml
|
|
```
|
|
|
|
## Ingest details
|
|
|
|
For each video file (`.mp4/.mkv/.avi/.mov/.m4v/.webm`) it looks for a sibling JSON named
|
|
`<file>_annotations.json`, `<file>.json`, or `<stem>.json` (video-annotator's export format —
|
|
`{ video, fixed_annotations[], range_annotations[] }`). From it the server derives per-video
|
|
annotation count, `annotated_by` (primary), `annotation_time_ms`, and `annotated_at` (latest
|
|
annotation timestamp, used for the throughput chart), and stores the full doc in `videos.raw_json`
|
|
for Phase 3 pull. Videos without a JSON are recorded as `pending`.
|
|
|
|
## API
|
|
|
|
### Clients → projects
|
|
|
|
A **client** (e.g. *IRB*) owns one or more **projects**; each project points at a folder via
|
|
`source_path` + `source_kind`. Project names are unique **per client**. Use the dashboard's
|
|
**Admin** tab: sign in (token) → create client (admin) → create project → **Sync**. A `default`
|
|
client/project is seeded **only on a fresh DB** (local dev); once real clients exist it never reappears.
|
|
|
|
**Sync from NFS (`source_kind='nfs'`).** A project's `source_path` may be a **bare export path**
|
|
(`/volume4/Saudi_Video_Sync` — the host comes from the `NFS_HOST` env, so the **IP stays server-side and
|
|
is never shown/returned**), or include the host explicitly as `host:/volume4/Share` or
|
|
`nfs://host/volume4/Share`. On sync the server mounts the share on demand (a fast TCP probe to port 2049
|
|
fails an unreachable NAS in ~3 s; the mount is bounded so a stuck NAS can't hang the request).
|
|
`source_kind='local'` keeps the old behaviour (a path inside the container, incl. a compose-managed NFS
|
|
volume). Point a project at a **subfolder** (e.g. `/volume4/Saudi_Video_Sync/IRB_Master_VIdeos`), not a
|
|
huge share root.
|
|
|
|
**Subfolders.** Ingest walks the folder **recursively**, so videos nested in subfolders are included;
|
|
each video's `rel_path` carries its subfolder, which the dashboard shows in a **Folder** column.
|
|
|
|
> In-app NFS mounting needs `nfs-common` (in the image) and the server container's `cap_add: SYS_ADMIN`
|
|
> + `security_opt: apparmor:unconfined` (in `docker-compose.yml`; use `privileged: true` if your host's
|
|
> `mount.nfs` still refuses). `local`-only deployments need none of this.
|
|
>
|
|
> The server service also sets `init: true` — it spawns exiftool/mount.nfs children as PID 1, and
|
|
> without an init their orphans become unreapable zombies that block `docker stop`
|
|
> ("PID … is zombie and can not be killed"). Keep it if you copy the service elsewhere.
|
|
|
|
### Roles
|
|
|
|
- **admin** (role `admin`, or the `ADMIN_TOKEN`): everything — create clients, manage collaborators, etc.
|
|
- **collaborator** (role `worker`): can **add a project** and **change a project's sync directory**, but
|
|
**not** create clients or manage users.
|
|
|
|
### Dashboard read APIs (open — the dashboard is manager-only on the VM)
|
|
|
|
- `GET /api/clients` — clients + project counts
|
|
- `GET /api/projects` — projects + counts (each carries `client_id`, `client_name`, `source_kind`)
|
|
- `GET /api/projects/:id/videos` — per-video rows (incl. claim/verify columns)
|
|
- `GET /api/projects/:id/stats` — overview, annotator leaderboard, `verifiers`, throughput, eta_days
|
|
- `GET /api/projects/:id/activity` — live claims + recent `video_events`
|
|
- `GET /api/audit?limit=N` — org audit log (client/project created, sync dir changed, synced, user created)
|
|
- `POST /api/projects/:id/ingest` — re-scan the project's folder ("Sync"); attributes the actor if a token is sent
|
|
|
|
## Phase 3 — worker pull/push + token auth
|
|
|
|
Workers claim an annotated video, verify it in the desktop app, and push the verified
|
|
pass back. All Phase 3 routes require `Authorization: Bearer <token>`.
|
|
|
|
- **Auth model:** tokens are random 256-bit hex strings; only their SHA-256 hash is
|
|
stored (`users.token_hash`). An admin provisions users and hands out the token (shown
|
|
**once**). A master `ADMIN_TOKEN` env var grants admin (bootstrap / break-glass).
|
|
- **Status vs. claim:** the annotation `status` (`pending`/`annotated`/`verified`) is
|
|
unchanged; claiming is a separate dimension (`claimed_by` + `lease_expires_at`). A video
|
|
is claimable when `annotated` and unclaimed-or-lease-expired. A push flips it to
|
|
`verified`, records `completed_by`/`verify_time_ms`, and clears the claim. Re-ingest then
|
|
**skips** verified videos (the pushed pass is authoritative).
|
|
- **Leases auto-release:** a 60 s background task returns claims whose lease lapsed
|
|
(`LEASE_SECS`, default 900) to the pool and logs an `auto_release` event.
|
|
|
|
### Admin (require admin / `ADMIN_TOKEN`)
|
|
|
|
- `POST /api/users` `{username, display_name?, role?}` → `{username, role, token}` (token shown once)
|
|
- `GET /api/users` — list users (no tokens)
|
|
- `POST /api/clients` `{name}` → the new client (unique name; 409 on dup)
|
|
|
|
### Create / manage projects (require any valid token — workers included)
|
|
|
|
- `POST /api/projects` `{client_id, name, source_path, source_kind}` — create/repoint a project
|
|
under a client (idempotent on `(client_id, name)`); `source_kind` = `local`|`nfs`
|
|
- `POST /api/projects/:id/source` `{source_path, source_kind}` — change a project's sync directory
|
|
|
|
### Worker (require any valid token)
|
|
|
|
- `GET /api/auth/whoami` — validate token → `{username, role}`
|
|
- `POST /api/videos/:id/claim` — atomic claim (annotated + unclaimed/expired); returns
|
|
metadata + `download_url` + preloaded `annotations`
|
|
- `GET /api/videos/:id/download` — stream the source bytes (claimant/admin only)
|
|
- `POST /api/videos/:id/heartbeat` — extend the lease while verifying
|
|
- `POST /api/videos/:id/release` — abandon the claim, back to the pool
|
|
- `POST /api/videos/:id/push` — body = export doc + `verify_time_ms`; replace annotations,
|
|
mark `verified`, record verifier + time, clear claim
|
|
|
|
### Map / GPS / road_type / chainage
|
|
|
|
The server image bundles exiftool; a serial background sweep extracts each video's GPS
|
|
track (`gps_status`: `pending → ok | none | error`). The raw track is immutable —
|
|
a kept "Viterbi" correction lives beside it and is preferred by the map + chainage.
|
|
`road_type` + the four chainage values (`chainage_start`, `chainage_end`,
|
|
`sr_chainage_start`, `sr_chainage_end`) are injected into every claim/export doc and
|
|
re-stamped on push (server-authoritative). **Every annotation** additionally gets its
|
|
**frame-exact** `lat`/`lon`/`chainage`/`sr_chainage` (range annotations also `end_*`):
|
|
frame 15 of a 30 fps video is t = 0.5 s — halfway between two 1 Hz GPS samples — so it
|
|
gets the halfway position/chainage, not a second/video boundary.
|
|
**Chainage unit convention (everywhere — JSON + UI): kilometres with 4 decimals**,
|
|
e.g. `22.4456` (`chainage` = client-calibrated, `sr_chainage` = GPS-measured).
|
|
Distances are WGS84-ellipsoidal (no spherical bias); calibration anchors persist in
|
|
`chainage_calibrations` so this works at any time after a recompute.
|
|
|
|
- `GET /api/projects/:id/map` — per-video map rows: track (≤400 pts), status,
|
|
road_type, chainage, `gps_status` (member/admin)
|
|
- `GET /api/projects/:id/tracks.kml?token=` — all tracks as KML LineStrings colored by
|
|
status + calibration points (Google Earth)
|
|
- `POST /api/videos/:id/gps/rescan` — re-queue a video for the exiftool sweep (admin)
|
|
- `POST /api/projects/:id/gps/rescan_all` — re-queue every video (admin; background)
|
|
- `POST /api/videos/:id/gps/correct` `{action: preview|keep|discard}` — spike smoothing +
|
|
missing-start stitch from the previous file (by GPS
|
|
timestamp, duration-guarded); the scan sweep applies
|
|
the same pipeline automatically as 'auto' (raw never
|
|
changes; discard blocks re-auto-correction) (admin)
|
|
- `GET/POST /api/projects/:id/road_types` — list / replace label set (`{names}`, admin)
|
|
- `POST /api/projects/:id/road_type_assign` `{video_ids, road_type}` — bulk-assign ('' clears, admin)
|
|
- `GET/POST /api/projects/:id/chainage/points` — client calibration points (POST admin:
|
|
`{lat, lon, chainage_m, note?, road_type?}`)
|
|
- `POST /api/projects/:id/chainage/points/bulk` `{points: […]}` — transactional bulk insert;
|
|
the dashboard's **KML import** uses this (placemark
|
|
names like `12+400` or `12.4` become chainage) (admin)
|
|
- `DELETE /api/chainage/points/:id` — remove a point (admin)
|
|
- `POST /api/chainage/points/:id` `{note}` — edit a point's note in place; position/chainage
|
|
stay immutable (admin). Points carry an `origin` field:
|
|
`''` = manual, `'quick'` = generated by the quick assign
|
|
(replaced wholesale on the next quick run — identity is
|
|
the column, so renaming the note is safe)
|
|
- `GET /api/projects/:id/chainage/calibrations` — the saved calibrations, one per road-type group
|
|
(`road_type, route_len_m, anchor_count, computed_by,
|
|
computed_at, has_summary`) — every recompute persists one,
|
|
so any user sees what's already calibrated (member/admin)
|
|
- `GET /api/projects/:id/chainage/summary?road_type=` — the SAVED full result of the group's last
|
|
recompute (route, spans, per-video chainage, warnings),
|
|
null if never computed; the dashboard auto-loads it so a
|
|
newly signed-in admin sees the existing compute
|
|
(member/admin)
|
|
- `POST /api/projects/:id/chainage/recompute` `{video_ids, road_type}` — order the videos into a
|
|
route, snap the points, distribute the client-vs-GPS error
|
|
piecewise-linearly; writes `chainage_*` (client) +
|
|
`sr_chainage_*` (measured, immutable) + `route_seq` (admin).
|
|
Summary reports `majority_flipped` + per-video
|
|
`against_flow` (opposite to the route's dominant direction —
|
|
the only direction flag the UI shows). Warnings include
|
|
mixed directions (≥25% against the majority → probably both
|
|
carriageways in one group) and ambiguous untagged points
|
|
(used by this route but also within 150 m of another
|
|
road-type group's videos — tag them)
|
|
- `POST /api/projects/:id/chainage/quick` `{video_ids, start_m, end_m, road_type}` — the
|
|
two-point case: points at the route's two geometric ends
|
|
(notes `quick: chain start/end`, tagged with the group) +
|
|
same recompute (admin)
|
|
|
|
### Env
|
|
|
|
| var | default | meaning |
|
|
|---------------|------------------------------------------|----------------------------------------------------|
|
|
| `ADMIN_TOKEN` | `dev-admin-token` | master admin token — **change in production** |
|
|
| `LEASE_SECS` | `900` | claim lease length; auto-released after this |
|
|
| `NFS_HOST` | _(empty)_ | NAS host/IP for `nfs`-kind projects (server-side only, never returned) |
|
|
| `NFS_OPTS` | `nfsvers=4,ro,soft,timeo=30,retrans=2,retry=0` | mount options for `nfs`-kind projects |
|