Centralised_VideoVerification_Dashboard/README.md

Central Verification Dashboard (Phase 1)

Dockerized pipeline that ingests a folder of videos + their sibling export-JSONs into Postgres and serves a manager dashboard: completion overview, per-user leaderboard, throughput + ETA, and a per-video table. Standalone — it does not touch the desktop video-annotator app.

Phase 1 = the dashboard pipeline. Worker pull/push (claim a video, verify, push) + token auth are Phase 3 and not built yet. So in this phase the dashboard shows the annotation snapshot ingested from the NAS JSONs: pending (no JSON) vs annotated (has a JSON) vs verified (a pushed pass — 0 until Phase 3).

Run it

cd central
docker compose up --build

• Dashboard → http://localhost:8081
• API → http://localhost:8080 (e.g. curl localhost:8080/api/projects)
• Postgres → localhost:5433 (user/pass/db = central)

On startup the server creates a default project pointing at INGEST_DIR (/data, mounted from ./sample-data) and ingests it once. Click Re-ingest in the dashboard to re-scan after files change. Re-ingest is idempotent and never downgrades a verified video.

Point it at the NAS (on the VM)

The ingest folder is just a path/volume. On the VM, mount the Synology share and map it to /data:

# docker-compose.override.yml on the VM
services:
  server:
    volumes:
      - /mnt/synology/<client-folder>:/data:ro

Then POST /api/projects/<id>/ingest (or restart the server). No code changes.

Database backups (→ NAS)

A db-backup sidecar (prodrigestivill/postgres-backup-local) runs pg_dump on a schedule and writes gzipped dumps straight to the NAS, pruning old ones automatically.

• Schedule: daily at 02:00 IST (SCHEDULE + TZ: Asia/Kolkata in docker-compose.yml).
• Destination: the nasbackups NFS volume → :/volume4/Saudi_Video_Sync/Verification_Dashboard_Backup on the NAS (NFS_HOST, default 192.168.1.199). Create that folder on the NAS and allow rw NFS from the Docker host; the host needs an NFS client (nfs-common).
• Retention: BACKUP_KEEP_DAYS=30, BACKUP_KEEP_WEEKS=8, BACKUP_KEEP_MONTHS=6 (tune to taste). Dumps land under daily/, weekly/, monthly/ as central-YYYYMMDD-HHMMSS.sql.gz.

docker compose up -d db-backup          # start it
docker compose exec db-backup /backup.sh # run an immediate backup (verify it writes to the NAS)
docker compose logs db-backup            # check the schedule/last run

Restore a dump into the running DB:

gunzip -c /path/on/nas/daily/central-YYYYMMDD-HHMMSS.sql.gz \
  | docker compose exec -T db psql -U central -d central

(For a clean restore, drop+recreate the DB first, or restore into a scratch DB and compare.) Test a restore periodically — an untested backup isn't a backup.

Layout

central/
  db/init.sql            # Postgres schema (projects, videos, annotations)
  server/                # Rust/Axum API + folder ingest  (embeds db/init.sql)
  dashboard/             # React + Vite, served by nginx (proxies /api → server)
  sample-data/           # demo: one annotated video JSON + two empty .mp4 placeholders
  docker-compose.yml

Ingest details

For each video file (.mp4/.mkv/.avi/.mov/.m4v/.webm) it looks for a sibling JSON named <file>_annotations.json, <file>.json, or <stem>.json (video-annotator's export format — { video, fixed_annotations[], range_annotations[] }). From it the server derives per-video annotation count, annotated_by (primary), annotation_time_ms, and annotated_at (latest annotation timestamp, used for the throughput chart), and stores the full doc in videos.raw_json for Phase 3 pull. Videos without a JSON are recorded as pending.

API

Clients → projects

A client (e.g. IRB) owns one or more projects; each project points at a folder via source_path + source_kind. Project names are unique per client. Use the dashboard's Admin tab: sign in (token) → create client (admin) → create project → Sync. A default client/project is seeded only on a fresh DB (local dev); once real clients exist it never reappears.

Sync from NFS (source_kind='nfs'). A project's source_path may be a bare export path (/volume4/Saudi_Video_Sync — the host comes from the NFS_HOST env, so the IP stays server-side and is never shown/returned), or include the host explicitly as host:/volume4/Share or nfs://host/volume4/Share. On sync the server mounts the share on demand (a fast TCP probe to port 2049 fails an unreachable NAS in ~3 s; the mount is bounded so a stuck NAS can't hang the request). source_kind='local' keeps the old behaviour (a path inside the container, incl. a compose-managed NFS volume). Point a project at a subfolder (e.g. /volume4/Saudi_Video_Sync/IRB_Master_VIdeos), not a huge share root.

Subfolders. Ingest walks the folder recursively, so videos nested in subfolders are included; each video's rel_path carries its subfolder, which the dashboard shows in a Folder column.

In-app NFS mounting needs nfs-common (in the image) and the server container's cap_add: SYS_ADMIN

• security_opt: apparmor:unconfined (in docker-compose.yml; use privileged: true if your host's mount.nfs still refuses). local-only deployments need none of this.

Roles

• admin (role admin, or the ADMIN_TOKEN): everything — create clients, manage collaborators, etc.
• collaborator (role worker): can add a project and change a project's sync directory, but not create clients or manage users.

Dashboard read APIs (open — the dashboard is manager-only on the VM)

• GET /api/clients — clients + project counts
• GET /api/projects — projects + counts (each carries client_id, client_name, source_kind)
• GET /api/projects/:id/videos — per-video rows (incl. claim/verify columns)
• GET /api/projects/:id/stats — overview, annotator leaderboard, verifiers, throughput, eta_days
• GET /api/projects/:id/activity — live claims + recent video_events
• GET /api/audit?limit=N — org audit log (client/project created, sync dir changed, synced, user created)
• POST /api/projects/:id/ingest — re-scan the project's folder ("Sync"); attributes the actor if a token is sent

Phase 3 — worker pull/push + token auth

Workers claim an annotated video, verify it in the desktop app, and push the verified pass back. All Phase 3 routes require Authorization: Bearer <token>.

• Auth model: tokens are random 256-bit hex strings; only their SHA-256 hash is stored (users.token_hash). An admin provisions users and hands out the token (shown once). A master ADMIN_TOKEN env var grants admin (bootstrap / break-glass).
• Status vs. claim: the annotation status (pending/annotated/verified) is unchanged; claiming is a separate dimension (claimed_by + lease_expires_at). A video is claimable when annotated and unclaimed-or-lease-expired. A push flips it to verified, records completed_by/verify_time_ms, and clears the claim. Re-ingest then skips verified videos (the pushed pass is authoritative).
• Leases auto-release: a 60 s background task returns claims whose lease lapsed (LEASE_SECS, default 900) to the pool and logs an auto_release event.

Admin (require admin / ADMIN_TOKEN)

• POST /api/users {username, display_name?, role?} → {username, role, token} (token shown once)
• GET /api/users — list users (no tokens)
• POST /api/clients {name} → the new client (unique name; 409 on dup)

Create / manage projects (require any valid token — workers included)

• POST /api/projects {client_id, name, source_path, source_kind} — create/repoint a project under a client (idempotent on (client_id, name)); source_kind = local|nfs
• POST /api/projects/:id/source {source_path, source_kind} — change a project's sync directory

Worker (require any valid token)

• GET /api/auth/whoami — validate token → {username, role}
• POST /api/videos/:id/claim — atomic claim (annotated + unclaimed/expired); returns metadata + download_url + preloaded annotations
• GET /api/videos/:id/download — stream the source bytes (claimant/admin only)
• POST /api/videos/:id/heartbeat — extend the lease while verifying
• POST /api/videos/:id/release — abandon the claim, back to the pool
• POST /api/videos/:id/push — body = export doc + verify_time_ms; replace annotations, mark verified, record verifier + time, clear claim

Map / GPS / road_type / chainage

The server image bundles exiftool; a serial background sweep extracts each video's GPS track (gps_status: pending → ok | none | error). The raw track is immutable — a kept "Viterbi" correction lives beside it and is preferred by the map + chainage. road_type + the four chainage values (chainage_start, chainage_end, sr_chainage_start, sr_chainage_end) are injected into every claim/export doc and re-stamped on push (server-authoritative). Every annotation additionally gets its frame-exact lat/lon/chainage/sr_chainage (range annotations also end_*): frame 15 of a 30 fps video is t = 0.5 s — halfway between two 1 Hz GPS samples — so it gets the halfway position/chainage, not a second/video boundary. Chainage unit convention (everywhere — JSON + UI): kilometres with 4 decimals, e.g. 22.4456 (chainage = client-calibrated, sr_chainage = GPS-measured). Distances are WGS84-ellipsoidal (no spherical bias); calibration anchors persist in chainage_calibrations so this works at any time after a recompute.

• GET /api/projects/:id/map — per-video map rows: track (≤400 pts), status, road_type, chainage, gps_status (member/admin)
• GET /api/projects/:id/tracks.kml?token= — all tracks as KML LineStrings colored by status + calibration points (Google Earth)
• POST /api/videos/:id/gps/rescan — re-queue a video for the exiftool sweep (admin)
• POST /api/projects/:id/gps/rescan_all — re-queue every video (admin; background)
• POST /api/videos/:id/gps/correct {action: preview|keep|discard} — spike smoothing + missing-start stitch from the previous file (by GPS timestamp, duration-guarded); the scan sweep applies the same pipeline automatically as 'auto' (raw never changes; discard blocks re-auto-correction) (admin)
• GET/POST /api/projects/:id/road_types — list / replace label set ({names}, admin)
• POST /api/projects/:id/road_type_assign {video_ids, road_type} — bulk-assign ('' clears, admin)
• GET/POST /api/projects/:id/chainage/points — client calibration points (POST admin: {lat, lon, chainage_m, note?, road_type?})
• POST /api/projects/:id/chainage/points/bulk {points: […]} — transactional bulk insert; the dashboard's KML import uses this (placemark names like 12+400 or 12.4 become chainage) (admin)
• DELETE /api/chainage/points/:id — remove a point (admin)
• POST /api/projects/:id/chainage/recompute {video_ids, road_type} — order the videos into a route, snap the points, distribute the client-vs-GPS error piecewise-linearly; writes chainage_* (client) + sr_chainage_* (measured, immutable) + route_seq (admin)
• POST /api/projects/:id/chainage/quick {video_ids, start_m, end_m, road_type} — the two-point case: points at the route ends + same recompute (admin)

Env

var	default	meaning
ADMIN_TOKEN	dev-admin-token	master admin token — change in production
LEASE_SECS	900	claim lease length; auto-released after this
NFS_HOST	(empty)	NAS host/IP for nfs-kind projects (server-side only, never returned)
NFS_OPTS	nfsvers=4,ro,soft,timeo=30,retrans=2,retry=0	mount options for nfs-kind projects