# Central Verification Dashboard (Phase 1)

Dockerized pipeline that ingests a folder of videos + their sibling export-JSONs into Postgres and
serves a manager dashboard: completion overview, per-user leaderboard, throughput + ETA, and a
per-video table. Standalone — it does **not** touch the desktop video-annotator app.

> Phase 1 = the dashboard pipeline. Worker pull/push (claim a video, verify, push) + token auth are
> **Phase 3** and not built yet. So in this phase the dashboard shows the **annotation snapshot**
> ingested from the NAS JSONs: `pending` (no JSON) vs `annotated` (has a JSON) vs `verified`
> (a pushed pass — 0 until Phase 3).

## Run it

```bash
cd central
docker compose up --build
```

- Dashboard → http://localhost:8081
- API       → http://localhost:8080  (e.g. `curl localhost:8080/api/projects`)
- Postgres  → localhost:5433  (user/pass/db = `central`)

On startup the server creates a `default` project pointing at `INGEST_DIR` (`/data`, mounted from
`./sample-data`) and ingests it once. Click **Re-ingest** in the dashboard to re-scan after files
change. Re-ingest is idempotent and never downgrades a `verified` video.

## Point it at the NAS (on the VM)

The ingest folder is just a path/volume. On the VM, mount the Synology share and map it to `/data`:

```yaml
# docker-compose.override.yml on the VM
services:
  server:
    volumes:
      - /mnt/synology/<client-folder>:/data:ro
```

Then `POST /api/projects/<id>/ingest` (or restart the server). No code changes.

## Database backups (→ NAS)

A `db-backup` sidecar (`prodrigestivill/postgres-backup-local`) runs `pg_dump` on a schedule
and writes gzipped dumps straight to the NAS, pruning old ones automatically.

- **Schedule:** daily at 02:00 IST (`SCHEDULE` + `TZ: Asia/Kolkata` in `docker-compose.yml`).
- **Destination:** the `nasbackups` NFS volume → `:/volume4/Saudi_Video_Sync/Verification_Dashboard_Backup`
  on the NAS (`NFS_HOST`, default `192.168.1.199`). **Create that folder on the NAS** and allow
  rw NFS from the Docker host; the host needs an NFS client (`nfs-common`).
- **Retention:** `BACKUP_KEEP_DAYS=30`, `BACKUP_KEEP_WEEKS=8`, `BACKUP_KEEP_MONTHS=6` (tune to taste).
  Dumps land under `daily/`, `weekly/`, `monthly/` as `central-YYYYMMDD-HHMMSS.sql.gz`.

```bash
docker compose up -d db-backup          # start it
docker compose exec db-backup /backup.sh # run an immediate backup (verify it writes to the NAS)
docker compose logs db-backup            # check the schedule/last run
```

**Restore** a dump into the running DB:

```bash
gunzip -c /path/on/nas/daily/central-YYYYMMDD-HHMMSS.sql.gz \
  | docker compose exec -T db psql -U central -d central
```

(For a clean restore, drop+recreate the DB first, or restore into a scratch DB and compare.)
Test a restore periodically — an untested backup isn't a backup.

## Layout

```
central/
  db/init.sql            # Postgres schema (projects, videos, annotations)
  server/                # Rust/Axum API + folder ingest  (embeds db/init.sql)
  dashboard/             # React + Vite, served by nginx (proxies /api → server)
  sample-data/           # demo: one annotated video JSON + two empty .mp4 placeholders
  docker-compose.yml
```

## Ingest details

For each video file (`.mp4/.mkv/.avi/.mov/.m4v/.webm`) it looks for a sibling JSON named
`<file>_annotations.json`, `<file>.json`, or `<stem>.json` (video-annotator's export format —
`{ video, fixed_annotations[], range_annotations[] }`). From it the server derives per-video
annotation count, `annotated_by` (primary), `annotation_time_ms`, and `annotated_at` (latest
annotation timestamp, used for the throughput chart), and stores the full doc in `videos.raw_json`
for Phase 3 pull. Videos without a JSON are recorded as `pending`.

## API

### Clients → projects

A **client** (e.g. *IRB*) owns one or more **projects**; each project points at a folder via
`source_path` + `source_kind`. Project names are unique **per client**. Use the dashboard's
**Admin** tab: sign in (token) → create client (admin) → create project → **Sync**. A `default`
client/project is seeded **only on a fresh DB** (local dev); once real clients exist it never reappears.

**Sync from NFS (`source_kind='nfs'`).** A project's `source_path` may be a **bare export path**
(`/volume4/Saudi_Video_Sync` — the host comes from the `NFS_HOST` env, so the **IP stays server-side and
is never shown/returned**), or include the host explicitly as `host:/volume4/Share` or
`nfs://host/volume4/Share`. On sync the server mounts the share on demand (a fast TCP probe to port 2049
fails an unreachable NAS in ~3 s; the mount is bounded so a stuck NAS can't hang the request).
`source_kind='local'` keeps the old behaviour (a path inside the container, incl. a compose-managed NFS
volume). Point a project at a **subfolder** (e.g. `/volume4/Saudi_Video_Sync/IRB_Master_VIdeos`), not a
huge share root.

**Subfolders.** Ingest walks the folder **recursively**, so videos nested in subfolders are included;
each video's `rel_path` carries its subfolder, which the dashboard shows in a **Folder** column.

> In-app NFS mounting needs `nfs-common` (in the image) and the server container's `cap_add: SYS_ADMIN`
> + `security_opt: apparmor:unconfined` (in `docker-compose.yml`; use `privileged: true` if your host's
> `mount.nfs` still refuses). `local`-only deployments need none of this.

### Roles

- **admin** (role `admin`, or the `ADMIN_TOKEN`): everything — create clients, manage collaborators, etc.
- **collaborator** (role `worker`): can **add a project** and **change a project's sync directory**, but
  **not** create clients or manage users.

### Dashboard read APIs (open — the dashboard is manager-only on the VM)

- `GET  /api/clients`                   — clients + project counts
- `GET  /api/projects`                  — projects + counts (each carries `client_id`, `client_name`, `source_kind`)
- `GET  /api/projects/:id/videos`       — per-video rows (incl. claim/verify columns)
- `GET  /api/projects/:id/stats`        — overview, annotator leaderboard, `verifiers`, throughput, eta_days
- `GET  /api/projects/:id/activity`     — live claims + recent `video_events`
- `GET  /api/audit?limit=N`             — org audit log (client/project created, sync dir changed, synced, user created)
- `POST /api/projects/:id/ingest`       — re-scan the project's folder ("Sync"); attributes the actor if a token is sent

## Phase 3 — worker pull/push + token auth

Workers claim an annotated video, verify it in the desktop app, and push the verified
pass back. All Phase 3 routes require `Authorization: Bearer <token>`.

- **Auth model:** tokens are random 256-bit hex strings; only their SHA-256 hash is
  stored (`users.token_hash`). An admin provisions users and hands out the token (shown
  **once**). A master `ADMIN_TOKEN` env var grants admin (bootstrap / break-glass).
- **Status vs. claim:** the annotation `status` (`pending`/`annotated`/`verified`) is
  unchanged; claiming is a separate dimension (`claimed_by` + `lease_expires_at`). A video
  is claimable when `annotated` and unclaimed-or-lease-expired. A push flips it to
  `verified`, records `completed_by`/`verify_time_ms`, and clears the claim. Re-ingest then
  **skips** verified videos (the pushed pass is authoritative).
- **Leases auto-release:** a 60 s background task returns claims whose lease lapsed
  (`LEASE_SECS`, default 900) to the pool and logs an `auto_release` event.

### Admin (require admin / `ADMIN_TOKEN`)

- `POST /api/users`    `{username, display_name?, role?}` → `{username, role, token}` (token shown once)
- `GET  /api/users`    — list users (no tokens)
- `POST /api/clients`  `{name}` → the new client (unique name; 409 on dup)

### Create / manage projects (require any valid token — workers included)

- `POST /api/projects`          `{client_id, name, source_path, source_kind}` — create/repoint a project
                                under a client (idempotent on `(client_id, name)`); `source_kind` = `local`|`nfs`
- `POST /api/projects/:id/source` `{source_path, source_kind}` — change a project's sync directory

### Worker (require any valid token)

- `GET  /api/auth/whoami`            — validate token → `{username, role}`
- `POST /api/videos/:id/claim`       — atomic claim (annotated + unclaimed/expired); returns
                                       metadata + `download_url` + preloaded `annotations`
- `GET  /api/videos/:id/download`    — stream the source bytes (claimant/admin only)
- `POST /api/videos/:id/heartbeat`   — extend the lease while verifying
- `POST /api/videos/:id/release`     — abandon the claim, back to the pool
- `POST /api/videos/:id/push`        — body = export doc + `verify_time_ms`; replace annotations,
                                       mark `verified`, record verifier + time, clear claim

### Map / GPS / road_type / chainage

The server image bundles exiftool; a serial background sweep extracts each video's GPS
track (`gps_status`: `pending → ok | none | error`). The raw track is immutable —
a kept "Viterbi" correction lives beside it and is preferred by the map + chainage.
`road_type` + the four chainage values are injected into every claim/export doc and
re-stamped on push (server-authoritative). **Every annotation** additionally gets its
**frame-exact** `lat`/`lon`/`chainage_m`/`sr_chainage_m` (range annotations also
`end_*`): frame 15 of a 30 fps video is t = 0.5 s — halfway between two 1 Hz GPS
samples — so it gets the halfway position/chainage, not a second/video boundary.
Distances are WGS84-ellipsoidal (no spherical bias); calibration anchors persist in
`chainage_calibrations` so this works at any time after a recompute.

- `GET  /api/projects/:id/map`         — per-video map rows: track (≤400 pts), status,
                                         road_type, chainage, `gps_status` (member/admin)
- `GET  /api/projects/:id/tracks.kml?token=` — all tracks as KML LineStrings colored by
                                         status + calibration points (Google Earth)
- `POST /api/videos/:id/gps/rescan`    — re-queue a video for the exiftool sweep (admin)
- `POST /api/projects/:id/gps/rescan_all` — re-queue every video (admin; background)
- `POST /api/videos/:id/gps/correct`   `{action: preview|keep|discard}` — spike smoothing +
                                         missing-start stitch from the previous file (by GPS
                                         timestamp, duration-guarded); the scan sweep applies
                                         the same pipeline automatically as 'auto' (raw never
                                         changes; discard blocks re-auto-correction) (admin)
- `GET/POST /api/projects/:id/road_types`      — list / replace label set (`{names}`, admin)
- `POST /api/projects/:id/road_type_assign`    `{video_ids, road_type}` — bulk-assign ('' clears, admin)
- `GET/POST /api/projects/:id/chainage/points` — client calibration points (POST admin:
                                         `{lat, lon, chainage_m, note?, road_type?}`)
- `POST /api/projects/:id/chainage/points/bulk` `{points: […]}` — transactional bulk insert;
                                         the dashboard's **KML import** uses this (placemark
                                         names like `12+400` or `12.4` become chainage) (admin)
- `DELETE /api/chainage/points/:id`    — remove a point (admin)
- `POST /api/projects/:id/chainage/recompute`  `{video_ids, road_type}` — order the videos into a
                                         route, snap the points, distribute the client-vs-GPS error
                                         piecewise-linearly; writes `chainage_*` (client) +
                                         `sr_chainage_*` (measured, immutable) + `route_seq` (admin)
- `POST /api/projects/:id/chainage/quick`      `{video_ids, start_m, end_m, road_type}` — the
                                         two-point case: points at the route ends + same recompute (admin)

### Env

| var           | default                                  | meaning                                            |
|---------------|------------------------------------------|----------------------------------------------------|
| `ADMIN_TOKEN` | `dev-admin-token`                        | master admin token — **change in production**      |
| `LEASE_SECS`  | `900`                                     | claim lease length; auto-released after this        |
| `NFS_HOST`    | _(empty)_                                 | NAS host/IP for `nfs`-kind projects (server-side only, never returned) |
| `NFS_OPTS`    | `nfsvers=4,ro,soft,timeo=30,retrans=2,retry=0` | mount options for `nfs`-kind projects        |