Central Verification Dashboard
This commit is contained in:
176
README.md
Normal file
176
README.md
Normal file
@@ -0,0 +1,176 @@
|
||||
# Central Verification Dashboard (Phase 1)
|
||||
|
||||
Dockerized pipeline that ingests a folder of videos + their sibling export-JSONs into Postgres and
|
||||
serves a manager dashboard: completion overview, per-user leaderboard, throughput + ETA, and a
|
||||
per-video table. Standalone — it does **not** touch the desktop video-annotator app.
|
||||
|
||||
> Phase 1 = the dashboard pipeline. Worker pull/push (claim a video, verify, push) + token auth are
|
||||
> **Phase 3** and not built yet. So in this phase the dashboard shows the **annotation snapshot**
|
||||
> ingested from the NAS JSONs: `pending` (no JSON) vs `annotated` (has a JSON) vs `verified`
|
||||
> (a pushed pass — 0 until Phase 3).
|
||||
|
||||
## Run it
|
||||
|
||||
```bash
|
||||
cd central
|
||||
docker compose up --build
|
||||
```
|
||||
|
||||
- Dashboard → http://localhost:8081
|
||||
- API → http://localhost:8080 (e.g. `curl localhost:8080/api/projects`)
|
||||
- Postgres → localhost:5433 (user/pass/db = `central`)
|
||||
|
||||
On startup the server creates a `default` project pointing at `INGEST_DIR` (`/data`, mounted from
|
||||
`./sample-data`) and ingests it once. Click **Re-ingest** in the dashboard to re-scan after files
|
||||
change. Re-ingest is idempotent and never downgrades a `verified` video.
|
||||
|
||||
## Point it at the NAS (on the VM)
|
||||
|
||||
The ingest folder is just a path/volume. On the VM, mount the Synology share and map it to `/data`:
|
||||
|
||||
```yaml
|
||||
# docker-compose.override.yml on the VM
|
||||
services:
|
||||
server:
|
||||
volumes:
|
||||
- /mnt/synology/<client-folder>:/data:ro
|
||||
```
|
||||
|
||||
Then `POST /api/projects/<id>/ingest` (or restart the server). No code changes.
|
||||
|
||||
## Database backups (→ NAS)
|
||||
|
||||
A `db-backup` sidecar (`prodrigestivill/postgres-backup-local`) runs `pg_dump` on a schedule
|
||||
and writes gzipped dumps straight to the NAS, pruning old ones automatically.
|
||||
|
||||
- **Schedule:** daily at 02:00 IST (`SCHEDULE` + `TZ: Asia/Kolkata` in `docker-compose.yml`).
|
||||
- **Destination:** the `nasbackups` NFS volume → `:/volume4/Saudi_Video_Sync/Verification_Dashboard_Backup`
|
||||
on the NAS (`NFS_HOST`, default `192.168.1.199`). **Create that folder on the NAS** and allow
|
||||
rw NFS from the Docker host; the host needs an NFS client (`nfs-common`).
|
||||
- **Retention:** `BACKUP_KEEP_DAYS=30`, `BACKUP_KEEP_WEEKS=8`, `BACKUP_KEEP_MONTHS=6` (tune to taste).
|
||||
Dumps land under `daily/`, `weekly/`, `monthly/` as `central-YYYYMMDD-HHMMSS.sql.gz`.
|
||||
|
||||
```bash
|
||||
docker compose up -d db-backup # start it
|
||||
docker compose exec db-backup /backup.sh # run an immediate backup (verify it writes to the NAS)
|
||||
docker compose logs db-backup # check the schedule/last run
|
||||
```
|
||||
|
||||
**Restore** a dump into the running DB:
|
||||
|
||||
```bash
|
||||
gunzip -c /path/on/nas/daily/central-YYYYMMDD-HHMMSS.sql.gz \
|
||||
| docker compose exec -T db psql -U central -d central
|
||||
```
|
||||
|
||||
(For a clean restore, drop+recreate the DB first, or restore into a scratch DB and compare.)
|
||||
Test a restore periodically — an untested backup isn't a backup.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
central/
|
||||
db/init.sql # Postgres schema (projects, videos, annotations)
|
||||
server/ # Rust/Axum API + folder ingest (embeds db/init.sql)
|
||||
dashboard/ # React + Vite, served by nginx (proxies /api → server)
|
||||
sample-data/ # demo: one annotated video JSON + two empty .mp4 placeholders
|
||||
docker-compose.yml
|
||||
```
|
||||
|
||||
## Ingest details
|
||||
|
||||
For each video file (`.mp4/.mkv/.avi/.mov/.m4v/.webm`) it looks for a sibling JSON named
|
||||
`<file>_annotations.json`, `<file>.json`, or `<stem>.json` (video-annotator's export format —
|
||||
`{ video, fixed_annotations[], range_annotations[] }`). From it the server derives per-video
|
||||
annotation count, `annotated_by` (primary), `annotation_time_ms`, and `annotated_at` (latest
|
||||
annotation timestamp, used for the throughput chart), and stores the full doc in `videos.raw_json`
|
||||
for Phase 3 pull. Videos without a JSON are recorded as `pending`.
|
||||
|
||||
## API
|
||||
|
||||
### Clients → projects
|
||||
|
||||
A **client** (e.g. *IRB*) owns one or more **projects**; each project points at a folder via
|
||||
`source_path` + `source_kind`. Project names are unique **per client**. Use the dashboard's
|
||||
**Admin** tab: sign in (token) → create client (admin) → create project → **Sync**. A `default`
|
||||
client/project is seeded **only on a fresh DB** (local dev); once real clients exist it never reappears.
|
||||
|
||||
**Sync from NFS (`source_kind='nfs'`).** A project's `source_path` may be a **bare export path**
|
||||
(`/volume4/Saudi_Video_Sync` — the host comes from the `NFS_HOST` env, so the **IP stays server-side and
|
||||
is never shown/returned**), or include the host explicitly as `host:/volume4/Share` or
|
||||
`nfs://host/volume4/Share`. On sync the server mounts the share on demand (a fast TCP probe to port 2049
|
||||
fails an unreachable NAS in ~3 s; the mount is bounded so a stuck NAS can't hang the request).
|
||||
`source_kind='local'` keeps the old behaviour (a path inside the container, incl. a compose-managed NFS
|
||||
volume). Point a project at a **subfolder** (e.g. `/volume4/Saudi_Video_Sync/IRB_Master_VIdeos`), not a
|
||||
huge share root.
|
||||
|
||||
**Subfolders.** Ingest walks the folder **recursively**, so videos nested in subfolders are included;
|
||||
each video's `rel_path` carries its subfolder, which the dashboard shows in a **Folder** column.
|
||||
|
||||
> In-app NFS mounting needs `nfs-common` (in the image) and the server container's `cap_add: SYS_ADMIN`
|
||||
> + `security_opt: apparmor:unconfined` (in `docker-compose.yml`; use `privileged: true` if your host's
|
||||
> `mount.nfs` still refuses). `local`-only deployments need none of this.
|
||||
|
||||
### Roles
|
||||
|
||||
- **admin** (role `admin`, or the `ADMIN_TOKEN`): everything — create clients, manage collaborators, etc.
|
||||
- **collaborator** (role `worker`): can **add a project** and **change a project's sync directory**, but
|
||||
**not** create clients or manage users.
|
||||
|
||||
### Dashboard read APIs (open — the dashboard is manager-only on the VM)
|
||||
|
||||
- `GET /api/clients` — clients + project counts
|
||||
- `GET /api/projects` — projects + counts (each carries `client_id`, `client_name`, `source_kind`)
|
||||
- `GET /api/projects/:id/videos` — per-video rows (incl. claim/verify columns)
|
||||
- `GET /api/projects/:id/stats` — overview, annotator leaderboard, `verifiers`, throughput, eta_days
|
||||
- `GET /api/projects/:id/activity` — live claims + recent `video_events`
|
||||
- `GET /api/audit?limit=N` — org audit log (client/project created, sync dir changed, synced, user created)
|
||||
- `POST /api/projects/:id/ingest` — re-scan the project's folder ("Sync"); attributes the actor if a token is sent
|
||||
|
||||
## Phase 3 — worker pull/push + token auth
|
||||
|
||||
Workers claim an annotated video, verify it in the desktop app, and push the verified
|
||||
pass back. All Phase 3 routes require `Authorization: Bearer <token>`.
|
||||
|
||||
- **Auth model:** tokens are random 256-bit hex strings; only their SHA-256 hash is
|
||||
stored (`users.token_hash`). An admin provisions users and hands out the token (shown
|
||||
**once**). A master `ADMIN_TOKEN` env var grants admin (bootstrap / break-glass).
|
||||
- **Status vs. claim:** the annotation `status` (`pending`/`annotated`/`verified`) is
|
||||
unchanged; claiming is a separate dimension (`claimed_by` + `lease_expires_at`). A video
|
||||
is claimable when `annotated` and unclaimed-or-lease-expired. A push flips it to
|
||||
`verified`, records `completed_by`/`verify_time_ms`, and clears the claim. Re-ingest then
|
||||
**skips** verified videos (the pushed pass is authoritative).
|
||||
- **Leases auto-release:** a 60 s background task returns claims whose lease lapsed
|
||||
(`LEASE_SECS`, default 900) to the pool and logs an `auto_release` event.
|
||||
|
||||
### Admin (require admin / `ADMIN_TOKEN`)
|
||||
|
||||
- `POST /api/users` `{username, display_name?, role?}` → `{username, role, token}` (token shown once)
|
||||
- `GET /api/users` — list users (no tokens)
|
||||
- `POST /api/clients` `{name}` → the new client (unique name; 409 on dup)
|
||||
|
||||
### Create / manage projects (require any valid token — workers included)
|
||||
|
||||
- `POST /api/projects` `{client_id, name, source_path, source_kind}` — create/repoint a project
|
||||
under a client (idempotent on `(client_id, name)`); `source_kind` = `local`|`nfs`
|
||||
- `POST /api/projects/:id/source` `{source_path, source_kind}` — change a project's sync directory
|
||||
|
||||
### Worker (require any valid token)
|
||||
|
||||
- `GET /api/auth/whoami` — validate token → `{username, role}`
|
||||
- `POST /api/videos/:id/claim` — atomic claim (annotated + unclaimed/expired); returns
|
||||
metadata + `download_url` + preloaded `annotations`
|
||||
- `GET /api/videos/:id/download` — stream the source bytes (claimant/admin only)
|
||||
- `POST /api/videos/:id/heartbeat` — extend the lease while verifying
|
||||
- `POST /api/videos/:id/release` — abandon the claim, back to the pool
|
||||
- `POST /api/videos/:id/push` — body = export doc + `verify_time_ms`; replace annotations,
|
||||
mark `verified`, record verifier + time, clear claim
|
||||
|
||||
### Env
|
||||
|
||||
| var | default | meaning |
|
||||
|---------------|------------------------------------------|----------------------------------------------------|
|
||||
| `ADMIN_TOKEN` | `dev-admin-token` | master admin token — **change in production** |
|
||||
| `LEASE_SECS` | `900` | claim lease length; auto-released after this |
|
||||
| `NFS_HOST` | _(empty)_ | NAS host/IP for `nfs`-kind projects (server-side only, never returned) |
|
||||
| `NFS_OPTS` | `nfsvers=4,ro,soft,timeo=30,retrans=2,retry=0` | mount options for `nfs`-kind projects |
|
||||
Reference in New Issue
Block a user