From 7043d39ae077ea47ea04e5b96e480c702c63c4a7 Mon Sep 17 00:00:00 2001 From: bchanot Date: Sun, 1 Nov 2026 17:38:07 +0100 Subject: [PATCH] docs(docs): V1-2.3 verification worker, M6, sandbox e2e and rehearsals --- CLAUDE.md | 8 ++-- DEPLOY.md | 5 ++- crates/worker/src/log.rs | 2 +- docs/development/RUST.md | 41 ++++++++++++++++++++ docs/development/STORAGE.md | 44 +++++++++++++++++++++- docs/development/SUPABASE.md | 1 + docs/development/V1-2_ANATOMICAL_SHEETS.md | 15 +++++++- 7 files changed, 106 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9fe3bb0f..1770442b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -129,7 +129,7 @@ pnpm db:reset # replay from zero pnpm db:test # pgTAP on the real stack pnpm db:types # regenerate packages/supabase/src/database.types.ts pnpm db:types:check # CI diff -pnpm db:sandbox up fresh|up upgrade []|test-db|types [--check]|test|e2e-shell|down # throwaway stack (SUPABASE.md §5.1): migrations tested without touching DEV +pnpm db:sandbox up fresh|up upgrade []|test-db|types [--check]|test|e2e-shell|verify-e2e|down # throwaway stack (SUPABASE.md §5.1): migrations tested without touching DEV # Reference sheet in the app: the dev server bundles the curated draft; a build does not pnpm build:preview # SHEET_BUNDLE=draft build (manual validation, preview deployments) @@ -274,7 +274,7 @@ faunosteo/ │ └─ tests/{client,anatomical-sheets}.test.ts, tests/db/ ← PGlite harness, shim, pgTAP runner, schema↔types test ├─ supabase/ │ ├─ config.toml, seed.sql ← CLI config (public schema only), local dev accounts -│ ├─ migrations/*.sql ← forward-only; foundation, profiles, platform_roles, anatomical_sheets, storage, sheet_document_schema_v2, profiles_professional, files_foundation (assets, buckets, avatar policies, purge queue, RPCs), workspaces (V1-1.1: personal workspace per account, access helpers), clients (V1-1.2: client RPCs and search, write budget per workspace), animals_relations (V1-1.4: species rows, animals, relations with history and voiding, their RPCs and reads) +│ ├─ migrations/*.sql ← forward-only; foundation, profiles, platform_roles, anatomical_sheets, storage, sheet_document_schema_v2, profiles_professional, files_foundation (assets, buckets, avatar policies, purge queue, RPCs), workspaces (V1-1.1: personal workspace per account, access helpers), clients (V1-1.2: client RPCs and search, write budget per workspace), animals_relations (V1-1.4: species rows, animals, relations with history and voiding, their RPCs and reads), sheet_access_hardening + sheet_review_workflow (V1-2.1 M4, M5), sheet_verification_jobs (V1-2.3 M6: the worker's verdict RPCs, service role only) │ ├─ seed/anatomical-sheets/equine/ │ │ ├─ legacy-sources/ ← the historical Horse.svg, Horse_head.svg, pathologies.json, vertebrae.ts (importer input, never edited) │ │ ├─ legacy-import/ ← GENERATED baseline: document, id map, IMPORT-REPORT, SVG copies (no SQL) @@ -286,10 +286,10 @@ faunosteo/ ├─ crates/ ← Rust workspace (Cargo.toml, rust-toolchain.toml, deny.toml, clippy.toml, .cargo/config.toml at the root) │ ├─ core/ ← faunosteo-core: domain/{js,sheet,report,archive,editor,session} (contracts and rules; sheet/{size,keys,caps,review,report,verdict,preview,edit} since V1-2.2: server document measure, review groups and digests, the server verdict, PreviewSheet, the editor's commands, RUST.md §12), anatomy/{svg/{prescan,tree,sanitize,refs,number,serialize,clean},layer,compose} (R3: the one SVG reader, the layer, canvas + highlight composition), pdf/{read,json_js,import,write,compare} (R4: reader, writer and session, the archive check tool behind feature `cli`), ops.rs + facade.rs (`call`, `call_with_sheet`), json (JS number printing), curation/{text,ids,prettier_json,markdown,report_guard,legacy/*,draft/*} (R5, feature `curation`: Prettier layout, the legacy importer, the changeset applier, migration map, stats, seed SQL, CHANGELOG, REVIEW-REPORT, review drawings), curation/upload.rs (R5.5, feature `upload`: the upload rules, pure) + bin/report_compare.rs (`pnpm report:compare`) + bin/import_equine.rs (`pnpm r5:import-equine`) + bin/upload_sheet_assets/ (`pnpm assets:upload`, ureq; the three tools allowed std::fs); tests/{oracles,conformance,fixtures,pdf/LEDGER.tsv} frozen from the TypeScript (oracles/r3, oracles/r4, fixtures/svg/hostile); no I/O, no clock (lint-enforced) │ ├─ wasm/ ← faunosteo-wasm: wasm-bindgen facade (core_version, call, SheetHandle, PreviewBuilder → PreviewSheet, clean_svg; trap exports in the proofs build only) -│ └─ worker/ ← faunosteo-worker: plain main; `decode` sandboxed child (rlimits, seccomp allowlist, image → libwebp), parent Spawner + RIFF check; R1 `run` loop: config, log (JSON lines, allowlist), keys, backend trait + http (ureq, PostgREST rpc + Storage objects), job, sweep, watchdog; Dockerfile, scripts/derive-seccomp.sh +│ └─ worker/ ← faunosteo-worker: plain main; `decode` sandboxed child (rlimits, seccomp allowlist, image → libwebp), parent Spawner + RIFF check; V1-2.3 `verify` sandboxed child (framed input → the core's `compute_verdict`) and `run --jobs sheet-verify` (verify_job, VerifyBackend: M6 RPCs + drawing downloads); R1 `run` loop: config, log (JSON lines, allowlist), keys, backend trait + http (ureq, PostgREST rpc + Storage objects), job, sweep, watchdog; Dockerfile, scripts/derive-seccomp.sh ├─ scripts/{rust-env,rust-check,wasm-build,check-all,avatar-e2e,report-compare,import-equine,upload-sheet-assets,sheet-assets-e2e,floor-clarification-check}.sh, wasm-stamp.mjs ← native Rust gate, wasm build (pkg + pkg-proofs + stamp), the pre-merge gate (`pnpm check:all`), the real-stack avatar e2e, the archive check tool (`pnpm report:compare`), the Rust pipeline (`pnpm r5:import-equine`) ├─ scripts/{dev-env,supabase,dev}.sh ← the DEV origin (LAN address) in one place; CLI wrapper; `pnpm dev` on that origin -├─ docker-compose.yml ← `app` service; `worker` service under the `worker` profile (read-only, cap_drop ALL, external Supabase network, `.env.worker`) +├─ docker-compose.yml ← `app` service; `worker` and `worker-verify` (V1-2.3, `run --jobs sheet-verify`) services under the `worker` profile (read-only, cap_drop ALL, external Supabase network, `.env.worker`) ├─ docker/rust-builder.Dockerfile, rust-run.sh, csp-proof/ ← reproducible Rust builder (alternative) + the CSP proofs (R0 module, R2 app image: app-image.sh) ├─ .gitea/workflows/ci.yml ← PREPARED, never executed (no runner): rust job (builder image, scripts/rust-check.sh, types drift, wasm), install, typecheck, test, lint, build, docker (app + worker test stage + runtime) ├─ .gitea/workflows/{rust-advisories,release-images}.yml ← nightly RustSec advisories; manual digest-tagged image publication (no deploy) diff --git a/DEPLOY.md b/DEPLOY.md index 223269e9..cb03f50c 100644 --- a/DEPLOY.md +++ b/DEPLOY.md @@ -76,7 +76,10 @@ Rust (plan R0/R1, `docs/development/RUST.md`): the repository also builds a worker image (`crates/worker/Dockerfile`, static musl binary on `distroless/static`, container `faunosteo_worker`, service `worker` under the Compose profile `worker`). Since Step 8 it carries the files job loop (avatar -processing, purge sweeps: `docs/development/STORAGE.md` §3). It is not deployed +processing, purge sweeps: `docs/development/STORAGE.md` §3); since V1-2.3 the +same image also runs the sheet verification mode, service `worker-verify` +(`run --jobs sheet-verify`, STORAGE.md §3.6), never started on DEV without the +owner's go. It is not deployed yet: it needs the self-hosted Supabase stack beside it, which production does not run today; the live deployment above is unchanged. The production VPS never needs a Rust toolchain: Rust is compiled on the Ubuntu diff --git a/crates/worker/src/log.rs b/crates/worker/src/log.rs index 0d33e4fb..b1b68cce 100644 --- a/crates/worker/src/log.rs +++ b/crates/worker/src/log.rs @@ -1,4 +1,4 @@ -//! JSON-lines logging of the parent (STORAGE.md §3.6): one object per line on stdout, an +//! JSON-lines logging of the parent (STORAGE.md §3.7): one object per line on stdout, an //! allowlist of fields, values authored by the worker only. Text a child wrote is escaped //! by the JSON encoder and truncated; no key, no token, no user content ever reaches a line. diff --git a/docs/development/RUST.md b/docs/development/RUST.md index 2f6e806d..7a63e58a 100644 --- a/docs/development/RUST.md +++ b/docs/development/RUST.md @@ -4454,3 +4454,44 @@ Ignored cargo tests: the three are one test, `pdf_serialize::l_report_with_real_ once per cargo run whose features build `pdf_serialize` (workspace, `cli`, `ts,pdf-read,pdf-write`). A timing print for §10.7, 40 s in debug; run on request in release (2026-11-01: passed, 120 views, 46.5 MB, 5.2 s). The L report's export itself is covered unignored (`pdf_budget`); it masks no required scenario. + +## 13. V1-2.3: the server verdict in the worker (2026-11-01) + +Owner brief « Validation définitive V1-2.2 » (V1-2.2 VERIFIED at `c73415a`): V1-2.3 along the V1-2 contract §16.4. +Contract `.claude/tasks/contracts/2026-11-01-v1-2-3-worker-verification.md`, plan +`.claude/tasks/plans/2026-11-01-v1-2-3-worker.md`. Sandbox only; DEV untouched. + +| Part | What | Where | +|---|---|---| +| M6 | `claim_sheet_verifications`, `get_sheet_verification_input`, `complete_sheet_verification`, `fail_sheet_verification`, service role only (claims checked too); the verdict checked for its binding (stale) and its structure (malformed), nothing written otherwise | `supabase/migrations/20261101100000_sheet_verification_jobs.sql`, pgTAP `021` (51) | +| Link | the worker links `faunosteo-core` (default features: no PDF, no curation); `version` prints the core version | `crates/worker/Cargo.toml` | +| Child | `verify`: framed stdin (`[u32 LE header length][header JSON][document][base][drawings]`), `compute_verdict`, verdict JSON out; same sandbox as `decode`, its own limits (12 MiB in, 4 MiB out, 1 GiB, 60 s CPU, 90 s); a frame the child cannot read is a broken parent (`frame_invalid`), never content | `verify_frame.rs`, `verify.rs`, `limits.rs` | +| Spawner | per command: `Spawner::for_decode` (unchanged values), `Spawner::for_verify`; a verify output counts only when it parses as a JSON object | `spawn.rs` | +| Mode | `run --jobs sheet-verify`: no sweep; drawings fetched under the archive caps, anything else passed without bytes for the core to name; §6.5 rows mapped to closed worker codes; watchdog stage `VERIFY` 95 s | `run.rs`, `verify_job.rs`, `backend.rs`, `http.rs` | +| Compose | service `worker-verify`, profile `worker`, same hardening as `worker` | `docker-compose.yml` | + +Seccomp: the verify child at the caps was killed by SIGSYS in the Alpine stage; the derivation +(`derive-seccomp.sh`, extended to trace `verify --no-seccomp` on the clean and at-caps frames) traced `brk` +(musl mallocng growing the heap for the core's many small allocations) and nothing else new. `brk` was added to the +allowlist shared by both children; it only moves the program break, bounded by `RLIMIT_AS`. A filter per command +would keep `decode` without it, for more code: not done, listed for the owner. + +Proofs (2026-11-01): +- `pnpm worker:test`: unit 28, `job_loop` 11 (unchanged), `verify_loop` 15, sandbox 23 under the full filter + (clean sheet passed, at-caps sheet within the limits, empty environment, over-cap input, malformed frame, + violation killed), derivation « allowlist covers the trace ». `pnpm worker:image` ok. `pnpm rust:check` green. +- `pnpm db:sandbox verify-e2e` (real worker container, gateway on the sandbox network, no host port): 7 tests, + cases a to h of the contract: clean equine draft opened (about 60 ms in the worker); `svg_not_canonical` and + `drawing_missing` with their exits; `no_geometry` and `out_of_frame`; no decision or publication without a current + passed verdict, then approval and publication after it; stale and malformed completions leave the round + byte-identical; avatar and purge rows untouched by the verification mode; the working copy verified in place on + the shared drawing object; the at-caps document (1 059 150 bytes of text, 129 groups) verified in 185 ms by the + worker, 842 ms from submission (budget 120 s), on a 10 × 10 drawing (on the horse frame its markers fail + `marker_too_small`, the rule working). +- Upgrade rehearsals on the sandbox: from the state before M6, and from the state before M4 (the DEV state, M4 to + M6 in one `migration up`): pgTAP 1 859 PASS, `verify-e2e` 7/7, `types --check` equal. +- Not measured: the largest input the policy allows (a 2 MiB document, its 2 MiB base and four drawings, 6 MiB). + +Choices for the owner: worker codes, attempt limit five, lease 10 minutes, verdict cap 4 MiB, service name +`worker-verify`, the shared `brk`, and that an object larger than the policy is passed without bytes (named +`drawing_missing` by the core, not `size_mismatch`). diff --git a/docs/development/STORAGE.md b/docs/development/STORAGE.md index 8dd34dc8..057ed093 100644 --- a/docs/development/STORAGE.md +++ b/docs/development/STORAGE.md @@ -234,10 +234,50 @@ Docker's restart policy brings it back. Those are the only deletions the worker ever makes: a key it did not get from the queue or the leftover query is never deleted. -### 3.6 Logs +### 3.6 Sheet verification mode (V1-2.3) + +`faunosteo-worker run --jobs sheet-verify`, service `worker-verify` under the +Compose profile `worker` (same image, environment file and hardening as +`worker`), runs the server verdict of submitted sheet rounds (V1-2 contract +§6.4, §6.5) and nothing else: no avatar job, no sweep. The default `run` is +unchanged. + +- Parent: `claim_sheet_verifications(1)` (M6, service role only), then + `get_sheet_verification_input` (the frozen document and the base as text, + the drawing rows), then each drawing from `anatomical-sheet-assets` at its + key `.svg`. A row not uploaded, a key that is not its own sha256, + an object missing (404) or over the policy (at most 4 drawings, 2 MiB each, + 6 MiB in all: the archive caps) is passed to the core without bytes, so the + verdict names it (`drawing_missing`, caps): a business failure, never a + worker failure. The parent never parses the document or a drawing. +- Child `faunosteo-worker verify`: same sandbox as `decode` (empty + environment, one thread, no core dump, the seccomp allowlist) with its own + limits: framed stdin input up to 12 MiB, `RLIMIT_AS` 1 GiB, `RLIMIT_CPU` + 60 s, 90 s deadline, verdict output up to 4 MiB (the SQL cap). It runs the + core's `compute_verdict` and writes the verdict JSON; a failed verdict is + still a normal exit. The seccomp derivation of 2026-11-01 traced `brk` in + the verify child at the caps (musl mallocng growing its heap): added to the + allowlist, the only change. +- Completion: `complete_sheet_verification` answers `passed`, `failed`, + `stale` (nothing written, logged) or `malformed` (reported + `verification.worker.verdict_refused`). Child timeout → + `verification.worker.timeout`; a signal, an output overflow or an input over + its cap → `verification.worker.resource_limit`; a transport error → + `verification.worker.unavailable`; a broken child → `unavailable`, then the + worker stops (as the avatar loop). A dead lease (10 minutes) is claimed + again as `verification.worker.lease_expired`; the fifth attempt closes the + round `verification_failed` / `unavailable`. +- Measured (2026-11-01): the at-caps document (64 views, 1 024 annotations, + 8 192 nodes) verified in 185 ms by the worker on the sandbox, verdict + 24 689 bytes; the equine draft in about 60 ms. +- Sandbox only: `pnpm db:sandbox verify-e2e` starts a gateway and this + worker on the sandbox network (SUPABASE.md §5.1). Never started on DEV + (V1-2 contract §16.11 item 3). + +### 3.7 Logs One JSON object per line on stdout, fields from an allowlist only: `ts`, -`level`, `event`, `worker_id`, `asset_id`, `stage`, `code`, `signal`, +`level`, `event`, `worker_id`, `asset_id`, `review_id`, `stage`, `code`, `signal`, `http_status`, `duration_ms`, `count`, `detail` (worker-authored text only). The child's stderr is child-controlled: it is logged under `child_stderr` JSON-escaped and truncated to 200 bytes, never raw. No key, no URL with a diff --git a/docs/development/SUPABASE.md b/docs/development/SUPABASE.md index 8adae5f2..e290451d 100644 --- a/docs/development/SUPABASE.md +++ b/docs/development/SUPABASE.md @@ -800,6 +800,7 @@ Docker. | `pnpm db:sandbox types [--check]` | `supabase gen types --db-url ... --schema public` through Prettier, written or diffed with the committed file | | `pnpm db:sandbox test` | `packages/supabase/tests/sandbox/*.sandbox.ts`: real GoTrue, PostgREST and Storage sessions, concurrent SQL sessions (two-session lock order, V1-2.1), timing at the caps, the PGlite shim against the image; re-runnable on one stack | | `pnpm db:sandbox e2e-shell` (= `pnpm e2e:shell`) | the authenticated `/app` shell in Chromium (`apps/app/e2e/shell`, `docs/development/BROWSER_TESTS.md`): a loopback dev server whose Supabase calls are replayed on the sandbox, fictitious accounts per test | +| `pnpm db:sandbox verify-e2e` | V1-2.3: a gateway (`nginx:1.31-alpine`, sandbox network only, no host port) and the real verification worker `faunosteo_worker:dev run --jobs sheet-verify` against the sandbox, then `packages/supabase/tests/sandbox/sheet-verification.sandbox.ts`; both containers removed at the end, the service key passed through a 600 file deleted at start | | `pnpm db:sandbox sql` / `down` | psql as postgres on stdin / remove the stack | Every CLI call passes `--db-url` with `sslmode=disable` (the sandbox has no diff --git a/docs/development/V1-2_ANATOMICAL_SHEETS.md b/docs/development/V1-2_ANATOMICAL_SHEETS.md index 9ffa1429..29241201 100644 --- a/docs/development/V1-2_ANATOMICAL_SHEETS.md +++ b/docs/development/V1-2_ANATOMICAL_SHEETS.md @@ -926,6 +926,16 @@ own (§16.11), because a started DEV worker also processes the avatar upload waiting since the V1-1 acceptance (AVATAR-01) and the due rows of the purge queue. +Implemented in V1-2.3 (2026-11-01, migration `20261101100000_sheet_verification_jobs.sql`, +`crates/worker` mode `sheet-verify`, STORAGE.md §3.6): lease 10 minutes, attempt limit +five, verdict text at most 4 MiB; the completion also refuses an unknown verdict format +or code, a duplicated group key and a drawing whose result is not `ok`, and accepts a +failed verdict without groups or drawings; worker codes `verification.worker.unavailable`, +`.timeout`, `.resource_limit`, `.lease_expired` (a dead lease), `.verdict_refused` (a +completion answered `malformed`), a closed list in SQL (`private.sheet_worker_codes`). +These values and codes are the implementer's, for the owner's confirmation. The +service role executes these four RPCs besides the two upload ones (pgTAP 018). + Rejected: a verdict computed in browsers (contrary to the owner's rule), a synchronous HTTP verification service (a public endpoint holding the service key, a sandbox to rebuild; not in the MVP), an operator command-line @@ -945,7 +955,7 @@ did not pass; the closed codes keep the categories apart. | Timeout or resource limit | child over its wall-clock or memory limit | stays `verifying`, code recorded | yes, up to the limit | reduce the document or the drawing if it repeats | | Interruption | worker crash, lease expired | claimable again; counts as an attempt | yes, up to the limit | automatic | | Stale result | completion with an old token, after a withdrawal, or naming another revision | nothing written, logged | no | none needed | -| Definitive failure | attempt limit reached (five, proposal, tuned in V1-2.3) | `verification_failed`, outcome `unavailable`, last code | no | submit again later | +| Definitive failure | attempt limit reached (five; V1-2.3 implements five, for the owner's confirmation) | `verification_failed`, outcome `unavailable`, last code | no | submit again later | | Core change | release that changes rules or reader | rounds withdrawn by the release migration | no | submit again | - Withdrawal is always available: the submitter or any editor withdraws a @@ -1695,7 +1705,7 @@ of the document's JSON text in one form, the same for every layer: |---|---|---|---| | M4 `sheet_access_hardening` | V1-2.1 | `anatomical_catalog_grants` and its RPCs; `has_mfa`, `grant_active`, `caller_has`; `can_read_official_catalog` and the reader row predicate; `can_edit_sheet` redefined (F14); AR-1 B on the platform role RPCs and the private write functions (F19, §5.7), F17, F18; F1, F2, F3 (fork closed), F5, F6, F8, F9, F12 (every catalogue table), F13, F15; asset-row immutability trigger (F16); historical-sheet protection trigger (§14.2); reader column grants narrowed; personal creation and visibility changes refused | no row changes | | M5 `sheet_review_workflow` | V1-2.1 | version columns `revision`, `authors`, `source_version_id`, `published_review_id`, `document_fingerprint`; sheet column `working_copy_source_version_id`; revision, author and freeze triggers, the historical-sheet write rule; extended immutability guard (§6.8); rounds, comments, acknowledgments; `create_official_sheet`, `create_sheet_draft` adapted, `save_sheet_draft`, submit, withdraw, comment, acknowledge, decide, publish; `create_sheet_working_copy`; species RPCs; catalogue and version read RPCs; upload: `register_sheet_asset` extended, `confirm_sheet_asset_upload` with the storage check (F7), INSERT policy, bucket limit 2 MiB; mirror removal and status check (§12.2); superseded signatures dropped; the content-size CHECK replaced by the text measure (`octet_length(content::text)`, §12.2) | new columns take their defaults on the equine v1 row (`revision` 0, empty author set); its document, status, number and assets unchanged; the new size CHECK is validated against the existing rows when M5 is applied (the equine document measures 184 105 bytes as `content::text`, measured on the sandbox); bucket limit lowered; no round, no grant | -| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only | none | +| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only; written as `20261101100000_sheet_verification_jobs.sql`, pgTAP `021`, rehearsed on the sandbox from the state before M4 (M4 to M6 in one `migration up`) and from the state before M6 | none | Each migration ships its pgTAP file (allowed and denied cases), regenerated `database.types.ts`, a `db:sandbox up upgrade` run from the DEV state (with a @@ -2154,6 +2164,7 @@ V1-2.7 on. verification time at the caps within budget. - Rollback: branch revert; the worker image exists on the sandbox only. - Checkpoint: owner validates the server workflow (V1-2.1 to V1-2.3). +- State 2026-11-01: built and tested on the sandbox (RUST.md §13); DEV untouched. ### 16.5 V1-2.4 Wireframes (parallel to V1-2.1 to V1-2.3)