docs(docs): V1-2.3 verification worker, M6, sandbox e2e and rehearsals
This commit is contained in:
1 parent
5afb34ed0f
commit
7043d39ae0
7 files changed
+106
-10
No files matched your search
@@ -129,7 +129,7 @@ pnpm db:reset # replay from zero
|
||||
pnpm db:test # pgTAP on the real stack
|
||||
pnpm db:types # regenerate packages/supabase/src/database.types.ts
|
||||
pnpm db:types:check # CI diff
|
||||
pnpm db:sandbox up fresh|up upgrade <version> [<then>]|test-db|types [--check]|test|e2e-shell|down # throwaway stack (SUPABASE.md §5.1): migrations tested without touching DEV
|
||||
pnpm db:sandbox up fresh|up upgrade <version> [<then>]|test-db|types [--check]|test|e2e-shell|verify-e2e|down # throwaway stack (SUPABASE.md §5.1): migrations tested without touching DEV
|
||||
|
||||
# Reference sheet in the app: the dev server bundles the curated draft; a build does not
|
||||
pnpm build:preview # SHEET_BUNDLE=draft build (manual validation, preview deployments)
|
||||
@@ -274,7 +274,7 @@ faunosteo/
|
||||
│ └─ tests/{client,anatomical-sheets}.test.ts, tests/db/ ← PGlite harness, shim, pgTAP runner, schema↔types test
|
||||
├─ supabase/
|
||||
│ ├─ config.toml, seed.sql ← CLI config (public schema only), local dev accounts
|
||||
│ ├─ migrations/*.sql ← forward-only; foundation, profiles, platform_roles, anatomical_sheets, storage, sheet_document_schema_v2, profiles_professional, files_foundation (assets, buckets, avatar policies, purge queue, RPCs), workspaces (V1-1.1: personal workspace per account, access helpers), clients (V1-1.2: client RPCs and search, write budget per workspace), animals_relations (V1-1.4: species rows, animals, relations with history and voiding, their RPCs and reads)
|
||||
│ ├─ migrations/*.sql ← forward-only; foundation, profiles, platform_roles, anatomical_sheets, storage, sheet_document_schema_v2, profiles_professional, files_foundation (assets, buckets, avatar policies, purge queue, RPCs), workspaces (V1-1.1: personal workspace per account, access helpers), clients (V1-1.2: client RPCs and search, write budget per workspace), animals_relations (V1-1.4: species rows, animals, relations with history and voiding, their RPCs and reads), sheet_access_hardening + sheet_review_workflow (V1-2.1 M4, M5), sheet_verification_jobs (V1-2.3 M6: the worker's verdict RPCs, service role only)
|
||||
│ ├─ seed/anatomical-sheets/equine/
|
||||
│ │ ├─ legacy-sources/ ← the historical Horse.svg, Horse_head.svg, pathologies.json, vertebrae.ts (importer input, never edited)
|
||||
│ │ ├─ legacy-import/ ← GENERATED baseline: document, id map, IMPORT-REPORT, SVG copies (no SQL)
|
||||
@@ -286,10 +286,10 @@ faunosteo/
|
||||
├─ crates/ ← Rust workspace (Cargo.toml, rust-toolchain.toml, deny.toml, clippy.toml, .cargo/config.toml at the root)
|
||||
│ ├─ core/ ← faunosteo-core: domain/{js,sheet,report,archive,editor,session} (contracts and rules; sheet/{size,keys,caps,review,report,verdict,preview,edit} since V1-2.2: server document measure, review groups and digests, the server verdict, PreviewSheet, the editor's commands, RUST.md §12), anatomy/{svg/{prescan,tree,sanitize,refs,number,serialize,clean},layer,compose} (R3: the one SVG reader, the layer, canvas + highlight composition), pdf/{read,json_js,import,write,compare} (R4: reader, writer and session, the archive check tool behind feature `cli`), ops.rs + facade.rs (`call`, `call_with_sheet`), json (JS number printing), curation/{text,ids,prettier_json,markdown,report_guard,legacy/*,draft/*} (R5, feature `curation`: Prettier layout, the legacy importer, the changeset applier, migration map, stats, seed SQL, CHANGELOG, REVIEW-REPORT, review drawings), curation/upload.rs (R5.5, feature `upload`: the upload rules, pure) + bin/report_compare.rs (`pnpm report:compare`) + bin/import_equine.rs (`pnpm r5:import-equine`) + bin/upload_sheet_assets/ (`pnpm assets:upload`, ureq; the three tools allowed std::fs); tests/{oracles,conformance,fixtures,pdf/LEDGER.tsv} frozen from the TypeScript (oracles/r3, oracles/r4, fixtures/svg/hostile); no I/O, no clock (lint-enforced)
|
||||
│ ├─ wasm/ ← faunosteo-wasm: wasm-bindgen facade (core_version, call, SheetHandle, PreviewBuilder → PreviewSheet, clean_svg; trap exports in the proofs build only)
|
||||
│ └─ worker/ ← faunosteo-worker: plain main; `decode` sandboxed child (rlimits, seccomp allowlist, image → libwebp), parent Spawner + RIFF check; R1 `run` loop: config, log (JSON lines, allowlist), keys, backend trait + http (ureq, PostgREST rpc + Storage objects), job, sweep, watchdog; Dockerfile, scripts/derive-seccomp.sh
|
||||
│ └─ worker/ ← faunosteo-worker: plain main; `decode` sandboxed child (rlimits, seccomp allowlist, image → libwebp), parent Spawner + RIFF check; V1-2.3 `verify` sandboxed child (framed input → the core's `compute_verdict`) and `run --jobs sheet-verify` (verify_job, VerifyBackend: M6 RPCs + drawing downloads); R1 `run` loop: config, log (JSON lines, allowlist), keys, backend trait + http (ureq, PostgREST rpc + Storage objects), job, sweep, watchdog; Dockerfile, scripts/derive-seccomp.sh
|
||||
├─ scripts/{rust-env,rust-check,wasm-build,check-all,avatar-e2e,report-compare,import-equine,upload-sheet-assets,sheet-assets-e2e,floor-clarification-check}.sh, wasm-stamp.mjs ← native Rust gate, wasm build (pkg + pkg-proofs + stamp), the pre-merge gate (`pnpm check:all`), the real-stack avatar e2e, the archive check tool (`pnpm report:compare`), the Rust pipeline (`pnpm r5:import-equine`)
|
||||
├─ scripts/{dev-env,supabase,dev}.sh ← the DEV origin (LAN address) in one place; CLI wrapper; `pnpm dev` on that origin
|
||||
├─ docker-compose.yml ← `app` service; `worker` service under the `worker` profile (read-only, cap_drop ALL, external Supabase network, `.env.worker`)
|
||||
├─ docker-compose.yml ← `app` service; `worker` and `worker-verify` (V1-2.3, `run --jobs sheet-verify`) services under the `worker` profile (read-only, cap_drop ALL, external Supabase network, `.env.worker`)
|
||||
├─ docker/rust-builder.Dockerfile, rust-run.sh, csp-proof/ ← reproducible Rust builder (alternative) + the CSP proofs (R0 module, R2 app image: app-image.sh)
|
||||
├─ .gitea/workflows/ci.yml ← PREPARED, never executed (no runner): rust job (builder image, scripts/rust-check.sh, types drift, wasm), install, typecheck, test, lint, build, docker (app + worker test stage + runtime)
|
||||
├─ .gitea/workflows/{rust-advisories,release-images}.yml ← nightly RustSec advisories; manual digest-tagged image publication (no deploy)
|
||||
|
||||
@@ -76,7 +76,10 @@ Rust (plan R0/R1, `docs/development/RUST.md`): the repository also builds a
|
||||
worker image (`crates/worker/Dockerfile`, static musl binary on
|
||||
`distroless/static`, container `faunosteo_worker`, service `worker` under the
|
||||
Compose profile `worker`). Since Step 8 it carries the files job loop (avatar
|
||||
processing, purge sweeps: `docs/development/STORAGE.md` §3). It is not deployed
|
||||
processing, purge sweeps: `docs/development/STORAGE.md` §3); since V1-2.3 the
|
||||
same image also runs the sheet verification mode, service `worker-verify`
|
||||
(`run --jobs sheet-verify`, STORAGE.md §3.6), never started on DEV without the
|
||||
owner's go. It is not deployed
|
||||
yet: it needs the self-hosted Supabase stack beside it, which production does
|
||||
not run today; the live deployment above is unchanged. The
|
||||
production VPS never needs a Rust toolchain: Rust is compiled on the Ubuntu
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
//! JSON-lines logging of the parent (STORAGE.md §3.6): one object per line on stdout, an
|
||||
//! JSON-lines logging of the parent (STORAGE.md §3.7): one object per line on stdout, an
|
||||
//! allowlist of fields, values authored by the worker only. Text a child wrote is escaped
|
||||
//! by the JSON encoder and truncated; no key, no token, no user content ever reaches a line.
|
||||
|
||||
|
||||
@@ -4454,3 +4454,44 @@ Ignored cargo tests: the three are one test, `pdf_serialize::l_report_with_real_
|
||||
once per cargo run whose features build `pdf_serialize` (workspace, `cli`, `ts,pdf-read,pdf-write`). A timing print for
|
||||
§10.7, 40 s in debug; run on request in release (2026-11-01: passed, 120 views, 46.5 MB, 5.2 s). The L report's export
|
||||
itself is covered unignored (`pdf_budget`); it masks no required scenario.
|
||||
|
||||
## 13. V1-2.3: the server verdict in the worker (2026-11-01)
|
||||
|
||||
Owner brief « Validation définitive V1-2.2 » (V1-2.2 VERIFIED at `c73415a`): V1-2.3 along the V1-2 contract §16.4.
|
||||
Contract `.claude/tasks/contracts/2026-11-01-v1-2-3-worker-verification.md`, plan
|
||||
`.claude/tasks/plans/2026-11-01-v1-2-3-worker.md`. Sandbox only; DEV untouched.
|
||||
|
||||
| Part | What | Where |
|
||||
|---|---|---|
|
||||
| M6 | `claim_sheet_verifications`, `get_sheet_verification_input`, `complete_sheet_verification`, `fail_sheet_verification`, service role only (claims checked too); the verdict checked for its binding (stale) and its structure (malformed), nothing written otherwise | `supabase/migrations/20261101100000_sheet_verification_jobs.sql`, pgTAP `021` (51) |
|
||||
| Link | the worker links `faunosteo-core` (default features: no PDF, no curation); `version` prints the core version | `crates/worker/Cargo.toml` |
|
||||
| Child | `verify`: framed stdin (`[u32 LE header length][header JSON][document][base][drawings]`), `compute_verdict`, verdict JSON out; same sandbox as `decode`, its own limits (12 MiB in, 4 MiB out, 1 GiB, 60 s CPU, 90 s); a frame the child cannot read is a broken parent (`frame_invalid`), never content | `verify_frame.rs`, `verify.rs`, `limits.rs` |
|
||||
| Spawner | per command: `Spawner::for_decode` (unchanged values), `Spawner::for_verify`; a verify output counts only when it parses as a JSON object | `spawn.rs` |
|
||||
| Mode | `run --jobs sheet-verify`: no sweep; drawings fetched under the archive caps, anything else passed without bytes for the core to name; §6.5 rows mapped to closed worker codes; watchdog stage `VERIFY` 95 s | `run.rs`, `verify_job.rs`, `backend.rs`, `http.rs` |
|
||||
| Compose | service `worker-verify`, profile `worker`, same hardening as `worker` | `docker-compose.yml` |
|
||||
|
||||
Seccomp: the verify child at the caps was killed by SIGSYS in the Alpine stage; the derivation
|
||||
(`derive-seccomp.sh`, extended to trace `verify --no-seccomp` on the clean and at-caps frames) traced `brk`
|
||||
(musl mallocng growing the heap for the core's many small allocations) and nothing else new. `brk` was added to the
|
||||
allowlist shared by both children; it only moves the program break, bounded by `RLIMIT_AS`. A filter per command
|
||||
would keep `decode` without it, for more code: not done, listed for the owner.
|
||||
|
||||
Proofs (2026-11-01):
|
||||
- `pnpm worker:test`: unit 28, `job_loop` 11 (unchanged), `verify_loop` 15, sandbox 23 under the full filter
|
||||
(clean sheet passed, at-caps sheet within the limits, empty environment, over-cap input, malformed frame,
|
||||
violation killed), derivation « allowlist covers the trace ». `pnpm worker:image` ok. `pnpm rust:check` green.
|
||||
- `pnpm db:sandbox verify-e2e` (real worker container, gateway on the sandbox network, no host port): 7 tests,
|
||||
cases a to h of the contract: clean equine draft opened (about 60 ms in the worker); `svg_not_canonical` and
|
||||
`drawing_missing` with their exits; `no_geometry` and `out_of_frame`; no decision or publication without a current
|
||||
passed verdict, then approval and publication after it; stale and malformed completions leave the round
|
||||
byte-identical; avatar and purge rows untouched by the verification mode; the working copy verified in place on
|
||||
the shared drawing object; the at-caps document (1 059 150 bytes of text, 129 groups) verified in 185 ms by the
|
||||
worker, 842 ms from submission (budget 120 s), on a 10 × 10 drawing (on the horse frame its markers fail
|
||||
`marker_too_small`, the rule working).
|
||||
- Upgrade rehearsals on the sandbox: from the state before M6, and from the state before M4 (the DEV state, M4 to
|
||||
M6 in one `migration up`): pgTAP 1 859 PASS, `verify-e2e` 7/7, `types --check` equal.
|
||||
- Not measured: the largest input the policy allows (a 2 MiB document, its 2 MiB base and four drawings, 6 MiB).
|
||||
|
||||
Choices for the owner: worker codes, attempt limit five, lease 10 minutes, verdict cap 4 MiB, service name
|
||||
`worker-verify`, the shared `brk`, and that an object larger than the policy is passed without bytes (named
|
||||
`drawing_missing` by the core, not `size_mismatch`).
|
||||
@@ -234,10 +234,50 @@ Docker's restart policy brings it back.
|
||||
Those are the only deletions the worker ever makes: a key it did not get
|
||||
from the queue or the leftover query is never deleted.
|
||||
|
||||
### 3.6 Logs
|
||||
### 3.6 Sheet verification mode (V1-2.3)
|
||||
|
||||
`faunosteo-worker run --jobs sheet-verify`, service `worker-verify` under the
|
||||
Compose profile `worker` (same image, environment file and hardening as
|
||||
`worker`), runs the server verdict of submitted sheet rounds (V1-2 contract
|
||||
§6.4, §6.5) and nothing else: no avatar job, no sweep. The default `run` is
|
||||
unchanged.
|
||||
|
||||
- Parent: `claim_sheet_verifications(1)` (M6, service role only), then
|
||||
`get_sheet_verification_input` (the frozen document and the base as text,
|
||||
the drawing rows), then each drawing from `anatomical-sheet-assets` at its
|
||||
key `<sha256>.svg`. A row not uploaded, a key that is not its own sha256,
|
||||
an object missing (404) or over the policy (at most 4 drawings, 2 MiB each,
|
||||
6 MiB in all: the archive caps) is passed to the core without bytes, so the
|
||||
verdict names it (`drawing_missing`, caps): a business failure, never a
|
||||
worker failure. The parent never parses the document or a drawing.
|
||||
- Child `faunosteo-worker verify`: same sandbox as `decode` (empty
|
||||
environment, one thread, no core dump, the seccomp allowlist) with its own
|
||||
limits: framed stdin input up to 12 MiB, `RLIMIT_AS` 1 GiB, `RLIMIT_CPU`
|
||||
60 s, 90 s deadline, verdict output up to 4 MiB (the SQL cap). It runs the
|
||||
core's `compute_verdict` and writes the verdict JSON; a failed verdict is
|
||||
still a normal exit. The seccomp derivation of 2026-11-01 traced `brk` in
|
||||
the verify child at the caps (musl mallocng growing its heap): added to the
|
||||
allowlist, the only change.
|
||||
- Completion: `complete_sheet_verification` answers `passed`, `failed`,
|
||||
`stale` (nothing written, logged) or `malformed` (reported
|
||||
`verification.worker.verdict_refused`). Child timeout →
|
||||
`verification.worker.timeout`; a signal, an output overflow or an input over
|
||||
its cap → `verification.worker.resource_limit`; a transport error →
|
||||
`verification.worker.unavailable`; a broken child → `unavailable`, then the
|
||||
worker stops (as the avatar loop). A dead lease (10 minutes) is claimed
|
||||
again as `verification.worker.lease_expired`; the fifth attempt closes the
|
||||
round `verification_failed` / `unavailable`.
|
||||
- Measured (2026-11-01): the at-caps document (64 views, 1 024 annotations,
|
||||
8 192 nodes) verified in 185 ms by the worker on the sandbox, verdict
|
||||
24 689 bytes; the equine draft in about 60 ms.
|
||||
- Sandbox only: `pnpm db:sandbox verify-e2e` starts a gateway and this
|
||||
worker on the sandbox network (SUPABASE.md §5.1). Never started on DEV
|
||||
(V1-2 contract §16.11 item 3).
|
||||
|
||||
### 3.7 Logs
|
||||
|
||||
One JSON object per line on stdout, fields from an allowlist only: `ts`,
|
||||
`level`, `event`, `worker_id`, `asset_id`, `stage`, `code`, `signal`,
|
||||
`level`, `event`, `worker_id`, `asset_id`, `review_id`, `stage`, `code`, `signal`,
|
||||
`http_status`, `duration_ms`, `count`, `detail` (worker-authored text only).
|
||||
The child's stderr is child-controlled: it is logged under `child_stderr`
|
||||
JSON-escaped and truncated to 200 bytes, never raw. No key, no URL with a
|
||||
|
||||
@@ -800,6 +800,7 @@ Docker.
|
||||
| `pnpm db:sandbox types [--check]` | `supabase gen types --db-url ... --schema public` through Prettier, written or diffed with the committed file |
|
||||
| `pnpm db:sandbox test` | `packages/supabase/tests/sandbox/*.sandbox.ts`: real GoTrue, PostgREST and Storage sessions, concurrent SQL sessions (two-session lock order, V1-2.1), timing at the caps, the PGlite shim against the image; re-runnable on one stack |
|
||||
| `pnpm db:sandbox e2e-shell` (= `pnpm e2e:shell`) | the authenticated `/app` shell in Chromium (`apps/app/e2e/shell`, `docs/development/BROWSER_TESTS.md`): a loopback dev server whose Supabase calls are replayed on the sandbox, fictitious accounts per test |
|
||||
| `pnpm db:sandbox verify-e2e` | V1-2.3: a gateway (`nginx:1.31-alpine`, sandbox network only, no host port) and the real verification worker `faunosteo_worker:dev run --jobs sheet-verify` against the sandbox, then `packages/supabase/tests/sandbox/sheet-verification.sandbox.ts`; both containers removed at the end, the service key passed through a 600 file deleted at start |
|
||||
| `pnpm db:sandbox sql` / `down` | psql as postgres on stdin / remove the stack |
|
||||
|
||||
Every CLI call passes `--db-url` with `sslmode=disable` (the sandbox has no
|
||||
|
||||
@@ -926,6 +926,16 @@ own (§16.11), because a started DEV worker also processes the avatar upload
|
||||
waiting since the V1-1 acceptance (AVATAR-01) and the due rows of the purge
|
||||
queue.
|
||||
|
||||
Implemented in V1-2.3 (2026-11-01, migration `20261101100000_sheet_verification_jobs.sql`,
|
||||
`crates/worker` mode `sheet-verify`, STORAGE.md §3.6): lease 10 minutes, attempt limit
|
||||
five, verdict text at most 4 MiB; the completion also refuses an unknown verdict format
|
||||
or code, a duplicated group key and a drawing whose result is not `ok`, and accepts a
|
||||
failed verdict without groups or drawings; worker codes `verification.worker.unavailable`,
|
||||
`.timeout`, `.resource_limit`, `.lease_expired` (a dead lease), `.verdict_refused` (a
|
||||
completion answered `malformed`), a closed list in SQL (`private.sheet_worker_codes`).
|
||||
These values and codes are the implementer's, for the owner's confirmation. The
|
||||
service role executes these four RPCs besides the two upload ones (pgTAP 018).
|
||||
|
||||
Rejected: a verdict computed in browsers (contrary to the owner's rule), a
|
||||
synchronous HTTP verification service (a public endpoint holding the service
|
||||
key, a sandbox to rebuild; not in the MVP), an operator command-line
|
||||
@@ -945,7 +955,7 @@ did not pass; the closed codes keep the categories apart.
|
||||
| Timeout or resource limit | child over its wall-clock or memory limit | stays `verifying`, code recorded | yes, up to the limit | reduce the document or the drawing if it repeats |
|
||||
| Interruption | worker crash, lease expired | claimable again; counts as an attempt | yes, up to the limit | automatic |
|
||||
| Stale result | completion with an old token, after a withdrawal, or naming another revision | nothing written, logged | no | none needed |
|
||||
| Definitive failure | attempt limit reached (five, proposal, tuned in V1-2.3) | `verification_failed`, outcome `unavailable`, last code | no | submit again later |
|
||||
| Definitive failure | attempt limit reached (five; V1-2.3 implements five, for the owner's confirmation) | `verification_failed`, outcome `unavailable`, last code | no | submit again later |
|
||||
| Core change | release that changes rules or reader | rounds withdrawn by the release migration | no | submit again |
|
||||
|
||||
- Withdrawal is always available: the submitter or any editor withdraws a
|
||||
@@ -1695,7 +1705,7 @@ of the document's JSON text in one form, the same for every layer:
|
||||
|---|---|---|---|
|
||||
| M4 `sheet_access_hardening` | V1-2.1 | `anatomical_catalog_grants` and its RPCs; `has_mfa`, `grant_active`, `caller_has`; `can_read_official_catalog` and the reader row predicate; `can_edit_sheet` redefined (F14); AR-1 B on the platform role RPCs and the private write functions (F19, §5.7), F17, F18; F1, F2, F3 (fork closed), F5, F6, F8, F9, F12 (every catalogue table), F13, F15; asset-row immutability trigger (F16); historical-sheet protection trigger (§14.2); reader column grants narrowed; personal creation and visibility changes refused | no row changes |
|
||||
| M5 `sheet_review_workflow` | V1-2.1 | version columns `revision`, `authors`, `source_version_id`, `published_review_id`, `document_fingerprint`; sheet column `working_copy_source_version_id`; revision, author and freeze triggers, the historical-sheet write rule; extended immutability guard (§6.8); rounds, comments, acknowledgments; `create_official_sheet`, `create_sheet_draft` adapted, `save_sheet_draft`, submit, withdraw, comment, acknowledge, decide, publish; `create_sheet_working_copy`; species RPCs; catalogue and version read RPCs; upload: `register_sheet_asset` extended, `confirm_sheet_asset_upload` with the storage check (F7), INSERT policy, bucket limit 2 MiB; mirror removal and status check (§12.2); superseded signatures dropped; the content-size CHECK replaced by the text measure (`octet_length(content::text)`, §12.2) | new columns take their defaults on the equine v1 row (`revision` 0, empty author set); its document, status, number and assets unchanged; the new size CHECK is validated against the existing rows when M5 is applied (the equine document measures 184 105 bytes as `content::text`, measured on the sandbox); bucket limit lowered; no round, no grant |
|
||||
| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only | none |
|
||||
| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only; written as `20261101100000_sheet_verification_jobs.sql`, pgTAP `021`, rehearsed on the sandbox from the state before M4 (M4 to M6 in one `migration up`) and from the state before M6 | none |
|
||||
|
||||
Each migration ships its pgTAP file (allowed and denied cases), regenerated
|
||||
`database.types.ts`, a `db:sandbox up upgrade` run from the DEV state (with a
|
||||
@@ -2154,6 +2164,7 @@ V1-2.7 on.
|
||||
verification time at the caps within budget.
|
||||
- Rollback: branch revert; the worker image exists on the sandbox only.
|
||||
- Checkpoint: owner validates the server workflow (V1-2.1 to V1-2.3).
|
||||
- State 2026-11-01: built and tested on the sandbox (RUST.md §13); DEV untouched.
|
||||
|
||||
### 16.5 V1-2.4 Wireframes (parallel to V1-2.1 to V1-2.3)
|
||||
|
||||
|
||||
Reference in new issue
Block a user