chore(memory): V1-2.3 worker plan

This commit is contained in:
bchanot committed 2026-11-01 16:51:31 +01:00
1 parent 1cd4344363
commit 345dde59e6
1 file changed
+104
@@ -0,0 +1,104 @@
# Plan: V1-2.3 worker verification mode (P2 child `verify`, P3 `run --jobs sheet-verify`)
Contract `.claude/tasks/contracts/2026-11-01-v1-2-3-worker-verification.md` (W2-W5). SQL side
(M6) is `supabase/migrations/20261101100000_sheet_verification_jobs.sql`: read it first, the RPC
names, argument names, return shapes and the closed worker codes are fixed there.
Style: follow `crates/worker` as it stands (plain `main`, no async, closed outcomes, `log`
allowlist, `Backend` trait + fake in tests, `#[allow(unsafe_code)]` never added). Functions
small, one responsibility, comments only for the why. No new dependency beyond `faunosteo-core`
(path dependency, default features: no `pdf-read`, no `curation`).
## P2 child `verify`
1. `crates/worker/Cargo.toml`: `faunosteo-core = { path = "../core" }`. Update `main.rs::version`
and `faunosteo_core_version` to print `faunosteo_core::CORE_VERSION` (the comment saying the
worker links no core becomes false: remove it).
2. `limits.rs`, per command (decode constants unchanged):
- `VERIFY_MAX_INPUT_BYTES` 12 MiB (document 2 MiB + base 2 MiB, SQL CHECK; four drawings,
6 MiB total, `SnapshotCaps::ASSETS*`; header and margin);
- `VERIFY_MAX_HEADER_BYTES` 64 KiB;
- `VERIFY_MAX_OUTPUT_BYTES` 4 MiB (= SQL `sheet_verification_max_verdict_bytes`);
- `VERIFY_ADDRESS_SPACE_BYTES` 1 GiB, `VERIFY_CPU_SECONDS` 60, `VERIFY_DEADLINE` 90 s
(contract budget: verification under 120 s at the caps);
- drawing download policy: at most `SnapshotCaps::ASSETS` drawings fetched, each at most
`SnapshotCaps::ASSET_BYTES`, total at most `SnapshotCaps::ASSETS_TOTAL_BYTES`.
3. `verify_frame.rs` (shared by parent and child): the framed stdin input.
`[u32 LE header length][header JSON][document][base?][drawing bytes in header order]`.
Header: `{sheet_id, version_id, revision, review_id, document_len, base_len|null,
drawings:[{asset_id, sha256, byte_size, bytes_len|null}]}`. `encode(&FrameInput) -> Vec<u8>`
and `decode(&[u8]) -> Result<Frame<'_>, FrameError>` (borrowed slices; header over its cap,
lengths not summing exactly to the rest, document or base not UTF-8 → `FrameError`). Unit
tests: round trip, every malformed case, empty base, missing drawing bytes.
4. `verify.rs`, `pub fn main(args, stdin, stdout, stderr) -> i32` mirroring `decode::main`:
empty environment check; `--no-seccomp` test hook like decode (only under `test-hooks`);
`sandbox::apply(&Limits { VERIFY_ADDRESS_SPACE_BYTES, VERIFY_CPU_SECONDS }, seccomp)`;
read stdin `take(cap + 1)`; over cap → `INPUT_TOO_LARGE`; frame error → new exit code
`FRAME_INVALID` (a parent bug: Broken, not content); build
`faunosteo_core::domain::sheet::verdict::VerdictInput` and run `compute_verdict`;
`serde_json::to_vec(&verdict)` to stdout. A failing verdict is still exit 0 (it is content,
judged by the core).
5. `exit_codes.rs`: add `FRAME_INVALID` (next free number), its name; NOT a content rejection.
6. `main.rs`: `Some("verify") => verify::main(...)`; usage line updated.
7. `spawn.rs`: the spawner takes the command and its limits: add a `ChildCommand` (or fields)
so `Spawner::for_decode(exe)` keeps today's values and `Spawner::for_verify(exe)` uses
`VERIFY_DEADLINE`, `VERIFY_MAX_OUTPUT_BYTES`, subcommand `verify`. Output classification per
command: decode keeps `webp_check`; verify exit 0 → `Produced(bytes)` only if the bytes parse
as a JSON object (`serde_json::from_slice::<serde_json::Map<..>>`), else `Failed`. Keep the
decode path byte-for-byte equivalent in behaviour (existing tests must pass unchanged).
8. Sandbox / seccomp: the verify child must run under the same filter
(`sandbox::ALLOWED_SYSCALLS`). Extend `crates/worker/scripts/derive-seccomp.sh` to also trace
`verify --no-seccomp` on a test-hooks fixture (add `fixture verify-clean` and
`fixture verify-caps` to `fixtures.rs`: framed inputs built from a small valid document +
drawing, and the at-caps document of `crates/core/tests/sheet_verdict.rs::at_caps`); if a
syscall is missing from the allowlist, add it ONLY if the trace proves it and say so in the
report (do not guess).
9. `tests/sandbox.rs` (Docker test stage): verify child exits 0 on the clean fixture with a
passed verdict, refuses a non-empty environment, kills on over-cap input, and a seccomp
violation still kills (reuse the decode patterns).
## P3 mode `run --jobs sheet-verify`
10. `run.rs::main`: accept `--jobs sheet-verify` (and keep `run` / `run --check` unchanged:
default mode = avatars + sweep, exactly as today). Verification mode: no sweep, its own loop
`verify_loop(config, backend, verifier, watchdog)`; `--check` may combine (`run --jobs
sheet-verify --check`) and prints the mode.
11. `backend.rs`: new trait `VerifyBackend` (keep `Backend` untouched): `claim_verifications(max)
-> Vec<VerifyJob>`, `verification_input(review_id, token) -> Option<VerifyInput>`,
`download_drawing(bucket, key, max_bytes) -> Vec<u8>` (404 → `Status(404)`; over max →
`Overflow`), `complete_verification(review_id, token, verdict_json) -> CompleteAnswer`
(`passed|failed|stale|malformed`), `fail_verification(review_id, token, code) -> FailAnswer`
(`retry|unavailable|stale`). Structs deserialize the M6 shapes exactly.
12. `http.rs`: implement it on `HttpBackend` (same client, auth headers, error mapping). Body
caps per call: input RPC response 16 MiB (two 2 MiB documents escaped in JSON can double),
other RPCs keep `MAX_RPC_BODY`; complete RPC request body up to `VERIFY_MAX_OUTPUT_BYTES` +
small envelope; drawing download capped at `max_bytes + 1`.
13. `verify_job.rs`: one job: validate ids are UUIDs (`keys::is_uuid`), read the input (None →
log `verify_stale_input`, end); drawings: bucket must be `anatomical-sheet-assets`, key must
be `<64 hex>.svg` equal to `sha256 + ".svg"` (else that drawing is passed without bytes);
rows not `uploaded`, over the per-drawing cap, past the count or total cap → passed without
bytes (the core reports missing / caps: a business failure, never a worker failure);
404 or `Overflow` → without bytes, logged; transport error → `fail(verification.worker.unavailable)`. Build the frame, spawn,
classify: `Produced` → complete; complete `malformed` → `fail(verification.worker.verdict_refused)`;
`stale` → log, end. Child `Timeout` → `fail(timeout)`; `Signalled` (SIGXCPU, SIGKILL,
SIGABRT, SIGSYS) and `OutputOverflow` and `Rejected` (input too large) →
`fail(resource_limit)`; `Broken` / `Failed` → `fail(unavailable)` then stop the loop (exit
`CHILD_BROKEN`), as the avatar loop does. Every log line through `log` with the allowlist
(add `review` field if needed, never document text, never child stderr raw).
14. `tests/verify_loop.rs` (fake `VerifyBackend`, real child binary like `tests/job_loop.rs`
does): passed verdict completes; failed verdict (unreadable document) completes `failed`;
stale input; complete answers stale/malformed; drawing 404 → verdict failed with
`drawing_missing` (not a worker failure); transport error → `fail(unavailable)`; timeout
(test-hooks hang like decode, if present; else a deadline of 0 in a test spawner) →
`fail(timeout)`; output overflow → `fail(resource_limit)`; broken child → stop.
15. `docker-compose.yml`: service `worker-verify` under profile `worker`, same image, same
hardening as `worker` (read-only, `cap_drop: ALL`, no-new-privileges, user, tmpfs if any,
network), `command: ["run", "--jobs", "sheet-verify"]`, same `.env.worker`. Do not start it.
## Gates the executor runs
`pnpm rust:check` (fmt, clippy -D warnings, cargo test core + wasm + worker unit, ts-rs, deny);
`cargo test -p faunosteo-worker --features test-hooks` natively for the non-sandbox tests. Docker
stages (`pnpm worker:test`, `pnpm worker:image`) are run by the orchestrator, not the executor.
Never start a worker against any stack. Never touch `supabase/`, `apps/`, `packages/`,
`crates/core/src` (read only), DEV or production. No commit.