chore(memory): V1-2.3 worker plan
This commit is contained in:
1 parent
1cd4344363
commit
345dde59e6
1 file changed
+104
@@ -0,0 +1,104 @@
|
||||
# Plan: V1-2.3 worker verification mode (P2 child `verify`, P3 `run --jobs sheet-verify`)
|
||||
|
||||
Contract `.claude/tasks/contracts/2026-11-01-v1-2-3-worker-verification.md` (W2-W5). SQL side
|
||||
(M6) is `supabase/migrations/20261101100000_sheet_verification_jobs.sql`: read it first, the RPC
|
||||
names, argument names, return shapes and the closed worker codes are fixed there.
|
||||
|
||||
Style: follow `crates/worker` as it stands (plain `main`, no async, closed outcomes, `log`
|
||||
allowlist, `Backend` trait + fake in tests, `#[allow(unsafe_code)]` never added). Functions
|
||||
small, one responsibility, comments only for the why. No new dependency beyond `faunosteo-core`
|
||||
(path dependency, default features: no `pdf-read`, no `curation`).
|
||||
|
||||
## P2 child `verify`
|
||||
|
||||
1. `crates/worker/Cargo.toml`: `faunosteo-core = { path = "../core" }`. Update `main.rs::version`
|
||||
and `faunosteo_core_version` to print `faunosteo_core::CORE_VERSION` (the comment saying the
|
||||
worker links no core becomes false: remove it).
|
||||
2. `limits.rs`, per command (decode constants unchanged):
|
||||
- `VERIFY_MAX_INPUT_BYTES` 12 MiB (document 2 MiB + base 2 MiB, SQL CHECK; four drawings,
|
||||
6 MiB total, `SnapshotCaps::ASSETS*`; header and margin);
|
||||
- `VERIFY_MAX_HEADER_BYTES` 64 KiB;
|
||||
- `VERIFY_MAX_OUTPUT_BYTES` 4 MiB (= SQL `sheet_verification_max_verdict_bytes`);
|
||||
- `VERIFY_ADDRESS_SPACE_BYTES` 1 GiB, `VERIFY_CPU_SECONDS` 60, `VERIFY_DEADLINE` 90 s
|
||||
(contract budget: verification under 120 s at the caps);
|
||||
- drawing download policy: at most `SnapshotCaps::ASSETS` drawings fetched, each at most
|
||||
`SnapshotCaps::ASSET_BYTES`, total at most `SnapshotCaps::ASSETS_TOTAL_BYTES`.
|
||||
3. `verify_frame.rs` (shared by parent and child): the framed stdin input.
|
||||
`[u32 LE header length][header JSON][document][base?][drawing bytes in header order]`.
|
||||
Header: `{sheet_id, version_id, revision, review_id, document_len, base_len|null,
|
||||
drawings:[{asset_id, sha256, byte_size, bytes_len|null}]}`. `encode(&FrameInput) -> Vec<u8>`
|
||||
and `decode(&[u8]) -> Result<Frame<'_>, FrameError>` (borrowed slices; header over its cap,
|
||||
lengths not summing exactly to the rest, document or base not UTF-8 → `FrameError`). Unit
|
||||
tests: round trip, every malformed case, empty base, missing drawing bytes.
|
||||
4. `verify.rs`, `pub fn main(args, stdin, stdout, stderr) -> i32` mirroring `decode::main`:
|
||||
empty environment check; `--no-seccomp` test hook like decode (only under `test-hooks`);
|
||||
`sandbox::apply(&Limits { VERIFY_ADDRESS_SPACE_BYTES, VERIFY_CPU_SECONDS }, seccomp)`;
|
||||
read stdin `take(cap + 1)`; over cap → `INPUT_TOO_LARGE`; frame error → new exit code
|
||||
`FRAME_INVALID` (a parent bug: Broken, not content); build
|
||||
`faunosteo_core::domain::sheet::verdict::VerdictInput` and run `compute_verdict`;
|
||||
`serde_json::to_vec(&verdict)` to stdout. A failing verdict is still exit 0 (it is content,
|
||||
judged by the core).
|
||||
5. `exit_codes.rs`: add `FRAME_INVALID` (next free number), its name; NOT a content rejection.
|
||||
6. `main.rs`: `Some("verify") => verify::main(...)`; usage line updated.
|
||||
7. `spawn.rs`: the spawner takes the command and its limits: add a `ChildCommand` (or fields)
|
||||
so `Spawner::for_decode(exe)` keeps today's values and `Spawner::for_verify(exe)` uses
|
||||
`VERIFY_DEADLINE`, `VERIFY_MAX_OUTPUT_BYTES`, subcommand `verify`. Output classification per
|
||||
command: decode keeps `webp_check`; verify exit 0 → `Produced(bytes)` only if the bytes parse
|
||||
as a JSON object (`serde_json::from_slice::<serde_json::Map<..>>`), else `Failed`. Keep the
|
||||
decode path byte-for-byte equivalent in behaviour (existing tests must pass unchanged).
|
||||
8. Sandbox / seccomp: the verify child must run under the same filter
|
||||
(`sandbox::ALLOWED_SYSCALLS`). Extend `crates/worker/scripts/derive-seccomp.sh` to also trace
|
||||
`verify --no-seccomp` on a test-hooks fixture (add `fixture verify-clean` and
|
||||
`fixture verify-caps` to `fixtures.rs`: framed inputs built from a small valid document +
|
||||
drawing, and the at-caps document of `crates/core/tests/sheet_verdict.rs::at_caps`); if a
|
||||
syscall is missing from the allowlist, add it ONLY if the trace proves it and say so in the
|
||||
report (do not guess).
|
||||
9. `tests/sandbox.rs` (Docker test stage): verify child exits 0 on the clean fixture with a
|
||||
passed verdict, refuses a non-empty environment, kills on over-cap input, and a seccomp
|
||||
violation still kills (reuse the decode patterns).
|
||||
|
||||
## P3 mode `run --jobs sheet-verify`
|
||||
|
||||
10. `run.rs::main`: accept `--jobs sheet-verify` (and keep `run` / `run --check` unchanged:
|
||||
default mode = avatars + sweep, exactly as today). Verification mode: no sweep, its own loop
|
||||
`verify_loop(config, backend, verifier, watchdog)`; `--check` may combine (`run --jobs
|
||||
sheet-verify --check`) and prints the mode.
|
||||
11. `backend.rs`: new trait `VerifyBackend` (keep `Backend` untouched): `claim_verifications(max)
|
||||
-> Vec<VerifyJob>`, `verification_input(review_id, token) -> Option<VerifyInput>`,
|
||||
`download_drawing(bucket, key, max_bytes) -> Vec<u8>` (404 → `Status(404)`; over max →
|
||||
`Overflow`), `complete_verification(review_id, token, verdict_json) -> CompleteAnswer`
|
||||
(`passed|failed|stale|malformed`), `fail_verification(review_id, token, code) -> FailAnswer`
|
||||
(`retry|unavailable|stale`). Structs deserialize the M6 shapes exactly.
|
||||
12. `http.rs`: implement it on `HttpBackend` (same client, auth headers, error mapping). Body
|
||||
caps per call: input RPC response 16 MiB (two 2 MiB documents escaped in JSON can double),
|
||||
other RPCs keep `MAX_RPC_BODY`; complete RPC request body up to `VERIFY_MAX_OUTPUT_BYTES` +
|
||||
small envelope; drawing download capped at `max_bytes + 1`.
|
||||
13. `verify_job.rs`: one job: validate ids are UUIDs (`keys::is_uuid`), read the input (None →
|
||||
log `verify_stale_input`, end); drawings: bucket must be `anatomical-sheet-assets`, key must
|
||||
be `<64 hex>.svg` equal to `sha256 + ".svg"` (else that drawing is passed without bytes);
|
||||
rows not `uploaded`, over the per-drawing cap, past the count or total cap → passed without
|
||||
bytes (the core reports missing / caps: a business failure, never a worker failure);
|
||||
404 or `Overflow` → without bytes, logged; transport error → `fail(verification.worker.unavailable)`. Build the frame, spawn,
|
||||
classify: `Produced` → complete; complete `malformed` → `fail(verification.worker.verdict_refused)`;
|
||||
`stale` → log, end. Child `Timeout` → `fail(timeout)`; `Signalled` (SIGXCPU, SIGKILL,
|
||||
SIGABRT, SIGSYS) and `OutputOverflow` and `Rejected` (input too large) →
|
||||
`fail(resource_limit)`; `Broken` / `Failed` → `fail(unavailable)` then stop the loop (exit
|
||||
`CHILD_BROKEN`), as the avatar loop does. Every log line through `log` with the allowlist
|
||||
(add `review` field if needed, never document text, never child stderr raw).
|
||||
14. `tests/verify_loop.rs` (fake `VerifyBackend`, real child binary like `tests/job_loop.rs`
|
||||
does): passed verdict completes; failed verdict (unreadable document) completes `failed`;
|
||||
stale input; complete answers stale/malformed; drawing 404 → verdict failed with
|
||||
`drawing_missing` (not a worker failure); transport error → `fail(unavailable)`; timeout
|
||||
(test-hooks hang like decode, if present; else a deadline of 0 in a test spawner) →
|
||||
`fail(timeout)`; output overflow → `fail(resource_limit)`; broken child → stop.
|
||||
15. `docker-compose.yml`: service `worker-verify` under profile `worker`, same image, same
|
||||
hardening as `worker` (read-only, `cap_drop: ALL`, no-new-privileges, user, tmpfs if any,
|
||||
network), `command: ["run", "--jobs", "sheet-verify"]`, same `.env.worker`. Do not start it.
|
||||
|
||||
## Gates the executor runs
|
||||
|
||||
`pnpm rust:check` (fmt, clippy -D warnings, cargo test core + wasm + worker unit, ts-rs, deny);
|
||||
`cargo test -p faunosteo-worker --features test-hooks` natively for the non-sandbox tests. Docker
|
||||
stages (`pnpm worker:test`, `pnpm worker:image`) are run by the orchestrator, not the executor.
|
||||
Never start a worker against any stack. Never touch `supabase/`, `apps/`, `packages/`,
|
||||
`crates/core/src` (read only), DEV or production. No commit.
|
||||
Reference in new issue
Block a user