From 5adad3709f89777dcab615fa77dcaeb3e0fcc8c1 Mon Sep 17 00:00:00 2001 From: bchanot Date: Sun, 1 Nov 2026 21:09:00 +0100 Subject: [PATCH] feat(supabase): retry delay of 2, 5, 10, 20 s for technical verification failures M6 amended in place (applied only to the throwaway sandbox): a hidden not_before on the round, set by fail_sheet_verification and honoured by the claim; RPC signatures unchanged. pgTAP 021 (68), sandbox cases i (delay with the real worker), j (inputs at the caps, 2 MiB base), k (just beyond). --- docs/development/RUST.md | 35 ++- docs/development/STORAGE.md | 11 +- docs/development/V1-2_ANATOMICAL_SHEETS.md | 35 ++- packages/supabase/src/database.types.ts | 3 + .../supabase/tests/db/schema-types.test.ts | 1 + .../sandbox/sheet-verification.sandbox.ts | 249 ++++++++++++++++++ ...20261101100000_sheet_verification_jobs.sql | 35 ++- .../tests/021_sheet_verification_jobs.sql | 58 +++- 8 files changed, 398 insertions(+), 29 deletions(-) diff --git a/docs/development/RUST.md b/docs/development/RUST.md index 6e8a9e0e..e9443fd7 100644 --- a/docs/development/RUST.md +++ b/docs/development/RUST.md @@ -175,7 +175,8 @@ cmake on Alpine). The decode child's traced syscall set did not change with the parent's TLS code (§5.4 re-run at R1: the same seven). Image 11.6 MB (8.7 MB at R0: ring + rustls). Tests: `crates/worker/tests/job_loop.rs` (11, fake backend + fake decoder) and the module unit tests run natively; the -real-stack end-to-end runs through `pnpm e2e:avatar` (`STORAGE.md` §6). +real-stack end-to-end runs through `pnpm db:sandbox avatar-e2e` on the throwaway stack since V1-2.3 +(`pnpm e2e:avatar` on DEV, the owner's call; `STORAGE.md` §6). The sandboxed child, unchanged from R0: @@ -4501,8 +4502,7 @@ Reviews (fresh, 2026-11-01, on `7043d39`): password literal in `scripts/db-sandbox.sh`). LOWs fixed: the verdict cap measured twice (child output now 3 MiB, under the 4 MiB SQL measure of `jsonb::text`); the child's 1 GiB address space above the container's 512 MiB (`worker-verify` and the sandbox worker now 1 280 MiB, so the child's limit binds first). LOWs open: no - backoff between attempts (the §6.4 design « without backoff »: an outage of a few seconds can spend the five - attempts, the round closing `unavailable`; fail-closed, availability only); a document naming several hundred + backoff between attempts (closed afterwards by the owner decision of 2026-11-01: SQL delay of 2, 5, 10, 20 s, V1-2 contract §6.4; the finding as written on `7043d39` is left); a document naming several hundred registered drawings overflows the 64 KiB frame header and is reported `resource_limit` instead of the core's `caps` (wrong exit message, no security effect); SQL interpolation of generated ids in the sandbox test only. - Verifier: ECARTS(2). W1-W5, W7 met. W6: the hostile corpus through the child was missing (added, above); `pnpm @@ -4525,3 +4525,32 @@ WASM 580 522 B gzip in the app image, unchanged by V1-2.3); `pnpm e2e:shell` 86 Second reviews on `6aa85fe`: verifier ECARTS(1), the one item owner-gated (`pnpm e2e:avatar`, a worker on DEV), the hostile corpus met; security PASS (MEDIUM pre-existing sandbox password literal; LOWs: the 3 MiB / 4 MiB relation is empirical, a refused verdict ending fail-closed `verdict_refused`; DEPLOY.md memory note, fixed). + +### 13.1 Reserves Q3 and Q4: inputs at the caps, `brk` bounded (2026-11-01) + +Contract `.claude/tasks/contracts/2026-11-01-v1-2-3-reserves.md`. Fixtures (`fixtures.rs`, `test-hooks`): `verify_max_frame()`, +also `fixture verify-max`: document and base of exactly 2 097 152 bytes (the `jsonb::text` spelling, `": "` and `", "`, +filler in `attribution.notes`; the core measures `document_bytes` 2 097 152), four drawings of 2 097 152, 2 097 152, +1 048 576 and 1 048 576 bytes (6 291 456 in all), a frame of 10 486 658 bytes. A drawing is the stored prolog, a 10 × 10 +root, 10 000 `` and a last path whose `d` ends in as many spaces as the size needs. Unit tests run the +core on each fixture: the maximum is `passed`; four drawings of 2 MiB (no base), five drawings, one drawing of 2 097 153 +bytes and a document of 2 097 153 bytes are `failed` with `verification.validation.caps` (content, exit 0). + +Through the full sandbox (`tests/sandbox.rs`, now 31 tests): `the_maximum_input_is_verified_within_the_limits` (`passed`, +verdict 2 531 bytes, under the 3 MiB cap and the 90 s deadline), `inputs_just_beyond_the_caps_fail_as_caps_through_the_sandbox`, +`the_verify_input_cap_is_exact` (a frame of exactly 12 MiB is read, here refused by the frame check; one byte more is refused +unread by the existing test). Measured on the development host, release build, glibc, sandbox applied: the maximum 104 ms, +the sheet at every other cap (2 118 765 bytes in, 24 689 out) 142 ms. The Docker `test` stage passes the same tests; its +log does not print the timings (captured output). + +`brk` in the verify child (`tests/sandbox.rs`, 1 GiB address space): `--alloc-gib 2` and the new hook +`--alloc-small-gib ` (4 KiB chunks kept alive, the small-allocation path) both end in SIGABRT and the survival +message never appears; `--mmap-exec` and `--signal-parent` die with SIGSYS; `--violate` was already covered. The CPU and +deadline hooks stay on the decode child. No syscall added to `ALLOWED_SYSCALLS`. + +Derivation: `derive-seccomp.sh` also traces `fixture verify-max` through `verify --no-seccomp`; the traced set is +unchanged (`brk exit_group getrandom mmap mremap munmap read write`), the allowlist covers it. + +The two misclassifications stay tracked, not fixed (`tests/verify_loop.rs`): a frame header over 64 KiB is reported +`resource_limit`; an object over the policy is passed without bytes and the verdict lists `caps` then `drawing_missing` +(its declared size already exceeds the cap), never `size_mismatch`. diff --git a/docs/development/STORAGE.md b/docs/development/STORAGE.md index e873dca5..0ef7c77a 100644 --- a/docs/development/STORAGE.md +++ b/docs/development/STORAGE.md @@ -267,6 +267,14 @@ unchanged. worker stops (as the avatar loop). A dead lease (10 minutes) is claimed again as `verification.worker.lease_expired`; the fifth attempt closes the round `verification_failed` / `unavailable`. +- Retry delay (owner decision 2026-11-01): a failure reported by + `fail_sheet_verification` makes the round claimable again after 2 s, 5 s, + 10 s, 20 s (after attempts 1 to 4), kept in SQL (`not_before`, hidden from + the API roles), so the worker never sleeps for it and the claim of another + round is not blocked; the effective delay is that delay plus up to one poll + interval (`WORKER_POLL_SECONDS`, 2 s). A verdict (failed or not) is never + retried; a dead lease is claimed again at once. The worker code is the + same as before. - Measured (2026-11-01): the at-caps document (64 views, 1 024 annotations, 8 192 nodes) verified in 185 ms by the worker on the sandbox, verdict 24 689 bytes; the equine draft in about 60 ms. @@ -369,7 +377,8 @@ docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges:true -e SUPABASE_SERVICE_ROLE_KEY="$(pnpm -s supabase status -o env | sed -n 's/^SERVICE_ROLE_KEY="\(.*\)"/\1/p')" \ faunosteo_worker:dev run pnpm --filter @faunosteo/supabase test:e2e # real-stack end-to-end (needs the worker above) -pnpm e2e:avatar # scripts/avatar-e2e.sh: the three lines above in one, worker started with a 5 s sweep and no quarantine grace +pnpm e2e:avatar # scripts/avatar-e2e.sh: the three lines above in one, worker started with a 5 s sweep and no quarantine grace (DEV: the owner's call) +pnpm db:sandbox avatar-e2e # the same suite and worker on the throwaway stack (SUPABASE.md §5.1): the check of a storage or worker change ``` `docker compose --profile worker up worker` runs the same container from diff --git a/docs/development/V1-2_ANATOMICAL_SHEETS.md b/docs/development/V1-2_ANATOMICAL_SHEETS.md index 29241201..061afa0d 100644 --- a/docs/development/V1-2_ANATOMICAL_SHEETS.md +++ b/docs/development/V1-2_ANATOMICAL_SHEETS.md @@ -871,12 +871,13 @@ fingerprint or a document hash from a page. Job. The round row is the job, with the avatar lease pattern of `20261013100000_files_foundation.sql` (lease token, `locked_at`, fixed lease, -attempts below a limit), without backoff. Submission sets the round -`verifying`. Worker RPCs, service role only (M6): +attempts below a limit), plus a progressive delay for technical retries only +(owner decision 2026-11-01). Submission sets the round `verifying`. Worker RPCs, service role only (M6): - `claim_sheet_verifications(limit)`: `for update skip locked` over rounds - `verifying` with no live lease; sets the token and `locked_at`, increments - `attempts`; returns round, version, sheet, revision and token. + `verifying` with no live lease and no pending delay (`not_before` null or + reached); sets the token and `locked_at`, increments `attempts`, clears + `not_before`; returns round, version, sheet, revision and token. - `get_sheet_verification_input(review_id, lease_token)`: the document at the round's revision (frozen) and the base document, both as text; the drawings the document references among the rows registered for the sheet (asset id, @@ -889,8 +890,17 @@ attempts below a limit), without backoff. Submission sets the round else answers `stale` or `malformed` and writes nothing. Moves the round to `open` or to `verification_failed` (§6.5) and audits. - `fail_sheet_verification(review_id, lease_token, code)`: a technical - failure; the round stays claimable until the attempt limit, then becomes - `verification_failed` with outcome `unavailable`. + failure; the round stays `verifying` and becomes claimable after a delay + that grows with the attempt (2 s after attempt 1, 5 s after 2, 10 s after 3, + 20 s after 4; `private.sheet_verification_retry_delay`), kept in the round's + `not_before` (a column the API roles cannot read, set here, cleared by the + claim); the fifth failure becomes `verification_failed` with outcome + `unavailable` and clears it. Nothing is written for a stale call. The delay + is persisted, so no worker sleeps and any worker honours it; the effective + wait is the delay plus up to one poll interval of the worker (2 s by + default). Only these technical failures wait: a validation or integrity + refusal is a verdict and is never retried, and a dead lease is claimed again + at once (the 10-minute lease is its wait). Processing. Verification runs as its own mode of the worker image (`run --jobs sheet-verify`), in its own container under the `worker` compose @@ -951,9 +961,9 @@ did not pass; the closed codes keep the categories apart. |---|---|---|---|---| | Business validation | integrity error, caps exceeded, annotation without geometry, annotation outside its view's frame | `verification_failed`, outcome `failed`, codes `verification.validation.*` and the issues | no | fix the draft, submit again | | Integrity | strict parse refused, drawing missing, size or sha256 mismatch, SVG refused, needing a removal or outside the stored form (§9.2), `object_conflict` | `verification_failed`, outcome `failed`, codes `verification.integrity.*` | no | fix or upload again; `object_conflict` follows §9.3 | -| Transient worker error | database or Storage unreachable, worker restart | stays `verifying`, code recorded | yes, up to the limit | automatic | -| Timeout or resource limit | child over its wall-clock or memory limit | stays `verifying`, code recorded | yes, up to the limit | reduce the document or the drawing if it repeats | -| Interruption | worker crash, lease expired | claimable again; counts as an attempt | yes, up to the limit | automatic | +| Transient worker error | database or Storage unreachable, worker restart | stays `verifying`, code recorded | yes, after 2 s, 5 s, 10 s, 20 s, up to the limit | automatic | +| Timeout or resource limit | child over its wall-clock or memory limit | stays `verifying`, code recorded | yes, same delays, up to the limit | reduce the document or the drawing if it repeats | +| Interruption | worker crash, lease expired | claimable again at once (the lease was the wait); counts as an attempt | yes, up to the limit | automatic | | Stale result | completion with an old token, after a withdrawal, or naming another revision | nothing written, logged | no | none needed | | Definitive failure | attempt limit reached (five; V1-2.3 implements five, for the owner's confirmation) | `verification_failed`, outcome `unavailable`, last code | no | submit again later | | Core change | release that changes rules or reader | rounds withdrawn by the release migration | no | submit again | @@ -1705,7 +1715,7 @@ of the document's JSON text in one form, the same for every layer: |---|---|---|---| | M4 `sheet_access_hardening` | V1-2.1 | `anatomical_catalog_grants` and its RPCs; `has_mfa`, `grant_active`, `caller_has`; `can_read_official_catalog` and the reader row predicate; `can_edit_sheet` redefined (F14); AR-1 B on the platform role RPCs and the private write functions (F19, §5.7), F17, F18; F1, F2, F3 (fork closed), F5, F6, F8, F9, F12 (every catalogue table), F13, F15; asset-row immutability trigger (F16); historical-sheet protection trigger (§14.2); reader column grants narrowed; personal creation and visibility changes refused | no row changes | | M5 `sheet_review_workflow` | V1-2.1 | version columns `revision`, `authors`, `source_version_id`, `published_review_id`, `document_fingerprint`; sheet column `working_copy_source_version_id`; revision, author and freeze triggers, the historical-sheet write rule; extended immutability guard (§6.8); rounds, comments, acknowledgments; `create_official_sheet`, `create_sheet_draft` adapted, `save_sheet_draft`, submit, withdraw, comment, acknowledge, decide, publish; `create_sheet_working_copy`; species RPCs; catalogue and version read RPCs; upload: `register_sheet_asset` extended, `confirm_sheet_asset_upload` with the storage check (F7), INSERT policy, bucket limit 2 MiB; mirror removal and status check (§12.2); superseded signatures dropped; the content-size CHECK replaced by the text measure (`octet_length(content::text)`, §12.2) | new columns take their defaults on the equine v1 row (`revision` 0, empty author set); its document, status, number and assets unchanged; the new size CHECK is validated against the existing rows when M5 is applied (the equine document measures 184 105 bytes as `content::text`, measured on the sandbox); bucket limit lowered; no round, no grant | -| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only; written as `20261101100000_sheet_verification_jobs.sql`, pgTAP `021`, rehearsed on the sandbox from the state before M4 (M4 to M6 in one `migration up`) and from the state before M6 | none | +| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only, with the retry delay (hidden `not_before` column on the round, `private.sheet_verification_retry_delay`: owner decision 2026-11-01, replaces « without backoff »); written as `20261101100000_sheet_verification_jobs.sql`, pgTAP `021`, rehearsed on the sandbox from the state before M4 (M4 to M6 in one `migration up`) and from the state before M6 | none | Each migration ships its pgTAP file (allowed and denied cases), regenerated `database.types.ts`, a `db:sandbox up upgrade` run from the DEV state (with a @@ -2021,8 +2031,9 @@ The final approval needs all 30 groups acknowledged, no `to_confirm` and no - Worker: the verification mode, verdicts accepted and each failure, the hostile SVG corpus in the sandboxed child, per-command limits and watchdog, the purge guard by bucket, sandbox tests and seccomp derivation, an e2e on - the sandbox stack with the verification worker running; `pnpm e2e:avatar` - unchanged in outcome; `pnpm e2e:sheet-assets` targets DEV only, its rerun + the sandbox stack with the verification worker running; the avatar suite + unchanged in outcome on the sandbox (`pnpm db:sandbox avatar-e2e`, owner + decision 2026-11-01: `pnpm e2e:avatar` targets DEV and is not run); `pnpm e2e:sheet-assets` targets DEV only, its rerun is an owner step (§16.11). - Release replay: a script loads every published version and drawing of a stack through the new core and fails on any regression. diff --git a/packages/supabase/src/database.types.ts b/packages/supabase/src/database.types.ts index a1aca73c..ac6a44a0 100644 --- a/packages/supabase/src/database.types.ts +++ b/packages/supabase/src/database.types.ts @@ -227,6 +227,7 @@ export type Database = { id: string lease_token: string | null locked_at: string | null + not_before: string | null revision: number round: number status: Database['public']['Enums']['sheet_round_status'] @@ -256,6 +257,7 @@ export type Database = { id?: string lease_token?: string | null locked_at?: string | null + not_before?: string | null revision: number round: number status?: Database['public']['Enums']['sheet_round_status'] @@ -284,6 +286,7 @@ export type Database = { id?: string lease_token?: string | null locked_at?: string | null + not_before?: string | null revision?: number round?: number status?: Database['public']['Enums']['sheet_round_status'] diff --git a/packages/supabase/tests/db/schema-types.test.ts b/packages/supabase/tests/db/schema-types.test.ts index 9732444e..18edfa3c 100644 --- a/packages/supabase/tests/db/schema-types.test.ts +++ b/packages/supabase/tests/db/schema-types.test.ts @@ -100,6 +100,7 @@ const expectedColumns = { 'id', 'lease_token', 'locked_at', + 'not_before', 'revision', 'round', 'status', diff --git a/packages/supabase/tests/sandbox/sheet-verification.sandbox.ts b/packages/supabase/tests/sandbox/sheet-verification.sandbox.ts index d451eb48..9af9cb61 100644 --- a/packages/supabase/tests/sandbox/sheet-verification.sandbox.ts +++ b/packages/supabase/tests/sandbox/sheet-verification.sandbox.ts @@ -229,6 +229,112 @@ const avatarsAndPurges = () => || ' ' || (select count(*) || ':' || md5(coalesce(string_agg(row_to_json(q)::text, ',' order by q.id), '')) from private.object_purge_queue q)`) +const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/ +/** An id read back from the stack, checked before it is interpolated into SQL. */ +function uuidOf(value: unknown): string { + if (typeof value !== 'string' || !UUID.test(value)) throw new Error(`not a uuid: ${String(value)}`) + return value +} + +/** `octet_length(value::jsonb::text)` of a value whose strings need no JSON escape (crates/core/src/domain/sheet/size.rs). */ +function jsonbBytes(value: unknown): number { + if (value === null) return 4 + if (typeof value === 'boolean') return value ? 4 : 5 + if (typeof value === 'number') return String(value).length + if (typeof value === 'string') return Buffer.byteLength(value) + 2 + if (Array.isArray(value)) return 2 + value.reduce((n: number, v) => n + jsonbBytes(v), 0) + 2 * Math.max(0, value.length - 1) + const entries = Object.entries(value as Record) + return 2 + entries.reduce((n, [k, v]) => n + jsonbBytes(k) + 2 + jsonbBytes(v), 0) + 2 * Math.max(0, entries.length - 1) +} + +/** + * A drawing of exactly `size` bytes in the stored form (`serialize::stored_form`): the UTF-8 + * prolog and one `` holding one `` whose `d` is padded with `L5 5` segments and + * spaces. Canonical because it is what the core writes back byte for byte: ASCII only, the + * namespace declared on the root, attributes in a fixed order, every value free of `& < > "` + * and of tab or line feed, no comment, no text node, the element self-closed; the pad adds + * no element, attribute, `<` or `=` (SVG_CAPS: 20 000 elements, 100 000 `<`, 200 000 `=`). + */ +function sizedDrawing(size: number, tag: string): Buffer { + const head = '\n' + + `' + const pad = size - head.length - tail.length + const text = head + ' L5 5'.repeat(Math.floor(pad / 5)) + ' '.repeat(pad % 5) + tail + expect(text.length).toBe(size) + return Buffer.from(text, 'ascii') +} + +/** `main` on the first drawing, one detail view per other drawing, each opened by a trigger on `main`. */ +function multiDoc(assets: string[], descriptionFr = 'd') { + const doc = smallDoc(assets[0]) as Record + doc.description = { fr: descriptionFr, en: 'd' } + const views = doc.views as unknown[] + const annotations = doc.annotations as unknown[] + assets.slice(1).forEach((asset, i) => { + views.push({ id: `d${i}`, kind: 'detail', name: name('D'), svg_asset_id: asset, view_box: null, parent_view_id: 'main' }) + annotations.push({ id: `t${i}`, kind: 'detail_trigger', view_id: 'main', category_id: 'c', laterality: 'unpaired', + geometry: { type: 'polygon', points: [[0, 0], [1, 0], [1, 1]] }, review_status: 'reviewed', name: null, + opens_view_id: `d${i}` }) + }) + return doc +} + +/** `multiDoc` whose `octet_length(content::text)` is exactly `bytes`; `fill` makes two such documents differ. */ +function documentOfBytes(assets: string[], bytes: number, fill = 'x'): unknown { + const missing = bytes - jsonbBytes(multiDoc(assets)) + expect(missing).toBeGreaterThanOrEqual(0) + return multiDoc(assets, 'd' + fill.repeat(missing)) +} + +/** Submits `d` until its round is `verifying` with `attempts` claims already spent and a lease held for another worker (SQL stand-in). */ +async function holdLease(editor: SteppedAccount, d: Draft, attempts: number): Promise<{ review: string; token: string }> { + const token = randomUUID() + for (let turn = 0; turn < 5; turn++) { + const id = uuidOf(await submit(editor, d)) + const held = await sql(`update public.anatomical_sheet_reviews set lease_token = '${token}', locked_at = now(), attempts = ${attempts} + where id = '${id}' and status = 'verifying' and lease_token is null returning id`) + if (held === id) return { review: id, token } + await settled(id) + ok(await rpc('withdraw_sheet_review', { review_id: id, note: 'raced' }, editor.aal2)) + } + throw new Error('lease not held before the worker claimed') +} + +/** Polls a one-value query until it answers `expected`. */ +async function waitFor(query: string, expected: string, timeoutMs: number): Promise { + const start = performance.now() + let last = '' + while (performance.now() - start < timeoutMs) { + last = await sql(query) + if (last === expected) return + await new Promise((resolve) => setTimeout(resolve, 200)) + } + throw new Error(`expected ${expected}, last ${last}`) +} + +/** Registers, uploads and confirms one drawing per size on `sheet`; the ids in order. */ +async function putDrawings(editor: SteppedAccount, sheet: string, sizes: number[]): Promise<{ id: string; bytes: Buffer }[]> { + const out: { id: string; bytes: Buffer }[] = [] + for (const [i, size] of sizes.entries()) { + const bytes = sizedDrawing(size, `${i}${randomUUID().replace(/-/g, '')}`) + out.push({ id: await putDrawing(editor, sheet, bytes), bytes }) + } + return out +} + +/** Acknowledges every group of the open round, approves it and publishes (the control of case d). */ +async function publishRound(editor: SteppedAccount, reviewer: SteppedAccount, review: string, r: Round): Promise { + for (const group of r.verdict?.groups ?? []) + ok(await rpc('acknowledge_sheet_review_group', + { review_id: review, group_key: group.key, group_digest: group.digest }, reviewer.aal2)) + ok(await rpc('decide_sheet_review', { review_id: review, approve: true, note: 'ok' }, reviewer.aal2)) + ok(await rpc('publish_sheet_version', { review_id: review }, editor.aal2)) +} + +const verdictBytes = async (review: string) => + Number(await sql(`select octet_length(verdict::text) from public.anatomical_sheet_reviews where id = '${review}'`)) + const timings: Record = {} describe.skipIf(!WORKER)('sheet verification by the real worker (V1-2.3)', () => { @@ -471,6 +577,149 @@ describe.skipIf(!WORKER)('sheet verification by the real worker (V1-2.3)', () => expect(ms).toBeLessThan(CAPS_BUDGET_MS) }, CAPS_BUDGET_MS + 60_000) + it('i: a technical failure is retried no earlier than its delay, the worker serving another round meanwhile', async () => { + const d = await newSheet(E) + const bytes = storedDrawing() + const asset = await putDrawing(E, d.sheet, bytes) + await save(E, d, smallDoc(asset)) + // Three attempts spent and the lease held for another worker: the failure is the third (10 s delay). + const { review, token } = await holdLease(E, d, 3) + const failed = await rpc('fail_sheet_verification', + { review_id: review, lease_token: token, code: 'verification.worker.timeout' }, stack.serviceKey) + expect(failed.body).toBe('retry') + // not_before = now() of the failing transaction + 10 s: the failure time in SQL's clock. + const failAt = Number(await sql(`select extract(epoch from not_before - interval '10 seconds') + from public.anatomical_sheet_reviews where id = '${review}'`)) + expect(failAt).toBeGreaterThan(0) + + // Meanwhile the worker serves another round: a clean sheet opens. + const other = await newSheet(E) + const otherBytes = storedDrawing() + const otherAsset = await putDrawing(E, other.sheet, otherBytes) + await save(E, other, smallDoc(otherAsset)) + const served = await settled(await submit(E, other)) + expectPassedFor(served.round, other, [{ id: otherAsset, sha: sha256(otherBytes) }]) + // The first round waits: verifying, no lease, three attempts, delay not over. + expect(await sql(`select status || '/' || attempts || '/' || (lease_token is null) || '/' || (clock_timestamp() < not_before) + from public.anatomical_sheet_reviews where id = '${review}'`)).toBe('verifying/3/true/true') + + const retried = await settled(review, 40_000) + expectPassedFor(retried.round, d, [{ id: asset, sha: sha256(bytes) }]) + expect(retried.round.attempts).toBe(4) + const gap = Number(await sql(`select extract(epoch from verified_at) - ${failAt} + from public.anatomical_sheet_reviews where id = '${review}'`)) + expect(gap).toBeGreaterThanOrEqual(10) + timings.i_retry = `claimed and verified ${gap.toFixed(1)} s after the failure (delay 10 s, poll 2 s), other round opened in ${served.ms} ms` + }, 90_000) + + describe('input limits (Q3)', () => { + let L: SteppedAccount + const MIB = 1024 * 1024 + + beforeAll(async () => { + L = await holder('vf-limits', 'admin', ['sheet_editor', 'sheet_publisher']) + }, 60_000) + + it('j: the largest input the policy allows (document 2 MiB, drawings 2+2+1+1 MiB, base 2 MiB) is verified within the budget', async () => { + const d = await newSheet(L) + const drawings = await putDrawings(L, d.sheet, [2 * MIB, 2 * MIB, MIB, MIB]) + const ids = drawings.map((x) => x.id) + const first = documentOfBytes(ids, 2 * MIB, 'x') + await save(L, d, first) + expect(await sql(`select octet_length(content::text) from public.anatomical_sheet_versions where id = '${d.draft}'`)).toBe(String(2 * MIB)) + const review = await submit(L, d) + const { round: r, ms } = await settled(review, CAPS_BUDGET_MS) + // The verdict lists the drawings in document order: both sides are compared sorted by asset id. + const expected = drawings.map((x) => ({ id: x.id, sha: sha256(x.bytes) })).sort((a, b) => a.id.localeCompare(b.id)) + const byAsset = (x: Round): Round => ({ ...x, verdict: { ...(x.verdict as Verdict), + drawings: [...(x.verdict as Verdict).drawings].sort((a, b) => a.asset_id.localeCompare(b.asset_id)) } }) + expectPassedFor(byAsset(r), d, expected) + const v = r.verdict as Verdict + expect(v.groups.map((g) => g.key)).toEqual(expect.arrayContaining(['general', 'view:main', 'view:d0', 'view:d1', 'view:d2'])) + const worker = await workerDuration(review) + const size = await verdictBytes(review) + expect(size).toBeLessThan(3_145_728) + timings.j_at_limits = `${ms} ms from submission (worker ${worker} ms), document ${2 * MIB} bytes, drawings ${drawings.map((x) => x.bytes.length).join('+')} bytes, verdict ${size} bytes` + expect(ms).toBeLessThan(CAPS_BUDGET_MS) + + // Second round: its base is the published 2 MiB document (frame: 2 + 2 + 6 MiB). + await publishRound(L, V, review, r) + const draft = uuidOf(ok(await rpc('create_sheet_draft', { sheet_id: d.sheet, from_version_id: null, change_note: 'next' }, L.aal2))) + const next: Draft = { sheet: d.sheet, draft, revision: Number(await sql(`select revision from public.anatomical_sheet_versions where id = '${draft}'`)) } + expect(await sql(`select (source_version_id = '${d.draft}')::text from public.anatomical_sheet_versions where id = '${draft}'`)).toBe('true') + await save(L, next, documentOfBytes(ids, 2 * MIB, 'y')) + const second = await submit(L, next) + const again = await settled(second, CAPS_BUDGET_MS) + expectPassedFor(byAsset(again.round), next, expected) + const size2 = await verdictBytes(second) + expect(size2).toBeLessThan(3_145_728) + timings.j_with_base = `${again.ms} ms from submission (worker ${await workerDuration(second)} ms), document and base ${2 * MIB} bytes each, drawings 6 MiB, verdict ${size2} bytes` + expect(again.ms).toBeLessThan(CAPS_BUDGET_MS) + }, 3 * CAPS_BUDGET_MS) + + it('k: just beyond the limits: refused before the worker, or a business failure; the tracked misclassification', async () => { + // A document of 2 097 153 bytes is refused at save. + const d = await newSheet(L) + const small = await putDrawing(L, d.sheet, storedDrawing()) + const save1 = await rpc('save_sheet_draft', + { version_id: d.draft, expected_revision: d.revision, document: documentOfBytes([small], 2 * MIB + 1), change_note: 'e2e' }, L.aal2) + expect(message(save1)).toBe('document.too_large') + + // A drawing of 2 097 153 bytes is refused at registration, and by the bucket whatever the row says. + const big = sizedDrawing(2 * MIB + 1, randomUUID().replace(/-/g, '')) + expect(message(await rpc('register_sheet_asset', + { sheet_id: d.sheet, sha256: sha256(big), byte_size: big.length, file_name: 'big.svg' }, L.aal2))).toBe('invalid_input') + // Registered with the policy's size, the object itself over it: the bucket's file size limit refuses it. + ok(await rpc('register_sheet_asset', + { sheet_id: d.sheet, sha256: sha256(big), byte_size: 2 * MIB, file_name: 'big.svg' }, L.aal2)) + for (const token of [L.aal2, stack.serviceKey]) { + const res = await fetch(`${stack.storageUrl}/object/${BUCKET}/${sha256(big)}.svg`, { + method: 'POST', + headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'image/svg+xml' }, + body: new Uint8Array(big), + }) + const body = await res.text() + console.log(`bucket upload of 2 097 153 bytes: HTTP ${res.status} ${body}`) + expect([res.status, JSON.parse(body).statusCode]).toEqual([400, '413']) + } + expect(await objectExists(sha256(big))).toBe(false) + + // Four drawings of 2 MiB (8 MiB, over the 6 MiB total): a business failure, no worker code. + const four = await putDrawings(L, d.sheet, [2 * MIB, 2 * MIB, 2 * MIB, 2 * MIB]) + await save(L, d, multiDoc(four.map((x) => x.id))) + const overTotal = await settled(await submit(L, d), CAPS_BUDGET_MS) + expect(overTotal.round.status).toBe('verification_failed') + expect([overTotal.round.failure_outcome, overTotal.round.attempts]).toEqual(['failed', 1]) + expect(overTotal.round.failure_code).not.toMatch(/^verification\.worker\./) + expect(overTotal.round.verdict?.failures).toContain('verification.validation.caps') + timings.k_over_total = `${overTotal.ms} ms, codes ${overTotal.round.verdict?.failures.join(',')}` + + // Five drawings (over the count of four): the same business failure. + const five = await putDrawings(L, d.sheet, [300, 301, 302, 303, 304]) + await save(L, d, multiDoc(five.map((x) => x.id))) + const overCount = await settled(await submit(L, d)) + expect(overCount.round.status).toBe('verification_failed') + expect([overCount.round.failure_outcome, overCount.round.attempts]).toEqual(['failed', 1]) + expect(overCount.round.verdict?.failures).toContain('verification.validation.caps') + timings.k_over_count = `${overCount.ms} ms, codes ${overCount.round.verdict?.failures.join(',')}` + + // Tracked misclassification (owner brief 2026-11-01: caps expected; not fixed): a document naming + // so many registered drawings (600 rows, about 160 bytes each in the header) that the frame header passes 64 KiB is reported resource_limit. + const many = await newSheet(L) + const rows = (await sql(`insert into public.anatomical_sheet_assets (sheet_id, sha256, byte_size, file_name, created_by, uploaded_at) + select '${uuidOf(many.sheet)}', md5(g::text) || md5(g::text || 'h'), 900, 'h' || g || '.svg', '${uuidOf(L.id)}', now() + from generate_series(1, 600) g returning id`)).split('\n').map(uuidOf) + expect(rows).toHaveLength(600) + await save(L, many, multiDoc(rows)) + const header = uuidOf(await submit(L, many)) + await waitFor(`select failure_code || '/' || attempts from public.anatomical_sheet_reviews where id = '${header}'`, + 'verification.worker.resource_limit/1', 30_000) + // Withdrawn so that the worker stops retrying it (its next claim is 2 s away). + ok(await rpc('withdraw_sheet_review', { review_id: header, note: 'tracked misclassification' }, L.aal2)) + expect((await round(header)).status).toBe('withdrawn') + }, 5 * CAPS_BUDGET_MS) + }) + it('f: avatar rows and the purge queue are untouched by the verification worker', async () => { expect(await avatarsAndPurges()).toBe(before) expect(await sql(`select status || ' ' || attempts || ' ' || (locked_at is null)::text from public.assets diff --git a/supabase/migrations/20261101100000_sheet_verification_jobs.sql b/supabase/migrations/20261101100000_sheet_verification_jobs.sql index 6b28f746..970cc2f5 100644 --- a/supabase/migrations/20261101100000_sheet_verification_jobs.sql +++ b/supabase/migrations/20261101100000_sheet_verification_jobs.sql @@ -4,7 +4,10 @@ -- -- The round row (M5 `anatomical_sheet_reviews`) is the job, with the avatar -- lease pattern of 20261013100000_files_foundation.sql (lease token, --- `locked_at`, a fixed lease, attempts below a limit), without backoff: +-- `locked_at`, a fixed lease, attempts below a limit), plus a progressive +-- delay for technical retries only (owner decision 2026-11-01): a failure +-- reported by the worker sets `not_before`, the claim skips the round until +-- then, so no worker sleeps and any worker honours it: -- * claim_sheet_verifications: rounds `verifying` with no live lease; -- * get_sheet_verification_input: the frozen document, the base, the rows of -- the drawings the document references; @@ -17,11 +20,28 @@ -- =========================================================================== -- 1. Constants -- =========================================================================== +-- The earliest time a technical retry may be claimed (null = now). Hidden: +-- the M5 grant to `authenticated` is column-level and does not list it. +alter table public.anatomical_sheet_reviews add column not_before timestamptz; +comment on column public.anatomical_sheet_reviews.not_before is + 'Earliest time a technical retry may be claimed; null = now. Set by fail_sheet_verification, cleared by the claim. Service role only.'; + -- Attempts before a round closes `verification_failed` / `unavailable` -- (§6.5 « five, proposal, tuned in V1-2.3 »). create or replace function private.sheet_verification_attempt_limit() returns integer language sql immutable set search_path = '' as $$ select 5 $$; +-- Wait before the next claim after a technical failure at attempt n: 2 s, 5 s, +-- 10 s, then 20 s (attempt 1 itself is claimed at once). Validation and +-- integrity refusals are verdicts, never retried. +create or replace function private.sheet_verification_retry_delay(attempts integer) +returns interval language sql immutable set search_path = '' as $$ + select case when attempts <= 1 then interval '2 seconds' + when attempts = 2 then interval '5 seconds' + when attempts = 3 then interval '10 seconds' + else interval '20 seconds' end +$$; + -- A lease older than this is dead: the worker crashed or lost the database. -- The child's wall-clock deadline is far below it (crates/worker limits). create or replace function private.sheet_verification_lease() @@ -191,12 +211,14 @@ begin where r.status = 'verifying' and (r.locked_at is null or r.locked_at < now() - private.sheet_verification_lease()) and r.attempts < private.sheet_verification_attempt_limit() + and (r.not_before is null or r.not_before <= now()) order by r.submitted_at limit v_limit for update skip locked ), leased as ( update public.anatomical_sheet_reviews r set lease_token = gen_random_uuid(), locked_at = now(), attempts = r.attempts + 1, + not_before = null, -- A dead lease left no code: record the interruption it was. failure_code = case when r.locked_at is not null then 'verification.worker.lease_expired' else r.failure_code end from picked @@ -344,8 +366,8 @@ end; $$; -- A technical failure of a leased round (§6.5): the round stays `verifying` --- and claimable, its code recorded, until the attempt limit closes it --- `verification_failed` / `unavailable`. Answers `retry`, `unavailable` or +-- and claimable after its retry delay, its code recorded, until the attempt +-- limit closes it `verification_failed` / `unavailable`. Answers `retry`, `unavailable` or -- `stale` (nothing written). create or replace function public.fail_sheet_verification(review_id uuid, lease_token uuid, code text) returns text @@ -370,6 +392,8 @@ begin v_final := v_review.attempts >= private.sheet_verification_attempt_limit(); update public.anatomical_sheet_reviews r set failure_code = code, lease_token = null, locked_at = null, + not_before = case when v_final then null + else now() + private.sheet_verification_retry_delay(v_review.attempts) end, status = case when v_final then 'verification_failed'::public.sheet_round_status else r.status end, failure_outcome = case when v_final then 'unavailable'::public.sheet_verification_outcome else null end where r.id = v_review.id; @@ -377,7 +401,9 @@ begin case when v_final then 'anatomical_sheet_review.verification_failed' else 'anatomical_sheet_review.verification_attempt_failed' end, 'anatomical_sheet_review', v_review.id, null, null, jsonb_build_object('code', code, 'attempts', v_review.attempts, - 'outcome', case when v_final then 'unavailable' end)); + 'outcome', case when v_final then 'unavailable' end, + 'retry_delay_seconds', case when not v_final then + extract(epoch from private.sheet_verification_retry_delay(v_review.attempts))::integer end)); return case when v_final then 'unavailable' else 'retry' end; end; $$; @@ -386,6 +412,7 @@ $$; -- 4. Privileges: the service role only -- =========================================================================== revoke all on function private.sheet_verification_attempt_limit() from public; +revoke all on function private.sheet_verification_retry_delay(integer) from public; revoke all on function private.sheet_verification_lease() from public; revoke all on function private.sheet_verification_max_verdict_bytes() from public; revoke all on function private.sheet_verdict_codes() from public; diff --git a/supabase/tests/021_sheet_verification_jobs.sql b/supabase/tests/021_sheet_verification_jobs.sql index 5e94472f..5049b584 100644 --- a/supabase/tests/021_sheet_verification_jobs.sql +++ b/supabase/tests/021_sheet_verification_jobs.sql @@ -1,9 +1,10 @@ -- V1-2.3 M6: the worker RPCs of the server verdict (contract §6.4, §6.5). -- Service role only; claim with a lease, input bound to the live token, -- completion checked for its binding and structure (stale / malformed write --- nothing), technical failures up to the attempt limit, dead leases. +-- nothing), technical failures up to the attempt limit with their progressive +-- delay (`not_before`), dead leases. begin; -select plan(51); +select plan(68); insert into auth.users (id, email, email_confirmed_at) values ('f0000000-0000-0000-0000-000000000001', 'own@vj.local', now()), @@ -75,6 +76,20 @@ create function pg_temp.token(p text) returns uuid language sql security definer select lease_token from public.anatomical_sheet_reviews where id = pg_temp.id(p); $$; grant execute on function pg_temp.token(text) to service_role; +-- `not_before` is hidden from the API roles: the helpers read and move it as +-- the test owner. now() is the transaction's, so the delay compares exactly. +create function pg_temp.delayed(p text, p_delay interval) returns boolean language sql security definer as $$ + select not_before = now() + p_delay from public.anatomical_sheet_reviews where id = pg_temp.id(p); +$$; +grant execute on function pg_temp.delayed(text, interval) to service_role; +create function pg_temp.not_before_is_null(p text) returns boolean language sql security definer as $$ + select not_before is null from public.anatomical_sheet_reviews where id = pg_temp.id(p); +$$; +grant execute on function pg_temp.not_before_is_null(text) to service_role; +create function pg_temp.set_not_before(p text, p_offset interval) returns void language sql security definer as $$ + update public.anatomical_sheet_reviews set not_before = now() + p_offset where id = pg_temp.id(p); +$$; +grant execute on function pg_temp.set_not_before(text, interval) to service_role; -- A sheet with one confirmed drawing; three drafts submitted (r1, r2, r3). set local role authenticated; @@ -103,6 +118,12 @@ select ok(has_function_privilege('service_role', 'public.claim_sheet_verificatio and has_function_privilege('service_role', 'public.complete_sheet_verification(uuid, uuid, jsonb)', 'execute') and has_function_privilege('service_role', 'public.fail_sheet_verification(uuid, uuid, text)', 'execute'), 'the service role executes the four RPCs'); +select ok(not has_column_privilege('authenticated', 'public.anatomical_sheet_reviews', 'not_before', 'select') + and not has_column_privilege('anon', 'public.anatomical_sheet_reviews', 'not_before', 'select'), + 'the API roles cannot read not_before'); +select ok(not has_function_privilege('authenticated', 'private.sheet_verification_retry_delay(integer)', 'execute'), 'authenticated cannot execute the delay constant'); +select is((select array_agg(extract(epoch from private.sheet_verification_retry_delay(n))::int order by n) from generate_series(1,5) n), + array[2,5,10,20,20], 'the delay after attempts 1 to 5: 2, 5, 10, 20, 20 seconds'); set local role authenticated; select pg_temp.act('f0000000-0000-0000-0000-000000000001','f1000000-0000-0000-0000-000000000001'); select throws_ok($$ select public.claim_sheet_verifications(1) $$, '42501', null, 'the owner (authenticated) cannot claim'); @@ -175,6 +196,7 @@ select is((select count(*)::int from public.claim_sheet_verifications(1)), 1, 's select is(public.complete_sheet_verification(pg_temp.id('r2'), pg_temp.token('r2'), pg_temp.verdict(pg_temp.id('r2'),'failed','verification.integrity.document')), 'failed', 'a failed verdict without groups is written'); select is(pg_temp.status('r2'), 'verification_failed/failed/verification.integrity.document', 'round closed failed with its code'); +select is((select count(*)::int from public.claim_sheet_verifications(10)), 0, 'a round completed with a failed verdict is never claimed again'); -- 7. Technical failures: retry below the limit, unavailable at it. reset role; @@ -190,19 +212,37 @@ select throws_ok(format('select public.fail_sheet_verification(%L, %L, %L)', pg_ select is(public.fail_sheet_verification(pg_temp.id('r3'), gen_random_uuid(), 'verification.worker.timeout'), 'stale', 'a failure with another token is stale'); select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.timeout'), 'retry', 'a timeout below the limit is retried'); select is(pg_temp.status('r3') || '/' || (pg_temp.token('r3') is null)::text, 'verifying/-/verification.worker.timeout/true', 'round stays verifying, code recorded, lease released'); --- A dead lease is claimed again and recorded as an interruption. -select is((select attempts from public.claim_sheet_verifications(1)), 2, 'reclaimed: second attempt'); +select ok(pg_temp.delayed('r3', interval '2 seconds'), 'after attempt 1 the retry waits 2 s'); +reset role; +select ok(exists (select 1 from audit.audit_logs where action = 'anatomical_sheet_review.verification_attempt_failed' and resource_id = pg_temp.id('r3') + and metadata ->> 'retry_delay_seconds' = '2'), 'attempt failure audited with its delay'); +set local role service_role; select pg_temp.worker(); +select is(public.fail_sheet_verification(pg_temp.id('r3'), gen_random_uuid(), 'verification.worker.timeout'), 'stale', 'a stale failure again'); +select ok(pg_temp.delayed('r3', interval '2 seconds'), 'a stale failure leaves the delay unchanged'); +select is((select count(*)::int from public.claim_sheet_verifications(1)), 0, 'no claim before the delay is over'); +-- The delay over exactly now: claimable. +select pg_temp.set_not_before('r3', interval '0'); +select is((select attempts from public.claim_sheet_verifications(1)), 2, 'claimed once not_before is reached: second attempt'); +select ok(pg_temp.not_before_is_null('r3'), 'the claim clears the delay'); +select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.unavailable'), 'retry', 'a second failure is retried'); +select ok(pg_temp.delayed('r3', interval '5 seconds'), 'after attempt 2 the retry waits 5 s'); +select pg_temp.set_not_before('r3', interval '-1 second'); +select is((select attempts from public.claim_sheet_verifications(1)), 3, 'claimed after 5 s: third attempt'); +-- A dead lease is claimed again at once and recorded as an interruption. reset role; update public.anatomical_sheet_reviews set locked_at = now() - interval '11 minutes' where id = pg_temp.id('r3'); set local role service_role; select pg_temp.worker(); -select is((select attempts from public.claim_sheet_verifications(1)), 3, 'a dead lease is claimed again'); +select is((select attempts from public.claim_sheet_verifications(1)), 4, 'a dead lease is claimed again'); select is(pg_temp.status('r3'), 'verifying/-/verification.worker.lease_expired', 'the interruption is recorded'); --- At the limit, a technical failure closes the round unavailable. -reset role; -update public.anatomical_sheet_reviews set attempts = 5 where id = pg_temp.id('r3'); -set local role service_role; select pg_temp.worker(); +select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.resource_limit'), 'retry', 'a fourth failure is retried'); +select ok(pg_temp.delayed('r3', interval '20 seconds'), 'after attempt 4 the retry waits 20 s'); +select pg_temp.set_not_before('r3', interval '-1 second'); +select is((select attempts from public.claim_sheet_verifications(1)), 5, 'claimed after 20 s: fifth attempt'); +-- At the limit, a technical failure closes the round unavailable, delay cleared. +select pg_temp.set_not_before('r3', interval '1 hour'); select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.resource_limit'), 'unavailable', 'the last attempt closes the round'); select is(pg_temp.status('r3'), 'verification_failed/unavailable/verification.worker.resource_limit', 'round closed unavailable with the last code'); +select ok(pg_temp.not_before_is_null('r3'), 'the fifth failure clears the delay'); reset role; select ok(exists (select 1 from audit.audit_logs where action = 'anatomical_sheet_review.verification_attempt_failed' and resource_id = pg_temp.id('r3')), 'attempt failure audited');