feat(supabase): retry delay of 2, 5, 10, 20 s for technical verification failures

M6 amended in place (applied only to the throwaway sandbox): a hidden
not_before on the round, set by fail_sheet_verification and honoured by the
claim; RPC signatures unchanged. pgTAP 021 (68), sandbox cases i (delay with
the real worker), j (inputs at the caps, 2 MiB base), k (just beyond).
This commit is contained in:
bchanot committed 2026-11-01 21:09:00 +01:00
1 parent e967ce6736
commit 5adad3709f
8 files changed
+398 -29

No files matched your search

+32 -3
View File
@@ -175,7 +175,8 @@ cmake on Alpine). The decode child's traced syscall set did not change with
the parent's TLS code (§5.4 re-run at R1: the same seven). Image 11.6 MB the parent's TLS code (§5.4 re-run at R1: the same seven). Image 11.6 MB
(8.7 MB at R0: ring + rustls). Tests: `crates/worker/tests/job_loop.rs` (11, (8.7 MB at R0: ring + rustls). Tests: `crates/worker/tests/job_loop.rs` (11,
fake backend + fake decoder) and the module unit tests run natively; the fake backend + fake decoder) and the module unit tests run natively; the
real-stack end-to-end runs through `pnpm e2e:avatar` (`STORAGE.md` §6). real-stack end-to-end runs through `pnpm db:sandbox avatar-e2e` on the throwaway stack since V1-2.3
(`pnpm e2e:avatar` on DEV, the owner's call; `STORAGE.md` §6).
The sandboxed child, unchanged from R0: The sandboxed child, unchanged from R0:
@@ -4501,8 +4502,7 @@ Reviews (fresh, 2026-11-01, on `7043d39`):
password literal in `scripts/db-sandbox.sh`). LOWs fixed: the verdict cap measured twice (child output now password literal in `scripts/db-sandbox.sh`). LOWs fixed: the verdict cap measured twice (child output now
3 MiB, under the 4 MiB SQL measure of `jsonb::text`); the child's 1 GiB address space above the container's 3 MiB, under the 4 MiB SQL measure of `jsonb::text`); the child's 1 GiB address space above the container's
512 MiB (`worker-verify` and the sandbox worker now 1 280 MiB, so the child's limit binds first). LOWs open: no 512 MiB (`worker-verify` and the sandbox worker now 1 280 MiB, so the child's limit binds first). LOWs open: no
backoff between attempts (the §6.4 design « without backoff »: an outage of a few seconds can spend the five backoff between attempts (closed afterwards by the owner decision of 2026-11-01: SQL delay of 2, 5, 10, 20 s, V1-2 contract §6.4; the finding as written on `7043d39` is left); a document naming several hundred
attempts, the round closing `unavailable`; fail-closed, availability only); a document naming several hundred
registered drawings overflows the 64 KiB frame header and is reported `resource_limit` instead of the core's registered drawings overflows the 64 KiB frame header and is reported `resource_limit` instead of the core's
`caps` (wrong exit message, no security effect); SQL interpolation of generated ids in the sandbox test only. `caps` (wrong exit message, no security effect); SQL interpolation of generated ids in the sandbox test only.
- Verifier: ECARTS(2). W1-W5, W7 met. W6: the hostile corpus through the child was missing (added, above); `pnpm - Verifier: ECARTS(2). W1-W5, W7 met. W6: the hostile corpus through the child was missing (added, above); `pnpm
@@ -4525,3 +4525,32 @@ WASM 580 522 B gzip in the app image, unchanged by V1-2.3); `pnpm e2e:shell` 86
Second reviews on `6aa85fe`: verifier ECARTS(1), the one item owner-gated (`pnpm e2e:avatar`, a worker on DEV), the Second reviews on `6aa85fe`: verifier ECARTS(1), the one item owner-gated (`pnpm e2e:avatar`, a worker on DEV), the
hostile corpus met; security PASS (MEDIUM pre-existing sandbox password literal; LOWs: the 3 MiB / 4 MiB relation is hostile corpus met; security PASS (MEDIUM pre-existing sandbox password literal; LOWs: the 3 MiB / 4 MiB relation is
empirical, a refused verdict ending fail-closed `verdict_refused`; DEPLOY.md memory note, fixed). empirical, a refused verdict ending fail-closed `verdict_refused`; DEPLOY.md memory note, fixed).
### 13.1 Reserves Q3 and Q4: inputs at the caps, `brk` bounded (2026-11-01)
Contract `.claude/tasks/contracts/2026-11-01-v1-2-3-reserves.md`. Fixtures (`fixtures.rs`, `test-hooks`): `verify_max_frame()`,
also `fixture verify-max`: document and base of exactly 2 097 152 bytes (the `jsonb::text` spelling, `": "` and `", "`,
filler in `attribution.notes`; the core measures `document_bytes` 2 097 152), four drawings of 2 097 152, 2 097 152,
1 048 576 and 1 048 576 bytes (6 291 456 in all), a frame of 10 486 658 bytes. A drawing is the stored prolog, a 10 × 10
root, 10 000 `<path d="M0 0L5 5"/>` and a last path whose `d` ends in as many spaces as the size needs. Unit tests run the
core on each fixture: the maximum is `passed`; four drawings of 2 MiB (no base), five drawings, one drawing of 2 097 153
bytes and a document of 2 097 153 bytes are `failed` with `verification.validation.caps` (content, exit 0).
Through the full sandbox (`tests/sandbox.rs`, now 31 tests): `the_maximum_input_is_verified_within_the_limits` (`passed`,
verdict 2 531 bytes, under the 3 MiB cap and the 90 s deadline), `inputs_just_beyond_the_caps_fail_as_caps_through_the_sandbox`,
`the_verify_input_cap_is_exact` (a frame of exactly 12 MiB is read, here refused by the frame check; one byte more is refused
unread by the existing test). Measured on the development host, release build, glibc, sandbox applied: the maximum 104 ms,
the sheet at every other cap (2 118 765 bytes in, 24 689 out) 142 ms. The Docker `test` stage passes the same tests; its
log does not print the timings (captured output).
`brk` in the verify child (`tests/sandbox.rs`, 1 GiB address space): `--alloc-gib 2` and the new hook
`--alloc-small-gib <n>` (4 KiB chunks kept alive, the small-allocation path) both end in SIGABRT and the survival
message never appears; `--mmap-exec` and `--signal-parent` die with SIGSYS; `--violate` was already covered. The CPU and
deadline hooks stay on the decode child. No syscall added to `ALLOWED_SYSCALLS`.
Derivation: `derive-seccomp.sh` also traces `fixture verify-max` through `verify --no-seccomp`; the traced set is
unchanged (`brk exit_group getrandom mmap mremap munmap read write`), the allowlist covers it.
The two misclassifications stay tracked, not fixed (`tests/verify_loop.rs`): a frame header over 64 KiB is reported
`resource_limit`; an object over the policy is passed without bytes and the verdict lists `caps` then `drawing_missing`
(its declared size already exceeds the cap), never `size_mismatch`.
+10 -1
View File
@@ -267,6 +267,14 @@ unchanged.
worker stops (as the avatar loop). A dead lease (10 minutes) is claimed worker stops (as the avatar loop). A dead lease (10 minutes) is claimed
again as `verification.worker.lease_expired`; the fifth attempt closes the again as `verification.worker.lease_expired`; the fifth attempt closes the
round `verification_failed` / `unavailable`. round `verification_failed` / `unavailable`.
- Retry delay (owner decision 2026-11-01): a failure reported by
`fail_sheet_verification` makes the round claimable again after 2 s, 5 s,
10 s, 20 s (after attempts 1 to 4), kept in SQL (`not_before`, hidden from
the API roles), so the worker never sleeps for it and the claim of another
round is not blocked; the effective delay is that delay plus up to one poll
interval (`WORKER_POLL_SECONDS`, 2 s). A verdict (failed or not) is never
retried; a dead lease is claimed again at once. The worker code is the
same as before.
- Measured (2026-11-01): the at-caps document (64 views, 1 024 annotations, - Measured (2026-11-01): the at-caps document (64 views, 1 024 annotations,
8 192 nodes) verified in 185 ms by the worker on the sandbox, verdict 8 192 nodes) verified in 185 ms by the worker on the sandbox, verdict
24 689 bytes; the equine draft in about 60 ms. 24 689 bytes; the equine draft in about 60 ms.
@@ -369,7 +377,8 @@ docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges:true
-e SUPABASE_SERVICE_ROLE_KEY="$(pnpm -s supabase status -o env | sed -n 's/^SERVICE_ROLE_KEY="\(.*\)"/\1/p')" \ -e SUPABASE_SERVICE_ROLE_KEY="$(pnpm -s supabase status -o env | sed -n 's/^SERVICE_ROLE_KEY="\(.*\)"/\1/p')" \
faunosteo_worker:dev run faunosteo_worker:dev run
pnpm --filter @faunosteo/supabase test:e2e # real-stack end-to-end (needs the worker above) pnpm --filter @faunosteo/supabase test:e2e # real-stack end-to-end (needs the worker above)
pnpm e2e:avatar # scripts/avatar-e2e.sh: the three lines above in one, worker started with a 5 s sweep and no quarantine grace pnpm e2e:avatar # scripts/avatar-e2e.sh: the three lines above in one, worker started with a 5 s sweep and no quarantine grace (DEV: the owner's call)
pnpm db:sandbox avatar-e2e # the same suite and worker on the throwaway stack (SUPABASE.md §5.1): the check of a storage or worker change
``` ```
`docker compose --profile worker up worker` runs the same container from `docker compose --profile worker up worker` runs the same container from
+23 -12
View File
@@ -871,12 +871,13 @@ fingerprint or a document hash from a page.
Job. The round row is the job, with the avatar lease pattern of Job. The round row is the job, with the avatar lease pattern of
`20261013100000_files_foundation.sql` (lease token, `locked_at`, fixed lease, `20261013100000_files_foundation.sql` (lease token, `locked_at`, fixed lease,
attempts below a limit), without backoff. Submission sets the round attempts below a limit), plus a progressive delay for technical retries only
`verifying`. Worker RPCs, service role only (M6): (owner decision 2026-11-01). Submission sets the round `verifying`. Worker RPCs, service role only (M6):
- `claim_sheet_verifications(limit)`: `for update skip locked` over rounds - `claim_sheet_verifications(limit)`: `for update skip locked` over rounds
`verifying` with no live lease; sets the token and `locked_at`, increments `verifying` with no live lease and no pending delay (`not_before` null or
`attempts`; returns round, version, sheet, revision and token. reached); sets the token and `locked_at`, increments `attempts`, clears
`not_before`; returns round, version, sheet, revision and token.
- `get_sheet_verification_input(review_id, lease_token)`: the document at the - `get_sheet_verification_input(review_id, lease_token)`: the document at the
round's revision (frozen) and the base document, both as text; the drawings round's revision (frozen) and the base document, both as text; the drawings
the document references among the rows registered for the sheet (asset id, the document references among the rows registered for the sheet (asset id,
@@ -889,8 +890,17 @@ attempts below a limit), without backoff. Submission sets the round
else answers `stale` or `malformed` and writes nothing. Moves the round to else answers `stale` or `malformed` and writes nothing. Moves the round to
`open` or to `verification_failed` (§6.5) and audits. `open` or to `verification_failed` (§6.5) and audits.
- `fail_sheet_verification(review_id, lease_token, code)`: a technical - `fail_sheet_verification(review_id, lease_token, code)`: a technical
failure; the round stays claimable until the attempt limit, then becomes failure; the round stays `verifying` and becomes claimable after a delay
`verification_failed` with outcome `unavailable`. that grows with the attempt (2 s after attempt 1, 5 s after 2, 10 s after 3,
20 s after 4; `private.sheet_verification_retry_delay`), kept in the round's
`not_before` (a column the API roles cannot read, set here, cleared by the
claim); the fifth failure becomes `verification_failed` with outcome
`unavailable` and clears it. Nothing is written for a stale call. The delay
is persisted, so no worker sleeps and any worker honours it; the effective
wait is the delay plus up to one poll interval of the worker (2 s by
default). Only these technical failures wait: a validation or integrity
refusal is a verdict and is never retried, and a dead lease is claimed again
at once (the 10-minute lease is its wait).
Processing. Verification runs as its own mode of the worker image (`run Processing. Verification runs as its own mode of the worker image (`run
--jobs sheet-verify`), in its own container under the `worker` compose --jobs sheet-verify`), in its own container under the `worker` compose
@@ -951,9 +961,9 @@ did not pass; the closed codes keep the categories apart.
|---|---|---|---|---| |---|---|---|---|---|
| Business validation | integrity error, caps exceeded, annotation without geometry, annotation outside its view's frame | `verification_failed`, outcome `failed`, codes `verification.validation.*` and the issues | no | fix the draft, submit again | | Business validation | integrity error, caps exceeded, annotation without geometry, annotation outside its view's frame | `verification_failed`, outcome `failed`, codes `verification.validation.*` and the issues | no | fix the draft, submit again |
| Integrity | strict parse refused, drawing missing, size or sha256 mismatch, SVG refused, needing a removal or outside the stored form (§9.2), `object_conflict` | `verification_failed`, outcome `failed`, codes `verification.integrity.*` | no | fix or upload again; `object_conflict` follows §9.3 | | Integrity | strict parse refused, drawing missing, size or sha256 mismatch, SVG refused, needing a removal or outside the stored form (§9.2), `object_conflict` | `verification_failed`, outcome `failed`, codes `verification.integrity.*` | no | fix or upload again; `object_conflict` follows §9.3 |
| Transient worker error | database or Storage unreachable, worker restart | stays `verifying`, code recorded | yes, up to the limit | automatic | | Transient worker error | database or Storage unreachable, worker restart | stays `verifying`, code recorded | yes, after 2 s, 5 s, 10 s, 20 s, up to the limit | automatic |
| Timeout or resource limit | child over its wall-clock or memory limit | stays `verifying`, code recorded | yes, up to the limit | reduce the document or the drawing if it repeats | | Timeout or resource limit | child over its wall-clock or memory limit | stays `verifying`, code recorded | yes, same delays, up to the limit | reduce the document or the drawing if it repeats |
| Interruption | worker crash, lease expired | claimable again; counts as an attempt | yes, up to the limit | automatic | | Interruption | worker crash, lease expired | claimable again at once (the lease was the wait); counts as an attempt | yes, up to the limit | automatic |
| Stale result | completion with an old token, after a withdrawal, or naming another revision | nothing written, logged | no | none needed | | Stale result | completion with an old token, after a withdrawal, or naming another revision | nothing written, logged | no | none needed |
| Definitive failure | attempt limit reached (five; V1-2.3 implements five, for the owner's confirmation) | `verification_failed`, outcome `unavailable`, last code | no | submit again later | | Definitive failure | attempt limit reached (five; V1-2.3 implements five, for the owner's confirmation) | `verification_failed`, outcome `unavailable`, last code | no | submit again later |
| Core change | release that changes rules or reader | rounds withdrawn by the release migration | no | submit again | | Core change | release that changes rules or reader | rounds withdrawn by the release migration | no | submit again |
@@ -1705,7 +1715,7 @@ of the document's JSON text in one form, the same for every layer:
|---|---|---|---| |---|---|---|---|
| M4 `sheet_access_hardening` | V1-2.1 | `anatomical_catalog_grants` and its RPCs; `has_mfa`, `grant_active`, `caller_has`; `can_read_official_catalog` and the reader row predicate; `can_edit_sheet` redefined (F14); AR-1 B on the platform role RPCs and the private write functions (F19, §5.7), F17, F18; F1, F2, F3 (fork closed), F5, F6, F8, F9, F12 (every catalogue table), F13, F15; asset-row immutability trigger (F16); historical-sheet protection trigger (§14.2); reader column grants narrowed; personal creation and visibility changes refused | no row changes | | M4 `sheet_access_hardening` | V1-2.1 | `anatomical_catalog_grants` and its RPCs; `has_mfa`, `grant_active`, `caller_has`; `can_read_official_catalog` and the reader row predicate; `can_edit_sheet` redefined (F14); AR-1 B on the platform role RPCs and the private write functions (F19, §5.7), F17, F18; F1, F2, F3 (fork closed), F5, F6, F8, F9, F12 (every catalogue table), F13, F15; asset-row immutability trigger (F16); historical-sheet protection trigger (§14.2); reader column grants narrowed; personal creation and visibility changes refused | no row changes |
| M5 `sheet_review_workflow` | V1-2.1 | version columns `revision`, `authors`, `source_version_id`, `published_review_id`, `document_fingerprint`; sheet column `working_copy_source_version_id`; revision, author and freeze triggers, the historical-sheet write rule; extended immutability guard (§6.8); rounds, comments, acknowledgments; `create_official_sheet`, `create_sheet_draft` adapted, `save_sheet_draft`, submit, withdraw, comment, acknowledge, decide, publish; `create_sheet_working_copy`; species RPCs; catalogue and version read RPCs; upload: `register_sheet_asset` extended, `confirm_sheet_asset_upload` with the storage check (F7), INSERT policy, bucket limit 2 MiB; mirror removal and status check (§12.2); superseded signatures dropped; the content-size CHECK replaced by the text measure (`octet_length(content::text)`, §12.2) | new columns take their defaults on the equine v1 row (`revision` 0, empty author set); its document, status, number and assets unchanged; the new size CHECK is validated against the existing rows when M5 is applied (the equine document measures 184 105 bytes as `content::text`, measured on the sandbox); bucket limit lowered; no round, no grant | | M5 `sheet_review_workflow` | V1-2.1 | version columns `revision`, `authors`, `source_version_id`, `published_review_id`, `document_fingerprint`; sheet column `working_copy_source_version_id`; revision, author and freeze triggers, the historical-sheet write rule; extended immutability guard (§6.8); rounds, comments, acknowledgments; `create_official_sheet`, `create_sheet_draft` adapted, `save_sheet_draft`, submit, withdraw, comment, acknowledge, decide, publish; `create_sheet_working_copy`; species RPCs; catalogue and version read RPCs; upload: `register_sheet_asset` extended, `confirm_sheet_asset_upload` with the storage check (F7), INSERT policy, bucket limit 2 MiB; mirror removal and status check (§12.2); superseded signatures dropped; the content-size CHECK replaced by the text measure (`octet_length(content::text)`, §12.2) | new columns take their defaults on the equine v1 row (`revision` 0, empty author set); its document, status, number and assets unchanged; the new size CHECK is validated against the existing rows when M5 is applied (the equine document measures 184 105 bytes as `content::text`, measured on the sandbox); bucket limit lowered; no round, no grant |
| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only; written as `20261101100000_sheet_verification_jobs.sql`, pgTAP `021`, rehearsed on the sandbox from the state before M4 (M4 to M6 in one `migration up`) and from the state before M6 | none | | M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only, with the retry delay (hidden `not_before` column on the round, `private.sheet_verification_retry_delay`: owner decision 2026-11-01, replaces « without backoff »); written as `20261101100000_sheet_verification_jobs.sql`, pgTAP `021`, rehearsed on the sandbox from the state before M4 (M4 to M6 in one `migration up`) and from the state before M6 | none |
Each migration ships its pgTAP file (allowed and denied cases), regenerated Each migration ships its pgTAP file (allowed and denied cases), regenerated
`database.types.ts`, a `db:sandbox up upgrade` run from the DEV state (with a `database.types.ts`, a `db:sandbox up upgrade` run from the DEV state (with a
@@ -2021,8 +2031,9 @@ The final approval needs all 30 groups acknowledged, no `to_confirm` and no
- Worker: the verification mode, verdicts accepted and each failure, the - Worker: the verification mode, verdicts accepted and each failure, the
hostile SVG corpus in the sandboxed child, per-command limits and watchdog, hostile SVG corpus in the sandboxed child, per-command limits and watchdog,
the purge guard by bucket, sandbox tests and seccomp derivation, an e2e on the purge guard by bucket, sandbox tests and seccomp derivation, an e2e on
the sandbox stack with the verification worker running; `pnpm e2e:avatar` the sandbox stack with the verification worker running; the avatar suite
unchanged in outcome; `pnpm e2e:sheet-assets` targets DEV only, its rerun unchanged in outcome on the sandbox (`pnpm db:sandbox avatar-e2e`, owner
decision 2026-11-01: `pnpm e2e:avatar` targets DEV and is not run); `pnpm e2e:sheet-assets` targets DEV only, its rerun
is an owner step (§16.11). is an owner step (§16.11).
- Release replay: a script loads every published version and drawing of a - Release replay: a script loads every published version and drawing of a
stack through the new core and fails on any regression. stack through the new core and fails on any regression.
+3
View File
@@ -227,6 +227,7 @@ export type Database = {
id: string id: string
lease_token: string | null lease_token: string | null
locked_at: string | null locked_at: string | null
not_before: string | null
revision: number revision: number
round: number round: number
status: Database['public']['Enums']['sheet_round_status'] status: Database['public']['Enums']['sheet_round_status']
@@ -256,6 +257,7 @@ export type Database = {
id?: string id?: string
lease_token?: string | null lease_token?: string | null
locked_at?: string | null locked_at?: string | null
not_before?: string | null
revision: number revision: number
round: number round: number
status?: Database['public']['Enums']['sheet_round_status'] status?: Database['public']['Enums']['sheet_round_status']
@@ -284,6 +286,7 @@ export type Database = {
id?: string id?: string
lease_token?: string | null lease_token?: string | null
locked_at?: string | null locked_at?: string | null
not_before?: string | null
revision?: number revision?: number
round?: number round?: number
status?: Database['public']['Enums']['sheet_round_status'] status?: Database['public']['Enums']['sheet_round_status']
@@ -100,6 +100,7 @@ const expectedColumns = {
'id', 'id',
'lease_token', 'lease_token',
'locked_at', 'locked_at',
'not_before',
'revision', 'revision',
'round', 'round',
'status', 'status',
@@ -229,6 +229,112 @@ const avatarsAndPurges = () =>
|| ' ' || || ' ' ||
(select count(*) || ':' || md5(coalesce(string_agg(row_to_json(q)::text, ',' order by q.id), '')) from private.object_purge_queue q)`) (select count(*) || ':' || md5(coalesce(string_agg(row_to_json(q)::text, ',' order by q.id), '')) from private.object_purge_queue q)`)
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/
/** An id read back from the stack, checked before it is interpolated into SQL. */
function uuidOf(value: unknown): string {
if (typeof value !== 'string' || !UUID.test(value)) throw new Error(`not a uuid: ${String(value)}`)
return value
}
/** `octet_length(value::jsonb::text)` of a value whose strings need no JSON escape (crates/core/src/domain/sheet/size.rs). */
function jsonbBytes(value: unknown): number {
if (value === null) return 4
if (typeof value === 'boolean') return value ? 4 : 5
if (typeof value === 'number') return String(value).length
if (typeof value === 'string') return Buffer.byteLength(value) + 2
if (Array.isArray(value)) return 2 + value.reduce((n: number, v) => n + jsonbBytes(v), 0) + 2 * Math.max(0, value.length - 1)
const entries = Object.entries(value as Record<string, unknown>)
return 2 + entries.reduce((n, [k, v]) => n + jsonbBytes(k) + 2 + jsonbBytes(v), 0) + 2 * Math.max(0, entries.length - 1)
}
/**
* A drawing of exactly `size` bytes in the stored form (`serialize::stored_form`): the UTF-8
* prolog and one `<svg>` holding one `<path>` whose `d` is padded with `L5 5` segments and
* spaces. Canonical because it is what the core writes back byte for byte: ASCII only, the
* namespace declared on the root, attributes in a fixed order, every value free of `& < > "`
* and of tab or line feed, no comment, no text node, the element self-closed; the pad adds
* no element, attribute, `<` or `=` (SVG_CAPS: 20 000 elements, 100 000 `<`, 200 000 `=`).
*/
function sizedDrawing(size: number, tag: string): Buffer {
const head = '<?xml version="1.0" encoding="UTF-8"?>\n<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 10 10">' +
`<path id="p${tag}" d="M0 0L5 5`
const tail = '"/></svg>'
const pad = size - head.length - tail.length
const text = head + ' L5 5'.repeat(Math.floor(pad / 5)) + ' '.repeat(pad % 5) + tail
expect(text.length).toBe(size)
return Buffer.from(text, 'ascii')
}
/** `main` on the first drawing, one detail view per other drawing, each opened by a trigger on `main`. */
function multiDoc(assets: string[], descriptionFr = 'd') {
const doc = smallDoc(assets[0]) as Record<string, unknown>
doc.description = { fr: descriptionFr, en: 'd' }
const views = doc.views as unknown[]
const annotations = doc.annotations as unknown[]
assets.slice(1).forEach((asset, i) => {
views.push({ id: `d${i}`, kind: 'detail', name: name('D'), svg_asset_id: asset, view_box: null, parent_view_id: 'main' })
annotations.push({ id: `t${i}`, kind: 'detail_trigger', view_id: 'main', category_id: 'c', laterality: 'unpaired',
geometry: { type: 'polygon', points: [[0, 0], [1, 0], [1, 1]] }, review_status: 'reviewed', name: null,
opens_view_id: `d${i}` })
})
return doc
}
/** `multiDoc` whose `octet_length(content::text)` is exactly `bytes`; `fill` makes two such documents differ. */
function documentOfBytes(assets: string[], bytes: number, fill = 'x'): unknown {
const missing = bytes - jsonbBytes(multiDoc(assets))
expect(missing).toBeGreaterThanOrEqual(0)
return multiDoc(assets, 'd' + fill.repeat(missing))
}
/** Submits `d` until its round is `verifying` with `attempts` claims already spent and a lease held for another worker (SQL stand-in). */
async function holdLease(editor: SteppedAccount, d: Draft, attempts: number): Promise<{ review: string; token: string }> {
const token = randomUUID()
for (let turn = 0; turn < 5; turn++) {
const id = uuidOf(await submit(editor, d))
const held = await sql(`update public.anatomical_sheet_reviews set lease_token = '${token}', locked_at = now(), attempts = ${attempts}
where id = '${id}' and status = 'verifying' and lease_token is null returning id`)
if (held === id) return { review: id, token }
await settled(id)
ok(await rpc('withdraw_sheet_review', { review_id: id, note: 'raced' }, editor.aal2))
}
throw new Error('lease not held before the worker claimed')
}
/** Polls a one-value query until it answers `expected`. */
async function waitFor(query: string, expected: string, timeoutMs: number): Promise<void> {
const start = performance.now()
let last = ''
while (performance.now() - start < timeoutMs) {
last = await sql(query)
if (last === expected) return
await new Promise((resolve) => setTimeout(resolve, 200))
}
throw new Error(`expected ${expected}, last ${last}`)
}
/** Registers, uploads and confirms one drawing per size on `sheet`; the ids in order. */
async function putDrawings(editor: SteppedAccount, sheet: string, sizes: number[]): Promise<{ id: string; bytes: Buffer }[]> {
const out: { id: string; bytes: Buffer }[] = []
for (const [i, size] of sizes.entries()) {
const bytes = sizedDrawing(size, `${i}${randomUUID().replace(/-/g, '')}`)
out.push({ id: await putDrawing(editor, sheet, bytes), bytes })
}
return out
}
/** Acknowledges every group of the open round, approves it and publishes (the control of case d). */
async function publishRound(editor: SteppedAccount, reviewer: SteppedAccount, review: string, r: Round): Promise<void> {
for (const group of r.verdict?.groups ?? [])
ok(await rpc('acknowledge_sheet_review_group',
{ review_id: review, group_key: group.key, group_digest: group.digest }, reviewer.aal2))
ok(await rpc('decide_sheet_review', { review_id: review, approve: true, note: 'ok' }, reviewer.aal2))
ok(await rpc('publish_sheet_version', { review_id: review }, editor.aal2))
}
const verdictBytes = async (review: string) =>
Number(await sql(`select octet_length(verdict::text) from public.anatomical_sheet_reviews where id = '${review}'`))
const timings: Record<string, string> = {} const timings: Record<string, string> = {}
describe.skipIf(!WORKER)('sheet verification by the real worker (V1-2.3)', () => { describe.skipIf(!WORKER)('sheet verification by the real worker (V1-2.3)', () => {
@@ -471,6 +577,149 @@ describe.skipIf(!WORKER)('sheet verification by the real worker (V1-2.3)', () =>
expect(ms).toBeLessThan(CAPS_BUDGET_MS) expect(ms).toBeLessThan(CAPS_BUDGET_MS)
}, CAPS_BUDGET_MS + 60_000) }, CAPS_BUDGET_MS + 60_000)
it('i: a technical failure is retried no earlier than its delay, the worker serving another round meanwhile', async () => {
const d = await newSheet(E)
const bytes = storedDrawing()
const asset = await putDrawing(E, d.sheet, bytes)
await save(E, d, smallDoc(asset))
// Three attempts spent and the lease held for another worker: the failure is the third (10 s delay).
const { review, token } = await holdLease(E, d, 3)
const failed = await rpc('fail_sheet_verification',
{ review_id: review, lease_token: token, code: 'verification.worker.timeout' }, stack.serviceKey)
expect(failed.body).toBe('retry')
// not_before = now() of the failing transaction + 10 s: the failure time in SQL's clock.
const failAt = Number(await sql(`select extract(epoch from not_before - interval '10 seconds')
from public.anatomical_sheet_reviews where id = '${review}'`))
expect(failAt).toBeGreaterThan(0)
// Meanwhile the worker serves another round: a clean sheet opens.
const other = await newSheet(E)
const otherBytes = storedDrawing()
const otherAsset = await putDrawing(E, other.sheet, otherBytes)
await save(E, other, smallDoc(otherAsset))
const served = await settled(await submit(E, other))
expectPassedFor(served.round, other, [{ id: otherAsset, sha: sha256(otherBytes) }])
// The first round waits: verifying, no lease, three attempts, delay not over.
expect(await sql(`select status || '/' || attempts || '/' || (lease_token is null) || '/' || (clock_timestamp() < not_before)
from public.anatomical_sheet_reviews where id = '${review}'`)).toBe('verifying/3/true/true')
const retried = await settled(review, 40_000)
expectPassedFor(retried.round, d, [{ id: asset, sha: sha256(bytes) }])
expect(retried.round.attempts).toBe(4)
const gap = Number(await sql(`select extract(epoch from verified_at) - ${failAt}
from public.anatomical_sheet_reviews where id = '${review}'`))
expect(gap).toBeGreaterThanOrEqual(10)
timings.i_retry = `claimed and verified ${gap.toFixed(1)} s after the failure (delay 10 s, poll 2 s), other round opened in ${served.ms} ms`
}, 90_000)
describe('input limits (Q3)', () => {
let L: SteppedAccount
const MIB = 1024 * 1024
beforeAll(async () => {
L = await holder('vf-limits', 'admin', ['sheet_editor', 'sheet_publisher'])
}, 60_000)
it('j: the largest input the policy allows (document 2 MiB, drawings 2+2+1+1 MiB, base 2 MiB) is verified within the budget', async () => {
const d = await newSheet(L)
const drawings = await putDrawings(L, d.sheet, [2 * MIB, 2 * MIB, MIB, MIB])
const ids = drawings.map((x) => x.id)
const first = documentOfBytes(ids, 2 * MIB, 'x')
await save(L, d, first)
expect(await sql(`select octet_length(content::text) from public.anatomical_sheet_versions where id = '${d.draft}'`)).toBe(String(2 * MIB))
const review = await submit(L, d)
const { round: r, ms } = await settled(review, CAPS_BUDGET_MS)
// The verdict lists the drawings in document order: both sides are compared sorted by asset id.
const expected = drawings.map((x) => ({ id: x.id, sha: sha256(x.bytes) })).sort((a, b) => a.id.localeCompare(b.id))
const byAsset = (x: Round): Round => ({ ...x, verdict: { ...(x.verdict as Verdict),
drawings: [...(x.verdict as Verdict).drawings].sort((a, b) => a.asset_id.localeCompare(b.asset_id)) } })
expectPassedFor(byAsset(r), d, expected)
const v = r.verdict as Verdict
expect(v.groups.map((g) => g.key)).toEqual(expect.arrayContaining(['general', 'view:main', 'view:d0', 'view:d1', 'view:d2']))
const worker = await workerDuration(review)
const size = await verdictBytes(review)
expect(size).toBeLessThan(3_145_728)
timings.j_at_limits = `${ms} ms from submission (worker ${worker} ms), document ${2 * MIB} bytes, drawings ${drawings.map((x) => x.bytes.length).join('+')} bytes, verdict ${size} bytes`
expect(ms).toBeLessThan(CAPS_BUDGET_MS)
// Second round: its base is the published 2 MiB document (frame: 2 + 2 + 6 MiB).
await publishRound(L, V, review, r)
const draft = uuidOf(ok(await rpc('create_sheet_draft', { sheet_id: d.sheet, from_version_id: null, change_note: 'next' }, L.aal2)))
const next: Draft = { sheet: d.sheet, draft, revision: Number(await sql(`select revision from public.anatomical_sheet_versions where id = '${draft}'`)) }
expect(await sql(`select (source_version_id = '${d.draft}')::text from public.anatomical_sheet_versions where id = '${draft}'`)).toBe('true')
await save(L, next, documentOfBytes(ids, 2 * MIB, 'y'))
const second = await submit(L, next)
const again = await settled(second, CAPS_BUDGET_MS)
expectPassedFor(byAsset(again.round), next, expected)
const size2 = await verdictBytes(second)
expect(size2).toBeLessThan(3_145_728)
timings.j_with_base = `${again.ms} ms from submission (worker ${await workerDuration(second)} ms), document and base ${2 * MIB} bytes each, drawings 6 MiB, verdict ${size2} bytes`
expect(again.ms).toBeLessThan(CAPS_BUDGET_MS)
}, 3 * CAPS_BUDGET_MS)
it('k: just beyond the limits: refused before the worker, or a business failure; the tracked misclassification', async () => {
// A document of 2 097 153 bytes is refused at save.
const d = await newSheet(L)
const small = await putDrawing(L, d.sheet, storedDrawing())
const save1 = await rpc('save_sheet_draft',
{ version_id: d.draft, expected_revision: d.revision, document: documentOfBytes([small], 2 * MIB + 1), change_note: 'e2e' }, L.aal2)
expect(message(save1)).toBe('document.too_large')
// A drawing of 2 097 153 bytes is refused at registration, and by the bucket whatever the row says.
const big = sizedDrawing(2 * MIB + 1, randomUUID().replace(/-/g, ''))
expect(message(await rpc('register_sheet_asset',
{ sheet_id: d.sheet, sha256: sha256(big), byte_size: big.length, file_name: 'big.svg' }, L.aal2))).toBe('invalid_input')
// Registered with the policy's size, the object itself over it: the bucket's file size limit refuses it.
ok(await rpc('register_sheet_asset',
{ sheet_id: d.sheet, sha256: sha256(big), byte_size: 2 * MIB, file_name: 'big.svg' }, L.aal2))
for (const token of [L.aal2, stack.serviceKey]) {
const res = await fetch(`${stack.storageUrl}/object/${BUCKET}/${sha256(big)}.svg`, {
method: 'POST',
headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'image/svg+xml' },
body: new Uint8Array(big),
})
const body = await res.text()
console.log(`bucket upload of 2 097 153 bytes: HTTP ${res.status} ${body}`)
expect([res.status, JSON.parse(body).statusCode]).toEqual([400, '413'])
}
expect(await objectExists(sha256(big))).toBe(false)
// Four drawings of 2 MiB (8 MiB, over the 6 MiB total): a business failure, no worker code.
const four = await putDrawings(L, d.sheet, [2 * MIB, 2 * MIB, 2 * MIB, 2 * MIB])
await save(L, d, multiDoc(four.map((x) => x.id)))
const overTotal = await settled(await submit(L, d), CAPS_BUDGET_MS)
expect(overTotal.round.status).toBe('verification_failed')
expect([overTotal.round.failure_outcome, overTotal.round.attempts]).toEqual(['failed', 1])
expect(overTotal.round.failure_code).not.toMatch(/^verification\.worker\./)
expect(overTotal.round.verdict?.failures).toContain('verification.validation.caps')
timings.k_over_total = `${overTotal.ms} ms, codes ${overTotal.round.verdict?.failures.join(',')}`
// Five drawings (over the count of four): the same business failure.
const five = await putDrawings(L, d.sheet, [300, 301, 302, 303, 304])
await save(L, d, multiDoc(five.map((x) => x.id)))
const overCount = await settled(await submit(L, d))
expect(overCount.round.status).toBe('verification_failed')
expect([overCount.round.failure_outcome, overCount.round.attempts]).toEqual(['failed', 1])
expect(overCount.round.verdict?.failures).toContain('verification.validation.caps')
timings.k_over_count = `${overCount.ms} ms, codes ${overCount.round.verdict?.failures.join(',')}`
// Tracked misclassification (owner brief 2026-11-01: caps expected; not fixed): a document naming
// so many registered drawings (600 rows, about 160 bytes each in the header) that the frame header passes 64 KiB is reported resource_limit.
const many = await newSheet(L)
const rows = (await sql(`insert into public.anatomical_sheet_assets (sheet_id, sha256, byte_size, file_name, created_by, uploaded_at)
select '${uuidOf(many.sheet)}', md5(g::text) || md5(g::text || 'h'), 900, 'h' || g || '.svg', '${uuidOf(L.id)}', now()
from generate_series(1, 600) g returning id`)).split('\n').map(uuidOf)
expect(rows).toHaveLength(600)
await save(L, many, multiDoc(rows))
const header = uuidOf(await submit(L, many))
await waitFor(`select failure_code || '/' || attempts from public.anatomical_sheet_reviews where id = '${header}'`,
'verification.worker.resource_limit/1', 30_000)
// Withdrawn so that the worker stops retrying it (its next claim is 2 s away).
ok(await rpc('withdraw_sheet_review', { review_id: header, note: 'tracked misclassification' }, L.aal2))
expect((await round(header)).status).toBe('withdrawn')
}, 5 * CAPS_BUDGET_MS)
})
it('f: avatar rows and the purge queue are untouched by the verification worker', async () => { it('f: avatar rows and the purge queue are untouched by the verification worker', async () => {
expect(await avatarsAndPurges()).toBe(before) expect(await avatarsAndPurges()).toBe(before)
expect(await sql(`select status || ' ' || attempts || ' ' || (locked_at is null)::text from public.assets expect(await sql(`select status || ' ' || attempts || ' ' || (locked_at is null)::text from public.assets
@@ -4,7 +4,10 @@
-- --
-- The round row (M5 `anatomical_sheet_reviews`) is the job, with the avatar -- The round row (M5 `anatomical_sheet_reviews`) is the job, with the avatar
-- lease pattern of 20261013100000_files_foundation.sql (lease token, -- lease pattern of 20261013100000_files_foundation.sql (lease token,
-- `locked_at`, a fixed lease, attempts below a limit), without backoff: -- `locked_at`, a fixed lease, attempts below a limit), plus a progressive
-- delay for technical retries only (owner decision 2026-11-01): a failure
-- reported by the worker sets `not_before`, the claim skips the round until
-- then, so no worker sleeps and any worker honours it:
-- * claim_sheet_verifications: rounds `verifying` with no live lease; -- * claim_sheet_verifications: rounds `verifying` with no live lease;
-- * get_sheet_verification_input: the frozen document, the base, the rows of -- * get_sheet_verification_input: the frozen document, the base, the rows of
-- the drawings the document references; -- the drawings the document references;
@@ -17,11 +20,28 @@
-- =========================================================================== -- ===========================================================================
-- 1. Constants -- 1. Constants
-- =========================================================================== -- ===========================================================================
-- The earliest time a technical retry may be claimed (null = now). Hidden:
-- the M5 grant to `authenticated` is column-level and does not list it.
alter table public.anatomical_sheet_reviews add column not_before timestamptz;
comment on column public.anatomical_sheet_reviews.not_before is
'Earliest time a technical retry may be claimed; null = now. Set by fail_sheet_verification, cleared by the claim. Service role only.';
-- Attempts before a round closes `verification_failed` / `unavailable` -- Attempts before a round closes `verification_failed` / `unavailable`
-- (§6.5 « five, proposal, tuned in V1-2.3 »). -- (§6.5 « five, proposal, tuned in V1-2.3 »).
create or replace function private.sheet_verification_attempt_limit() create or replace function private.sheet_verification_attempt_limit()
returns integer language sql immutable set search_path = '' as $$ select 5 $$; returns integer language sql immutable set search_path = '' as $$ select 5 $$;
-- Wait before the next claim after a technical failure at attempt n: 2 s, 5 s,
-- 10 s, then 20 s (attempt 1 itself is claimed at once). Validation and
-- integrity refusals are verdicts, never retried.
create or replace function private.sheet_verification_retry_delay(attempts integer)
returns interval language sql immutable set search_path = '' as $$
select case when attempts <= 1 then interval '2 seconds'
when attempts = 2 then interval '5 seconds'
when attempts = 3 then interval '10 seconds'
else interval '20 seconds' end
$$;
-- A lease older than this is dead: the worker crashed or lost the database. -- A lease older than this is dead: the worker crashed or lost the database.
-- The child's wall-clock deadline is far below it (crates/worker limits). -- The child's wall-clock deadline is far below it (crates/worker limits).
create or replace function private.sheet_verification_lease() create or replace function private.sheet_verification_lease()
@@ -191,12 +211,14 @@ begin
where r.status = 'verifying' where r.status = 'verifying'
and (r.locked_at is null or r.locked_at < now() - private.sheet_verification_lease()) and (r.locked_at is null or r.locked_at < now() - private.sheet_verification_lease())
and r.attempts < private.sheet_verification_attempt_limit() and r.attempts < private.sheet_verification_attempt_limit()
and (r.not_before is null or r.not_before <= now())
order by r.submitted_at order by r.submitted_at
limit v_limit limit v_limit
for update skip locked for update skip locked
), leased as ( ), leased as (
update public.anatomical_sheet_reviews r update public.anatomical_sheet_reviews r
set lease_token = gen_random_uuid(), locked_at = now(), attempts = r.attempts + 1, set lease_token = gen_random_uuid(), locked_at = now(), attempts = r.attempts + 1,
not_before = null,
-- A dead lease left no code: record the interruption it was. -- A dead lease left no code: record the interruption it was.
failure_code = case when r.locked_at is not null then 'verification.worker.lease_expired' else r.failure_code end failure_code = case when r.locked_at is not null then 'verification.worker.lease_expired' else r.failure_code end
from picked from picked
@@ -344,8 +366,8 @@ end;
$$; $$;
-- A technical failure of a leased round (§6.5): the round stays `verifying` -- A technical failure of a leased round (§6.5): the round stays `verifying`
-- and claimable, its code recorded, until the attempt limit closes it -- and claimable after its retry delay, its code recorded, until the attempt
-- `verification_failed` / `unavailable`. Answers `retry`, `unavailable` or -- limit closes it `verification_failed` / `unavailable`. Answers `retry`, `unavailable` or
-- `stale` (nothing written). -- `stale` (nothing written).
create or replace function public.fail_sheet_verification(review_id uuid, lease_token uuid, code text) create or replace function public.fail_sheet_verification(review_id uuid, lease_token uuid, code text)
returns text returns text
@@ -370,6 +392,8 @@ begin
v_final := v_review.attempts >= private.sheet_verification_attempt_limit(); v_final := v_review.attempts >= private.sheet_verification_attempt_limit();
update public.anatomical_sheet_reviews r update public.anatomical_sheet_reviews r
set failure_code = code, lease_token = null, locked_at = null, set failure_code = code, lease_token = null, locked_at = null,
not_before = case when v_final then null
else now() + private.sheet_verification_retry_delay(v_review.attempts) end,
status = case when v_final then 'verification_failed'::public.sheet_round_status else r.status end, status = case when v_final then 'verification_failed'::public.sheet_round_status else r.status end,
failure_outcome = case when v_final then 'unavailable'::public.sheet_verification_outcome else null end failure_outcome = case when v_final then 'unavailable'::public.sheet_verification_outcome else null end
where r.id = v_review.id; where r.id = v_review.id;
@@ -377,7 +401,9 @@ begin
case when v_final then 'anatomical_sheet_review.verification_failed' else 'anatomical_sheet_review.verification_attempt_failed' end, case when v_final then 'anatomical_sheet_review.verification_failed' else 'anatomical_sheet_review.verification_attempt_failed' end,
'anatomical_sheet_review', v_review.id, null, null, 'anatomical_sheet_review', v_review.id, null, null,
jsonb_build_object('code', code, 'attempts', v_review.attempts, jsonb_build_object('code', code, 'attempts', v_review.attempts,
'outcome', case when v_final then 'unavailable' end)); 'outcome', case when v_final then 'unavailable' end,
'retry_delay_seconds', case when not v_final then
extract(epoch from private.sheet_verification_retry_delay(v_review.attempts))::integer end));
return case when v_final then 'unavailable' else 'retry' end; return case when v_final then 'unavailable' else 'retry' end;
end; end;
$$; $$;
@@ -386,6 +412,7 @@ $$;
-- 4. Privileges: the service role only -- 4. Privileges: the service role only
-- =========================================================================== -- ===========================================================================
revoke all on function private.sheet_verification_attempt_limit() from public; revoke all on function private.sheet_verification_attempt_limit() from public;
revoke all on function private.sheet_verification_retry_delay(integer) from public;
revoke all on function private.sheet_verification_lease() from public; revoke all on function private.sheet_verification_lease() from public;
revoke all on function private.sheet_verification_max_verdict_bytes() from public; revoke all on function private.sheet_verification_max_verdict_bytes() from public;
revoke all on function private.sheet_verdict_codes() from public; revoke all on function private.sheet_verdict_codes() from public;
+49 -9
View File
@@ -1,9 +1,10 @@
-- V1-2.3 M6: the worker RPCs of the server verdict (contract §6.4, §6.5). -- V1-2.3 M6: the worker RPCs of the server verdict (contract §6.4, §6.5).
-- Service role only; claim with a lease, input bound to the live token, -- Service role only; claim with a lease, input bound to the live token,
-- completion checked for its binding and structure (stale / malformed write -- completion checked for its binding and structure (stale / malformed write
-- nothing), technical failures up to the attempt limit, dead leases. -- nothing), technical failures up to the attempt limit with their progressive
-- delay (`not_before`), dead leases.
begin; begin;
select plan(51); select plan(68);
insert into auth.users (id, email, email_confirmed_at) values insert into auth.users (id, email, email_confirmed_at) values
('f0000000-0000-0000-0000-000000000001', 'own@vj.local', now()), ('f0000000-0000-0000-0000-000000000001', 'own@vj.local', now()),
@@ -75,6 +76,20 @@ create function pg_temp.token(p text) returns uuid language sql security definer
select lease_token from public.anatomical_sheet_reviews where id = pg_temp.id(p); select lease_token from public.anatomical_sheet_reviews where id = pg_temp.id(p);
$$; $$;
grant execute on function pg_temp.token(text) to service_role; grant execute on function pg_temp.token(text) to service_role;
-- `not_before` is hidden from the API roles: the helpers read and move it as
-- the test owner. now() is the transaction's, so the delay compares exactly.
create function pg_temp.delayed(p text, p_delay interval) returns boolean language sql security definer as $$
select not_before = now() + p_delay from public.anatomical_sheet_reviews where id = pg_temp.id(p);
$$;
grant execute on function pg_temp.delayed(text, interval) to service_role;
create function pg_temp.not_before_is_null(p text) returns boolean language sql security definer as $$
select not_before is null from public.anatomical_sheet_reviews where id = pg_temp.id(p);
$$;
grant execute on function pg_temp.not_before_is_null(text) to service_role;
create function pg_temp.set_not_before(p text, p_offset interval) returns void language sql security definer as $$
update public.anatomical_sheet_reviews set not_before = now() + p_offset where id = pg_temp.id(p);
$$;
grant execute on function pg_temp.set_not_before(text, interval) to service_role;
-- A sheet with one confirmed drawing; three drafts submitted (r1, r2, r3). -- A sheet with one confirmed drawing; three drafts submitted (r1, r2, r3).
set local role authenticated; set local role authenticated;
@@ -103,6 +118,12 @@ select ok(has_function_privilege('service_role', 'public.claim_sheet_verificatio
and has_function_privilege('service_role', 'public.complete_sheet_verification(uuid, uuid, jsonb)', 'execute') and has_function_privilege('service_role', 'public.complete_sheet_verification(uuid, uuid, jsonb)', 'execute')
and has_function_privilege('service_role', 'public.fail_sheet_verification(uuid, uuid, text)', 'execute'), and has_function_privilege('service_role', 'public.fail_sheet_verification(uuid, uuid, text)', 'execute'),
'the service role executes the four RPCs'); 'the service role executes the four RPCs');
select ok(not has_column_privilege('authenticated', 'public.anatomical_sheet_reviews', 'not_before', 'select')
and not has_column_privilege('anon', 'public.anatomical_sheet_reviews', 'not_before', 'select'),
'the API roles cannot read not_before');
select ok(not has_function_privilege('authenticated', 'private.sheet_verification_retry_delay(integer)', 'execute'), 'authenticated cannot execute the delay constant');
select is((select array_agg(extract(epoch from private.sheet_verification_retry_delay(n))::int order by n) from generate_series(1,5) n),
array[2,5,10,20,20], 'the delay after attempts 1 to 5: 2, 5, 10, 20, 20 seconds');
set local role authenticated; set local role authenticated;
select pg_temp.act('f0000000-0000-0000-0000-000000000001','f1000000-0000-0000-0000-000000000001'); select pg_temp.act('f0000000-0000-0000-0000-000000000001','f1000000-0000-0000-0000-000000000001');
select throws_ok($$ select public.claim_sheet_verifications(1) $$, '42501', null, 'the owner (authenticated) cannot claim'); select throws_ok($$ select public.claim_sheet_verifications(1) $$, '42501', null, 'the owner (authenticated) cannot claim');
@@ -175,6 +196,7 @@ select is((select count(*)::int from public.claim_sheet_verifications(1)), 1, 's
select is(public.complete_sheet_verification(pg_temp.id('r2'), pg_temp.token('r2'), select is(public.complete_sheet_verification(pg_temp.id('r2'), pg_temp.token('r2'),
pg_temp.verdict(pg_temp.id('r2'),'failed','verification.integrity.document')), 'failed', 'a failed verdict without groups is written'); pg_temp.verdict(pg_temp.id('r2'),'failed','verification.integrity.document')), 'failed', 'a failed verdict without groups is written');
select is(pg_temp.status('r2'), 'verification_failed/failed/verification.integrity.document', 'round closed failed with its code'); select is(pg_temp.status('r2'), 'verification_failed/failed/verification.integrity.document', 'round closed failed with its code');
select is((select count(*)::int from public.claim_sheet_verifications(10)), 0, 'a round completed with a failed verdict is never claimed again');
-- 7. Technical failures: retry below the limit, unavailable at it. -- 7. Technical failures: retry below the limit, unavailable at it.
reset role; reset role;
@@ -190,19 +212,37 @@ select throws_ok(format('select public.fail_sheet_verification(%L, %L, %L)', pg_
select is(public.fail_sheet_verification(pg_temp.id('r3'), gen_random_uuid(), 'verification.worker.timeout'), 'stale', 'a failure with another token is stale'); select is(public.fail_sheet_verification(pg_temp.id('r3'), gen_random_uuid(), 'verification.worker.timeout'), 'stale', 'a failure with another token is stale');
select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.timeout'), 'retry', 'a timeout below the limit is retried'); select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.timeout'), 'retry', 'a timeout below the limit is retried');
select is(pg_temp.status('r3') || '/' || (pg_temp.token('r3') is null)::text, 'verifying/-/verification.worker.timeout/true', 'round stays verifying, code recorded, lease released'); select is(pg_temp.status('r3') || '/' || (pg_temp.token('r3') is null)::text, 'verifying/-/verification.worker.timeout/true', 'round stays verifying, code recorded, lease released');
-- A dead lease is claimed again and recorded as an interruption. select ok(pg_temp.delayed('r3', interval '2 seconds'), 'after attempt 1 the retry waits 2 s');
select is((select attempts from public.claim_sheet_verifications(1)), 2, 'reclaimed: second attempt'); reset role;
select ok(exists (select 1 from audit.audit_logs where action = 'anatomical_sheet_review.verification_attempt_failed' and resource_id = pg_temp.id('r3')
and metadata ->> 'retry_delay_seconds' = '2'), 'attempt failure audited with its delay');
set local role service_role; select pg_temp.worker();
select is(public.fail_sheet_verification(pg_temp.id('r3'), gen_random_uuid(), 'verification.worker.timeout'), 'stale', 'a stale failure again');
select ok(pg_temp.delayed('r3', interval '2 seconds'), 'a stale failure leaves the delay unchanged');
select is((select count(*)::int from public.claim_sheet_verifications(1)), 0, 'no claim before the delay is over');
-- The delay over exactly now: claimable.
select pg_temp.set_not_before('r3', interval '0');
select is((select attempts from public.claim_sheet_verifications(1)), 2, 'claimed once not_before is reached: second attempt');
select ok(pg_temp.not_before_is_null('r3'), 'the claim clears the delay');
select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.unavailable'), 'retry', 'a second failure is retried');
select ok(pg_temp.delayed('r3', interval '5 seconds'), 'after attempt 2 the retry waits 5 s');
select pg_temp.set_not_before('r3', interval '-1 second');
select is((select attempts from public.claim_sheet_verifications(1)), 3, 'claimed after 5 s: third attempt');
-- A dead lease is claimed again at once and recorded as an interruption.
reset role; reset role;
update public.anatomical_sheet_reviews set locked_at = now() - interval '11 minutes' where id = pg_temp.id('r3'); update public.anatomical_sheet_reviews set locked_at = now() - interval '11 minutes' where id = pg_temp.id('r3');
set local role service_role; select pg_temp.worker(); set local role service_role; select pg_temp.worker();
select is((select attempts from public.claim_sheet_verifications(1)), 3, 'a dead lease is claimed again'); select is((select attempts from public.claim_sheet_verifications(1)), 4, 'a dead lease is claimed again');
select is(pg_temp.status('r3'), 'verifying/-/verification.worker.lease_expired', 'the interruption is recorded'); select is(pg_temp.status('r3'), 'verifying/-/verification.worker.lease_expired', 'the interruption is recorded');
-- At the limit, a technical failure closes the round unavailable. select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.resource_limit'), 'retry', 'a fourth failure is retried');
reset role; select ok(pg_temp.delayed('r3', interval '20 seconds'), 'after attempt 4 the retry waits 20 s');
update public.anatomical_sheet_reviews set attempts = 5 where id = pg_temp.id('r3'); select pg_temp.set_not_before('r3', interval '-1 second');
set local role service_role; select pg_temp.worker(); select is((select attempts from public.claim_sheet_verifications(1)), 5, 'claimed after 20 s: fifth attempt');
-- At the limit, a technical failure closes the round unavailable, delay cleared.
select pg_temp.set_not_before('r3', interval '1 hour');
select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.resource_limit'), 'unavailable', 'the last attempt closes the round'); select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.resource_limit'), 'unavailable', 'the last attempt closes the round');
select is(pg_temp.status('r3'), 'verification_failed/unavailable/verification.worker.resource_limit', 'round closed unavailable with the last code'); select is(pg_temp.status('r3'), 'verification_failed/unavailable/verification.worker.resource_limit', 'round closed unavailable with the last code');
select ok(pg_temp.not_before_is_null('r3'), 'the fifth failure clears the delay');
reset role; reset role;
select ok(exists (select 1 from audit.audit_logs where action = 'anatomical_sheet_review.verification_attempt_failed' and resource_id = pg_temp.id('r3')), 'attempt failure audited'); select ok(exists (select 1 from audit.audit_logs where action = 'anatomical_sheet_review.verification_attempt_failed' and resource_id = pg_temp.id('r3')), 'attempt failure audited');