feat(supabase): verification retry by effective cause

resource_limit (intrinsic to the frozen input: frame, input or output cap,
SIGXCPU, SIGABRT, SIGSYS) and verdict_refused close the round at the first
occurrence; timeout, unavailable (now also an external SIGKILL) and
lease_expired keep the 2/5/10/20 s retries up to five attempts. pgTAP 021: 89;
sandbox case k closes at attempt 1.
This commit is contained in:
bchanot committed 2026-11-01 23:01:30 +01:00
1 parent c98a9c8f82
commit a5844b9d56
8 files changed
+156 -46

No files matched your search

+26
View File
@@ -4567,3 +4567,29 @@ through `is distinct from`, service role only) fixed in the three round RPCs, pg
sandbox `test-db` 1 880 PASS, `types --check` equal and `verify-e2e` 10/10 on a fresh stack and on the upgrades from
before M6 and from before M4 (the DEV path); `avatar-e2e` 5/5; `test` 87 and `e2e:shell` 86 (rerun on the final M6
after the verifier's ECARTS(2): the earlier run predated the fix); DEV untouched.
Retry by effective cause (owner brief 2026-11-01, closure C1). SQL holds the code policy
(`private.sheet_worker_final_codes`: `resource_limit`, `verdict_refused` close the round at the first
occurrence, `closed_by` `definitive` in the audit; the others retry 2, 5, 10, 20 s up to five attempts,
`closed_by` `attempt_limit`); the worker maps the cause to the code (`verify_job::classify_child`,
`signal_code`), as implemented:
| Child end | Code | Policy |
|---|---|---|
| frame refused by the parent (header over 64 KiB) | `resource_limit` | definitive |
| `Rejected` (input over the child cap) | `resource_limit` | definitive |
| `OutputOverflow` | `resource_limit` | definitive |
| SIGXCPU (CPU budget; the hard limit is two seconds after the soft one, so the budget shows as SIGXCPU) | `resource_limit` | definitive |
| SIGABRT (memory budget, abort) | `resource_limit` | definitive |
| SIGSYS (seccomp filter) | `resource_limit` | definitive |
| SIGKILL or any other signal (OOM killer, stop: from outside; the parent's own kills are `Timeout` and `OutputOverflow`) | `unavailable` | transient |
| `Timeout` (wall clock) | `timeout` | transient |
| transport, Storage | `unavailable` | transient |
| `Broken`, `Failed` | `unavailable`, then the worker stops | transient |
| verdict answered `malformed` | `verdict_refused` | definitive |
| dead lease | `lease_expired` (set by the claim) | transient, immediate |
What changes for the codes: a SIGKILL is now reported `unavailable` (it was `resource_limit`); the
outcome (`unavailable`) and the code list are otherwise unchanged. A definitive round closes after one
child run instead of five identical ones. Still tracked and open: the header over 64 KiB is reported
`resource_limit` where `caps` would name it (test `tracked_a_header_over_64_kib_is_reported_resource_limit`).
+14 -8
View File
@@ -261,20 +261,26 @@ unchanged.
- Completion: `complete_sheet_verification` answers `passed`, `failed`,
`stale` (nothing written, logged) or `malformed` (reported
`verification.worker.verdict_refused`). Child timeout →
`verification.worker.timeout`; a signal, an output overflow or an input over
its cap → `verification.worker.resource_limit`; a transport error →
`verification.worker.unavailable`; a broken child → `unavailable`, then the
worker stops (as the avatar loop). A dead lease (10 minutes) is claimed
again as `verification.worker.lease_expired`; the fifth attempt closes the
round `verification_failed` / `unavailable`.
`verification.worker.timeout` (transient); SIGXCPU, SIGABRT or SIGSYS (the
child's own CPU, memory and seccomp budgets), an output overflow or an input
over its cap → `verification.worker.resource_limit` (definitive: the frozen
input gives the same end every time); any other signal, SIGKILL first (the
OOM killer or a stop, from outside) and a transport error →
`verification.worker.unavailable` (transient); a broken child →
`unavailable`, then the worker stops (as the avatar loop). A dead lease (10
minutes) is claimed again as `verification.worker.lease_expired`. Retry by
cause: `resource_limit` and `verdict_refused` close the round
`verification_failed` / `unavailable` at the first occurrence; the other
codes retry up to the fifth attempt, which closes it the same way.
- Retry delay (owner decision 2026-11-01): a failure reported by
`fail_sheet_verification` makes the round claimable again after 2 s, 5 s,
10 s, 20 s (after attempts 1 to 4), kept in SQL (`not_before`, hidden from
the API roles), so the worker never sleeps for it and the claim of another
round is not blocked; the effective delay is that delay plus up to one poll
interval (`WORKER_POLL_SECONDS`, 2 s). A verdict (failed or not) is never
retried; a dead lease is claimed again at once. The worker code is the
same as before.
retried; a dead lease is claimed again at once. Only the transient codes
wait: the delay applies to `timeout`, `unavailable` and `lease_expired`
(SQL decides, `private.sheet_worker_final_codes`).
- Measured (2026-11-01): the at-caps document (64 views, 1 024 annotations,
8 192 nodes) verified in 185 ms by the worker on the sandbox, verdict
24 689 bytes; the equine draft in about 60 ms.
+18 -8
View File
@@ -890,12 +890,19 @@ attempts below a limit), plus a progressive delay for technical retries only
else answers `stale` or `malformed` and writes nothing. Moves the round to
`open` or to `verification_failed` (§6.5) and audits.
- `fail_sheet_verification(review_id, lease_token, code)`: a technical
failure; the round stays `verifying` and becomes claimable after a delay
failure, closed by cause (2026-11-01, `private.sheet_worker_final_codes`).
A definitive code (`verification.worker.resource_limit`, cause intrinsic to
the frozen input or the contract caps; `verification.worker.verdict_refused`)
closes the round at its first occurrence: `verification_failed`, outcome
`unavailable`, the code kept, no delay, never claimed again. A transient
code (`timeout`, `unavailable`, `lease_expired`): the round stays `verifying`
and becomes claimable after a delay
that grows with the attempt (2 s after attempt 1, 5 s after 2, 10 s after 3,
20 s after 4; `private.sheet_verification_retry_delay`), kept in the round's
`not_before` (a column the API roles cannot read, set here, cleared by the
claim); the fifth failure becomes `verification_failed` with outcome
`unavailable` and clears it. Nothing is written for a stale call. The delay
claim); the fifth failure of a transient code becomes `verification_failed`
with outcome `unavailable` and clears it. The audit payload says which rule
closed the round (`closed_by`: `definitive` or `attempt_limit`). Nothing is written for a stale call. The delay
is persisted, so no worker sleeps and any worker honours it; the effective
wait is the delay plus up to one poll interval of the worker (2 s by
default). Only these technical failures wait: a validation or integrity
@@ -942,7 +949,8 @@ five, verdict text at most 4 MiB; the completion also refuses an unknown verdict
or code, a duplicated group key and a drawing whose result is not `ok`, and accepts a
failed verdict without groups or drawings; worker codes `verification.worker.unavailable`,
`.timeout`, `.resource_limit`, `.lease_expired` (a dead lease), `.verdict_refused` (a
completion answered `malformed`), a closed list in SQL (`private.sheet_worker_codes`).
completion answered `malformed`), a closed list in SQL (`private.sheet_worker_codes`); the
definitive ones are `.resource_limit` and `.verdict_refused` (`private.sheet_worker_final_codes`).
These values and codes are the implementer's, for the owner's confirmation. The
service role executes these four RPCs besides the two upload ones (pgTAP 018).
@@ -961,11 +969,13 @@ did not pass; the closed codes keep the categories apart.
|---|---|---|---|---|
| Business validation | integrity error, caps exceeded, annotation without geometry, annotation outside its view's frame | `verification_failed`, outcome `failed`, codes `verification.validation.*` and the issues | no | fix the draft, submit again |
| Integrity | strict parse refused, drawing missing, size or sha256 mismatch, SVG refused, needing a removal or outside the stored form (§9.2), `object_conflict` | `verification_failed`, outcome `failed`, codes `verification.integrity.*` | no | fix or upload again; `object_conflict` follows §9.3 |
| Transient worker error | database or Storage unreachable, worker restart | stays `verifying`, code recorded | yes, after 2 s, 5 s, 10 s, 20 s, up to the limit | automatic |
| Timeout or resource limit | child over its wall-clock or memory limit | stays `verifying`, code recorded | yes, same delays, up to the limit | reduce the document or the drawing if it repeats |
| Transient worker error | database or Storage unreachable, broken child, child killed from outside (SIGKILL or any signal not listed below: OOM killer, stop), worker restart | `verification.worker.unavailable`, stays `verifying`, code recorded | yes, after 2 s, 5 s, 10 s, 20 s, up to the limit | automatic |
| Timeout | child over its wall-clock deadline | `verification.worker.timeout`, stays `verifying`, code recorded | yes, same delays, up to the limit | reduce the document or the drawing if it repeats |
| Resource limit intrinsic to the content | frame refused, input over the child cap, output over its cap, CPU budget (SIGXCPU), memory budget (SIGABRT), seccomp (SIGSYS) | `verification_failed`, outcome `unavailable`, `verification.worker.resource_limit`, closed at the first occurrence | no (the same input ends the same way) | reduce the document or the drawings, submit again |
| Verdict refused | completion answered `malformed` | `verification_failed`, outcome `unavailable`, `verification.worker.verdict_refused`, closed at the first occurrence | no | submit again later; reported to the operator |
| Interruption | worker crash, lease expired | claimable again at once (the lease was the wait); counts as an attempt | yes, up to the limit | automatic |
| Stale result | completion with an old token, after a withdrawal, or naming another revision | nothing written, logged | no | none needed |
| Definitive failure | attempt limit reached (five; V1-2.3 implements five, for the owner's confirmation) | `verification_failed`, outcome `unavailable`, last code | no | submit again later |
| Attempt limit | a transient code at its fifth attempt (five; V1-2.3 implements five, for the owner's confirmation) | `verification_failed`, outcome `unavailable`, last code | no | submit again later |
| Core change | release that changes rules or reader | rounds withdrawn by the release migration | no | submit again |
- Withdrawal is always available: the submitter or any editor withdraws a
@@ -1715,7 +1725,7 @@ of the document's JSON text in one form, the same for every layer:
|---|---|---|---|
| M4 `sheet_access_hardening` | V1-2.1 | `anatomical_catalog_grants` and its RPCs; `has_mfa`, `grant_active`, `caller_has`; `can_read_official_catalog` and the reader row predicate; `can_edit_sheet` redefined (F14); AR-1 B on the platform role RPCs and the private write functions (F19, §5.7), F17, F18; F1, F2, F3 (fork closed), F5, F6, F8, F9, F12 (every catalogue table), F13, F15; asset-row immutability trigger (F16); historical-sheet protection trigger (§14.2); reader column grants narrowed; personal creation and visibility changes refused | no row changes |
| M5 `sheet_review_workflow` | V1-2.1 | version columns `revision`, `authors`, `source_version_id`, `published_review_id`, `document_fingerprint`; sheet column `working_copy_source_version_id`; revision, author and freeze triggers, the historical-sheet write rule; extended immutability guard (§6.8); rounds, comments, acknowledgments; `create_official_sheet`, `create_sheet_draft` adapted, `save_sheet_draft`, submit, withdraw, comment, acknowledge, decide, publish; `create_sheet_working_copy`; species RPCs; catalogue and version read RPCs; upload: `register_sheet_asset` extended, `confirm_sheet_asset_upload` with the storage check (F7), INSERT policy, bucket limit 2 MiB; mirror removal and status check (§12.2); superseded signatures dropped; the content-size CHECK replaced by the text measure (`octet_length(content::text)`, §12.2) | new columns take their defaults on the equine v1 row (`revision` 0, empty author set); its document, status, number and assets unchanged; the new size CHECK is validated against the existing rows when M5 is applied (the equine document measures 184 105 bytes as `content::text`, measured on the sandbox); bucket limit lowered; no round, no grant |
| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only, with the retry delay (hidden `not_before` column on the round, `private.sheet_verification_retry_delay`: owner decision 2026-11-01, replaces « without backoff »); written as `20261101100000_sheet_verification_jobs.sql`, pgTAP `021`, rehearsed on the sandbox from the state before M4 (M4 to M6 in one `migration up`) and from the state before M6 | none |
| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only, with the retry delay (hidden `not_before` column on the round, `private.sheet_verification_retry_delay`: owner decision 2026-11-01, replaces « without backoff ») and the retry by cause (definitive codes close at the first occurrence, `private.sheet_worker_final_codes`); written as `20261101100000_sheet_verification_jobs.sql`, pgTAP `021`, rehearsed on the sandbox from the state before M4 (M4 to M6 in one `migration up`) and from the state before M6 | none |
Each migration ships its pgTAP file (allowed and denied cases), regenerated
`database.types.ts`, a `db:sandbox up upgrade` run from the DEV state (with a