feat(supabase): retry delay of 2, 5, 10, 20 s for technical verification failures
M6 amended in place (applied only to the throwaway sandbox): a hidden not_before on the round, set by fail_sheet_verification and honoured by the claim; RPC signatures unchanged. pgTAP 021 (68), sandbox cases i (delay with the real worker), j (inputs at the caps, 2 MiB base), k (just beyond).
This commit is contained in:
1 parent
e967ce6736
commit
5adad3709f
8 files changed
+398
-29
No files matched your search
@@ -175,7 +175,8 @@ cmake on Alpine). The decode child's traced syscall set did not change with
|
||||
the parent's TLS code (§5.4 re-run at R1: the same seven). Image 11.6 MB
|
||||
(8.7 MB at R0: ring + rustls). Tests: `crates/worker/tests/job_loop.rs` (11,
|
||||
fake backend + fake decoder) and the module unit tests run natively; the
|
||||
real-stack end-to-end runs through `pnpm e2e:avatar` (`STORAGE.md` §6).
|
||||
real-stack end-to-end runs through `pnpm db:sandbox avatar-e2e` on the throwaway stack since V1-2.3
|
||||
(`pnpm e2e:avatar` on DEV, the owner's call; `STORAGE.md` §6).
|
||||
|
||||
The sandboxed child, unchanged from R0:
|
||||
|
||||
@@ -4501,8 +4502,7 @@ Reviews (fresh, 2026-11-01, on `7043d39`):
|
||||
password literal in `scripts/db-sandbox.sh`). LOWs fixed: the verdict cap measured twice (child output now
|
||||
3 MiB, under the 4 MiB SQL measure of `jsonb::text`); the child's 1 GiB address space above the container's
|
||||
512 MiB (`worker-verify` and the sandbox worker now 1 280 MiB, so the child's limit binds first). LOWs open: no
|
||||
backoff between attempts (the §6.4 design « without backoff »: an outage of a few seconds can spend the five
|
||||
attempts, the round closing `unavailable`; fail-closed, availability only); a document naming several hundred
|
||||
backoff between attempts (closed afterwards by the owner decision of 2026-11-01: SQL delay of 2, 5, 10, 20 s, V1-2 contract §6.4; the finding as written on `7043d39` is left); a document naming several hundred
|
||||
registered drawings overflows the 64 KiB frame header and is reported `resource_limit` instead of the core's
|
||||
`caps` (wrong exit message, no security effect); SQL interpolation of generated ids in the sandbox test only.
|
||||
- Verifier: ECARTS(2). W1-W5, W7 met. W6: the hostile corpus through the child was missing (added, above); `pnpm
|
||||
@@ -4525,3 +4525,32 @@ WASM 580 522 B gzip in the app image, unchanged by V1-2.3); `pnpm e2e:shell` 86
|
||||
Second reviews on `6aa85fe`: verifier ECARTS(1), the one item owner-gated (`pnpm e2e:avatar`, a worker on DEV), the
|
||||
hostile corpus met; security PASS (MEDIUM pre-existing sandbox password literal; LOWs: the 3 MiB / 4 MiB relation is
|
||||
empirical, a refused verdict ending fail-closed `verdict_refused`; DEPLOY.md memory note, fixed).
|
||||
|
||||
### 13.1 Reserves Q3 and Q4: inputs at the caps, `brk` bounded (2026-11-01)
|
||||
|
||||
Contract `.claude/tasks/contracts/2026-11-01-v1-2-3-reserves.md`. Fixtures (`fixtures.rs`, `test-hooks`): `verify_max_frame()`,
|
||||
also `fixture verify-max`: document and base of exactly 2 097 152 bytes (the `jsonb::text` spelling, `": "` and `", "`,
|
||||
filler in `attribution.notes`; the core measures `document_bytes` 2 097 152), four drawings of 2 097 152, 2 097 152,
|
||||
1 048 576 and 1 048 576 bytes (6 291 456 in all), a frame of 10 486 658 bytes. A drawing is the stored prolog, a 10 × 10
|
||||
root, 10 000 `<path d="M0 0L5 5"/>` and a last path whose `d` ends in as many spaces as the size needs. Unit tests run the
|
||||
core on each fixture: the maximum is `passed`; four drawings of 2 MiB (no base), five drawings, one drawing of 2 097 153
|
||||
bytes and a document of 2 097 153 bytes are `failed` with `verification.validation.caps` (content, exit 0).
|
||||
|
||||
Through the full sandbox (`tests/sandbox.rs`, now 31 tests): `the_maximum_input_is_verified_within_the_limits` (`passed`,
|
||||
verdict 2 531 bytes, under the 3 MiB cap and the 90 s deadline), `inputs_just_beyond_the_caps_fail_as_caps_through_the_sandbox`,
|
||||
`the_verify_input_cap_is_exact` (a frame of exactly 12 MiB is read, here refused by the frame check; one byte more is refused
|
||||
unread by the existing test). Measured on the development host, release build, glibc, sandbox applied: the maximum 104 ms,
|
||||
the sheet at every other cap (2 118 765 bytes in, 24 689 out) 142 ms. The Docker `test` stage passes the same tests; its
|
||||
log does not print the timings (captured output).
|
||||
|
||||
`brk` in the verify child (`tests/sandbox.rs`, 1 GiB address space): `--alloc-gib 2` and the new hook
|
||||
`--alloc-small-gib <n>` (4 KiB chunks kept alive, the small-allocation path) both end in SIGABRT and the survival
|
||||
message never appears; `--mmap-exec` and `--signal-parent` die with SIGSYS; `--violate` was already covered. The CPU and
|
||||
deadline hooks stay on the decode child. No syscall added to `ALLOWED_SYSCALLS`.
|
||||
|
||||
Derivation: `derive-seccomp.sh` also traces `fixture verify-max` through `verify --no-seccomp`; the traced set is
|
||||
unchanged (`brk exit_group getrandom mmap mremap munmap read write`), the allowlist covers it.
|
||||
|
||||
The two misclassifications stay tracked, not fixed (`tests/verify_loop.rs`): a frame header over 64 KiB is reported
|
||||
`resource_limit`; an object over the policy is passed without bytes and the verdict lists `caps` then `drawing_missing`
|
||||
(its declared size already exceeds the cap), never `size_mismatch`.
|
||||
@@ -267,6 +267,14 @@ unchanged.
|
||||
worker stops (as the avatar loop). A dead lease (10 minutes) is claimed
|
||||
again as `verification.worker.lease_expired`; the fifth attempt closes the
|
||||
round `verification_failed` / `unavailable`.
|
||||
- Retry delay (owner decision 2026-11-01): a failure reported by
|
||||
`fail_sheet_verification` makes the round claimable again after 2 s, 5 s,
|
||||
10 s, 20 s (after attempts 1 to 4), kept in SQL (`not_before`, hidden from
|
||||
the API roles), so the worker never sleeps for it and the claim of another
|
||||
round is not blocked; the effective delay is that delay plus up to one poll
|
||||
interval (`WORKER_POLL_SECONDS`, 2 s). A verdict (failed or not) is never
|
||||
retried; a dead lease is claimed again at once. The worker code is the
|
||||
same as before.
|
||||
- Measured (2026-11-01): the at-caps document (64 views, 1 024 annotations,
|
||||
8 192 nodes) verified in 185 ms by the worker on the sandbox, verdict
|
||||
24 689 bytes; the equine draft in about 60 ms.
|
||||
@@ -369,7 +377,8 @@ docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges:true
|
||||
-e SUPABASE_SERVICE_ROLE_KEY="$(pnpm -s supabase status -o env | sed -n 's/^SERVICE_ROLE_KEY="\(.*\)"/\1/p')" \
|
||||
faunosteo_worker:dev run
|
||||
pnpm --filter @faunosteo/supabase test:e2e # real-stack end-to-end (needs the worker above)
|
||||
pnpm e2e:avatar # scripts/avatar-e2e.sh: the three lines above in one, worker started with a 5 s sweep and no quarantine grace
|
||||
pnpm e2e:avatar # scripts/avatar-e2e.sh: the three lines above in one, worker started with a 5 s sweep and no quarantine grace (DEV: the owner's call)
|
||||
pnpm db:sandbox avatar-e2e # the same suite and worker on the throwaway stack (SUPABASE.md §5.1): the check of a storage or worker change
|
||||
```
|
||||
|
||||
`docker compose --profile worker up worker` runs the same container from
|
||||
|
||||
@@ -871,12 +871,13 @@ fingerprint or a document hash from a page.
|
||||
|
||||
Job. The round row is the job, with the avatar lease pattern of
|
||||
`20261013100000_files_foundation.sql` (lease token, `locked_at`, fixed lease,
|
||||
attempts below a limit), without backoff. Submission sets the round
|
||||
`verifying`. Worker RPCs, service role only (M6):
|
||||
attempts below a limit), plus a progressive delay for technical retries only
|
||||
(owner decision 2026-11-01). Submission sets the round `verifying`. Worker RPCs, service role only (M6):
|
||||
|
||||
- `claim_sheet_verifications(limit)`: `for update skip locked` over rounds
|
||||
`verifying` with no live lease; sets the token and `locked_at`, increments
|
||||
`attempts`; returns round, version, sheet, revision and token.
|
||||
`verifying` with no live lease and no pending delay (`not_before` null or
|
||||
reached); sets the token and `locked_at`, increments `attempts`, clears
|
||||
`not_before`; returns round, version, sheet, revision and token.
|
||||
- `get_sheet_verification_input(review_id, lease_token)`: the document at the
|
||||
round's revision (frozen) and the base document, both as text; the drawings
|
||||
the document references among the rows registered for the sheet (asset id,
|
||||
@@ -889,8 +890,17 @@ attempts below a limit), without backoff. Submission sets the round
|
||||
else answers `stale` or `malformed` and writes nothing. Moves the round to
|
||||
`open` or to `verification_failed` (§6.5) and audits.
|
||||
- `fail_sheet_verification(review_id, lease_token, code)`: a technical
|
||||
failure; the round stays claimable until the attempt limit, then becomes
|
||||
`verification_failed` with outcome `unavailable`.
|
||||
failure; the round stays `verifying` and becomes claimable after a delay
|
||||
that grows with the attempt (2 s after attempt 1, 5 s after 2, 10 s after 3,
|
||||
20 s after 4; `private.sheet_verification_retry_delay`), kept in the round's
|
||||
`not_before` (a column the API roles cannot read, set here, cleared by the
|
||||
claim); the fifth failure becomes `verification_failed` with outcome
|
||||
`unavailable` and clears it. Nothing is written for a stale call. The delay
|
||||
is persisted, so no worker sleeps and any worker honours it; the effective
|
||||
wait is the delay plus up to one poll interval of the worker (2 s by
|
||||
default). Only these technical failures wait: a validation or integrity
|
||||
refusal is a verdict and is never retried, and a dead lease is claimed again
|
||||
at once (the 10-minute lease is its wait).
|
||||
|
||||
Processing. Verification runs as its own mode of the worker image (`run
|
||||
--jobs sheet-verify`), in its own container under the `worker` compose
|
||||
@@ -951,9 +961,9 @@ did not pass; the closed codes keep the categories apart.
|
||||
|---|---|---|---|---|
|
||||
| Business validation | integrity error, caps exceeded, annotation without geometry, annotation outside its view's frame | `verification_failed`, outcome `failed`, codes `verification.validation.*` and the issues | no | fix the draft, submit again |
|
||||
| Integrity | strict parse refused, drawing missing, size or sha256 mismatch, SVG refused, needing a removal or outside the stored form (§9.2), `object_conflict` | `verification_failed`, outcome `failed`, codes `verification.integrity.*` | no | fix or upload again; `object_conflict` follows §9.3 |
|
||||
| Transient worker error | database or Storage unreachable, worker restart | stays `verifying`, code recorded | yes, up to the limit | automatic |
|
||||
| Timeout or resource limit | child over its wall-clock or memory limit | stays `verifying`, code recorded | yes, up to the limit | reduce the document or the drawing if it repeats |
|
||||
| Interruption | worker crash, lease expired | claimable again; counts as an attempt | yes, up to the limit | automatic |
|
||||
| Transient worker error | database or Storage unreachable, worker restart | stays `verifying`, code recorded | yes, after 2 s, 5 s, 10 s, 20 s, up to the limit | automatic |
|
||||
| Timeout or resource limit | child over its wall-clock or memory limit | stays `verifying`, code recorded | yes, same delays, up to the limit | reduce the document or the drawing if it repeats |
|
||||
| Interruption | worker crash, lease expired | claimable again at once (the lease was the wait); counts as an attempt | yes, up to the limit | automatic |
|
||||
| Stale result | completion with an old token, after a withdrawal, or naming another revision | nothing written, logged | no | none needed |
|
||||
| Definitive failure | attempt limit reached (five; V1-2.3 implements five, for the owner's confirmation) | `verification_failed`, outcome `unavailable`, last code | no | submit again later |
|
||||
| Core change | release that changes rules or reader | rounds withdrawn by the release migration | no | submit again |
|
||||
@@ -1705,7 +1715,7 @@ of the document's JSON text in one form, the same for every layer:
|
||||
|---|---|---|---|
|
||||
| M4 `sheet_access_hardening` | V1-2.1 | `anatomical_catalog_grants` and its RPCs; `has_mfa`, `grant_active`, `caller_has`; `can_read_official_catalog` and the reader row predicate; `can_edit_sheet` redefined (F14); AR-1 B on the platform role RPCs and the private write functions (F19, §5.7), F17, F18; F1, F2, F3 (fork closed), F5, F6, F8, F9, F12 (every catalogue table), F13, F15; asset-row immutability trigger (F16); historical-sheet protection trigger (§14.2); reader column grants narrowed; personal creation and visibility changes refused | no row changes |
|
||||
| M5 `sheet_review_workflow` | V1-2.1 | version columns `revision`, `authors`, `source_version_id`, `published_review_id`, `document_fingerprint`; sheet column `working_copy_source_version_id`; revision, author and freeze triggers, the historical-sheet write rule; extended immutability guard (§6.8); rounds, comments, acknowledgments; `create_official_sheet`, `create_sheet_draft` adapted, `save_sheet_draft`, submit, withdraw, comment, acknowledge, decide, publish; `create_sheet_working_copy`; species RPCs; catalogue and version read RPCs; upload: `register_sheet_asset` extended, `confirm_sheet_asset_upload` with the storage check (F7), INSERT policy, bucket limit 2 MiB; mirror removal and status check (§12.2); superseded signatures dropped; the content-size CHECK replaced by the text measure (`octet_length(content::text)`, §12.2) | new columns take their defaults on the equine v1 row (`revision` 0, empty author set); its document, status, number and assets unchanged; the new size CHECK is validated against the existing rows when M5 is applied (the equine document measures 184 105 bytes as `content::text`, measured on the sandbox); bucket limit lowered; no round, no grant |
|
||||
| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only; written as `20261101100000_sheet_verification_jobs.sql`, pgTAP `021`, rehearsed on the sandbox from the state before M4 (M4 to M6 in one `migration up`) and from the state before M6 | none |
|
||||
| M6 `sheet_verification_jobs` | V1-2.3 | worker RPCs (claim, input, complete with the verdict structure check, fail), service role only, with the retry delay (hidden `not_before` column on the round, `private.sheet_verification_retry_delay`: owner decision 2026-11-01, replaces « without backoff »); written as `20261101100000_sheet_verification_jobs.sql`, pgTAP `021`, rehearsed on the sandbox from the state before M4 (M4 to M6 in one `migration up`) and from the state before M6 | none |
|
||||
|
||||
Each migration ships its pgTAP file (allowed and denied cases), regenerated
|
||||
`database.types.ts`, a `db:sandbox up upgrade` run from the DEV state (with a
|
||||
@@ -2021,8 +2031,9 @@ The final approval needs all 30 groups acknowledged, no `to_confirm` and no
|
||||
- Worker: the verification mode, verdicts accepted and each failure, the
|
||||
hostile SVG corpus in the sandboxed child, per-command limits and watchdog,
|
||||
the purge guard by bucket, sandbox tests and seccomp derivation, an e2e on
|
||||
the sandbox stack with the verification worker running; `pnpm e2e:avatar`
|
||||
unchanged in outcome; `pnpm e2e:sheet-assets` targets DEV only, its rerun
|
||||
the sandbox stack with the verification worker running; the avatar suite
|
||||
unchanged in outcome on the sandbox (`pnpm db:sandbox avatar-e2e`, owner
|
||||
decision 2026-11-01: `pnpm e2e:avatar` targets DEV and is not run); `pnpm e2e:sheet-assets` targets DEV only, its rerun
|
||||
is an owner step (§16.11).
|
||||
- Release replay: a script loads every published version and drawing of a
|
||||
stack through the new core and fails on any regression.
|
||||
|
||||
@@ -227,6 +227,7 @@ export type Database = {
|
||||
id: string
|
||||
lease_token: string | null
|
||||
locked_at: string | null
|
||||
not_before: string | null
|
||||
revision: number
|
||||
round: number
|
||||
status: Database['public']['Enums']['sheet_round_status']
|
||||
@@ -256,6 +257,7 @@ export type Database = {
|
||||
id?: string
|
||||
lease_token?: string | null
|
||||
locked_at?: string | null
|
||||
not_before?: string | null
|
||||
revision: number
|
||||
round: number
|
||||
status?: Database['public']['Enums']['sheet_round_status']
|
||||
@@ -284,6 +286,7 @@ export type Database = {
|
||||
id?: string
|
||||
lease_token?: string | null
|
||||
locked_at?: string | null
|
||||
not_before?: string | null
|
||||
revision?: number
|
||||
round?: number
|
||||
status?: Database['public']['Enums']['sheet_round_status']
|
||||
|
||||
@@ -100,6 +100,7 @@ const expectedColumns = {
|
||||
'id',
|
||||
'lease_token',
|
||||
'locked_at',
|
||||
'not_before',
|
||||
'revision',
|
||||
'round',
|
||||
'status',
|
||||
|
||||
@@ -229,6 +229,112 @@ const avatarsAndPurges = () =>
|
||||
|| ' ' ||
|
||||
(select count(*) || ':' || md5(coalesce(string_agg(row_to_json(q)::text, ',' order by q.id), '')) from private.object_purge_queue q)`)
|
||||
|
||||
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/
|
||||
/** An id read back from the stack, checked before it is interpolated into SQL. */
|
||||
function uuidOf(value: unknown): string {
|
||||
if (typeof value !== 'string' || !UUID.test(value)) throw new Error(`not a uuid: ${String(value)}`)
|
||||
return value
|
||||
}
|
||||
|
||||
/** `octet_length(value::jsonb::text)` of a value whose strings need no JSON escape (crates/core/src/domain/sheet/size.rs). */
|
||||
function jsonbBytes(value: unknown): number {
|
||||
if (value === null) return 4
|
||||
if (typeof value === 'boolean') return value ? 4 : 5
|
||||
if (typeof value === 'number') return String(value).length
|
||||
if (typeof value === 'string') return Buffer.byteLength(value) + 2
|
||||
if (Array.isArray(value)) return 2 + value.reduce((n: number, v) => n + jsonbBytes(v), 0) + 2 * Math.max(0, value.length - 1)
|
||||
const entries = Object.entries(value as Record<string, unknown>)
|
||||
return 2 + entries.reduce((n, [k, v]) => n + jsonbBytes(k) + 2 + jsonbBytes(v), 0) + 2 * Math.max(0, entries.length - 1)
|
||||
}
|
||||
|
||||
/**
|
||||
* A drawing of exactly `size` bytes in the stored form (`serialize::stored_form`): the UTF-8
|
||||
* prolog and one `<svg>` holding one `<path>` whose `d` is padded with `L5 5` segments and
|
||||
* spaces. Canonical because it is what the core writes back byte for byte: ASCII only, the
|
||||
* namespace declared on the root, attributes in a fixed order, every value free of `& < > "`
|
||||
* and of tab or line feed, no comment, no text node, the element self-closed; the pad adds
|
||||
* no element, attribute, `<` or `=` (SVG_CAPS: 20 000 elements, 100 000 `<`, 200 000 `=`).
|
||||
*/
|
||||
function sizedDrawing(size: number, tag: string): Buffer {
|
||||
const head = '<?xml version="1.0" encoding="UTF-8"?>\n<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 10 10">' +
|
||||
`<path id="p${tag}" d="M0 0L5 5`
|
||||
const tail = '"/></svg>'
|
||||
const pad = size - head.length - tail.length
|
||||
const text = head + ' L5 5'.repeat(Math.floor(pad / 5)) + ' '.repeat(pad % 5) + tail
|
||||
expect(text.length).toBe(size)
|
||||
return Buffer.from(text, 'ascii')
|
||||
}
|
||||
|
||||
/** `main` on the first drawing, one detail view per other drawing, each opened by a trigger on `main`. */
|
||||
function multiDoc(assets: string[], descriptionFr = 'd') {
|
||||
const doc = smallDoc(assets[0]) as Record<string, unknown>
|
||||
doc.description = { fr: descriptionFr, en: 'd' }
|
||||
const views = doc.views as unknown[]
|
||||
const annotations = doc.annotations as unknown[]
|
||||
assets.slice(1).forEach((asset, i) => {
|
||||
views.push({ id: `d${i}`, kind: 'detail', name: name('D'), svg_asset_id: asset, view_box: null, parent_view_id: 'main' })
|
||||
annotations.push({ id: `t${i}`, kind: 'detail_trigger', view_id: 'main', category_id: 'c', laterality: 'unpaired',
|
||||
geometry: { type: 'polygon', points: [[0, 0], [1, 0], [1, 1]] }, review_status: 'reviewed', name: null,
|
||||
opens_view_id: `d${i}` })
|
||||
})
|
||||
return doc
|
||||
}
|
||||
|
||||
/** `multiDoc` whose `octet_length(content::text)` is exactly `bytes`; `fill` makes two such documents differ. */
|
||||
function documentOfBytes(assets: string[], bytes: number, fill = 'x'): unknown {
|
||||
const missing = bytes - jsonbBytes(multiDoc(assets))
|
||||
expect(missing).toBeGreaterThanOrEqual(0)
|
||||
return multiDoc(assets, 'd' + fill.repeat(missing))
|
||||
}
|
||||
|
||||
/** Submits `d` until its round is `verifying` with `attempts` claims already spent and a lease held for another worker (SQL stand-in). */
|
||||
async function holdLease(editor: SteppedAccount, d: Draft, attempts: number): Promise<{ review: string; token: string }> {
|
||||
const token = randomUUID()
|
||||
for (let turn = 0; turn < 5; turn++) {
|
||||
const id = uuidOf(await submit(editor, d))
|
||||
const held = await sql(`update public.anatomical_sheet_reviews set lease_token = '${token}', locked_at = now(), attempts = ${attempts}
|
||||
where id = '${id}' and status = 'verifying' and lease_token is null returning id`)
|
||||
if (held === id) return { review: id, token }
|
||||
await settled(id)
|
||||
ok(await rpc('withdraw_sheet_review', { review_id: id, note: 'raced' }, editor.aal2))
|
||||
}
|
||||
throw new Error('lease not held before the worker claimed')
|
||||
}
|
||||
|
||||
/** Polls a one-value query until it answers `expected`. */
|
||||
async function waitFor(query: string, expected: string, timeoutMs: number): Promise<void> {
|
||||
const start = performance.now()
|
||||
let last = ''
|
||||
while (performance.now() - start < timeoutMs) {
|
||||
last = await sql(query)
|
||||
if (last === expected) return
|
||||
await new Promise((resolve) => setTimeout(resolve, 200))
|
||||
}
|
||||
throw new Error(`expected ${expected}, last ${last}`)
|
||||
}
|
||||
|
||||
/** Registers, uploads and confirms one drawing per size on `sheet`; the ids in order. */
|
||||
async function putDrawings(editor: SteppedAccount, sheet: string, sizes: number[]): Promise<{ id: string; bytes: Buffer }[]> {
|
||||
const out: { id: string; bytes: Buffer }[] = []
|
||||
for (const [i, size] of sizes.entries()) {
|
||||
const bytes = sizedDrawing(size, `${i}${randomUUID().replace(/-/g, '')}`)
|
||||
out.push({ id: await putDrawing(editor, sheet, bytes), bytes })
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
/** Acknowledges every group of the open round, approves it and publishes (the control of case d). */
|
||||
async function publishRound(editor: SteppedAccount, reviewer: SteppedAccount, review: string, r: Round): Promise<void> {
|
||||
for (const group of r.verdict?.groups ?? [])
|
||||
ok(await rpc('acknowledge_sheet_review_group',
|
||||
{ review_id: review, group_key: group.key, group_digest: group.digest }, reviewer.aal2))
|
||||
ok(await rpc('decide_sheet_review', { review_id: review, approve: true, note: 'ok' }, reviewer.aal2))
|
||||
ok(await rpc('publish_sheet_version', { review_id: review }, editor.aal2))
|
||||
}
|
||||
|
||||
const verdictBytes = async (review: string) =>
|
||||
Number(await sql(`select octet_length(verdict::text) from public.anatomical_sheet_reviews where id = '${review}'`))
|
||||
|
||||
const timings: Record<string, string> = {}
|
||||
|
||||
describe.skipIf(!WORKER)('sheet verification by the real worker (V1-2.3)', () => {
|
||||
@@ -471,6 +577,149 @@ describe.skipIf(!WORKER)('sheet verification by the real worker (V1-2.3)', () =>
|
||||
expect(ms).toBeLessThan(CAPS_BUDGET_MS)
|
||||
}, CAPS_BUDGET_MS + 60_000)
|
||||
|
||||
it('i: a technical failure is retried no earlier than its delay, the worker serving another round meanwhile', async () => {
|
||||
const d = await newSheet(E)
|
||||
const bytes = storedDrawing()
|
||||
const asset = await putDrawing(E, d.sheet, bytes)
|
||||
await save(E, d, smallDoc(asset))
|
||||
// Three attempts spent and the lease held for another worker: the failure is the third (10 s delay).
|
||||
const { review, token } = await holdLease(E, d, 3)
|
||||
const failed = await rpc('fail_sheet_verification',
|
||||
{ review_id: review, lease_token: token, code: 'verification.worker.timeout' }, stack.serviceKey)
|
||||
expect(failed.body).toBe('retry')
|
||||
// not_before = now() of the failing transaction + 10 s: the failure time in SQL's clock.
|
||||
const failAt = Number(await sql(`select extract(epoch from not_before - interval '10 seconds')
|
||||
from public.anatomical_sheet_reviews where id = '${review}'`))
|
||||
expect(failAt).toBeGreaterThan(0)
|
||||
|
||||
// Meanwhile the worker serves another round: a clean sheet opens.
|
||||
const other = await newSheet(E)
|
||||
const otherBytes = storedDrawing()
|
||||
const otherAsset = await putDrawing(E, other.sheet, otherBytes)
|
||||
await save(E, other, smallDoc(otherAsset))
|
||||
const served = await settled(await submit(E, other))
|
||||
expectPassedFor(served.round, other, [{ id: otherAsset, sha: sha256(otherBytes) }])
|
||||
// The first round waits: verifying, no lease, three attempts, delay not over.
|
||||
expect(await sql(`select status || '/' || attempts || '/' || (lease_token is null) || '/' || (clock_timestamp() < not_before)
|
||||
from public.anatomical_sheet_reviews where id = '${review}'`)).toBe('verifying/3/true/true')
|
||||
|
||||
const retried = await settled(review, 40_000)
|
||||
expectPassedFor(retried.round, d, [{ id: asset, sha: sha256(bytes) }])
|
||||
expect(retried.round.attempts).toBe(4)
|
||||
const gap = Number(await sql(`select extract(epoch from verified_at) - ${failAt}
|
||||
from public.anatomical_sheet_reviews where id = '${review}'`))
|
||||
expect(gap).toBeGreaterThanOrEqual(10)
|
||||
timings.i_retry = `claimed and verified ${gap.toFixed(1)} s after the failure (delay 10 s, poll 2 s), other round opened in ${served.ms} ms`
|
||||
}, 90_000)
|
||||
|
||||
describe('input limits (Q3)', () => {
|
||||
let L: SteppedAccount
|
||||
const MIB = 1024 * 1024
|
||||
|
||||
beforeAll(async () => {
|
||||
L = await holder('vf-limits', 'admin', ['sheet_editor', 'sheet_publisher'])
|
||||
}, 60_000)
|
||||
|
||||
it('j: the largest input the policy allows (document 2 MiB, drawings 2+2+1+1 MiB, base 2 MiB) is verified within the budget', async () => {
|
||||
const d = await newSheet(L)
|
||||
const drawings = await putDrawings(L, d.sheet, [2 * MIB, 2 * MIB, MIB, MIB])
|
||||
const ids = drawings.map((x) => x.id)
|
||||
const first = documentOfBytes(ids, 2 * MIB, 'x')
|
||||
await save(L, d, first)
|
||||
expect(await sql(`select octet_length(content::text) from public.anatomical_sheet_versions where id = '${d.draft}'`)).toBe(String(2 * MIB))
|
||||
const review = await submit(L, d)
|
||||
const { round: r, ms } = await settled(review, CAPS_BUDGET_MS)
|
||||
// The verdict lists the drawings in document order: both sides are compared sorted by asset id.
|
||||
const expected = drawings.map((x) => ({ id: x.id, sha: sha256(x.bytes) })).sort((a, b) => a.id.localeCompare(b.id))
|
||||
const byAsset = (x: Round): Round => ({ ...x, verdict: { ...(x.verdict as Verdict),
|
||||
drawings: [...(x.verdict as Verdict).drawings].sort((a, b) => a.asset_id.localeCompare(b.asset_id)) } })
|
||||
expectPassedFor(byAsset(r), d, expected)
|
||||
const v = r.verdict as Verdict
|
||||
expect(v.groups.map((g) => g.key)).toEqual(expect.arrayContaining(['general', 'view:main', 'view:d0', 'view:d1', 'view:d2']))
|
||||
const worker = await workerDuration(review)
|
||||
const size = await verdictBytes(review)
|
||||
expect(size).toBeLessThan(3_145_728)
|
||||
timings.j_at_limits = `${ms} ms from submission (worker ${worker} ms), document ${2 * MIB} bytes, drawings ${drawings.map((x) => x.bytes.length).join('+')} bytes, verdict ${size} bytes`
|
||||
expect(ms).toBeLessThan(CAPS_BUDGET_MS)
|
||||
|
||||
// Second round: its base is the published 2 MiB document (frame: 2 + 2 + 6 MiB).
|
||||
await publishRound(L, V, review, r)
|
||||
const draft = uuidOf(ok(await rpc('create_sheet_draft', { sheet_id: d.sheet, from_version_id: null, change_note: 'next' }, L.aal2)))
|
||||
const next: Draft = { sheet: d.sheet, draft, revision: Number(await sql(`select revision from public.anatomical_sheet_versions where id = '${draft}'`)) }
|
||||
expect(await sql(`select (source_version_id = '${d.draft}')::text from public.anatomical_sheet_versions where id = '${draft}'`)).toBe('true')
|
||||
await save(L, next, documentOfBytes(ids, 2 * MIB, 'y'))
|
||||
const second = await submit(L, next)
|
||||
const again = await settled(second, CAPS_BUDGET_MS)
|
||||
expectPassedFor(byAsset(again.round), next, expected)
|
||||
const size2 = await verdictBytes(second)
|
||||
expect(size2).toBeLessThan(3_145_728)
|
||||
timings.j_with_base = `${again.ms} ms from submission (worker ${await workerDuration(second)} ms), document and base ${2 * MIB} bytes each, drawings 6 MiB, verdict ${size2} bytes`
|
||||
expect(again.ms).toBeLessThan(CAPS_BUDGET_MS)
|
||||
}, 3 * CAPS_BUDGET_MS)
|
||||
|
||||
it('k: just beyond the limits: refused before the worker, or a business failure; the tracked misclassification', async () => {
|
||||
// A document of 2 097 153 bytes is refused at save.
|
||||
const d = await newSheet(L)
|
||||
const small = await putDrawing(L, d.sheet, storedDrawing())
|
||||
const save1 = await rpc('save_sheet_draft',
|
||||
{ version_id: d.draft, expected_revision: d.revision, document: documentOfBytes([small], 2 * MIB + 1), change_note: 'e2e' }, L.aal2)
|
||||
expect(message(save1)).toBe('document.too_large')
|
||||
|
||||
// A drawing of 2 097 153 bytes is refused at registration, and by the bucket whatever the row says.
|
||||
const big = sizedDrawing(2 * MIB + 1, randomUUID().replace(/-/g, ''))
|
||||
expect(message(await rpc('register_sheet_asset',
|
||||
{ sheet_id: d.sheet, sha256: sha256(big), byte_size: big.length, file_name: 'big.svg' }, L.aal2))).toBe('invalid_input')
|
||||
// Registered with the policy's size, the object itself over it: the bucket's file size limit refuses it.
|
||||
ok(await rpc('register_sheet_asset',
|
||||
{ sheet_id: d.sheet, sha256: sha256(big), byte_size: 2 * MIB, file_name: 'big.svg' }, L.aal2))
|
||||
for (const token of [L.aal2, stack.serviceKey]) {
|
||||
const res = await fetch(`${stack.storageUrl}/object/${BUCKET}/${sha256(big)}.svg`, {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'image/svg+xml' },
|
||||
body: new Uint8Array(big),
|
||||
})
|
||||
const body = await res.text()
|
||||
console.log(`bucket upload of 2 097 153 bytes: HTTP ${res.status} ${body}`)
|
||||
expect([res.status, JSON.parse(body).statusCode]).toEqual([400, '413'])
|
||||
}
|
||||
expect(await objectExists(sha256(big))).toBe(false)
|
||||
|
||||
// Four drawings of 2 MiB (8 MiB, over the 6 MiB total): a business failure, no worker code.
|
||||
const four = await putDrawings(L, d.sheet, [2 * MIB, 2 * MIB, 2 * MIB, 2 * MIB])
|
||||
await save(L, d, multiDoc(four.map((x) => x.id)))
|
||||
const overTotal = await settled(await submit(L, d), CAPS_BUDGET_MS)
|
||||
expect(overTotal.round.status).toBe('verification_failed')
|
||||
expect([overTotal.round.failure_outcome, overTotal.round.attempts]).toEqual(['failed', 1])
|
||||
expect(overTotal.round.failure_code).not.toMatch(/^verification\.worker\./)
|
||||
expect(overTotal.round.verdict?.failures).toContain('verification.validation.caps')
|
||||
timings.k_over_total = `${overTotal.ms} ms, codes ${overTotal.round.verdict?.failures.join(',')}`
|
||||
|
||||
// Five drawings (over the count of four): the same business failure.
|
||||
const five = await putDrawings(L, d.sheet, [300, 301, 302, 303, 304])
|
||||
await save(L, d, multiDoc(five.map((x) => x.id)))
|
||||
const overCount = await settled(await submit(L, d))
|
||||
expect(overCount.round.status).toBe('verification_failed')
|
||||
expect([overCount.round.failure_outcome, overCount.round.attempts]).toEqual(['failed', 1])
|
||||
expect(overCount.round.verdict?.failures).toContain('verification.validation.caps')
|
||||
timings.k_over_count = `${overCount.ms} ms, codes ${overCount.round.verdict?.failures.join(',')}`
|
||||
|
||||
// Tracked misclassification (owner brief 2026-11-01: caps expected; not fixed): a document naming
|
||||
// so many registered drawings (600 rows, about 160 bytes each in the header) that the frame header passes 64 KiB is reported resource_limit.
|
||||
const many = await newSheet(L)
|
||||
const rows = (await sql(`insert into public.anatomical_sheet_assets (sheet_id, sha256, byte_size, file_name, created_by, uploaded_at)
|
||||
select '${uuidOf(many.sheet)}', md5(g::text) || md5(g::text || 'h'), 900, 'h' || g || '.svg', '${uuidOf(L.id)}', now()
|
||||
from generate_series(1, 600) g returning id`)).split('\n').map(uuidOf)
|
||||
expect(rows).toHaveLength(600)
|
||||
await save(L, many, multiDoc(rows))
|
||||
const header = uuidOf(await submit(L, many))
|
||||
await waitFor(`select failure_code || '/' || attempts from public.anatomical_sheet_reviews where id = '${header}'`,
|
||||
'verification.worker.resource_limit/1', 30_000)
|
||||
// Withdrawn so that the worker stops retrying it (its next claim is 2 s away).
|
||||
ok(await rpc('withdraw_sheet_review', { review_id: header, note: 'tracked misclassification' }, L.aal2))
|
||||
expect((await round(header)).status).toBe('withdrawn')
|
||||
}, 5 * CAPS_BUDGET_MS)
|
||||
})
|
||||
|
||||
it('f: avatar rows and the purge queue are untouched by the verification worker', async () => {
|
||||
expect(await avatarsAndPurges()).toBe(before)
|
||||
expect(await sql(`select status || ' ' || attempts || ' ' || (locked_at is null)::text from public.assets
|
||||
|
||||
@@ -4,7 +4,10 @@
|
||||
--
|
||||
-- The round row (M5 `anatomical_sheet_reviews`) is the job, with the avatar
|
||||
-- lease pattern of 20261013100000_files_foundation.sql (lease token,
|
||||
-- `locked_at`, a fixed lease, attempts below a limit), without backoff:
|
||||
-- `locked_at`, a fixed lease, attempts below a limit), plus a progressive
|
||||
-- delay for technical retries only (owner decision 2026-11-01): a failure
|
||||
-- reported by the worker sets `not_before`, the claim skips the round until
|
||||
-- then, so no worker sleeps and any worker honours it:
|
||||
-- * claim_sheet_verifications: rounds `verifying` with no live lease;
|
||||
-- * get_sheet_verification_input: the frozen document, the base, the rows of
|
||||
-- the drawings the document references;
|
||||
@@ -17,11 +20,28 @@
|
||||
-- ===========================================================================
|
||||
-- 1. Constants
|
||||
-- ===========================================================================
|
||||
-- The earliest time a technical retry may be claimed (null = now). Hidden:
|
||||
-- the M5 grant to `authenticated` is column-level and does not list it.
|
||||
alter table public.anatomical_sheet_reviews add column not_before timestamptz;
|
||||
comment on column public.anatomical_sheet_reviews.not_before is
|
||||
'Earliest time a technical retry may be claimed; null = now. Set by fail_sheet_verification, cleared by the claim. Service role only.';
|
||||
|
||||
-- Attempts before a round closes `verification_failed` / `unavailable`
|
||||
-- (§6.5 « five, proposal, tuned in V1-2.3 »).
|
||||
create or replace function private.sheet_verification_attempt_limit()
|
||||
returns integer language sql immutable set search_path = '' as $$ select 5 $$;
|
||||
|
||||
-- Wait before the next claim after a technical failure at attempt n: 2 s, 5 s,
|
||||
-- 10 s, then 20 s (attempt 1 itself is claimed at once). Validation and
|
||||
-- integrity refusals are verdicts, never retried.
|
||||
create or replace function private.sheet_verification_retry_delay(attempts integer)
|
||||
returns interval language sql immutable set search_path = '' as $$
|
||||
select case when attempts <= 1 then interval '2 seconds'
|
||||
when attempts = 2 then interval '5 seconds'
|
||||
when attempts = 3 then interval '10 seconds'
|
||||
else interval '20 seconds' end
|
||||
$$;
|
||||
|
||||
-- A lease older than this is dead: the worker crashed or lost the database.
|
||||
-- The child's wall-clock deadline is far below it (crates/worker limits).
|
||||
create or replace function private.sheet_verification_lease()
|
||||
@@ -191,12 +211,14 @@ begin
|
||||
where r.status = 'verifying'
|
||||
and (r.locked_at is null or r.locked_at < now() - private.sheet_verification_lease())
|
||||
and r.attempts < private.sheet_verification_attempt_limit()
|
||||
and (r.not_before is null or r.not_before <= now())
|
||||
order by r.submitted_at
|
||||
limit v_limit
|
||||
for update skip locked
|
||||
), leased as (
|
||||
update public.anatomical_sheet_reviews r
|
||||
set lease_token = gen_random_uuid(), locked_at = now(), attempts = r.attempts + 1,
|
||||
not_before = null,
|
||||
-- A dead lease left no code: record the interruption it was.
|
||||
failure_code = case when r.locked_at is not null then 'verification.worker.lease_expired' else r.failure_code end
|
||||
from picked
|
||||
@@ -344,8 +366,8 @@ end;
|
||||
$$;
|
||||
|
||||
-- A technical failure of a leased round (§6.5): the round stays `verifying`
|
||||
-- and claimable, its code recorded, until the attempt limit closes it
|
||||
-- `verification_failed` / `unavailable`. Answers `retry`, `unavailable` or
|
||||
-- and claimable after its retry delay, its code recorded, until the attempt
|
||||
-- limit closes it `verification_failed` / `unavailable`. Answers `retry`, `unavailable` or
|
||||
-- `stale` (nothing written).
|
||||
create or replace function public.fail_sheet_verification(review_id uuid, lease_token uuid, code text)
|
||||
returns text
|
||||
@@ -370,6 +392,8 @@ begin
|
||||
v_final := v_review.attempts >= private.sheet_verification_attempt_limit();
|
||||
update public.anatomical_sheet_reviews r
|
||||
set failure_code = code, lease_token = null, locked_at = null,
|
||||
not_before = case when v_final then null
|
||||
else now() + private.sheet_verification_retry_delay(v_review.attempts) end,
|
||||
status = case when v_final then 'verification_failed'::public.sheet_round_status else r.status end,
|
||||
failure_outcome = case when v_final then 'unavailable'::public.sheet_verification_outcome else null end
|
||||
where r.id = v_review.id;
|
||||
@@ -377,7 +401,9 @@ begin
|
||||
case when v_final then 'anatomical_sheet_review.verification_failed' else 'anatomical_sheet_review.verification_attempt_failed' end,
|
||||
'anatomical_sheet_review', v_review.id, null, null,
|
||||
jsonb_build_object('code', code, 'attempts', v_review.attempts,
|
||||
'outcome', case when v_final then 'unavailable' end));
|
||||
'outcome', case when v_final then 'unavailable' end,
|
||||
'retry_delay_seconds', case when not v_final then
|
||||
extract(epoch from private.sheet_verification_retry_delay(v_review.attempts))::integer end));
|
||||
return case when v_final then 'unavailable' else 'retry' end;
|
||||
end;
|
||||
$$;
|
||||
@@ -386,6 +412,7 @@ $$;
|
||||
-- 4. Privileges: the service role only
|
||||
-- ===========================================================================
|
||||
revoke all on function private.sheet_verification_attempt_limit() from public;
|
||||
revoke all on function private.sheet_verification_retry_delay(integer) from public;
|
||||
revoke all on function private.sheet_verification_lease() from public;
|
||||
revoke all on function private.sheet_verification_max_verdict_bytes() from public;
|
||||
revoke all on function private.sheet_verdict_codes() from public;
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
-- V1-2.3 M6: the worker RPCs of the server verdict (contract §6.4, §6.5).
|
||||
-- Service role only; claim with a lease, input bound to the live token,
|
||||
-- completion checked for its binding and structure (stale / malformed write
|
||||
-- nothing), technical failures up to the attempt limit, dead leases.
|
||||
-- nothing), technical failures up to the attempt limit with their progressive
|
||||
-- delay (`not_before`), dead leases.
|
||||
begin;
|
||||
select plan(51);
|
||||
select plan(68);
|
||||
|
||||
insert into auth.users (id, email, email_confirmed_at) values
|
||||
('f0000000-0000-0000-0000-000000000001', 'own@vj.local', now()),
|
||||
@@ -75,6 +76,20 @@ create function pg_temp.token(p text) returns uuid language sql security definer
|
||||
select lease_token from public.anatomical_sheet_reviews where id = pg_temp.id(p);
|
||||
$$;
|
||||
grant execute on function pg_temp.token(text) to service_role;
|
||||
-- `not_before` is hidden from the API roles: the helpers read and move it as
|
||||
-- the test owner. now() is the transaction's, so the delay compares exactly.
|
||||
create function pg_temp.delayed(p text, p_delay interval) returns boolean language sql security definer as $$
|
||||
select not_before = now() + p_delay from public.anatomical_sheet_reviews where id = pg_temp.id(p);
|
||||
$$;
|
||||
grant execute on function pg_temp.delayed(text, interval) to service_role;
|
||||
create function pg_temp.not_before_is_null(p text) returns boolean language sql security definer as $$
|
||||
select not_before is null from public.anatomical_sheet_reviews where id = pg_temp.id(p);
|
||||
$$;
|
||||
grant execute on function pg_temp.not_before_is_null(text) to service_role;
|
||||
create function pg_temp.set_not_before(p text, p_offset interval) returns void language sql security definer as $$
|
||||
update public.anatomical_sheet_reviews set not_before = now() + p_offset where id = pg_temp.id(p);
|
||||
$$;
|
||||
grant execute on function pg_temp.set_not_before(text, interval) to service_role;
|
||||
|
||||
-- A sheet with one confirmed drawing; three drafts submitted (r1, r2, r3).
|
||||
set local role authenticated;
|
||||
@@ -103,6 +118,12 @@ select ok(has_function_privilege('service_role', 'public.claim_sheet_verificatio
|
||||
and has_function_privilege('service_role', 'public.complete_sheet_verification(uuid, uuid, jsonb)', 'execute')
|
||||
and has_function_privilege('service_role', 'public.fail_sheet_verification(uuid, uuid, text)', 'execute'),
|
||||
'the service role executes the four RPCs');
|
||||
select ok(not has_column_privilege('authenticated', 'public.anatomical_sheet_reviews', 'not_before', 'select')
|
||||
and not has_column_privilege('anon', 'public.anatomical_sheet_reviews', 'not_before', 'select'),
|
||||
'the API roles cannot read not_before');
|
||||
select ok(not has_function_privilege('authenticated', 'private.sheet_verification_retry_delay(integer)', 'execute'), 'authenticated cannot execute the delay constant');
|
||||
select is((select array_agg(extract(epoch from private.sheet_verification_retry_delay(n))::int order by n) from generate_series(1,5) n),
|
||||
array[2,5,10,20,20], 'the delay after attempts 1 to 5: 2, 5, 10, 20, 20 seconds');
|
||||
set local role authenticated;
|
||||
select pg_temp.act('f0000000-0000-0000-0000-000000000001','f1000000-0000-0000-0000-000000000001');
|
||||
select throws_ok($$ select public.claim_sheet_verifications(1) $$, '42501', null, 'the owner (authenticated) cannot claim');
|
||||
@@ -175,6 +196,7 @@ select is((select count(*)::int from public.claim_sheet_verifications(1)), 1, 's
|
||||
select is(public.complete_sheet_verification(pg_temp.id('r2'), pg_temp.token('r2'),
|
||||
pg_temp.verdict(pg_temp.id('r2'),'failed','verification.integrity.document')), 'failed', 'a failed verdict without groups is written');
|
||||
select is(pg_temp.status('r2'), 'verification_failed/failed/verification.integrity.document', 'round closed failed with its code');
|
||||
select is((select count(*)::int from public.claim_sheet_verifications(10)), 0, 'a round completed with a failed verdict is never claimed again');
|
||||
|
||||
-- 7. Technical failures: retry below the limit, unavailable at it.
|
||||
reset role;
|
||||
@@ -190,19 +212,37 @@ select throws_ok(format('select public.fail_sheet_verification(%L, %L, %L)', pg_
|
||||
select is(public.fail_sheet_verification(pg_temp.id('r3'), gen_random_uuid(), 'verification.worker.timeout'), 'stale', 'a failure with another token is stale');
|
||||
select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.timeout'), 'retry', 'a timeout below the limit is retried');
|
||||
select is(pg_temp.status('r3') || '/' || (pg_temp.token('r3') is null)::text, 'verifying/-/verification.worker.timeout/true', 'round stays verifying, code recorded, lease released');
|
||||
-- A dead lease is claimed again and recorded as an interruption.
|
||||
select is((select attempts from public.claim_sheet_verifications(1)), 2, 'reclaimed: second attempt');
|
||||
select ok(pg_temp.delayed('r3', interval '2 seconds'), 'after attempt 1 the retry waits 2 s');
|
||||
reset role;
|
||||
select ok(exists (select 1 from audit.audit_logs where action = 'anatomical_sheet_review.verification_attempt_failed' and resource_id = pg_temp.id('r3')
|
||||
and metadata ->> 'retry_delay_seconds' = '2'), 'attempt failure audited with its delay');
|
||||
set local role service_role; select pg_temp.worker();
|
||||
select is(public.fail_sheet_verification(pg_temp.id('r3'), gen_random_uuid(), 'verification.worker.timeout'), 'stale', 'a stale failure again');
|
||||
select ok(pg_temp.delayed('r3', interval '2 seconds'), 'a stale failure leaves the delay unchanged');
|
||||
select is((select count(*)::int from public.claim_sheet_verifications(1)), 0, 'no claim before the delay is over');
|
||||
-- The delay over exactly now: claimable.
|
||||
select pg_temp.set_not_before('r3', interval '0');
|
||||
select is((select attempts from public.claim_sheet_verifications(1)), 2, 'claimed once not_before is reached: second attempt');
|
||||
select ok(pg_temp.not_before_is_null('r3'), 'the claim clears the delay');
|
||||
select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.unavailable'), 'retry', 'a second failure is retried');
|
||||
select ok(pg_temp.delayed('r3', interval '5 seconds'), 'after attempt 2 the retry waits 5 s');
|
||||
select pg_temp.set_not_before('r3', interval '-1 second');
|
||||
select is((select attempts from public.claim_sheet_verifications(1)), 3, 'claimed after 5 s: third attempt');
|
||||
-- A dead lease is claimed again at once and recorded as an interruption.
|
||||
reset role;
|
||||
update public.anatomical_sheet_reviews set locked_at = now() - interval '11 minutes' where id = pg_temp.id('r3');
|
||||
set local role service_role; select pg_temp.worker();
|
||||
select is((select attempts from public.claim_sheet_verifications(1)), 3, 'a dead lease is claimed again');
|
||||
select is((select attempts from public.claim_sheet_verifications(1)), 4, 'a dead lease is claimed again');
|
||||
select is(pg_temp.status('r3'), 'verifying/-/verification.worker.lease_expired', 'the interruption is recorded');
|
||||
-- At the limit, a technical failure closes the round unavailable.
|
||||
reset role;
|
||||
update public.anatomical_sheet_reviews set attempts = 5 where id = pg_temp.id('r3');
|
||||
set local role service_role; select pg_temp.worker();
|
||||
select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.resource_limit'), 'retry', 'a fourth failure is retried');
|
||||
select ok(pg_temp.delayed('r3', interval '20 seconds'), 'after attempt 4 the retry waits 20 s');
|
||||
select pg_temp.set_not_before('r3', interval '-1 second');
|
||||
select is((select attempts from public.claim_sheet_verifications(1)), 5, 'claimed after 20 s: fifth attempt');
|
||||
-- At the limit, a technical failure closes the round unavailable, delay cleared.
|
||||
select pg_temp.set_not_before('r3', interval '1 hour');
|
||||
select is(public.fail_sheet_verification(pg_temp.id('r3'), pg_temp.token('r3'), 'verification.worker.resource_limit'), 'unavailable', 'the last attempt closes the round');
|
||||
select is(pg_temp.status('r3'), 'verification_failed/unavailable/verification.worker.resource_limit', 'round closed unavailable with the last code');
|
||||
select ok(pg_temp.not_before_is_null('r3'), 'the fifth failure clears the delay');
|
||||
reset role;
|
||||
select ok(exists (select 1 from audit.audit_logs where action = 'anatomical_sheet_review.verification_attempt_failed' and resource_id = pg_temp.id('r3')), 'attempt failure audited');
|
||||
|
||||
|
||||
Reference in new issue
Block a user