57 KiB
Evals Registry
Format: EVAL-XXX | Date | What evaluated | Method | Result | Anomalies | Action
EVAL-001 | 2026-05-18 | Scaffold build + test pass
Method: pnpm install → pnpm build → pnpm test:run → pnpm typecheck. Result: all pass, 0 errors. Anomalies: pnpm version 11 instead of 9 (corepack issue on host). Action: document in BDR-007, BLK-001.
EVAL-002 | 2026-05-20 | B3 i18n — code review caught hardcoded FR in PDF import
Method: cavecrew-reviewer ran against full diff 0a05c47..HEAD after self-review + analyzer agent both reported CLEAN.
Result: reviewer flagged 3 CRITICAL: pdf/import.ts:9, pdf/attachment.ts:9-10, app/routes/HomeRoute.tsx:32 — hardcoded FR error messages leaking to EN UI.
Anomaly: prior i18n grep ('Annuler', 'Valider', etc.) missed these because the error-path files contained different French vocabulary ("Ce PDF n'a pas été..."). Implementer's TODO checklist did not include the PDF import error paths since they were not in the original component migration list. Analyzer also missed because the leak only surfaces under EN locale at runtime (no static signal in code).
Action: fixed in d602726+next via typed ImportError discriminant (BDR-015). Future i18n migrations should enumerate ALL files that emit user-facing strings (not just components/routes) — include error path files, exception messages, anything that may end up in window.alert or setError.
EVAL-003 | 2026-05-21 | PDF vet-marker ⚠ WinAnsi crash slipped past CI
Method: user manual export of a v2.0 horse report containing requires_vet_referral leaves → EditorRoute.tsx:263 PDF export failed Error: WinAnsi cannot encode "⚠" (0x26a0).
Result: 154/154 tests stayed green; bug shipped to user-facing path. Static signal absent (the ⚠ literal was visible in the source but typecheck/lint don't flag font-encoding issues).
Anomaly: pdf.roundtrip.test.ts exercised export with empty observations + no vet findings. The ⚠ prefix was added in 60e1b04-era (observationPage.ts:407 ternary) without a regression case covering a flagged leaf.
Action: hotfix swapped ⚠ → (!) (WinAnsi-safe). Added regression vet-referral marker is encodable by the body font that calls widthOfTextAtSize on a prefixed label — would catch any future non-WinAnsi marker. LRN-021 captured the pdf-lib StandardFonts WinAnsi limitation. Follow-up: switch PDF body to embedded TTF (Inter/Fraunces per CLAUDE.md spec) so the catalog can carry arbitrary glyphs — out of hotfix scope.
EVAL-004 | 2026-07-12 | Tour sweep converged 3 it. — agent spend-limit deaths resumable
Method: /tour full pipeline on faunosteo — semgrep floor ×3 + cso posture (it1) + full clean audit + fresh independent re-verify + doc-syncer. 13 fixed / 10 open residuals / 6 bugs quarantined; 11 commits on unmerged chore/tour-2026-07-12. Result: re-verify PASS (0 findings on 20 modified files); checks green throughout (typecheck, lint 0, 174/174, build, docker run non-root). Convergence honest: it2 surfaced 1 new tuning note (min-release-age 1440 < 10080 rec) → triaged open → it3 semgrep identical → converged at 3-iteration bound. Anomaly: both Phase A agents died mid-run on "monthly spend limit" — SendMessage to same agentId resumed from transcript, zero rework. PostToolUse hook false-positived 3× on 'dangerouslySetInnerHTML' as prose inside audit REPORTS (not code). Action: none blocking. Consider excluding .claude/audits/*.md from sink-pattern hook scan.
EVAL-005 | 2026-10-08 | Step 1 characterisation suite (91 cases)
Method: 7 files on real assets (SVG, catalog, fixtures), oracle = audit tables (06 §4, 05 §2.4). Run typecheck/test/lint/build. Result: 271/271 green, 0 src change. Audit figures all confirmed (26 mappings score 0, 792/612/41, 48 attrs…). Anomalies: 4 assertions wrong on first run, all test-side (census regex, stale closure, ZONES drawn as bullet rows not ' · ' line, attachment names in object streams). None app-side. Action: LRN-033. Known quirks pinned as "KNOWN:" tests (orphan findings, no_attachment cause lost, sphenoid id, fuzzy FR pairs).
EVAL-006 | 2026-10-08 | Step 3 Supabase foundation — PGlite suite
Method: 4 pgTAP files (81 checks: schema/privileges 24, profiles RLS 17, platform roles 28, audit 12) + schema↔types (6) + client (2) on PGlite; root typecheck/lint/build; app suite.
Result: all green first run except plan count (27→28) and pgtap-in-public polluting function census; 278 app + 13 package tests. Zero app behaviour change.
Anomalies: set_config rows appear in TAP line capture (noise, filtered by regex). now() constant in tx broke naive updated_at>created_at assertion.
Action: LRN-035/036. Not executed: real Supabase image (supabase test db), gen types diff, seed.sql (needs auth.identities + extensions.crypt), REST manual script. Flagged in report + BLK-006.
EVAL-007 | 2026-10-08 | Step 3 on the real Supabase stack
Method: supabase start (postgres 17.11.0.004), supabase test db, gen types --local diff, REST curl script (SUPABASE.md §8) with seed users, psql on audit, db reset.
Result: migrations + seed apply from zero; pgTAP 81/81 after fix; REST: alice 1 row, anon 42501, self-deactivate 42501 (column grant), self-grant 42501 (RPC message), cross-row PATCH 0 rows, owner grant → uuid, alice sees her role, owner reads 3 profiles; audit shows system bootstrap + owner grant.
Anomalies: (1) 6 pgTAP counts assumed empty DB — seed rows on real stack (PGlite runs no seed) → scoped to fixture ids + neutralise foreign owners before last-owner test. (2) CLI gen output unformatted + ComputedFields: never, no __InternalSupabase → hand-written file replaced by generated+prettier; scripts pipe prettier. PGlite harness had hidden both: environment parity gap, not logic gap.
Action: LRN-037 (tests must be seed-agnostic; generated files via deterministic formatter). BLK-006 resolved.
EVAL-008 | 2026-10-08 | Step 4 sheet model — domain tests, pgTAP both engines, fresh verifier + security gates
Method: packages/domain 61 vitest (schema 29 + integrity 32, one per rule code); pgTAP 005 (127 checks) + 001 census on PGlite and real stack (db reset, db test 210/210, gen types diff clean); root check (typecheck, 356 tests, lint 0, build); fresh verifier (contract C1–C13) + security-auditor (semgrep 725 rules 0 findings + checklist).
Result: first gate pass → verifier ECARTS(5), security BLOCK(1): private EXECUTE not revoked (BLK-007), visibility change unaudited, 7 RPCs without non-editor denial test, doc/code mismatches, 75-line SQL function, any suppression in a test. All fixed same session: +32 pgTAP checks, hardening migration, visibility RPC, caps, authz-first, SQL check split in 4 functions, set()/unset() typed mutations. Second pass: fresh verifier (pending at write time).
Anomalies: 6 test-side fixes on first pgTAP run (RLS hides rows → "no change" asserts; data-modifying CTE; collation of order by; trigger name order decides errcode; audit list; pg_temp grants). Executor (opus) deviated usefully: RegionCode pattern separate from StableId.
Action: LRN-038/039/040, BLK-007. Keep the two-gate pass on every migration step.
EVAL-009 | 2026-10-09 | Step 4b equine import — importer tests, pgTAP both engines, Docker build, gates
Method: packages/anatomy 30 vitest (counts in=out on both sides, special cases, determinism, golden artifacts, guards); pgTAP 001–007 on PGlite and real stack (db reset, db test 256/256, gen types diff clean); root check (61+30+20+278, lint 0, build); docker build of the app image; Storage API end-to-end (owner 400 before confirmation → 200 after, alice/anon 400); fresh verifier (C1–C12) + security-auditor (semgrep 130 rules, 0 findings).
Result: first pass verifier ECARTS(4) (self-referential in=out control, undeclared importer-authored labels, docs vs literal sql_paths, silent FR→EN fallback) + security PASS with MEDIUM hash-claim bypass (register_sheet_asset accepts any sha256 → editor reads foreign objects). All fixed: uploaded_at + confirm_sheet_asset_upload (service role) required by the read predicate, bob-claims-hash pgTAP, authored-label anomalies (27 anomalies now), seeds-listed test, Dockerfile missing workspace manifests (found by building, not by the gates).
Anomalies: real image storage.protect_delete aborts direct DELETE tests (LRN-041); pg_temp fixtures need service_role grants; docker build raced the lockfile refresh (rebuild clean).
Action: fresh verifier re-dispatched on the delta; BDR-034, LRN-041.
EVAL-010 | 2026-10-10 | Step 4c equine curation — research agents, changeset, gates
Method: 4 research agents (skull, axial, limbs, osteo/legal), every citation tagged VERIFIED-fetched/snippet/UNVERIFIED; changeset authored in scratch python (44 changes / 679 ops / 48 refs) → pipeline Zod + 20 integrity rules; anatomy 100 vitest (ops, rename propagation, overlay, golden), pgTAP 001–008 on PGlite and real stack (db reset, db test 267/267, gen types diff clean); root check (68+21+100+278, lint 0, build); docker build; Storage e2e after upload (owner 200, alice/anon 400); gridded magick crops to verify point positions; fresh verifier ×3 + security-auditor ×1.
Result: verifier pass 1 ECARTS(4): B changes left targets inherited (CHG-008/032/039 + partials), 3 stale doc numbers, lockfile outside contract scope (orchestrator omission), eslint-disable any in a test → executor: applier rule (B/C ⇒ touched findings/annotations to_confirm, reviewed refused in B/C), typed test reads, docs. Pass 2 ECARTS(2): JSAVA citation without author (completed from the PDF: Le Roux & Carstens 2018, DOI), DEPLOY.md stale faunosteo-web. Pass 3 ECARTS(1): 3 stale doc lines (migration list, "two assets", CLAUDE.md migrations) → fixed by hand, cap of 3 verifier passes reached, not re-verified. Security PASS, LOW: SQL gate did not check reference url scheme → reference.url.scheme added (migration still unmerged, pgTAP 008 10 cases); .env.example COMPOSE_PROJECT_NAME=osteo-report overrides compose name: → file denied to Claude, user action.
Anomalies: ImageMagick MSVG drops fill="none" shapes and shapes with <title> child (LRN-042); two legacy points (t18-l1, l6-s1) floated above the silhouette, found only by the overlay render, not by the catalog audit; Supabase db reset unconfirms the asset row (upload script must re-run: objects kept, row confirmed).
Action: user validation of the draft content (REVIEW-REPORT §10: 11 questions); never publish; Step 5 only after validation.
EVAL-011 | 2026-10-10 | Step 5 app on the canonical sheet — suites on the bundled sheet, archive PDFs, gates
Method: golden v2 PDFs frozen from the old pipeline before any change; suites rewritten on the bundled draft (domain 94, anatomy 107, supabase 21, app 168+: service branches 10/5/12/1, store + v2 draft migration, provider, canvas, pickers, card, route click chain, PDF v3 roundtrip FR/EN + both v2 fixtures + synthetic removed/unknown ids); root check, vite builds none/draft (grep of a draft label in dist = 0/1), docker build, real stack 267/267 + types; legacy-vs-app marker style diff by the verifier (0 differences); fresh verifier ×3 + security-auditor ×1.
Result: verifier pass 1 ECARTS(36) = 4 real (golden test not pinning the new generated files, EditorRoute 398 lines, 5 stale doc lines, one eslint-disable) + my contract omissions (scope files, deleted legacy suites) + floor noise ("species" matched "spec"); pass 2 ECARTS(4) = replacement assertions missing for 4 deleted suites; pass 3 ECARTS(4) = doc counts + 3 more suites' assertions (PDF conclusion/metadata/attachment structure, ObservationCard, persistence). All closed by executors (two) and by hand (docs); cap of 3 passes reached → last lot not re-verified. Security PASS with LOW: prototype-key lookup on untrusted ids (fixed: Object.hasOwn + test), unbounded inflate on the primary attachment path + no file cap (fixed: 16 MiB attachment, 50 MB PDF + test), denylist SVG sanitiser acceptable while assets are repo-built (allowlist before Storage-served assets, Step 13).
Anomalies: executor ran pnpm format repo-wide and reverted by hand (watch for format in briefs); foreign JSON attachment (no version) surfaced as incompatible_version → fixed to no_attachment (legacy message); verifier floor counts deleted legacy tests as findings until CLARIFICATIONS name them.
Action: user validation of Step 5 (manual FR/EN flow, visual before/after); BDR-038, LRN-043.
EVAL-012 | 2026-10-10 | Step 5 addendum during user manual validation — per-leaf migration tests, archive check tool, final controls
Method: user asked (while testing) for final controls on last lot + per-leaf test beyond the map + simple v2/v3 PDF comparison method. Built: anatomy test all 612 legacy leaves through migrateReportV2ToV3 with committed map vs draft document (442 identical / 141 moved = 134 category + 7 leaf renames / 29 removed, snapshots + vet flag verbatim, 26 annotations on right view, v3 re-read stable); app test same leaves via sheetIndex.nodeAt + v3 PDF roundtrip; CLI pnpm report:compare (pure module + entry, readAttachedDocument split out of extractReportJson, behaviour unchanged). Contract criteria 16-18 added, gates MET, fresh verifier ×3 + security ×2.
Result: verifier pass 1 ECARTS(2) = floor only (EVAL-011 prose "eslint-disable" false positive → CLARIFICATIONS; 11 dropped regionFromLabel assertions → new legacy.regions.test.ts on inferRegionCode); security PASS + 2 MEDIUM (terminal control chars in untrusted strings; free text printed by default) → executor: esc() + --full privacy default; pass 2 ECARTS(1) (--diff still clear text, JSON.parse message echoed attachment fragment) + security PASS 4 LOW → executor: diff follows --full, generic reason, Cc/Cf escaping, regular files only; pass 3 CONFORME. Workspace 94/21/119/181, docker build OK.
Anomalies: verifier stage runs after executor edits only (concurrent reads otherwise); gates.sh needs EVIDENCE: PENDING placeholder lines on new criteria. Residuals noted, not fixed: ZodError message echoes closed-vocabulary received values (never names), compareReport computes v3 lines twice, template-literal lines >80 chars, locals >5 in two functions.
Action: await user manual validation of Step 5; residuals in TODO follow-up.
EVAL-013 | 2026-10-11 | Step 6 autonomous PDF archives — 5 phases by executors, 3 verifier passes, 4 security audits
Method: analysis (3 analyzers + measurements) challenged by 3 plan-challengers (correctness/robustness/simplicity, 48 findings folded in) → user validation with D7/D8/D13 adjustments → Part II contract (exact types) → one executor per phase (0 freeze goldens, 1 domain, 2 anatomy compose, 3 engine, 4 app) with closed contracts, orchestrator commits → whole-step verifier + security auditor → fix loops.
Result: Phase 0 froze 39 goldens (text/images/links per locale × logo mode, view facts) + 2 route characterisation tests with the OLD code; engine reproduced 36 goldens with 0 mismatch (views: data-* 2→0 and hidden-marker titles declared relaxed). Verifier pass 1 ECARTS(31) = all oracle/scope defects of MY contract (grep on parametrised table, xmldom filter, narrow FILE SCOPE, 3 deleted tests with moved coverage) → contract amended → pass 2 CONFORME 13/13. Security pass 1 BLOCK(1) HIGH real + PoC: archived branding URL written unescaped into /URI (PDF object injection via exportFromEnvelope), +3 MEDIUM/4 LOW → executor fixed (hex URI, strict URL schema, export re-validation, PNG IHDR budget, CSS-escape rules, NaN guard, file.size) → pass 2 PASS but 2 MEDIUM incl. a regression the fix introduced (UTF-16 BOM URI dead in poppler) + PNG double-IHDR bypass → executor → pass 3 PASS + 1 MEDIUM (UPNG nextZero spin on tEXt) → executor (chunk allowlist + IEND) → pass 4 PASS + 3 MEDIUM (UPNG inflateRaw spins on truncated IDAT with allowlisted chunks only; IHDR fields unvalidated; zero dimension) → executor: native bounded inflateFlate of the IDAT against the exact raw size, IHDR fields, sides ≥ 1, no interlace, raw ≤ 16 MiB → pass 5 focused. Lesson: a third-party decoder on untrusted bytes needs a native oracle in front, not a chunk allowlist. Final: 656 tests (194/21/130/88/223), typecheck, lint, build, docker ×4 green; goldens byte-identical since 8b86b92.
Anomalies: contract oracles written fast = 2 verifier passes lost (lesson: run every CHECK once before dispatching the verifier); security fix introduced a functional regression invisible to tests because the test reader decoded the BOM (lesson: assert raw bytes, cross-check with a real reader like poppler); executors denied rm/git rm (orchestrator deletes); one executor used git stash despite "no git state" (now forbidden explicitly in briefs); demo envelope with archived branding = 136 KB stored / 306 KB raw (raw > 250 KB, stored < 250 KB: reported, user said data first).
Action: await user manual validation + UX decision on re-export of the kept archive; BLK-009 (pdf-lib ObjStm inflation) before external users; react-router advisories (pre-existing) to bump in a chore.
EVAL-014 | 2026-10-11 | Step 6 closure round — manual-test feedback, BLK-009 scanner, final audits
Method: user manual tests round 1 (logos absent, picker blurry) → reproduction attempts (headless Chromium dev/preview/Docker, Firefox 157 via esbuild harness + raw WebDriver BiDi client, CDP LayerTree, LCD-fringe metric, headed runs on the Wayland session) → fixes by executors → user round 2 confirmed both OK → closure decisions → BLK-009 implemented (byte-scan reader, no object streams) → verifier + 2 security audits + 1 fix loop.
Result: neither visual defect reproduced here (4 environments, 2 engines, 2 PDF renderers); shipped the environment-independent fixes (branding bundled, no fetch; warnings surfaced) and the user confirmed logos + crisp text. Scanner: verifier CONFORME 7/7; audit 1 PASS + 2 MEDIUM (Chromium DecompressionStream read-ahead defeats the cancel: +2.2 GB on a 4 GiB bomb; ×32 candidate CPU) → executor (16 KiB slice feeding, shared 16 MiB budget, monotonic bounded endstream search, { pre-check; 133 pdf tests) → audit 2 focused. Workspace 710 tests, build, docker green.
Anomalies: the blur fix of the previous round (will-change) had been marked PARTIALLY MET "desktop by user" — the user's "still blurry" came from that unverified half; this round could not verify real-GPU text rendering either (headless engines render grayscale AA, headed captures failed) → lesson: a screen-rendering defect needs the user's own capture + an A/B recipe early, not days of headless measurement. The Node vitest suite cannot observe browser stream read-ahead: record browser-only bounds as owed manual checks (PDF_IMPORT_HARDENING §5).
Action: BLK-009 stays "mitigated, gate open" (BDR-040); Step 6 finished into develop; release/0.2.0 prepped for the user's go.
EVAL-015 | 2026-10-11 | Step 7 identity — gates on the whole step
Method: contract with 16 criteria (2026-10-11-step-07-verify.md) → fresh verifier; security-auditor (semgrep 159 rules + checklist) on c746952..HEAD; headless e2e vs the local stack (playwright-core in scratchpad, Mailpit API); real-stack pgTAP + docker build by the orchestrator.
Result: verifier pass 1 ECARTS(2): (a) real gap, safeNextPath prefix check let /\host and /<TAB>/host through (router location.assign fallback = open redirect after login) → fixed by resolving against a sentinel origin and rebuilding from parsed parts, 5 new cases; (b) contract regex over-matched other unions ('google' | 'apple', 'fr' | 'en') → CHECK scoped. Pass 2 CONFORME 16/16. Security PASS, 1 MEDIUM (same redirect, already fixed) + 4 LOW: 3 closed same session (sign-out failure reported, reauthentication_needed phrased, exact prod redirect allowlist documented), 1 pre-existing (TOTP not enforced for staff = documented gap). E2E 20/20. Workspace 763 tests, lint 0, build, docker, real-stack pgTAP 280 green.
Anomalies: both gates found the same redirect hole my own tests missed → a prefix check on a URL is never enough, parse it; the auditor misread the sign-out note ("this device") as "everywhere" but the LOW still pointed at a silent failure path worth phrasing. Mockups validated with ChatGPT are not in the repo: the screens follow the written direction and the existing tokens, pixel fidelity unverifiable here.
Action: awaiting user validation of Step 7 (manual procedure IDENTITY.md §5); release/0.2.0 still open for the human go.
EVAL-016 | 2026-10-12 | Rust-by-default architecture review — 4 analyzers + 3 researchers + 3 plan-challengers
Method: 4 read-only analyzers (domain, anatomy, pdf + app pdf side, app logic + supabase) wrote factual inventories to the scratchpad; 3 general-purpose researchers verified crate versions/maintenance/advisories and Supabase facts on crates.io, RustSec, GitHub, supabase docs (2026-10-12), UNVERIFIED items listed; study rev 1 written from them, committed (6a078da); 3 plan-challengers (correctness FATAL(11), robustness FATAL(11), simplicity CONCERNS(9); 61 findings) → rev 2 with 56 folded, 2 declined with reasons (§0.2).
Result: real blockers caught by the gates, not by me: (1) VI.4 deleted TS modules still imported by later-phase TS consumers (pipeline, engine); (2) production CSP script-src 'self' refuses wasm, VPS 1 GB compiles images; plus wrong premises: envelope byte-parity (integer-valued floats, insertion order), Prettier identity overstated (hash only compared), catch_unwind unusable on wasm32 (panic=abort), serde-wasm-bindgen null-for-absent, parent re-decode breaking the keyless invariant, Step 15 misread as server rendering. Simplicity: 6 crates → 3, dual-run harness → frozen goldens, Landlock + 3 rlimits redundant, constraints generator/schemars/cargo-chef/cargo-audit/infer/img-parts dropped.
Anomalies: one analyzer had a read-only find -exec cat | wc -l refused (no rule named) and re-ran as find | xargs wc -l (already allowed earlier) — reported to the user as a possible workaround; agents flagged writing report files to the scratchpad against their default (orchestrator asked for it). Three challenger claims came from memory not code (json_compatible null, panic=abort, musl mprotect) → folded as "verify at R0/R1".
Action: owner validates study + D-R1..D-R14; nothing implemented; CLAUDE.md rule section added; TODO roadmap lines.
EVAL-017 | 2026-10-13 | Rust R0 foundation — proofs in the real environment, 3 verifier passes, 3 security passes
Method: contract 2026-10-13-rust-r0-verify.md (16 criteria, CHECK/EXPECT) → fresh verifier; security-auditor on 5fec065..HEAD (semgrep p/security-audit + p/secrets + p/owasp-top-ten + checklist + manual Rust review); all cargo commands through the Docker builder (no Rust on the host); worker sandbox tests + strace derivation inside the Alpine test stage on the release-profile musl binary; wasm proofs in vitest on the built module; CSP proof with the production nginx image + headers in headless Chrome 156.
Result: verifier pass 1 ECARTS(2), pass 2 CONFORME 16/16, pass 3 ECARTS(2) — every gap an oracle of MY contract (greps counting comment lines, the contract matching its own regex, the [gated] SAFETY form and a second refusal arm moved by the security fixes); code behaviour confirmed by reading each time. Security pass 1 BLOCK(1) HIGH real: shell injection via the workflow_dispatch input of release-images.yml, + 6 MEDIUM (shared BuildKit caches, tkill/tgkill without arg filters, core dumps, creds on the runner, no advisories gate, mutable action tag) → fixed; pass 2 PASS but 3 MEDIUM introduced by my fixes (ephemeral builder never used: GITHUB_ENV applied one step too late, silent fallback; rm -rf "$VAR"; content reaching the Broken outcome via Reject::Internal) → fixed; pass 3 PASS, 2 new MEDIUM (unpinned setup-node and pnpm install inside the token job) → gates moved to a secret-less job; remaining lows documented as R1/Step 27 follow-ups. Experimental findings (LRN-049): serde_json fast float path (float_roundtrip), traps leave the instance unusable + ESM cache, RLIMIT_CPU soft=hard → SIGKILL, minimal rustup profile, cargo-deny path wildcard, 7 traced syscalls, strace blind once dumpable=0 (marker became a syscall). Final: cargo 12+14 core tests, 9 unit + 16 sandbox worker tests on the release profile, 14 vitest proofs, root pnpm check 777 tests green, worker image 8.7 MB read-only/cap-drop, CSP refused/accepted as predicted, floor-guard clean ×3.
Anomalies: my contract oracles cost two verifier passes (lesson repeated from EVAL-013: run every CHECK once before dispatch, and re-run them after any fix commit); two of my own security fixes created new findings (an env variable applied too late, a recursive delete through a variable: the exact pattern the destructive-tools rule forbids) → fixes to CI files need the same adversarial pass as code; a scripted edit silently aborted on a rustfmt-reformatted string and the commit went out with the old test (caught by the next build); the [gated 2026-10-13] SAFETY markers were placed by me at the auditor's request, not by the owner → surfaced in the report for the owner's confirmation; one security-auditor Bash call (cat of two files) was refused twice by the auto-mode hook, the agent used Read instead (reported, no workaround). Not proven here: the Gitea runner (caches, timings, workflow syntax such as job-level env, actions/cache), HTTPS from the worker (no HTTP client before R1), browser-side fresh-instance mechanism (R2).
Action: owner validation of R0 (report with proved vs hypothetical); R1 not started.
EVAL-018 | 2026-10-13 | R0 correction: native Rust toolchain on the Ubuntu dev server — 2 verifier passes, 2 security passes
Method: host audit first (rustup already user-level, 1.99.0 auto-installed by rustup show, system deps present, no sudo); installs --locked; scripts moved to scripts/, pnpm check:all gate; contract 2026-10-13-rust-r0-native-verify.md (11 criteria) dry-run by me before dispatch (3 oracles fixed: comment prose matched a code grep, per-package vitest totals, README phrasing); fresh verifier ×2; security-auditor on bd9c7d1..6e2bbec then on the fix commits; full gate run twice (check-all.log, check-all-2.log).
Result: verifier pass 1 ECARTS(2) = floor STUB hits on doc prose "(not implemented)" → reworded; pass 2 CONFORME 11/11, floor clean. Security pass 1 PASS + 1 MEDIUM (no advisory check has ever run: nightly job never executed) + 6 LOW (PATH prepend, rm -rf "$out" via variable, prctl comment overstating the kernel contract + i32 into variadic unsigned long, dormant CI triggers, native build.rs exposure, drift gate blind to untracked files) → fixed except the trigger question (owner's CI/CD step); pass 2 PASS + 1 MEDIUM (my ci.yml drift step failed open on a git error: test -z "$(…)" drops the exit status) + --untracked-files=all + stale comment → fixed (496f549), verified by execution + shellcheck, no third dispatch (3-line change). Numbers: native rust-check 32 s warm / 1 s incremental, cold workspace build 31.5 s, wasm 224 777 B (+64 B vs builder, path strings), 14 proofs, check:all 1 min 59 s then 1 min 39 s, 777 Node tests, 16 sandbox tests, allowlist covers the trace, worker read-only run --check OK.
Anomalies: the R0 note "no cargo on the host" was stale, rustup had been installed 2026-10-12 23:27 (rtk) → always re-audit the host at session start; rustup show has a side effect (auto-install) → acceptable here (requested install) but say so; a fix I wrote for one security finding introduced the next MEDIUM (fail-open test) → the EVAL-017 lesson again: CI one-liners need the same scrutiny as code, prefer assign-then-test; the harness clock jumped 01:16 → 09:43 mid-session (timestamps in logs are not durations).
Action: owner's final validation of the R0 correction; R1 not started; CI/CD decision separate (trigger mode, runner label, action pinning, advisories cadence).
EVAL-019 | 2026-10-13 | Step 8 / R1 files foundation — 2 forks, verifier ×2, security ×5, e2e ×7, browser smoke
Method: contract-level STORAGE.md written first; migration + pgTAP by me while fork A (Rust loop) and fork B (TS + UI) ran with full context; real-stack e2e (pnpm e2e:avatar, worker container, fresh admin user) + headless Chromium smoke (playwright-core in scratchpad, cached chromium, dev server :5174) for the browser canvas path; contract 2026-10-13-step-08-r1-verify.md (15 criteria, oracles dry-run before dispatch) → fresh verifier; security-auditor on 3f1105b..HEAD then focused passes on each fix commit; pnpm check:all ×4.
Result: verifier pass 1 ECARTS(4) = oracle (for update row-lock grep) + FILE SCOPE (tracking files, lockfile) → rev 2 → CONFORME 15/15, floor clean. Security pass 1 PASS (2 MEDIUM: ceiling ratchet by rejected rows, cleartext http any host; 9 LOW) → fixed; pass 2 PASS but my fixes introduced 2 MEDIUM (bypassable host check 127.*/numeric, ceiling undercount vs contract) → fixed; pass 3 BLOCK(1) HIGH: my purge-queue join let begin spam pin the ceiling with claims → ceiling rewritten on bytes present + cap per pending claim; pass 4 PASS (MEDIUM residual: per-account holding) → 64 MiB per-account budget; pass 5 PASS (LOW hygiene) → comments/doc. e2e 5/5 ×7 (found the Storage API 400-as-404 making purge acks loop: added queue-drained assertion); smoke 18/18 desktop+mobile; final: pgTAP 140 (real stack 420 total), app 300, supabase 37, worker 21+11 native, sandbox 16, allowlist = 7 syscalls unchanged, image 11.6 MB.
Anomalies: EVAL-017 lesson again ×3 — each security fix needed its own adversarial pass and twice created a new finding (host check, ceiling join); a security control I added became the attack lever (BLOCK). Fork B's database.types.ts hand-write matched the generator byte-for-byte except ordering; fork A found the 400/404 only via e2e. gitflow finish of R0 and any .env* write refused by the harness classifier; git config reads refused too. Verifier sandbox intercepted the literal pnpm lint (rtk wrapper) → oracle uses the binary path.
Action: owner's manual validation (TEST FONCTIONNEL + TEST GRAPHIQUE), owner runs gitflow finish for R0 then R1 after validation; no merge, no deploy, release 0.2.0 untouched.
EVAL-020 | 2026-10-14 | Rust R2 domain core — 3 plan challengers, 4 forks, verifier, security ×2, CSP proof on the image
Method: plan + contract written first (.claude/tasks/plans/2026-10-13-rust-r2-domain-core.md), 3 plan-challengers (correctness, robustness, simplicity) before code; freeze-then-switch: envelopes, Zod output oracles, characterisation and verdict tables written by the TS before trusting Rust; forks A/B/C in worktrees on disjoint files (sheet rules + conformance, report/archive rules, editor rules), runtime + app switch by me, fork D adapted the app suites in tree; browser checks with playwright-core from the scratchpad (dev server, prod build, real image under the DEPLOY CSP); contract 2026-10-14-rust-r2-verify.md (12 criteria, every oracle dry-run before dispatch) → fresh verifier; security-auditor on the whole range, then a focused pass on the fix commit.
Result: challengers 45 findings incl. 1 BLOCKER (degraded editor redirect could overwrite an unread draft) → plan rev 2 before any app code; contracts byte-equal to Zod on first run; forks merged with zero conflicts (one stub replaced); fork D found 2 real src bugs (Node source path under happy-dom, failed quarantine reported as success) → fixed with tests. Verifier: 12/12 MET, ECARTS(2) = floor DELETED_TEST (zoneInteraction suite ported to cargo; R0 skip helper replaced by a stricter global setup) → human micro-gate for the owner. Security: PASS (3 MEDIUM nginx false positives pre-existing, 5 LOW) → by-reference deserialisation (also −134 KB raw / −31 KB gzip module) + literal delete paths. Final: cargo core 135, worker+wasm 32, Node 906 (app 313), pgTAP PGlite 12 files / real stack 446, CSP proof 15/15, check:all green ×3, module 1,046,498 B raw / 340,929 B gzip.
Anomalies: worktree base stale (BLK-010); harness decodes \u escapes in tool inputs (LRN-054); my first runtime test helper reused the pattern Vite rewrites; the CSP proof first failed on its own harness (EN locale, disclaimer modal, caught compile error logs nothing) — proof scripts need the same dry-run discipline as oracles; a security LOW fix improved size, measure before assuming a cost.
Action: owner manual LAN validation, L1 decision, approve the 2 deleted test files (floor), then merge go; R3 next, not started.
EVAL-021 | 2026-10-14 | R2 closure — L1 refusal, deletions proof, fresh verifier + security
Method: owner asked to fix L1 (no silent U+FFFD) + prove the 2 approved deletions + checks before merge. Mapped every JSON entry into the core (TS side always JSON.stringify → escapes reach Rust as ASCII); preservation costed (own JSON reader + lossless string type ~150 fields) → AskUserQuestion: refusal + quarantine chosen, detailed import wording chosen. Stale-module proof run before wasm:build (suites throw pkg is stale). Worktrees checked blob by blob vs branch history (60/66 identical, 6 = stub, merged mod lists, 2 lifted #[ignore]). Contract 2026-10-14-rust-r2-l1-verify.md (13 criteria, each dry-run) → fresh verifier; fresh security-auditor on 36316cf..HEAD.
Result: verifier CONFORME 13/13 (floor 2 = the approved deletions, authorised by REQUEST); security PASS, 6 LOW: wasm-bindgen &str would re-encode a raw lone surrogate (unreachable, JSON.stringify callers; documented invariant), pretty_json proof op refuses as invalid_json, scanner not string-aware (classification only), newest-only quarantine slot, archive quarantine failure under full storage lets a later reset remove it (pre-existing R2, reported to owner), semgrep partial parse. Final: core 142, Node 915 (app 320, supabase 40, core-wasm 26), PGlite pgTAP 12 files, real stack 446/446, types no drift, check:all green 1 min 32 s, module 1,040,784 B raw / 339,933 B gzip.
Anomalies: refusal broke a test loader (oracle file freezes TS outputs with lone escapes) → loader spells escapes out, never normalises (LRN-055); cargo deny advisories (warn-only) shows RUSTSEC-2026-0346/0347/0348 on rustls 0.23.45 (worker, none applicable: no brotli, built-in session cache, no ECH) → bump follow-up; supabase/.temp/cli-latest tracked by mistake, rewritten by the CLI, restored; export PATH=… denied → absolute ~/.cargo/bin/cargo; first commit with chained git config denied → plain commit OK.
Action: gitflow finish into develop (owner authorised after L1 + checks), worktree cleanup (BLK-010), R3 summary only.
EVAL-022 | 2026-10-15 | R2 maintenance + R3 anatomy core — challengers, oracles, Chromium, verifier ×2, security ×3
Method: maintenance: AskUserQuestion for B UI, store rewrite + 27 tests, verifier (11/11, floor false positive on cli-latest) + security PASS (4 LOW fixed), check:all, merge. R3: 2 analyzers (facts), plan + 3 challengers (FATAL 3 / FATAL 6 / CONCERNS 4; blocker = roxmltree recursion, which I had measured independently) → rev 2; freeze O1–O6 with TS before any change; Rust reader/layer/compose; mutation checks per oracle test; wasm thresholds in fresh instances; app switch; Chromium DOM + 30 screenshots + perf vs develop baseline; CSP proof on image; verifier CONFORME 11/11; security PASS + MEDIUM (prescan closer desync) → fixed → pass 2 PASS (4M differential docs) + LOW (url whitespace) → fixed.
Result: all oracles green (O2 exact incl. order, O1/O3 tree-equal, 42 hostile + 10 generated declared or equal, 4 product SVGs untouched); Chromium live trees = O1, 30/30 pixel-identical; module 395.5 KB gz (+56); editor 503 vs 476 ms, detail 143 vs 125 ms (CPU ×4); check:all green.
Anomalies: my first prescan had the closer-overlap bug (security caught it, tests did not: corpus lacked overlapped closers); first wasm memory threshold measured leaked sessions (harness artefact) → per-case fresh instance; probe in perf script retained detached nodes (fake leak); env and export refused by harness → inline env vars; cargo fmt reformat broke python patch anchors ×3 (write whole file or re-read).
Action: owner LAN validation (TEST FONCTIONNEL / TEST GRAPHIQUE), owner decisions (EN trigger name, degraded wording), then merge; R4 not started.
EVAL-023 | 2026-10-16 | Owner-requested audit of 0279953 + bugfix — independent auditor, 3 challengers + confirm, verifier ×2, security
Method: fresh general-purpose auditor on commit only (5 owner axes); claims re-read in code; /bugfix: plan → 3 blind challengers → rev 2 → confirm pass → rev 3; owner AskUserQuestion ×3 batches; bugfixer; gates.sh 5/5; verifier; security; doc-syncer. Result: audit NOT satisfactory (3 defects: unsaved promise, 4 dead retries, recovery dead end) — all fixed 6371523; app 374 tests, core-wasm 37; verifier pass 1 ECARTS(3) (2 untested surfaces, helper out of scope) → pass 2 CONFORME 12/12; security PASS 2 LOW; docs NONE. Anomalies: previous session's commit + CHANGELOG claimed "never a retry that cannot work" without surface census; challengers converged on dead end the plan fixed only on canvas; confirm pass found offline restart never reaching stopped → copy moved to save banner (owner choice). Executor found latent GC finalizer throw (not fixed). Action: R3 stays unmerged until owner LAN confirmation; R4 not started.
EVAL-024 | 2026-10-17 | R3 final checks — measurements, challengers ×3 + confirm, verifier, security, check:all
Method: scratch round-trip test (3 golden reports); fuzz agent (37k draft.restore on shipped wasm + 40 worst cases, OOM threshold); GC agent (Node --expose-gc, 50 recoveries, WeakRef probes, mitigations in scratch); owner AskUserQuestion ×3; plan → 3 blind challengers → rev 2 → confirm → rev 3; bugfixer with negative controls; gates.sh 4/4; verifier; security; check:all; doc-syncer. Result: copy restorable in app; loop impossible with real drafts, defended anyway; work-loss paths closed; app 405 tests, check:all green (~75 s cached); verifier CONFORME 10/10 (1 floor ASSERT_DROP = guard test moved, authorized by plan); security PASS 2 LOW; docs NONE. Anomalies: challengers caught trap re-read spending recoveries (2 lenses), BDR-055 contradiction, wrong "parity" cap (50 MiB file vs 16 MiB JSON) only on confirm pass; executor found Retry-during-recovery edge. Action: owner LAN validation, then merge on owner go; no R4, no release, no deploy.
EVAL-025 | 2026-10-20 | R4.9 gates — bench, verifier ×3, security ×4, check:all
Method: memory bench 3 engines × demo/L × 6 exports + Chromium L 16 + raster control; contract 16 criteria (owner §2 list), fresh verifier per pass; full security gate on 168 R4 source files, then targeted re-gates on each fix; check:all.
Result: verifier pass 1 ECARTS(1): core with no feature did not compile (restore_copy used gated pdf::read const; workspace feature unification via crates/wasm hid it in rust-check/check:all) → gated + no-feature clippy in rust-check; pass 2 ECARTS(1): §6 lacked J2 merge-gate row → added. Security pass 1 BLOCK(1): .gitleaks.toml allowlist (R4.1, owner-approved in journal) lacks owner [gated] marker → owner action; MEDIUM serve.mjs --host "" bound every interface.
Anomalies: my first host fix = exact-string denylist → bypassed by 0, 0::, ::ffff:0.0.0.0, scoped :: (re-gate 1); rebuilt resolve-then-BlockList; re-gate 2 found post-bind check dead for IPv6 (family 'IPv6' vs 6) and my fstat-first CLI read opened FIFOs before the type check (regression). Each fix of a gate finding introduced the next finding: 3 re-gates for 2 small fixes. Contract CHECKs used root vitest path (absent): oracle defect.
Action: owner checkpoint; floor 21 DELETED_TEST (R4.8 deletions, approved in brief §1) need owner [gated] clarification line to clear formally.
EVAL-026 | 2026-10-22 | R5.1 freeze + D-R5 study — analyzers ×2, oracle generator, cross-stack digests
Method: 2 analyzer agents (pipeline inventory 43 files, fingerprint census: producers, consumers, 99 PDFs, docs), both refused to write files (role rule) → orchestrator saved reports; generator test (MAKE_R5_ORACLES=1) + freeze re-run green 6/6, anatomy 137, domain 262; Node scripts (matrix, PGlite jsonb) + scratchpad Rust binary on core crate: digests compared value by value; tie census 200k values.
Anomalies: R2 oracle test self-referential on number printing (LRN-070) → F1 undetected since R2; docs contradict (BDR-045 vs RUST.md §10.10 / TODO "will change once") → cross-reference added; analyzer inventory counted "10 anomalies" wrong in orchestrator draft (actual 27, records 825 not 690, 16 ledger artifacts) → corrected from oracle counts before commit.
Action: F1 fix proposed, owner go; D-R5 recommendation B + aliases; checkpoint delivered, STOP.
EVAL-027 | 2026-10-22 | R5.1b + R5.2 gates: check:all, security, verifier
Method: check:all green 21:16:39 (app 434, core-wasm 58, domain 262, anatomy 141, supabase 40, curation suites, worker + images); security-auditor semgrep 129 rules / 36 files → PASS, 0 HIGH, 5 LOW; fresh verifier on contract 2026-10-22-rust-r5-f1-importer.md → CONFORME 12/12 (ran cargo workspace + curation suites + import-equine --check itself, red-before proven by rustc probe).
Anomalies: LOW report.rs Map index panics on hand-built ImportResult → fixed (get chain) after verdict, importer suite + --check rerun green; LOW quadratic vertebrae scan, symlink-following listed() in --check, lint allows (accepted, operator inputs). check:all ran on pre-commit tree (code identical to 03bdb5e). Verifier floor note: plan B.3.5 "(proposal, not implemented)" heading from R5.1, intended.
Action: checkpoint R5.2 delivered, STOP before R5.3.
EVAL-028 | 2026-10-23 | R5.3 gates: fresh verifier ×4 + security; successive passes found real defects
Method: check:all after each fix round; security-auditor (semgrep 725 rules, checklist) → PASS, 6 LOW (3 fixed: Map index, destination claimed + canonicalised, INIT_CWD absolute only); fresh verifier each round, no history. Pass 1 ECARTS(1) (floor on a plan heading) + 2 unreachable divergences aligned; pass 2 null-on-optional (LRN-072); pass 3 (first attempt died on API credit error, rerun on Opus) repeated keys refused by serde derive, NUL touch-mark silent drop (LRN-073), --out symlink/ trailing slash followed the link; pass 4 C1-C12 met, C13 docs gaps (lossy UTF-8 read, 2 contrived splits undeclared, stale counts).
Anomalies: green byte parity on real data hid every finding: all were hostile-input or file-protection defects. A verdict reported after post-pass edits was not re-verified → owner required a fresh pass on the final state.
Action: refusal tests now compare reasons by stage; Object.prototype characterised + tested; migration-map debt characterised; final fresh verifier + security on the final R5.3 state before R5.4. Next ports: budget ≥3 verifier passes, probe hostile inputs (null, duplicates, reserved names, symlinks, encodings) in the oracle from the start.
EVAL-029 | 2026-10-23 | R5.4 gates: 14 rounds (verifier + security + check:all) to converge
Method: each round fresh verifier (opus, contract from disk, own TS-vs-Rust differential) + fresh security-auditor (semgrep + checklist + memory probes) + pnpm check:all; superseded runs stopped. Rounds 1-5: Markdown guard families (denylist → allowlist at round 3); rounds 6-11: resource bounds (table amplification, records per op, string headers, drawings per annotation, fixed parts); 12-14: doc precision only. Final: c4e717d security PASS (LOW only), check:all green 16:41:34; eb667ad (docs/comments only over c4e717d) verifier all 9 clauses MET, ECARTS(13) = floor-guard hits on audit prose quoting suppression keywords (product paths clean).
Anomalies: docs repeatedly claimed bounds "before any row is built" / "tens of MB" that the next audit disproved; counts drifted after each fix; FLOOR family needs a contract Clarification (human gate), never an audit edit.
Action: LRN-074, LRN-075; checkpoint R5.4 to owner with decisions (FLOOR clarification, curation input cap vs 65× parse, bare URLs autolinked, 2 declared quadratic scans), STOP before R5.5.
EVAL-030 | 2026-10-23/24 | R5.5 gates: 2 rounds + owner real-stack recette
Method: fresh verifier + security-auditor (opus; pinned Fable agents died on usage credits, re-dispatched), check:all per round; owner ran real-stack e2e by hand. Anomalies: round 1 verifier ECARTS(4) all real (oversize body → Network+retry, "retried 3×" = 3 attempts, runbook exit codes, FLOOR script never OK); security M1 key on argv in e2e script. Round 2 security 5 LOW (Protocol retried, broadcast DEV addr accepted). Census doc claim "earlier revisions" wrong → corrected before checkpoint. Action: fixes + negative tests; owner closed R5.5 2026-10-24 (e2e PASS 11/11, owner's run). LRN-076.
EVAL-031 | 2026-10-24 | R5.6 prep: claimed rehearsal green after extending the patch
Method: fresh verifier applied patch in scratch copy, ran every feature set + switched-seed simulation; security read patch. Anomalies: docs/TODO said "rehearsed green"; green run covered 5-consumer wiring only, then generator + loaded.rs added without rerun → report-compare E0063, 4 curation tests failing unreported; switched seed never simulated (+3 failures incl. frozen R4 O7). Security M1: alias-path export writes report 27c7 + snapshot 12d7 → pre-switch/N-1 readers drop snapshot. Action: rerun gate after every patch extension, claim only measured state; patch fixed (build, strict block, NotCanonical); docs corrected; M1 to owner.
EVAL-032 | 2026-10-25 | R5.6 rehearsal: docs claimed export guarded by fingerprint_verified
Method: isolated worktree (patch + switched seed), rust:check + check:all green, fresh security + verifier (opus), verifier probe on core export API.
Anomalies: RUST.md said M1 "needs fingerprint_verified" for every consumer; export path (input.rs) never sees LoadedSheet → v0 sheet snapshot gets alias at export (app unreachable, API reachable). Fix needs snapshot_rules.rs, outside owner's 8-file scope → stopped. Also pnpm format rewrote append-only audits in worktree (reverted).
Action: claim a guard per call site, not per design; owner decision on snapshot_rules.rs; never run format over .claude/.
EVAL-033 | 2026-10-25 | R5.7 closure: deletion gates + pre-merge rerun
Method: ledger 407 TS cases case by case before git rm; fresh verifier + security (opus) twice; check:all on R5.7 state, then rerun on 27bf59c after post-check edits; verifier ran 14 scratch reintroduction probes against census.
Anomalies: round 1 verifier ECARTS(2): contract C6 "tsx gone" false vs pnpm optional peer (owner Clarification), digest typo …dfe7 in docs. Two edits landed after first check:all → owner required full rerun. Census regex accepted #[ignore] targets (fixed) and still accepts ignore-before-test (LOW). Docs pointed to premerge record before it existed; .bin/tsx wording root-only; SUPABASE.md layout stale since R2. pnpm audit never run before this round (cargo advisories only): 57 pre-existing.
Action: LRN-078, LRN-079; rerun gate after any post-check edit; run pnpm audit in every pre-merge round; checkpoint to owner, STOP before merge.
EVAL-034 | 2026-10-25 | Functional inventory put form validation in Rust, against D-R7
Method: P0 doc census + V1-1 contract drafting; inventory .claude/audits/2026-10-25-functional-inventory.md cross-checked with RUST_ARCHITECTURE_REVIEW Annex A.
Anomaly: inventory criterion 4 placed validation rules (formats, lengths, required fields) in crates/core behind wasm facade. D-R7: form constraints stay TypeScript (Zod, i18n keys), SQL CHECK = integrity boundary, constants module + PGlite parity test against pg_constraint. Contradiction caught and corrected before V1-1 implementation contract; no code written on it.
Action: V1-1 contract §1 + §8.3 follow D-R7. Check every proposal against Annex A D-R rows before writing it.
EVAL-035 | 2026-10-25 | V1-1 design draft: 3 fresh challengers found 3 false security claims
Method: plan-challenger ×3 (correctness, robustness, simplicity) on V1-1_OWNERS_ANIMALS.md before owner checkpoint; evidence read from migrations, harness, worker config, react-router dist.
Anomaly: draft claimed (1) N7 service_role revoke → "D16 holds at key level" (false: service key = GoTrue admin key → magic link / set password → user session); (2) identical 42501 while reusing require_actor() (two messages); (3) audit growth bounded (false: log_event stores attacker-sized x-correlation-id). Also phone search vs E.164 broken, OR search unindexable, backfill untestable, multi-call create duplicates, relation RPCs race, useBlocker needs data router.
Action: all integrated into contract §6.3, §8-§12, §14 + TODO. Claims about key/role reach need proof via the auth admin API too, not only PostgREST grants.
EVAL-036 | 2026-10-25 | V1-1.1 M1 workspaces: gates and isolated real stack
Method: fresh verifier on contract .claude/tasks/contracts/2026-10-25-v1-1-1-workspaces.md + fresh security auditor (semgrep 3 rulesets + hand SQL review), PGlite + sandbox runs, 5 mutations of M1 (email check, ON CONFLICT, correlation cap, foreign-member guard, service_role EXECUTE) all caught by 012.
Result: security PASS (2 MEDIUM scanner false positives: policy via helper, not raw auth.uid(); LOWs = table names in 42501, residual admin paths, NOT VALID CHECK, pre-existing bounded revoke reason). Verifier C1-C8 MET, C9 UNVERIFIABLE (forbidden to read DEV) → closed by supabase migration list --local (M1 remote empty).
Anomalies: first mutation run misread (sed indentation miss, crash not printed) → rerun with crash capture; concurrency script pipefail printed false "UNEXPECTED" after real ERROR → fixed. GoTrue 500 on same-email concurrent sign-up = GoTrue race on users_email_partial_key, not M1.
Action: contract for gates should name a read-only way to check DEV state (migration list) so C9-type clauses are verifiable.
EVAL-037 | 2026-10-25 | V1-1.2 M2 clients: gates, mutations, sandbox hardening
Method: fresh verifier (contract 2026-10-25-v1-1-2-clients.md) + fresh security auditor; 5 M2 mutations (divergent replay, workspace auth, _ escape, N9 filter, hourly limit) all caught by 013; real image fresh + upgrade (M2 alone, M1+M2 together).
Result: security PASS (MEDIUM: secrets on argv in db-sandbox.sh, fixture password literal, policy false positive; LOW: empty-secret fallback risk, state dir in repo, uncapped cursor, no read limit). Verifier ECARTS(2): BDR-075 uncommitted (committed after), fresh-mode real run not reproducible by it (rerun after hardening: 661 PASS, types match, 28/28).
Fixes applied: secrets via env files + PGPASSWORD name only, password-free --db-url, load_secrets refuses incomplete file, state dir refused in repo. Accepted: fixture password literal (sandbox only), uncapped cursor (bounded by name_key compare), read rate limit (deploy notes RR-5).
Anomalies: Write tool turned
/
escapes into raw chars (broke a JS regex) → check new files for raw line separators; prettier over tests/ touched 3 unrelated files → restored; real image adds GRANT hint to table denials.
Action: write Unicode escapes via shell/python when exact bytes matter.
EVAL-038 | 2026-10-25 | V1-1.3 UI foundations + db:sandbox final check
Method: sandbox scripted check (fresh, upgrade, types, argv scan of /proc cmdlines, negative args/secrets/state dir, DEV fingerprint before/after); 6 mutations on dialog/drawer/avatar; fresh verifier (contract 2026-10-25-v1-1-3-ui-foundations.md) + fresh security auditor.
Result: sandbox 40/42 first pass → defect up bad mode/version started 4 containers + secrets before refusing → fixed 57922e0, 15/15 rerun, upgrade rerun green, DEV unchanged. Mutations: 5/6 caught at once; M5 (confirmation closed before removal) missed until RPC slowed. Verifier CONFORME 8/8. Security PASS (MEDIUM pre-existing fixture password literal; LOW seed SQL interpolation of generated values, env breadth to webServer, refused URLs unmasked → fixed). check:all green 23:41:29; e2e:ui 26, e2e:shell 6, vitest app 458.
Anomalies: axe pre-existing serious findings (switcher, top-bar link label-in-name, dashboard dl) → KNOWN list + RR-7; the shell suite found a real focus defect in AvatarSection.
Action: keep mutation pass for browser tests; slow async backends in focus tests.
EVAL-039 | 2026-10-26 | V1-1.4 M3 animals + relations: gates, mutations, four migration states
Method: 14 mutations of M3 (PGlite pgTAP; M9 swapped into the sandbox, restored, md5 checked); 4 sandbox states (fresh, pre-M1→M1..M3, pre-M1→M1→M2+M3, M1+M2→M3): pgTAP 919, types --check, sandbox suites 46; e2e:ui 26, e2e:shell 6; import:equine --check 0 diff; check:all green 01:11:05 (e2e:ui PASSED, e2e:shell NOT RUN line); fresh verifier (contract 2026-10-26-v1-1-4-animals-relations.md) + fresh security auditor.
Result: verifier CONFORME 11/11; security PASS (MEDIUM: sandbox fixture password literal pre-existing, policy-via-helper false positive; LOW: authz read then lock not re-checked (no membership change in V1-1), no-op/replay unmetered (as M2), reads reachable by GET (app convention), service_role ALL on species pre-existing). Mutations 12/14 caught first pass; M6 (voided-only link counted) + M9 (foreign client locked) survived → tests added (533a1c4), both caught after.
Anomalies: pgTAP 014 green at first run (suspicious) → mutations proved it bites; axe route regex assumed /app/dashboard, dashboard is /app → shell suite failed until fixed (counter-mutations confirm scoping); permission layer refused sourcing sandbox secrets in shell → rehearsal via SQL + container env.
Action: keep lock-wait probes for cross-tenant RPCs (LRN-084); check real route table before path-scoped test rules.
EVAL-040 | 2026-10-26 | V1-1.5 client screens: gates, three verifier rounds, report check
Method: unit (supabase 92, app 509), access layer vs real PostgREST (sandbox 54), Chromium e2e:shell 42 (36 client journeys), mutations on app (request id per submit, double-click guard, late-answer guard, account reset, re-read loading) all caught; visual pass desktop+375 via temp capture spec; impeccable detect 0; report scan (decoded embedded zip); check:all green twice (02:04:48, 02:15:51); import:equine 0 diff; pgTAP 919; types match. Fresh security auditor + verifier ×3.
Result: security PASS (LOW: test SQL interpolation of sandbox uuids); noted USER_UPDATED reset of workspace unmounting forms → fixed e1c1980. Verifier 1 ECARTS (not-found heading unfocused → fixed 567ad40), verifier 2 ECARTS (docs counts uncommitted; floor assert drop of moved signIn → CLARIFICATIONS), verifier 3 CONFORME 11/11.
Anomalies: Prettier over whole src reformatted 39 files → restored, edits re-applied on originals; duplicate h1 (pre-existing top bar) broke selectors; StrictMode ran listener cleanup; drawer exit kept page inert; shell HTML report held typed passwords (BROWSER_TESTS claim false since V1-1.3) → reporter removed; permission layer refused rm -rf via variable (correct) and secrets sourcing.
Action: format new files only; dispatch verifier after committing docs; scan reports by content.
EVAL-041 | 2026-10-26 | V1-1.6 animal screens and relations: gates, three verifier rounds
Method: access layer by fork (unit 35, real PostgREST sandbox 7); screens unit tests by fork (57); Chromium e2e:shell 82 (animals 40), 3 full runs; 12 unit + 2 browser mutations all caught; visual pass desktop+375 (one batch fix: relation actions bordered); impeccable detect 0; check:all green 17:19:26; import:equine 0 diff; pgTAP 919; types match; DEV restore step rehearsed on sandbox (counts equal, 26 expected privilege errors). Fresh security auditor + verifier ×3.
Result: security PASS (MEDIUM pre-existing sandbox literal pw; LOW trace override by argv). Verifier 1 ECARTS: client journey failed after passing (late replay of the animal-section reload added by V1-1.6) → dropLateReplays for every shell spec. Verifier 2 ECARTS: pgTAP 014 #237 regex 20[0-9]{2}- matched a workspace uuid on long-lived sandbox (latent V1-1.4 false positive; would stop DEV db:test) → metadata minus workspace_id. Verifier 3 CONFORME 12/12.
Anomalies: Card dropped aria-labelledby since Step 7 (TS ignores hyphenated props; LRN-087); useT unstable in deps (LRN-088); en substring matches (Male/Female, Save/saved) → exact; safety check refused rm inside sh -c (not bypassed; dump left for owner).
Action: run the whole shell suite (not just the new spec) before the verifier; scope pgTAP regexes away from uuids; never chain rm into sh -c.
EVAL-042 — Gate chain substituted test:run for db:sandbox test (2026-11-04)
V1-2.6 Lot M gate chain ran pnpm test:run as third step instead of contract's pnpm db:sandbox test (sandbox suites on same fresh stack after e2e:shell, LRN-113). Two chains reported green with wrong step; §16.7 doc carried wrong order. Method: fresh verifier against contract C22 read packages/supabase/vitest.sandbox.config.ts and caught it. Anomaly: orchestrator's own gate script, not a subagent. Action: re-ran exact order on 697cefa and 7a4ce9e (sandbox test 105 passed / 10 skipped); doc fixed. Lesson: copy gate commands verbatim from contract/LRN, never paraphrase a step name.