34 KiB
Evals Registry
Format: EVAL-XXX | Date | What evaluated | Method | Result | Anomalies | Action
EVAL-001 | 2026-05-18 | Scaffold build + test pass
Method: pnpm install → pnpm build → pnpm test:run → pnpm typecheck. Result: all pass, 0 errors. Anomalies: pnpm version 11 instead of 9 (corepack issue on host). Action: document in BDR-007, BLK-001.
EVAL-002 | 2026-05-20 | B3 i18n — code review caught hardcoded FR in PDF import
Method: cavecrew-reviewer ran against full diff 0a05c47..HEAD after self-review + analyzer agent both reported CLEAN.
Result: reviewer flagged 3 CRITICAL: pdf/import.ts:9, pdf/attachment.ts:9-10, app/routes/HomeRoute.tsx:32 — hardcoded FR error messages leaking to EN UI.
Anomaly: prior i18n grep ('Annuler', 'Valider', etc.) missed these because the error-path files contained different French vocabulary ("Ce PDF n'a pas été..."). Implementer's TODO checklist did not include the PDF import error paths since they were not in the original component migration list. Analyzer also missed because the leak only surfaces under EN locale at runtime (no static signal in code).
Action: fixed in d602726+next via typed ImportError discriminant (BDR-015). Future i18n migrations should enumerate ALL files that emit user-facing strings (not just components/routes) — include error path files, exception messages, anything that may end up in window.alert or setError.
EVAL-003 | 2026-05-21 | PDF vet-marker ⚠ WinAnsi crash slipped past CI
Method: user manual export of a v2.0 horse report containing requires_vet_referral leaves → EditorRoute.tsx:263 PDF export failed Error: WinAnsi cannot encode "⚠" (0x26a0).
Result: 154/154 tests stayed green; bug shipped to user-facing path. Static signal absent (the ⚠ literal was visible in the source but typecheck/lint don't flag font-encoding issues).
Anomaly: pdf.roundtrip.test.ts exercised export with empty observations + no vet findings. The ⚠ prefix was added in 60e1b04-era (observationPage.ts:407 ternary) without a regression case covering a flagged leaf.
Action: hotfix swapped ⚠ → (!) (WinAnsi-safe). Added regression vet-referral marker is encodable by the body font that calls widthOfTextAtSize on a prefixed label — would catch any future non-WinAnsi marker. LRN-021 captured the pdf-lib StandardFonts WinAnsi limitation. Follow-up: switch PDF body to embedded TTF (Inter/Fraunces per CLAUDE.md spec) so the catalog can carry arbitrary glyphs — out of hotfix scope.
EVAL-004 | 2026-07-12 | Tour sweep converged 3 it. — agent spend-limit deaths resumable
Method: /tour full pipeline on faunosteo — semgrep floor ×3 + cso posture (it1) + full clean audit + fresh independent re-verify + doc-syncer. 13 fixed / 10 open residuals / 6 bugs quarantined; 11 commits on unmerged chore/tour-2026-07-12. Result: re-verify PASS (0 findings on 20 modified files); checks green throughout (typecheck, lint 0, 174/174, build, docker run non-root). Convergence honest: it2 surfaced 1 new tuning note (min-release-age 1440 < 10080 rec) → triaged open → it3 semgrep identical → converged at 3-iteration bound. Anomaly: both Phase A agents died mid-run on "monthly spend limit" — SendMessage to same agentId resumed from transcript, zero rework. PostToolUse hook false-positived 3× on 'dangerouslySetInnerHTML' as prose inside audit REPORTS (not code). Action: none blocking. Consider excluding .claude/audits/*.md from sink-pattern hook scan.
EVAL-005 | 2026-10-08 | Step 1 characterisation suite (91 cases)
Method: 7 files on real assets (SVG, catalog, fixtures), oracle = audit tables (06 §4, 05 §2.4). Run typecheck/test/lint/build. Result: 271/271 green, 0 src change. Audit figures all confirmed (26 mappings score 0, 792/612/41, 48 attrs…). Anomalies: 4 assertions wrong on first run, all test-side (census regex, stale closure, ZONES drawn as bullet rows not ' · ' line, attachment names in object streams). None app-side. Action: LRN-033. Known quirks pinned as "KNOWN:" tests (orphan findings, no_attachment cause lost, sphenoid id, fuzzy FR pairs).
EVAL-006 | 2026-10-08 | Step 3 Supabase foundation — PGlite suite
Method: 4 pgTAP files (81 checks: schema/privileges 24, profiles RLS 17, platform roles 28, audit 12) + schema↔types (6) + client (2) on PGlite; root typecheck/lint/build; app suite.
Result: all green first run except plan count (27→28) and pgtap-in-public polluting function census; 278 app + 13 package tests. Zero app behaviour change.
Anomalies: set_config rows appear in TAP line capture (noise, filtered by regex). now() constant in tx broke naive updated_at>created_at assertion.
Action: LRN-035/036. Not executed: real Supabase image (supabase test db), gen types diff, seed.sql (needs auth.identities + extensions.crypt), REST manual script. Flagged in report + BLK-006.
EVAL-007 | 2026-10-08 | Step 3 on the real Supabase stack
Method: supabase start (postgres 17.11.0.004), supabase test db, gen types --local diff, REST curl script (SUPABASE.md §8) with seed users, psql on audit, db reset.
Result: migrations + seed apply from zero; pgTAP 81/81 after fix; REST: alice 1 row, anon 42501, self-deactivate 42501 (column grant), self-grant 42501 (RPC message), cross-row PATCH 0 rows, owner grant → uuid, alice sees her role, owner reads 3 profiles; audit shows system bootstrap + owner grant.
Anomalies: (1) 6 pgTAP counts assumed empty DB — seed rows on real stack (PGlite runs no seed) → scoped to fixture ids + neutralise foreign owners before last-owner test. (2) CLI gen output unformatted + ComputedFields: never, no __InternalSupabase → hand-written file replaced by generated+prettier; scripts pipe prettier. PGlite harness had hidden both: environment parity gap, not logic gap.
Action: LRN-037 (tests must be seed-agnostic; generated files via deterministic formatter). BLK-006 resolved.
EVAL-008 | 2026-10-08 | Step 4 sheet model — domain tests, pgTAP both engines, fresh verifier + security gates
Method: packages/domain 61 vitest (schema 29 + integrity 32, one per rule code); pgTAP 005 (127 checks) + 001 census on PGlite and real stack (db reset, db test 210/210, gen types diff clean); root check (typecheck, 356 tests, lint 0, build); fresh verifier (contract C1–C13) + security-auditor (semgrep 725 rules 0 findings + checklist).
Result: first gate pass → verifier ECARTS(5), security BLOCK(1): private EXECUTE not revoked (BLK-007), visibility change unaudited, 7 RPCs without non-editor denial test, doc/code mismatches, 75-line SQL function, any suppression in a test. All fixed same session: +32 pgTAP checks, hardening migration, visibility RPC, caps, authz-first, SQL check split in 4 functions, set()/unset() typed mutations. Second pass: fresh verifier (pending at write time).
Anomalies: 6 test-side fixes on first pgTAP run (RLS hides rows → "no change" asserts; data-modifying CTE; collation of order by; trigger name order decides errcode; audit list; pg_temp grants). Executor (opus) deviated usefully: RegionCode pattern separate from StableId.
Action: LRN-038/039/040, BLK-007. Keep the two-gate pass on every migration step.
EVAL-009 | 2026-10-09 | Step 4b equine import — importer tests, pgTAP both engines, Docker build, gates
Method: packages/anatomy 30 vitest (counts in=out on both sides, special cases, determinism, golden artifacts, guards); pgTAP 001–007 on PGlite and real stack (db reset, db test 256/256, gen types diff clean); root check (61+30+20+278, lint 0, build); docker build of the app image; Storage API end-to-end (owner 400 before confirmation → 200 after, alice/anon 400); fresh verifier (C1–C12) + security-auditor (semgrep 130 rules, 0 findings).
Result: first pass verifier ECARTS(4) (self-referential in=out control, undeclared importer-authored labels, docs vs literal sql_paths, silent FR→EN fallback) + security PASS with MEDIUM hash-claim bypass (register_sheet_asset accepts any sha256 → editor reads foreign objects). All fixed: uploaded_at + confirm_sheet_asset_upload (service role) required by the read predicate, bob-claims-hash pgTAP, authored-label anomalies (27 anomalies now), seeds-listed test, Dockerfile missing workspace manifests (found by building, not by the gates).
Anomalies: real image storage.protect_delete aborts direct DELETE tests (LRN-041); pg_temp fixtures need service_role grants; docker build raced the lockfile refresh (rebuild clean).
Action: fresh verifier re-dispatched on the delta; BDR-034, LRN-041.
EVAL-010 | 2026-10-10 | Step 4c equine curation — research agents, changeset, gates
Method: 4 research agents (skull, axial, limbs, osteo/legal), every citation tagged VERIFIED-fetched/snippet/UNVERIFIED; changeset authored in scratch python (44 changes / 679 ops / 48 refs) → pipeline Zod + 20 integrity rules; anatomy 100 vitest (ops, rename propagation, overlay, golden), pgTAP 001–008 on PGlite and real stack (db reset, db test 267/267, gen types diff clean); root check (68+21+100+278, lint 0, build); docker build; Storage e2e after upload (owner 200, alice/anon 400); gridded magick crops to verify point positions; fresh verifier ×3 + security-auditor ×1.
Result: verifier pass 1 ECARTS(4): B changes left targets inherited (CHG-008/032/039 + partials), 3 stale doc numbers, lockfile outside contract scope (orchestrator omission), eslint-disable any in a test → executor: applier rule (B/C ⇒ touched findings/annotations to_confirm, reviewed refused in B/C), typed test reads, docs. Pass 2 ECARTS(2): JSAVA citation without author (completed from the PDF: Le Roux & Carstens 2018, DOI), DEPLOY.md stale faunosteo-web. Pass 3 ECARTS(1): 3 stale doc lines (migration list, "two assets", CLAUDE.md migrations) → fixed by hand, cap of 3 verifier passes reached, not re-verified. Security PASS, LOW: SQL gate did not check reference url scheme → reference.url.scheme added (migration still unmerged, pgTAP 008 10 cases); .env.example COMPOSE_PROJECT_NAME=osteo-report overrides compose name: → file denied to Claude, user action.
Anomalies: ImageMagick MSVG drops fill="none" shapes and shapes with <title> child (LRN-042); two legacy points (t18-l1, l6-s1) floated above the silhouette, found only by the overlay render, not by the catalog audit; Supabase db reset unconfirms the asset row (upload script must re-run: objects kept, row confirmed).
Action: user validation of the draft content (REVIEW-REPORT §10: 11 questions); never publish; Step 5 only after validation.
EVAL-011 | 2026-10-10 | Step 5 app on the canonical sheet — suites on the bundled sheet, archive PDFs, gates
Method: golden v2 PDFs frozen from the old pipeline before any change; suites rewritten on the bundled draft (domain 94, anatomy 107, supabase 21, app 168+: service branches 10/5/12/1, store + v2 draft migration, provider, canvas, pickers, card, route click chain, PDF v3 roundtrip FR/EN + both v2 fixtures + synthetic removed/unknown ids); root check, vite builds none/draft (grep of a draft label in dist = 0/1), docker build, real stack 267/267 + types; legacy-vs-app marker style diff by the verifier (0 differences); fresh verifier ×3 + security-auditor ×1.
Result: verifier pass 1 ECARTS(36) = 4 real (golden test not pinning the new generated files, EditorRoute 398 lines, 5 stale doc lines, one eslint-disable) + my contract omissions (scope files, deleted legacy suites) + floor noise ("species" matched "spec"); pass 2 ECARTS(4) = replacement assertions missing for 4 deleted suites; pass 3 ECARTS(4) = doc counts + 3 more suites' assertions (PDF conclusion/metadata/attachment structure, ObservationCard, persistence). All closed by executors (two) and by hand (docs); cap of 3 passes reached → last lot not re-verified. Security PASS with LOW: prototype-key lookup on untrusted ids (fixed: Object.hasOwn + test), unbounded inflate on the primary attachment path + no file cap (fixed: 16 MiB attachment, 50 MB PDF + test), denylist SVG sanitiser acceptable while assets are repo-built (allowlist before Storage-served assets, Step 13).
Anomalies: executor ran pnpm format repo-wide and reverted by hand (watch for format in briefs); foreign JSON attachment (no version) surfaced as incompatible_version → fixed to no_attachment (legacy message); verifier floor counts deleted legacy tests as findings until CLARIFICATIONS name them.
Action: user validation of Step 5 (manual FR/EN flow, visual before/after); BDR-038, LRN-043.
EVAL-012 | 2026-10-10 | Step 5 addendum during user manual validation — per-leaf migration tests, archive check tool, final controls
Method: user asked (while testing) for final controls on last lot + per-leaf test beyond the map + simple v2/v3 PDF comparison method. Built: anatomy test all 612 legacy leaves through migrateReportV2ToV3 with committed map vs draft document (442 identical / 141 moved = 134 category + 7 leaf renames / 29 removed, snapshots + vet flag verbatim, 26 annotations on right view, v3 re-read stable); app test same leaves via sheetIndex.nodeAt + v3 PDF roundtrip; CLI pnpm report:compare (pure module + entry, readAttachedDocument split out of extractReportJson, behaviour unchanged). Contract criteria 16-18 added, gates MET, fresh verifier ×3 + security ×2.
Result: verifier pass 1 ECARTS(2) = floor only (EVAL-011 prose "eslint-disable" false positive → CLARIFICATIONS; 11 dropped regionFromLabel assertions → new legacy.regions.test.ts on inferRegionCode); security PASS + 2 MEDIUM (terminal control chars in untrusted strings; free text printed by default) → executor: esc() + --full privacy default; pass 2 ECARTS(1) (--diff still clear text, JSON.parse message echoed attachment fragment) + security PASS 4 LOW → executor: diff follows --full, generic reason, Cc/Cf escaping, regular files only; pass 3 CONFORME. Workspace 94/21/119/181, docker build OK.
Anomalies: verifier stage runs after executor edits only (concurrent reads otherwise); gates.sh needs EVIDENCE: PENDING placeholder lines on new criteria. Residuals noted, not fixed: ZodError message echoes closed-vocabulary received values (never names), compareReport computes v3 lines twice, template-literal lines >80 chars, locals >5 in two functions.
Action: await user manual validation of Step 5; residuals in TODO follow-up.
EVAL-013 | 2026-10-11 | Step 6 autonomous PDF archives — 5 phases by executors, 3 verifier passes, 4 security audits
Method: analysis (3 analyzers + measurements) challenged by 3 plan-challengers (correctness/robustness/simplicity, 48 findings folded in) → user validation with D7/D8/D13 adjustments → Part II contract (exact types) → one executor per phase (0 freeze goldens, 1 domain, 2 anatomy compose, 3 engine, 4 app) with closed contracts, orchestrator commits → whole-step verifier + security auditor → fix loops.
Result: Phase 0 froze 39 goldens (text/images/links per locale × logo mode, view facts) + 2 route characterisation tests with the OLD code; engine reproduced 36 goldens with 0 mismatch (views: data-* 2→0 and hidden-marker titles declared relaxed). Verifier pass 1 ECARTS(31) = all oracle/scope defects of MY contract (grep on parametrised table, xmldom filter, narrow FILE SCOPE, 3 deleted tests with moved coverage) → contract amended → pass 2 CONFORME 13/13. Security pass 1 BLOCK(1) HIGH real + PoC: archived branding URL written unescaped into /URI (PDF object injection via exportFromEnvelope), +3 MEDIUM/4 LOW → executor fixed (hex URI, strict URL schema, export re-validation, PNG IHDR budget, CSS-escape rules, NaN guard, file.size) → pass 2 PASS but 2 MEDIUM incl. a regression the fix introduced (UTF-16 BOM URI dead in poppler) + PNG double-IHDR bypass → executor → pass 3 PASS + 1 MEDIUM (UPNG nextZero spin on tEXt) → executor (chunk allowlist + IEND) → pass 4 PASS + 3 MEDIUM (UPNG inflateRaw spins on truncated IDAT with allowlisted chunks only; IHDR fields unvalidated; zero dimension) → executor: native bounded inflateFlate of the IDAT against the exact raw size, IHDR fields, sides ≥ 1, no interlace, raw ≤ 16 MiB → pass 5 focused. Lesson: a third-party decoder on untrusted bytes needs a native oracle in front, not a chunk allowlist. Final: 656 tests (194/21/130/88/223), typecheck, lint, build, docker ×4 green; goldens byte-identical since 8b86b92.
Anomalies: contract oracles written fast = 2 verifier passes lost (lesson: run every CHECK once before dispatching the verifier); security fix introduced a functional regression invisible to tests because the test reader decoded the BOM (lesson: assert raw bytes, cross-check with a real reader like poppler); executors denied rm/git rm (orchestrator deletes); one executor used git stash despite "no git state" (now forbidden explicitly in briefs); demo envelope with archived branding = 136 KB stored / 306 KB raw (raw > 250 KB, stored < 250 KB: reported, user said data first).
Action: await user manual validation + UX decision on re-export of the kept archive; BLK-009 (pdf-lib ObjStm inflation) before external users; react-router advisories (pre-existing) to bump in a chore.
EVAL-014 | 2026-10-11 | Step 6 closure round — manual-test feedback, BLK-009 scanner, final audits
Method: user manual tests round 1 (logos absent, picker blurry) → reproduction attempts (headless Chromium dev/preview/Docker, Firefox 157 via esbuild harness + raw WebDriver BiDi client, CDP LayerTree, LCD-fringe metric, headed runs on the Wayland session) → fixes by executors → user round 2 confirmed both OK → closure decisions → BLK-009 implemented (byte-scan reader, no object streams) → verifier + 2 security audits + 1 fix loop.
Result: neither visual defect reproduced here (4 environments, 2 engines, 2 PDF renderers); shipped the environment-independent fixes (branding bundled, no fetch; warnings surfaced) and the user confirmed logos + crisp text. Scanner: verifier CONFORME 7/7; audit 1 PASS + 2 MEDIUM (Chromium DecompressionStream read-ahead defeats the cancel: +2.2 GB on a 4 GiB bomb; ×32 candidate CPU) → executor (16 KiB slice feeding, shared 16 MiB budget, monotonic bounded endstream search, { pre-check; 133 pdf tests) → audit 2 focused. Workspace 710 tests, build, docker green.
Anomalies: the blur fix of the previous round (will-change) had been marked PARTIALLY MET "desktop by user" — the user's "still blurry" came from that unverified half; this round could not verify real-GPU text rendering either (headless engines render grayscale AA, headed captures failed) → lesson: a screen-rendering defect needs the user's own capture + an A/B recipe early, not days of headless measurement. The Node vitest suite cannot observe browser stream read-ahead: record browser-only bounds as owed manual checks (PDF_IMPORT_HARDENING §5).
Action: BLK-009 stays "mitigated, gate open" (BDR-040); Step 6 finished into develop; release/0.2.0 prepped for the user's go.
EVAL-015 | 2026-10-11 | Step 7 identity — gates on the whole step
Method: contract with 16 criteria (2026-10-11-step-07-verify.md) → fresh verifier; security-auditor (semgrep 159 rules + checklist) on c746952..HEAD; headless e2e vs the local stack (playwright-core in scratchpad, Mailpit API); real-stack pgTAP + docker build by the orchestrator.
Result: verifier pass 1 ECARTS(2): (a) real gap, safeNextPath prefix check let /\host and /<TAB>/host through (router location.assign fallback = open redirect after login) → fixed by resolving against a sentinel origin and rebuilding from parsed parts, 5 new cases; (b) contract regex over-matched other unions ('google' | 'apple', 'fr' | 'en') → CHECK scoped. Pass 2 CONFORME 16/16. Security PASS, 1 MEDIUM (same redirect, already fixed) + 4 LOW: 3 closed same session (sign-out failure reported, reauthentication_needed phrased, exact prod redirect allowlist documented), 1 pre-existing (TOTP not enforced for staff = documented gap). E2E 20/20. Workspace 763 tests, lint 0, build, docker, real-stack pgTAP 280 green.
Anomalies: both gates found the same redirect hole my own tests missed → a prefix check on a URL is never enough, parse it; the auditor misread the sign-out note ("this device") as "everywhere" but the LOW still pointed at a silent failure path worth phrasing. Mockups validated with ChatGPT are not in the repo: the screens follow the written direction and the existing tokens, pixel fidelity unverifiable here.
Action: awaiting user validation of Step 7 (manual procedure IDENTITY.md §5); release/0.2.0 still open for the human go.
EVAL-016 | 2026-10-12 | Rust-by-default architecture review — 4 analyzers + 3 researchers + 3 plan-challengers
Method: 4 read-only analyzers (domain, anatomy, pdf + app pdf side, app logic + supabase) wrote factual inventories to the scratchpad; 3 general-purpose researchers verified crate versions/maintenance/advisories and Supabase facts on crates.io, RustSec, GitHub, supabase docs (2026-10-12), UNVERIFIED items listed; study rev 1 written from them, committed (6a078da); 3 plan-challengers (correctness FATAL(11), robustness FATAL(11), simplicity CONCERNS(9); 61 findings) → rev 2 with 56 folded, 2 declined with reasons (§0.2).
Result: real blockers caught by the gates, not by me: (1) VI.4 deleted TS modules still imported by later-phase TS consumers (pipeline, engine); (2) production CSP script-src 'self' refuses wasm, VPS 1 GB compiles images; plus wrong premises: envelope byte-parity (integer-valued floats, insertion order), Prettier identity overstated (hash only compared), catch_unwind unusable on wasm32 (panic=abort), serde-wasm-bindgen null-for-absent, parent re-decode breaking the keyless invariant, Step 15 misread as server rendering. Simplicity: 6 crates → 3, dual-run harness → frozen goldens, Landlock + 3 rlimits redundant, constraints generator/schemars/cargo-chef/cargo-audit/infer/img-parts dropped.
Anomalies: one analyzer had a read-only find -exec cat | wc -l refused (no rule named) and re-ran as find | xargs wc -l (already allowed earlier) — reported to the user as a possible workaround; agents flagged writing report files to the scratchpad against their default (orchestrator asked for it). Three challenger claims came from memory not code (json_compatible null, panic=abort, musl mprotect) → folded as "verify at R0/R1".
Action: owner validates study + D-R1..D-R14; nothing implemented; CLAUDE.md rule section added; TODO roadmap lines.
EVAL-017 | 2026-10-13 | Rust R0 foundation — proofs in the real environment, 3 verifier passes, 3 security passes
Method: contract 2026-10-13-rust-r0-verify.md (16 criteria, CHECK/EXPECT) → fresh verifier; security-auditor on 5fec065..HEAD (semgrep p/security-audit + p/secrets + p/owasp-top-ten + checklist + manual Rust review); all cargo commands through the Docker builder (no Rust on the host); worker sandbox tests + strace derivation inside the Alpine test stage on the release-profile musl binary; wasm proofs in vitest on the built module; CSP proof with the production nginx image + headers in headless Chrome 156.
Result: verifier pass 1 ECARTS(2), pass 2 CONFORME 16/16, pass 3 ECARTS(2) — every gap an oracle of MY contract (greps counting comment lines, the contract matching its own regex, the [gated] SAFETY form and a second refusal arm moved by the security fixes); code behaviour confirmed by reading each time. Security pass 1 BLOCK(1) HIGH real: shell injection via the workflow_dispatch input of release-images.yml, + 6 MEDIUM (shared BuildKit caches, tkill/tgkill without arg filters, core dumps, creds on the runner, no advisories gate, mutable action tag) → fixed; pass 2 PASS but 3 MEDIUM introduced by my fixes (ephemeral builder never used: GITHUB_ENV applied one step too late, silent fallback; rm -rf "$VAR"; content reaching the Broken outcome via Reject::Internal) → fixed; pass 3 PASS, 2 new MEDIUM (unpinned setup-node and pnpm install inside the token job) → gates moved to a secret-less job; remaining lows documented as R1/Step 27 follow-ups. Experimental findings (LRN-049): serde_json fast float path (float_roundtrip), traps leave the instance unusable + ESM cache, RLIMIT_CPU soft=hard → SIGKILL, minimal rustup profile, cargo-deny path wildcard, 7 traced syscalls, strace blind once dumpable=0 (marker became a syscall). Final: cargo 12+14 core tests, 9 unit + 16 sandbox worker tests on the release profile, 14 vitest proofs, root pnpm check 777 tests green, worker image 8.7 MB read-only/cap-drop, CSP refused/accepted as predicted, floor-guard clean ×3.
Anomalies: my contract oracles cost two verifier passes (lesson repeated from EVAL-013: run every CHECK once before dispatch, and re-run them after any fix commit); two of my own security fixes created new findings (an env variable applied too late, a recursive delete through a variable: the exact pattern the destructive-tools rule forbids) → fixes to CI files need the same adversarial pass as code; a scripted edit silently aborted on a rustfmt-reformatted string and the commit went out with the old test (caught by the next build); the [gated 2026-10-13] SAFETY markers were placed by me at the auditor's request, not by the owner → surfaced in the report for the owner's confirmation; one security-auditor Bash call (cat of two files) was refused twice by the auto-mode hook, the agent used Read instead (reported, no workaround). Not proven here: the Gitea runner (caches, timings, workflow syntax such as job-level env, actions/cache), HTTPS from the worker (no HTTP client before R1), browser-side fresh-instance mechanism (R2).
Action: owner validation of R0 (report with proved vs hypothetical); R1 not started.
EVAL-018 | 2026-10-13 | R0 correction: native Rust toolchain on the Ubuntu dev server — 2 verifier passes, 2 security passes
Method: host audit first (rustup already user-level, 1.99.0 auto-installed by rustup show, system deps present, no sudo); installs --locked; scripts moved to scripts/, pnpm check:all gate; contract 2026-10-13-rust-r0-native-verify.md (11 criteria) dry-run by me before dispatch (3 oracles fixed: comment prose matched a code grep, per-package vitest totals, README phrasing); fresh verifier ×2; security-auditor on bd9c7d1..6e2bbec then on the fix commits; full gate run twice (check-all.log, check-all-2.log).
Result: verifier pass 1 ECARTS(2) = floor STUB hits on doc prose "(not implemented)" → reworded; pass 2 CONFORME 11/11, floor clean. Security pass 1 PASS + 1 MEDIUM (no advisory check has ever run: nightly job never executed) + 6 LOW (PATH prepend, rm -rf "$out" via variable, prctl comment overstating the kernel contract + i32 into variadic unsigned long, dormant CI triggers, native build.rs exposure, drift gate blind to untracked files) → fixed except the trigger question (owner's CI/CD step); pass 2 PASS + 1 MEDIUM (my ci.yml drift step failed open on a git error: test -z "$(…)" drops the exit status) + --untracked-files=all + stale comment → fixed (496f549), verified by execution + shellcheck, no third dispatch (3-line change). Numbers: native rust-check 32 s warm / 1 s incremental, cold workspace build 31.5 s, wasm 224 777 B (+64 B vs builder, path strings), 14 proofs, check:all 1 min 59 s then 1 min 39 s, 777 Node tests, 16 sandbox tests, allowlist covers the trace, worker read-only run --check OK.
Anomalies: the R0 note "no cargo on the host" was stale, rustup had been installed 2026-10-12 23:27 (rtk) → always re-audit the host at session start; rustup show has a side effect (auto-install) → acceptable here (requested install) but say so; a fix I wrote for one security finding introduced the next MEDIUM (fail-open test) → the EVAL-017 lesson again: CI one-liners need the same scrutiny as code, prefer assign-then-test; the harness clock jumped 01:16 → 09:43 mid-session (timestamps in logs are not durations).
Action: owner's final validation of the R0 correction; R1 not started; CI/CD decision separate (trigger mode, runner label, action pinning, advisories cadence).
EVAL-019 | 2026-10-13 | Step 8 / R1 files foundation — 2 forks, verifier ×2, security ×5, e2e ×7, browser smoke
Method: contract-level STORAGE.md written first; migration + pgTAP by me while fork A (Rust loop) and fork B (TS + UI) ran with full context; real-stack e2e (pnpm e2e:avatar, worker container, fresh admin user) + headless Chromium smoke (playwright-core in scratchpad, cached chromium, dev server :5174) for the browser canvas path; contract 2026-10-13-step-08-r1-verify.md (15 criteria, oracles dry-run before dispatch) → fresh verifier; security-auditor on 3f1105b..HEAD then focused passes on each fix commit; pnpm check:all ×4.
Result: verifier pass 1 ECARTS(4) = oracle (for update row-lock grep) + FILE SCOPE (tracking files, lockfile) → rev 2 → CONFORME 15/15, floor clean. Security pass 1 PASS (2 MEDIUM: ceiling ratchet by rejected rows, cleartext http any host; 9 LOW) → fixed; pass 2 PASS but my fixes introduced 2 MEDIUM (bypassable host check 127.*/numeric, ceiling undercount vs contract) → fixed; pass 3 BLOCK(1) HIGH: my purge-queue join let begin spam pin the ceiling with claims → ceiling rewritten on bytes present + cap per pending claim; pass 4 PASS (MEDIUM residual: per-account holding) → 64 MiB per-account budget; pass 5 PASS (LOW hygiene) → comments/doc. e2e 5/5 ×7 (found the Storage API 400-as-404 making purge acks loop: added queue-drained assertion); smoke 18/18 desktop+mobile; final: pgTAP 140 (real stack 420 total), app 300, supabase 37, worker 21+11 native, sandbox 16, allowlist = 7 syscalls unchanged, image 11.6 MB.
Anomalies: EVAL-017 lesson again ×3 — each security fix needed its own adversarial pass and twice created a new finding (host check, ceiling join); a security control I added became the attack lever (BLOCK). Fork B's database.types.ts hand-write matched the generator byte-for-byte except ordering; fork A found the 400/404 only via e2e. gitflow finish of R0 and any .env* write refused by the harness classifier; git config reads refused too. Verifier sandbox intercepted the literal pnpm lint (rtk wrapper) → oracle uses the binary path.
Action: owner's manual validation (TEST FONCTIONNEL + TEST GRAPHIQUE), owner runs gitflow finish for R0 then R1 after validation; no merge, no deploy, release 0.2.0 untouched.
EVAL-020 | 2026-10-14 | Rust R2 domain core — 3 plan challengers, 4 forks, verifier, security ×2, CSP proof on the image
Method: plan + contract written first (.claude/tasks/plans/2026-10-13-rust-r2-domain-core.md), 3 plan-challengers (correctness, robustness, simplicity) before code; freeze-then-switch: envelopes, Zod output oracles, characterisation and verdict tables written by the TS before trusting Rust; forks A/B/C in worktrees on disjoint files (sheet rules + conformance, report/archive rules, editor rules), runtime + app switch by me, fork D adapted the app suites in tree; browser checks with playwright-core from the scratchpad (dev server, prod build, real image under the DEPLOY CSP); contract 2026-10-14-rust-r2-verify.md (12 criteria, every oracle dry-run before dispatch) → fresh verifier; security-auditor on the whole range, then a focused pass on the fix commit.
Result: challengers 45 findings incl. 1 BLOCKER (degraded editor redirect could overwrite an unread draft) → plan rev 2 before any app code; contracts byte-equal to Zod on first run; forks merged with zero conflicts (one stub replaced); fork D found 2 real src bugs (Node source path under happy-dom, failed quarantine reported as success) → fixed with tests. Verifier: 12/12 MET, ECARTS(2) = floor DELETED_TEST (zoneInteraction suite ported to cargo; R0 skip helper replaced by a stricter global setup) → human micro-gate for the owner. Security: PASS (3 MEDIUM nginx false positives pre-existing, 5 LOW) → by-reference deserialisation (also −134 KB raw / −31 KB gzip module) + literal delete paths. Final: cargo core 135, worker+wasm 32, Node 906 (app 313), pgTAP PGlite 12 files / real stack 446, CSP proof 15/15, check:all green ×3, module 1,046,498 B raw / 340,929 B gzip.
Anomalies: worktree base stale (BLK-010); harness decodes \u escapes in tool inputs (LRN-054); my first runtime test helper reused the pattern Vite rewrites; the CSP proof first failed on its own harness (EN locale, disclaimer modal, caught compile error logs nothing) — proof scripts need the same dry-run discipline as oracles; a security LOW fix improved size, measure before assuming a cost.
Action: owner manual LAN validation, L1 decision, approve the 2 deleted test files (floor), then merge go; R3 next, not started.
EVAL-021 | 2026-10-14 | R2 closure — L1 refusal, deletions proof, fresh verifier + security
Method: owner asked to fix L1 (no silent U+FFFD) + prove the 2 approved deletions + checks before merge. Mapped every JSON entry into the core (TS side always JSON.stringify → escapes reach Rust as ASCII); preservation costed (own JSON reader + lossless string type ~150 fields) → AskUserQuestion: refusal + quarantine chosen, detailed import wording chosen. Stale-module proof run before wasm:build (suites throw pkg is stale). Worktrees checked blob by blob vs branch history (60/66 identical, 6 = stub, merged mod lists, 2 lifted #[ignore]). Contract 2026-10-14-rust-r2-l1-verify.md (13 criteria, each dry-run) → fresh verifier; fresh security-auditor on 36316cf..HEAD.
Result: verifier CONFORME 13/13 (floor 2 = the approved deletions, authorised by REQUEST); security PASS, 6 LOW: wasm-bindgen &str would re-encode a raw lone surrogate (unreachable, JSON.stringify callers; documented invariant), pretty_json proof op refuses as invalid_json, scanner not string-aware (classification only), newest-only quarantine slot, archive quarantine failure under full storage lets a later reset remove it (pre-existing R2, reported to owner), semgrep partial parse. Final: core 142, Node 915 (app 320, supabase 40, core-wasm 26), PGlite pgTAP 12 files, real stack 446/446, types no drift, check:all green 1 min 32 s, module 1,040,784 B raw / 339,933 B gzip.
Anomalies: refusal broke a test loader (oracle file freezes TS outputs with lone escapes) → loader spells escapes out, never normalises (LRN-055); cargo deny advisories (warn-only) shows RUSTSEC-2026-0346/0347/0348 on rustls 0.23.45 (worker, none applicable: no brotli, built-in session cache, no ECH) → bump follow-up; supabase/.temp/cli-latest tracked by mistake, rewritten by the CLI, restored; export PATH=… denied → absolute ~/.cargo/bin/cargo; first commit with chained git config denied → plain commit OK.
Action: gitflow finish into develop (owner authorised after L1 + checks), worktree cleanup (BLK-010), R3 summary only.