19 KiB
Evals Registry
Format: EVAL-XXX | Date | What evaluated | Method | Result | Anomalies | Action
EVAL-001 | 2026-05-18 | Scaffold build + test pass
Method: pnpm install → pnpm build → pnpm test:run → pnpm typecheck. Result: all pass, 0 errors. Anomalies: pnpm version 11 instead of 9 (corepack issue on host). Action: document in BDR-007, BLK-001.
EVAL-002 | 2026-05-20 | B3 i18n — code review caught hardcoded FR in PDF import
Method: cavecrew-reviewer ran against full diff 0a05c47..HEAD after self-review + analyzer agent both reported CLEAN.
Result: reviewer flagged 3 CRITICAL: pdf/import.ts:9, pdf/attachment.ts:9-10, app/routes/HomeRoute.tsx:32 — hardcoded FR error messages leaking to EN UI.
Anomaly: prior i18n grep ('Annuler', 'Valider', etc.) missed these because the error-path files contained different French vocabulary ("Ce PDF n'a pas été..."). Implementer's TODO checklist did not include the PDF import error paths since they were not in the original component migration list. Analyzer also missed because the leak only surfaces under EN locale at runtime (no static signal in code).
Action: fixed in d602726+next via typed ImportError discriminant (BDR-015). Future i18n migrations should enumerate ALL files that emit user-facing strings (not just components/routes) — include error path files, exception messages, anything that may end up in window.alert or setError.
EVAL-003 | 2026-05-21 | PDF vet-marker ⚠ WinAnsi crash slipped past CI
Method: user manual export of a v2.0 horse report containing requires_vet_referral leaves → EditorRoute.tsx:263 PDF export failed Error: WinAnsi cannot encode "⚠" (0x26a0).
Result: 154/154 tests stayed green; bug shipped to user-facing path. Static signal absent (the ⚠ literal was visible in the source but typecheck/lint don't flag font-encoding issues).
Anomaly: pdf.roundtrip.test.ts exercised export with empty observations + no vet findings. The ⚠ prefix was added in 60e1b04-era (observationPage.ts:407 ternary) without a regression case covering a flagged leaf.
Action: hotfix swapped ⚠ → (!) (WinAnsi-safe). Added regression vet-referral marker is encodable by the body font that calls widthOfTextAtSize on a prefixed label — would catch any future non-WinAnsi marker. LRN-021 captured the pdf-lib StandardFonts WinAnsi limitation. Follow-up: switch PDF body to embedded TTF (Inter/Fraunces per CLAUDE.md spec) so the catalog can carry arbitrary glyphs — out of hotfix scope.
EVAL-004 | 2026-07-12 | Tour sweep converged 3 it. — agent spend-limit deaths resumable
Method: /tour full pipeline on faunosteo — semgrep floor ×3 + cso posture (it1) + full clean audit + fresh independent re-verify + doc-syncer. 13 fixed / 10 open residuals / 6 bugs quarantined; 11 commits on unmerged chore/tour-2026-07-12. Result: re-verify PASS (0 findings on 20 modified files); checks green throughout (typecheck, lint 0, 174/174, build, docker run non-root). Convergence honest: it2 surfaced 1 new tuning note (min-release-age 1440 < 10080 rec) → triaged open → it3 semgrep identical → converged at 3-iteration bound. Anomaly: both Phase A agents died mid-run on "monthly spend limit" — SendMessage to same agentId resumed from transcript, zero rework. PostToolUse hook false-positived 3× on 'dangerouslySetInnerHTML' as prose inside audit REPORTS (not code). Action: none blocking. Consider excluding .claude/audits/*.md from sink-pattern hook scan.
EVAL-005 | 2026-10-08 | Step 1 characterisation suite (91 cases)
Method: 7 files on real assets (SVG, catalog, fixtures), oracle = audit tables (06 §4, 05 §2.4). Run typecheck/test/lint/build. Result: 271/271 green, 0 src change. Audit figures all confirmed (26 mappings score 0, 792/612/41, 48 attrs…). Anomalies: 4 assertions wrong on first run, all test-side (census regex, stale closure, ZONES drawn as bullet rows not ' · ' line, attachment names in object streams). None app-side. Action: LRN-033. Known quirks pinned as "KNOWN:" tests (orphan findings, no_attachment cause lost, sphenoid id, fuzzy FR pairs).
EVAL-006 | 2026-10-08 | Step 3 Supabase foundation — PGlite suite
Method: 4 pgTAP files (81 checks: schema/privileges 24, profiles RLS 17, platform roles 28, audit 12) + schema↔types (6) + client (2) on PGlite; root typecheck/lint/build; app suite.
Result: all green first run except plan count (27→28) and pgtap-in-public polluting function census; 278 app + 13 package tests. Zero app behaviour change.
Anomalies: set_config rows appear in TAP line capture (noise, filtered by regex). now() constant in tx broke naive updated_at>created_at assertion.
Action: LRN-035/036. Not executed: real Supabase image (supabase test db), gen types diff, seed.sql (needs auth.identities + extensions.crypt), REST manual script. Flagged in report + BLK-006.
EVAL-007 | 2026-10-08 | Step 3 on the real Supabase stack
Method: supabase start (postgres 17.11.0.004), supabase test db, gen types --local diff, REST curl script (SUPABASE.md §8) with seed users, psql on audit, db reset.
Result: migrations + seed apply from zero; pgTAP 81/81 after fix; REST: alice 1 row, anon 42501, self-deactivate 42501 (column grant), self-grant 42501 (RPC message), cross-row PATCH 0 rows, owner grant → uuid, alice sees her role, owner reads 3 profiles; audit shows system bootstrap + owner grant.
Anomalies: (1) 6 pgTAP counts assumed empty DB — seed rows on real stack (PGlite runs no seed) → scoped to fixture ids + neutralise foreign owners before last-owner test. (2) CLI gen output unformatted + ComputedFields: never, no __InternalSupabase → hand-written file replaced by generated+prettier; scripts pipe prettier. PGlite harness had hidden both: environment parity gap, not logic gap.
Action: LRN-037 (tests must be seed-agnostic; generated files via deterministic formatter). BLK-006 resolved.
EVAL-008 | 2026-10-08 | Step 4 sheet model — domain tests, pgTAP both engines, fresh verifier + security gates
Method: packages/domain 61 vitest (schema 29 + integrity 32, one per rule code); pgTAP 005 (127 checks) + 001 census on PGlite and real stack (db reset, db test 210/210, gen types diff clean); root check (typecheck, 356 tests, lint 0, build); fresh verifier (contract C1–C13) + security-auditor (semgrep 725 rules 0 findings + checklist).
Result: first gate pass → verifier ECARTS(5), security BLOCK(1): private EXECUTE not revoked (BLK-007), visibility change unaudited, 7 RPCs without non-editor denial test, doc/code mismatches, 75-line SQL function, any suppression in a test. All fixed same session: +32 pgTAP checks, hardening migration, visibility RPC, caps, authz-first, SQL check split in 4 functions, set()/unset() typed mutations. Second pass: fresh verifier (pending at write time).
Anomalies: 6 test-side fixes on first pgTAP run (RLS hides rows → "no change" asserts; data-modifying CTE; collation of order by; trigger name order decides errcode; audit list; pg_temp grants). Executor (opus) deviated usefully: RegionCode pattern separate from StableId.
Action: LRN-038/039/040, BLK-007. Keep the two-gate pass on every migration step.
EVAL-009 | 2026-10-09 | Step 4b equine import — importer tests, pgTAP both engines, Docker build, gates
Method: packages/anatomy 30 vitest (counts in=out on both sides, special cases, determinism, golden artifacts, guards); pgTAP 001–007 on PGlite and real stack (db reset, db test 256/256, gen types diff clean); root check (61+30+20+278, lint 0, build); docker build of the app image; Storage API end-to-end (owner 400 before confirmation → 200 after, alice/anon 400); fresh verifier (C1–C12) + security-auditor (semgrep 130 rules, 0 findings).
Result: first pass verifier ECARTS(4) (self-referential in=out control, undeclared importer-authored labels, docs vs literal sql_paths, silent FR→EN fallback) + security PASS with MEDIUM hash-claim bypass (register_sheet_asset accepts any sha256 → editor reads foreign objects). All fixed: uploaded_at + confirm_sheet_asset_upload (service role) required by the read predicate, bob-claims-hash pgTAP, authored-label anomalies (27 anomalies now), seeds-listed test, Dockerfile missing workspace manifests (found by building, not by the gates).
Anomalies: real image storage.protect_delete aborts direct DELETE tests (LRN-041); pg_temp fixtures need service_role grants; docker build raced the lockfile refresh (rebuild clean).
Action: fresh verifier re-dispatched on the delta; BDR-034, LRN-041.
EVAL-010 | 2026-10-10 | Step 4c equine curation — research agents, changeset, gates
Method: 4 research agents (skull, axial, limbs, osteo/legal), every citation tagged VERIFIED-fetched/snippet/UNVERIFIED; changeset authored in scratch python (44 changes / 679 ops / 48 refs) → pipeline Zod + 20 integrity rules; anatomy 100 vitest (ops, rename propagation, overlay, golden), pgTAP 001–008 on PGlite and real stack (db reset, db test 267/267, gen types diff clean); root check (68+21+100+278, lint 0, build); docker build; Storage e2e after upload (owner 200, alice/anon 400); gridded magick crops to verify point positions; fresh verifier ×3 + security-auditor ×1.
Result: verifier pass 1 ECARTS(4): B changes left targets inherited (CHG-008/032/039 + partials), 3 stale doc numbers, lockfile outside contract scope (orchestrator omission), eslint-disable any in a test → executor: applier rule (B/C ⇒ touched findings/annotations to_confirm, reviewed refused in B/C), typed test reads, docs. Pass 2 ECARTS(2): JSAVA citation without author (completed from the PDF: Le Roux & Carstens 2018, DOI), DEPLOY.md stale faunosteo-web. Pass 3 ECARTS(1): 3 stale doc lines (migration list, "two assets", CLAUDE.md migrations) → fixed by hand, cap of 3 verifier passes reached, not re-verified. Security PASS, LOW: SQL gate did not check reference url scheme → reference.url.scheme added (migration still unmerged, pgTAP 008 10 cases); .env.example COMPOSE_PROJECT_NAME=osteo-report overrides compose name: → file denied to Claude, user action.
Anomalies: ImageMagick MSVG drops fill="none" shapes and shapes with <title> child (LRN-042); two legacy points (t18-l1, l6-s1) floated above the silhouette, found only by the overlay render, not by the catalog audit; Supabase db reset unconfirms the asset row (upload script must re-run: objects kept, row confirmed).
Action: user validation of the draft content (REVIEW-REPORT §10: 11 questions); never publish; Step 5 only after validation.
EVAL-011 | 2026-10-10 | Step 5 app on the canonical sheet — suites on the bundled sheet, archive PDFs, gates
Method: golden v2 PDFs frozen from the old pipeline before any change; suites rewritten on the bundled draft (domain 94, anatomy 107, supabase 21, app 168+: service branches 10/5/12/1, store + v2 draft migration, provider, canvas, pickers, card, route click chain, PDF v3 roundtrip FR/EN + both v2 fixtures + synthetic removed/unknown ids); root check, vite builds none/draft (grep of a draft label in dist = 0/1), docker build, real stack 267/267 + types; legacy-vs-app marker style diff by the verifier (0 differences); fresh verifier ×3 + security-auditor ×1.
Result: verifier pass 1 ECARTS(36) = 4 real (golden test not pinning the new generated files, EditorRoute 398 lines, 5 stale doc lines, one eslint-disable) + my contract omissions (scope files, deleted legacy suites) + floor noise ("species" matched "spec"); pass 2 ECARTS(4) = replacement assertions missing for 4 deleted suites; pass 3 ECARTS(4) = doc counts + 3 more suites' assertions (PDF conclusion/metadata/attachment structure, ObservationCard, persistence). All closed by executors (two) and by hand (docs); cap of 3 passes reached → last lot not re-verified. Security PASS with LOW: prototype-key lookup on untrusted ids (fixed: Object.hasOwn + test), unbounded inflate on the primary attachment path + no file cap (fixed: 16 MiB attachment, 50 MB PDF + test), denylist SVG sanitiser acceptable while assets are repo-built (allowlist before Storage-served assets, Step 13).
Anomalies: executor ran pnpm format repo-wide and reverted by hand (watch for format in briefs); foreign JSON attachment (no version) surfaced as incompatible_version → fixed to no_attachment (legacy message); verifier floor counts deleted legacy tests as findings until CLARIFICATIONS name them.
Action: user validation of Step 5 (manual FR/EN flow, visual before/after); BDR-038, LRN-043.
EVAL-012 | 2026-10-10 | Step 5 addendum during user manual validation — per-leaf migration tests, archive check tool, final controls
Method: user asked (while testing) for final controls on last lot + per-leaf test beyond the map + simple v2/v3 PDF comparison method. Built: anatomy test all 612 legacy leaves through migrateReportV2ToV3 with committed map vs draft document (442 identical / 141 moved = 134 category + 7 leaf renames / 29 removed, snapshots + vet flag verbatim, 26 annotations on right view, v3 re-read stable); app test same leaves via sheetIndex.nodeAt + v3 PDF roundtrip; CLI pnpm report:compare (pure module + entry, readAttachedDocument split out of extractReportJson, behaviour unchanged). Contract criteria 16-18 added, gates MET, fresh verifier ×3 + security ×2.
Result: verifier pass 1 ECARTS(2) = floor only (EVAL-011 prose "eslint-disable" false positive → CLARIFICATIONS; 11 dropped regionFromLabel assertions → new legacy.regions.test.ts on inferRegionCode); security PASS + 2 MEDIUM (terminal control chars in untrusted strings; free text printed by default) → executor: esc() + --full privacy default; pass 2 ECARTS(1) (--diff still clear text, JSON.parse message echoed attachment fragment) + security PASS 4 LOW → executor: diff follows --full, generic reason, Cc/Cf escaping, regular files only; pass 3 CONFORME. Workspace 94/21/119/181, docker build OK.
Anomalies: verifier stage runs after executor edits only (concurrent reads otherwise); gates.sh needs EVIDENCE: PENDING placeholder lines on new criteria. Residuals noted, not fixed: ZodError message echoes closed-vocabulary received values (never names), compareReport computes v3 lines twice, template-literal lines >80 chars, locals >5 in two functions.
Action: await user manual validation of Step 5; residuals in TODO follow-up.
EVAL-013 | 2026-10-11 | Step 6 autonomous PDF archives — 5 phases by executors, 3 verifier passes, 4 security audits
Method: analysis (3 analyzers + measurements) challenged by 3 plan-challengers (correctness/robustness/simplicity, 48 findings folded in) → user validation with D7/D8/D13 adjustments → Part II contract (exact types) → one executor per phase (0 freeze goldens, 1 domain, 2 anatomy compose, 3 engine, 4 app) with closed contracts, orchestrator commits → whole-step verifier + security auditor → fix loops.
Result: Phase 0 froze 39 goldens (text/images/links per locale × logo mode, view facts) + 2 route characterisation tests with the OLD code; engine reproduced 36 goldens with 0 mismatch (views: data-* 2→0 and hidden-marker titles declared relaxed). Verifier pass 1 ECARTS(31) = all oracle/scope defects of MY contract (grep on parametrised table, xmldom filter, narrow FILE SCOPE, 3 deleted tests with moved coverage) → contract amended → pass 2 CONFORME 13/13. Security pass 1 BLOCK(1) HIGH real + PoC: archived branding URL written unescaped into /URI (PDF object injection via exportFromEnvelope), +3 MEDIUM/4 LOW → executor fixed (hex URI, strict URL schema, export re-validation, PNG IHDR budget, CSS-escape rules, NaN guard, file.size) → pass 2 PASS but 2 MEDIUM incl. a regression the fix introduced (UTF-16 BOM URI dead in poppler) + PNG double-IHDR bypass → executor → pass 3 PASS + 1 MEDIUM (UPNG nextZero spin on tEXt) → executor (chunk allowlist + IEND) → pass 4 PASS + 3 MEDIUM (UPNG inflateRaw spins on truncated IDAT with allowlisted chunks only; IHDR fields unvalidated; zero dimension) → executor: native bounded inflateFlate of the IDAT against the exact raw size, IHDR fields, sides ≥ 1, no interlace, raw ≤ 16 MiB → pass 5 focused. Lesson: a third-party decoder on untrusted bytes needs a native oracle in front, not a chunk allowlist. Final: 656 tests (194/21/130/88/223), typecheck, lint, build, docker ×4 green; goldens byte-identical since 8b86b92.
Anomalies: contract oracles written fast = 2 verifier passes lost (lesson: run every CHECK once before dispatching the verifier); security fix introduced a functional regression invisible to tests because the test reader decoded the BOM (lesson: assert raw bytes, cross-check with a real reader like poppler); executors denied rm/git rm (orchestrator deletes); one executor used git stash despite "no git state" (now forbidden explicitly in briefs); demo envelope with archived branding = 136 KB stored / 306 KB raw (raw > 250 KB, stored < 250 KB: reported, user said data first).
Action: await user manual validation + UX decision on re-export of the kept archive; BLK-009 (pdf-lib ObjStm inflation) before external users; react-router advisories (pre-existing) to bump in a chore.
EVAL-014 | 2026-10-11 | Step 6 closure round — manual-test feedback, BLK-009 scanner, final audits
Method: user manual tests round 1 (logos absent, picker blurry) → reproduction attempts (headless Chromium dev/preview/Docker, Firefox 157 via esbuild harness + raw WebDriver BiDi client, CDP LayerTree, LCD-fringe metric, headed runs on the Wayland session) → fixes by executors → user round 2 confirmed both OK → closure decisions → BLK-009 implemented (byte-scan reader, no object streams) → verifier + 2 security audits + 1 fix loop.
Result: neither visual defect reproduced here (4 environments, 2 engines, 2 PDF renderers); shipped the environment-independent fixes (branding bundled, no fetch; warnings surfaced) and the user confirmed logos + crisp text. Scanner: verifier CONFORME 7/7; audit 1 PASS + 2 MEDIUM (Chromium DecompressionStream read-ahead defeats the cancel: +2.2 GB on a 4 GiB bomb; ×32 candidate CPU) → executor (16 KiB slice feeding, shared 16 MiB budget, monotonic bounded endstream search, { pre-check; 133 pdf tests) → audit 2 focused. Workspace 710 tests, build, docker green.
Anomalies: the blur fix of the previous round (will-change) had been marked PARTIALLY MET "desktop by user" — the user's "still blurry" came from that unverified half; this round could not verify real-GPU text rendering either (headless engines render grayscale AA, headed captures failed) → lesson: a screen-rendering defect needs the user's own capture + an A/B recipe early, not days of headless measurement. The Node vitest suite cannot observe browser stream read-ahead: record browser-only bounds as owed manual checks (PDF_IMPORT_HARDENING §5).
Action: BLK-009 stays "mitigated, gate open" (BDR-040); Step 6 finished into develop; release/0.2.0 prepped for the user's go.