Files
faunosteo/.claude/memory/evals.md
T

7.1 KiB
Raw Blame History

Evals Registry

Format: EVAL-XXX | Date | What evaluated | Method | Result | Anomalies | Action


EVAL-001 | 2026-05-18 | Scaffold build + test pass

Method: pnpm install → pnpm build → pnpm test:run → pnpm typecheck. Result: all pass, 0 errors. Anomalies: pnpm version 11 instead of 9 (corepack issue on host). Action: document in BDR-007, BLK-001.


EVAL-002 | 2026-05-20 | B3 i18n — code review caught hardcoded FR in PDF import

Method: cavecrew-reviewer ran against full diff 0a05c47..HEAD after self-review + analyzer agent both reported CLEAN. Result: reviewer flagged 3 CRITICAL: pdf/import.ts:9, pdf/attachment.ts:9-10, app/routes/HomeRoute.tsx:32 — hardcoded FR error messages leaking to EN UI. Anomaly: prior i18n grep ('Annuler', 'Valider', etc.) missed these because the error-path files contained different French vocabulary ("Ce PDF n'a pas été..."). Implementer's TODO checklist did not include the PDF import error paths since they were not in the original component migration list. Analyzer also missed because the leak only surfaces under EN locale at runtime (no static signal in code). Action: fixed in d602726+next via typed ImportError discriminant (BDR-015). Future i18n migrations should enumerate ALL files that emit user-facing strings (not just components/routes) — include error path files, exception messages, anything that may end up in window.alert or setError.


EVAL-003 | 2026-05-21 | PDF vet-marker ⚠ WinAnsi crash slipped past CI

Method: user manual export of a v2.0 horse report containing requires_vet_referral leaves → EditorRoute.tsx:263 PDF export failed Error: WinAnsi cannot encode "⚠" (0x26a0). Result: 154/154 tests stayed green; bug shipped to user-facing path. Static signal absent (the ⚠ literal was visible in the source but typecheck/lint don't flag font-encoding issues). Anomaly: pdf.roundtrip.test.ts exercised export with empty observations + no vet findings. The ⚠ prefix was added in 60e1b04-era (observationPage.ts:407 ternary) without a regression case covering a flagged leaf. Action: hotfix swapped ⚠ → (!) (WinAnsi-safe). Added regression vet-referral marker is encodable by the body font that calls widthOfTextAtSize on a prefixed label — would catch any future non-WinAnsi marker. LRN-021 captured the pdf-lib StandardFonts WinAnsi limitation. Follow-up: switch PDF body to embedded TTF (Inter/Fraunces per CLAUDE.md spec) so the catalog can carry arbitrary glyphs — out of hotfix scope.

EVAL-004 | 2026-07-12 | Tour sweep converged 3 it. — agent spend-limit deaths resumable

Method: /tour full pipeline on faunosteo — semgrep floor ×3 + cso posture (it1) + full clean audit + fresh independent re-verify + doc-syncer. 13 fixed / 10 open residuals / 6 bugs quarantined; 11 commits on unmerged chore/tour-2026-07-12. Result: re-verify PASS (0 findings on 20 modified files); checks green throughout (typecheck, lint 0, 174/174, build, docker run non-root). Convergence honest: it2 surfaced 1 new tuning note (min-release-age 1440 < 10080 rec) → triaged open → it3 semgrep identical → converged at 3-iteration bound. Anomaly: both Phase A agents died mid-run on "monthly spend limit" — SendMessage to same agentId resumed from transcript, zero rework. PostToolUse hook false-positived 3× on 'dangerouslySetInnerHTML' as prose inside audit REPORTS (not code). Action: none blocking. Consider excluding .claude/audits/*.md from sink-pattern hook scan.

EVAL-005 | 2026-10-08 | Step 1 characterisation suite (91 cases)

Method: 7 files on real assets (SVG, catalog, fixtures), oracle = audit tables (06 §4, 05 §2.4). Run typecheck/test/lint/build. Result: 271/271 green, 0 src change. Audit figures all confirmed (26 mappings score 0, 792/612/41, 48 attrs…). Anomalies: 4 assertions wrong on first run, all test-side (census regex, stale closure, ZONES drawn as bullet rows not ' · ' line, attachment names in object streams). None app-side. Action: LRN-033. Known quirks pinned as "KNOWN:" tests (orphan findings, no_attachment cause lost, sphenoid id, fuzzy FR pairs).

EVAL-006 | 2026-10-08 | Step 3 Supabase foundation — PGlite suite

Method: 4 pgTAP files (81 checks: schema/privileges 24, profiles RLS 17, platform roles 28, audit 12) + schema↔types (6) + client (2) on PGlite; root typecheck/lint/build; app suite. Result: all green first run except plan count (27→28) and pgtap-in-public polluting function census; 278 app + 13 package tests. Zero app behaviour change. Anomalies: set_config rows appear in TAP line capture (noise, filtered by regex). now() constant in tx broke naive updated_at>created_at assertion. Action: LRN-035/036. Not executed: real Supabase image (supabase test db), gen types diff, seed.sql (needs auth.identities + extensions.crypt), REST manual script. Flagged in report + BLK-006.

EVAL-007 | 2026-10-08 | Step 3 on the real Supabase stack

Method: supabase start (postgres 17.11.0.004), supabase test db, gen types --local diff, REST curl script (SUPABASE.md §8) with seed users, psql on audit, db reset. Result: migrations + seed apply from zero; pgTAP 81/81 after fix; REST: alice 1 row, anon 42501, self-deactivate 42501 (column grant), self-grant 42501 (RPC message), cross-row PATCH 0 rows, owner grant → uuid, alice sees her role, owner reads 3 profiles; audit shows system bootstrap + owner grant. Anomalies: (1) 6 pgTAP counts assumed empty DB — seed rows on real stack (PGlite runs no seed) → scoped to fixture ids + neutralise foreign owners before last-owner test. (2) CLI gen output unformatted + ComputedFields: never, no __InternalSupabase → hand-written file replaced by generated+prettier; scripts pipe prettier. PGlite harness had hidden both: environment parity gap, not logic gap. Action: LRN-037 (tests must be seed-agnostic; generated files via deterministic formatter). BLK-006 resolved.

EVAL-008 | 2026-10-08 | Step 4 sheet model — domain tests, pgTAP both engines, fresh verifier + security gates

Method: packages/domain 61 vitest (schema 29 + integrity 32, one per rule code); pgTAP 005 (127 checks) + 001 census on PGlite and real stack (db reset, db test 210/210, gen types diff clean); root check (typecheck, 356 tests, lint 0, build); fresh verifier (contract C1–C13) + security-auditor (semgrep 725 rules 0 findings + checklist). Result: first gate pass → verifier ECARTS(5), security BLOCK(1): private EXECUTE not revoked (BLK-007), visibility change unaudited, 7 RPCs without non-editor denial test, doc/code mismatches, 75-line SQL function, any suppression in a test. All fixed same session: +32 pgTAP checks, hardening migration, visibility RPC, caps, authz-first, SQL check split in 4 functions, set()/unset() typed mutations. Second pass: fresh verifier (pending at write time). Anomalies: 6 test-side fixes on first pgTAP run (RLS hides rows → "no change" asserts; data-modifying CTE; collation of order by; trigger name order decides errcode; audit list; pg_temp grants). Executor (opus) deviated usefully: RegionCode pattern separate from StableId. Action: LRN-038/039/040, BLK-007. Keep the two-gate pass on every migration step.