Files
faunosteo/.claude/memory/evals.md
T

45 lines
4.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Evals Registry
Format: EVAL-XXX | Date | What evaluated | Method | Result | Anomalies | Action
---
## EVAL-001 | 2026-05-18 | Scaffold build + test pass
Method: pnpm install → pnpm build → pnpm test:run → pnpm typecheck.
Result: all pass, 0 errors.
Anomalies: pnpm version 11 instead of 9 (corepack issue on host).
Action: document in BDR-007, BLK-001.
---
## EVAL-002 | 2026-05-20 | B3 i18n — code review caught hardcoded FR in PDF import
Method: cavecrew-reviewer ran against full diff `0a05c47..HEAD` after self-review + analyzer agent both reported CLEAN.
Result: reviewer flagged 3 CRITICAL: pdf/import.ts:9, pdf/attachment.ts:9-10, app/routes/HomeRoute.tsx:32 — hardcoded FR error messages leaking to EN UI.
Anomaly: prior i18n grep ('Annuler', 'Valider', etc.) missed these because the error-path files contained different French vocabulary ("Ce PDF n'a pas été..."). Implementer's TODO checklist did not include the PDF import error paths since they were not in the original component migration list. Analyzer also missed because the leak only surfaces under EN locale at runtime (no static signal in code).
Action: fixed in d602726+next via typed ImportError discriminant (BDR-015). Future i18n migrations should enumerate ALL files that emit user-facing strings (not just components/routes) — include error path files, exception messages, anything that may end up in window.alert or setError.
---
## EVAL-003 | 2026-05-21 | PDF vet-marker ⚠ WinAnsi crash slipped past CI
Method: user manual export of a v2.0 horse report containing requires_vet_referral leaves → `EditorRoute.tsx:263 PDF export failed Error: WinAnsi cannot encode "⚠" (0x26a0)`.
Result: 154/154 tests stayed green; bug shipped to user-facing path. Static signal absent (the ⚠ literal was visible in the source but typecheck/lint don't flag font-encoding issues).
Anomaly: pdf.roundtrip.test.ts exercised export with empty observations + no vet findings. The ⚠ prefix was added in 60e1b04-era (`observationPage.ts:407` ternary) without a regression case covering a flagged leaf.
Action: hotfix swapped ⚠ → `(!) ` (WinAnsi-safe). Added regression `vet-referral marker is encodable by the body font` that calls `widthOfTextAtSize` on a prefixed label — would catch any future non-WinAnsi marker. LRN-021 captured the pdf-lib StandardFonts WinAnsi limitation. Follow-up: switch PDF body to embedded TTF (Inter/Fraunces per CLAUDE.md spec) so the catalog can carry arbitrary glyphs — out of hotfix scope.
## EVAL-004 | 2026-07-12 | Tour sweep converged 3 it. — agent spend-limit deaths resumable
Method: /tour full pipeline on faunosteo — semgrep floor ×3 + cso posture (it1) + full clean audit + fresh independent re-verify + doc-syncer. 13 fixed / 10 open residuals / 6 bugs quarantined; 11 commits on unmerged chore/tour-2026-07-12.
Result: re-verify PASS (0 findings on 20 modified files); checks green throughout (typecheck, lint 0, 174/174, build, docker run non-root). Convergence honest: it2 surfaced 1 new tuning note (min-release-age 1440 < 10080 rec) → triaged open → it3 semgrep identical → converged at 3-iteration bound.
Anomaly: both Phase A agents died mid-run on "monthly spend limit" — SendMessage to same agentId resumed from transcript, zero rework. PostToolUse hook false-positived 3× on 'dangerouslySetInnerHTML' as prose inside audit REPORTS (not code).
Action: none blocking. Consider excluding .claude/audits/*.md from sink-pattern hook scan.
## EVAL-005 | 2026-10-08 | Step 1 characterisation suite (91 cases)
Method: 7 files on real assets (SVG, catalog, fixtures), oracle = audit tables (06 §4, 05 §2.4). Run typecheck/test/lint/build.
Result: 271/271 green, 0 src change. Audit figures all confirmed (26 mappings score 0, 792/612/41, 48 attrs…).
Anomalies: 4 assertions wrong on first run, all test-side (census regex, stale closure, ZONES drawn as bullet rows not ' · ' line, attachment names in object streams). None app-side.
Action: LRN-033. Known quirks pinned as "KNOWN:" tests (orphan findings, no_attachment cause lost, sphenoid id, fuzzy FR pairs).