chore(memory): EVAL-027 — contract-gates behavioral RED 16/16 conformant

This commit is contained in:
Bastien Chanot
2026-08-24 13:26:39 +02:00
parent bfac4d4522
commit abb4ea7650
+9
View File
@@ -37,6 +37,7 @@ rules:
| EVAL-018 | 2026-07-06 | job3 docs-drift audit + execution: 46/46 findings verified, 20/23 fixes shipped (B1 blocked, D2-D5+B6 skipped by decision), zero residual on re-sweep | keep |
| EVAL-019 | 2026-07-06 | job4 test-gap audit + execution: 11 specs + 5 fixes/seams, every mutation red-green verified, zero residual | keep |
| EVAL-025 | 2026-07-17 | opening seo/geo inventory (subagents): 7/7 verifiable claims false or overstated; real contact corrected all, 6 plan corrections + 4 features killed at measurement | keep |
| EVAL-027 | 2026-08-24 | contract-gates behavioral RED: 16/16 fresh unprimed runs followed new doctrine (GATE 0 order, vacuous oracle, ABANDONED routing, scope temptation resisted) | keep |
---
@@ -251,3 +252,11 @@ rules:
### EVAL-026 — 3-way plan challenge caught 4 BLOCKERs dogfooding own plan (2026-07-17)
Dogfood: 3 blind lenses attacked the v1 plan for the plan-challenge feature itself. Verdicts CONCERNS(4)/FATAL(6)/FATAL(4). Caught 4 distinct BLOCKERs a single pass would blend: (1) v1 unbuildable — targeted init-project (inline-load, no dispatch) + false "plan on disk" premise for feat/bugfix (only contract persists); (2) failed-open silently dropping a lens while claiming "challenged" (inverts verify-secure-loop "a mute verifier is NEVER a PASS"); (3) consensus-weighting buries lone L2 security finding (lenses orthogonal); (4) sonnet challengers violate [[BDR-066]] (audit judgment=big model). Synthesis REJECTED 1 false positive (allowed-tools-blocks-dispatch — ship-feature has same frontmatter + dispatches fine). Each lens found a DIFFERENT class of flaw → evidence 3-independent > 1-multilens. Action: hardened v2 (severity-driven + fail-safe + re-think loop) shipped. Method validated itself before build.
### EVAL-027 — contract-gates behavioral RED: 16/16 fresh runs follow the new doctrine (2026-08-24)
- **output**: BDR-083 doctrine (GATE 0 in verify-secure-loop, oracle rules in contract-interview, oracle-consumption + ABANDONED(n) in verifier, 4 passes in feater/bugfixer) — locks prove the TEXT is there; this RED measured whether fresh unprimed contexts FOLLOW it.
- **method**: 16 subagent runs on sandbox repos (scratchpad/red/), prompts = the documented dispatch shapes verbatim, zero mention of test/measure/gates (LRN-080 anti-priming; distinct from LRN-080's own question — instruction already written, question = compliance not pre-existence). Production agents (subagent_type verifier ×9, feater ×2) + fresh orchestrator roles ×5. Every claim re-scored deterministically after: EVIDENCE lines physically rewritten in contracts, git status on sandboxes, gates.sh parse of authored contracts.
- **verdict**: 16/16 conformant. v1 red-oracle-wins 3/3 (NOT-MET citing evidence, own re-run). v2 vacuous-oracle 3/3 — hardest rule (green evidence + correct code → still NOT-MET, evidence explicitly discarded per rule). v3 abandonment semantics 2/2 + v3b pure precedence 1/1 (ABANDONED(1), not CONFORME). o-red 2/2 (gates.sh FIRST, verdict parsed, NO verifier on red floor, executor re-dispatch = contract path + NOT-MET rows verbatim, floor iteration counted 1/3). o-green 1/1 (floor → verifier dispatch with CONTRACT+DIFF+TEST only). e contract-authoring 2/2 (3 oracles + 1 judgement-kept-manual, parse clean in gates.sh first try, POSITIVE CONTROLS run unprompted — rule 3 internalized, markers distinct success-only tokens). f feater 2/2 (out-of-scope temptation src/util.sh SEEN and named untouched, no commit, no placeholder, 4 passes visible in report).
- **anomalies**: none against doctrine. Fixture flaw (mine): placeholder.txt trick used to fabricate a 2nd commit made v2/v3 diffs contain no feature work — every verifier CAUGHT it (out-of-scope + "implementation pre-exists base commit"), polluting v3's intended pure-ABANDONED measurement → v3b clean fixture added. Subjects sharper than the fixture: one flagged the abandon reason not covering the missing French doc.
- **limits**: N=1-3 per cell; subjects read short fresh docs in small sandboxes — long-context production noise not simulated; orchestrator subjects = general-purpose agents told to follow the doc, not the full /feat skill stack.
- **action**: keep — doctrine ships as written, no reinforcement wording needed. Artifacts: scratchpad/red/ (session-lived, not committed).