chore(memory): EVAL-058 — memory-feedback gate 1

This commit is contained in:
bchanot committed 2026-11-12 17:39:15 +01:00
1 parent f8b324136c
commit b7505d11f7
3 files changed
+13

No files matched your search

+7
View File
@@ -447,3 +447,10 @@ Dogfood: 3 blind lenses attacked the v1 plan for the plan-challenge feature itse
- **Result**: plan r1 FATAL(9) (note unprovable in `-p`) before code. Task reviews caught 5 Important (quadratic STATUS regex, heredoc commit form never matched, unguarded `fs.exists`, whole-command milestone regexes, fs.exists missing/unreadable). Final opus review still found 2 Important + 13 minors (all-placeholder grams on real registries, redaction gaps, `> \d+` letterless, success path reading the store per call). Security gate PASS ×3 but each coverage round found new shapes (cookie tail from the fix itself, `-b`, JSON affixes). Fix rounds introduced 2 regressions (quadratic JWT, cookie 400-cap), both caught by scoped re-reviews.
- **Anomalies**: controller ruling "letterless rule subsumed by ERRORISH" was wrong (`> \d+` alternative) and cost one revert; "1:1 port" claim in noise.py drifted as the TS rule grew; the verifier's second pass refused `gates.sh run` (it rewrites the contract) and re-ran the CHECKs read-only instead.
- **Action**: a ruling that removes a check lists the counterexample search over the alternatives first; reviewers measure regex cost by running; keep a scoped re-review after every fix round, a round of its own can regress; a port of a live rule carries the commit it was ported from.
## EVAL-058 — Memory recall gate 1: 57 % precision (n=75), idle 44 %, 3rd pick 35 %, tool channel empty (2026-11-12)
- **Checked**: phase 4c recall quality at first `/memory-feedback` prompt gate (BDR-140), window 2026-11-08 → 2026-11-12.
- **Method**: 28 records, 2 repos (config + 1 project), 75 injected entries labelled relevant/irrelevant against the ask; 5 labels spot-checked by user (0 flip); misses searched on 5 asks picked where a miss was suspected (biased, overstates the miss rate); gate output `FEEDBACK.md`.
- **Result**: prompt precision 57 % (n=75), under the 64 % (14/22) 4b-4d proof target. By trigger: fresh 78 % (n=23), idle 44 % (n=32), keyword 55 % (n=20). 3rd pick 35 % (n=23). Timeout + paused 0 %. Rerank p90 3355 ms. Tool gate 0/10 records. Misses: 7 on 5 asks, 4 no-overlap (BM25 cannot reach, title+lesson share no word with the prompt), 3 overlap (rerank dropped them). 1 noisy entry + 1 promotion candidate in the project repo, to run there via /prune-memory.
- **Anomalies**: idle trigger fires on short follow-up prompts ("ok on y va") where all 3 entries miss; tool channel logged 0 records in 4 days, unproven live; rerank p90 above the 3 s bound.
- **Action**: 4 TODO lines under Phase 4b-4d (idle trigger, cap at 2, tool channel check, no-overlap misses). Decisions deferred to the user.
+1
View File
@@ -727,3 +727,4 @@ rules:
- 2026-11-12: /feat dead-deny lock on feature/dead-deny-lock: 79f1a3f push-guard T41 drops dead Write lines, T42 justified list until 3.0.0 (T42c/d/e). 2 challenge rounds (r2 FATAL: base-keyed expiry no green path -> version expiry re-gated). Gates MET/CONFORME/PASS, MEDIUM fail-open fixed. User hand edit of settings.json pending (TODO:92).
- 2026-11-12: /doc 3.0.0 on chore/docs-3.0.0: ed67c52 MIGRATION 'Upgrading to 3.0.0' (7 breaking items), CHANGELOG gitflow deny 14 entries, Fixed bullet dead 2.0.0 deny forms, [2.0.0] Linux residual fixed in place, '(98 checks)' dropped. Next: /release-candidate.
- 2026-11-12: /reconcile before 3.0.0: 75 SHAs + 31 merges verified, nothing blocks the cut; TODO restructured (9 stale boxes fixed, shipped BDR-125 phase blocks to History, residuals grouped, post-3.0.0 items: empty push-guard removed-ok at next bump, D3 MEDIUM /bugfix). CHANGELOG internal headings -> [internal X.Y.Z].
- 2026-11-12: /memory-feedback gate 1 (prompt 26/20): 28 records, 75 labels, 5 spot-checked (0 flip), 7 misses on 5 asks. Precision 57 % under the 64 % target, idle 44 %, tool channel empty. EVAL-058 + 4 TODO lines on chore/memory-feedback-gate1.
+5
View File
@@ -29,6 +29,11 @@ Shared rules for every phase:
- [ ] BLK-044: trace origin of `oom_score_adj=-1000` on claude processes; set `OOMScoreAdjust` on the `cc` scope so a runaway claude dies first.
- [ ] safety-residuals gate LOWs (2026-11-06, bugfix/safety-residuals): `rtk read -n server.key` / `rtk read id_rsa` at repo root pass (`.key`/`.pem`/`id_*` rtk denies all need `*/`; pre-existing, bench before adding); session-start `memory.sh check` and digest unbounded when `timeout` is absent (stock macOS); pre-commit allowlist fails open if `git diff --cached` errors (pre-existing, Gitea backstop); `lib/memory.sh` `_mem_path_safe` line > 80 chars; `claude-stub.sh` interpolates the mktemp log path into the generated stub (test-only).
- Proof: recall precision on ≥20 real prompts and ≥10 real errors, target ≥ the 14/22 baseline; digest after a real compaction; redaction test on the ledger; `/context` delta; captured-failure demo: an error from session N recalled in session N+1.
- Gate 1 result (2026-11-12, EVAL-058): prompt precision 57 % (n=75), under the 64 % target; fresh 78 %, idle 44 %, keyword 55 %; tool gate 0 records.
- [ ] 4d gate 1 (2026-11-12, EVAL-058): idle trigger at 44 % precision: tighten (skip prompts under N words, or only after a real idle gap) or drop; decide before the next gate
- [ ] 4d gate 1: cap recall at 2 entries (3rd pick 35 %)
- [ ] 4d gate 1: tool channel logged 0 records in 4 days: check the PostToolUse error match actually fires (live error in a session, then memory-feedback)
- [ ] 4d gate 1: misses 4/7 no-overlap: BM25 over title+lesson cannot reach them; option = index body too, or accept
### BDR-125 residuals (after 3.0.0)
Open children of the shipped phases (History 2026-11-12), text verbatim.