From b7505d11f7c5462d16d8809d80dff348d180e4d8 Mon Sep 17 00:00:00 2001 From: bchanot Date: Thu, 12 Nov 2026 17:39:15 +0100 Subject: [PATCH] =?UTF-8?q?chore(memory):=20EVAL-058=20=E2=80=94=20memory-?= =?UTF-8?q?feedback=20gate=201?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .claude/memory/evals.md | 7 +++++++ .claude/memory/journal.md | 1 + .claude/tasks/TODO.md | 5 +++++ 3 files changed, 13 insertions(+) diff --git a/.claude/memory/evals.md b/.claude/memory/evals.md index 9e873ab..e60b934 100644 --- a/.claude/memory/evals.md +++ b/.claude/memory/evals.md @@ -447,3 +447,10 @@ Dogfood: 3 blind lenses attacked the v1 plan for the plan-challenge feature itse - **Result**: plan r1 FATAL(9) (note unprovable in `-p`) before code. Task reviews caught 5 Important (quadratic STATUS regex, heredoc commit form never matched, unguarded `fs.exists`, whole-command milestone regexes, fs.exists missing/unreadable). Final opus review still found 2 Important + 13 minors (all-placeholder grams on real registries, redaction gaps, `> \d+` letterless, success path reading the store per call). Security gate PASS ×3 but each coverage round found new shapes (cookie tail from the fix itself, `-b`, JSON affixes). Fix rounds introduced 2 regressions (quadratic JWT, cookie 400-cap), both caught by scoped re-reviews. - **Anomalies**: controller ruling "letterless rule subsumed by ERRORISH" was wrong (`> \d+` alternative) and cost one revert; "1:1 port" claim in noise.py drifted as the TS rule grew; the verifier's second pass refused `gates.sh run` (it rewrites the contract) and re-ran the CHECKs read-only instead. - **Action**: a ruling that removes a check lists the counterexample search over the alternatives first; reviewers measure regex cost by running; keep a scoped re-review after every fix round, a round of its own can regress; a port of a live rule carries the commit it was ported from. + +## EVAL-058 — Memory recall gate 1: 57 % precision (n=75), idle 44 %, 3rd pick 35 %, tool channel empty (2026-11-12) +- **Checked**: phase 4c recall quality at first `/memory-feedback` prompt gate (BDR-140), window 2026-11-08 → 2026-11-12. +- **Method**: 28 records, 2 repos (config + 1 project), 75 injected entries labelled relevant/irrelevant against the ask; 5 labels spot-checked by user (0 flip); misses searched on 5 asks picked where a miss was suspected (biased, overstates the miss rate); gate output `FEEDBACK.md`. +- **Result**: prompt precision 57 % (n=75), under the 64 % (14/22) 4b-4d proof target. By trigger: fresh 78 % (n=23), idle 44 % (n=32), keyword 55 % (n=20). 3rd pick 35 % (n=23). Timeout + paused 0 %. Rerank p90 3355 ms. Tool gate 0/10 records. Misses: 7 on 5 asks, 4 no-overlap (BM25 cannot reach, title+lesson share no word with the prompt), 3 overlap (rerank dropped them). 1 noisy entry + 1 promotion candidate in the project repo, to run there via /prune-memory. +- **Anomalies**: idle trigger fires on short follow-up prompts ("ok on y va") where all 3 entries miss; tool channel logged 0 records in 4 days, unproven live; rerank p90 above the 3 s bound. +- **Action**: 4 TODO lines under Phase 4b-4d (idle trigger, cap at 2, tool channel check, no-overlap misses). Decisions deferred to the user. diff --git a/.claude/memory/journal.md b/.claude/memory/journal.md index f80dd15..eb84d24 100644 --- a/.claude/memory/journal.md +++ b/.claude/memory/journal.md @@ -727,3 +727,4 @@ rules: - 2026-11-12: /feat dead-deny lock on feature/dead-deny-lock: 79f1a3f push-guard T41 drops dead Write lines, T42 justified list until 3.0.0 (T42c/d/e). 2 challenge rounds (r2 FATAL: base-keyed expiry no green path -> version expiry re-gated). Gates MET/CONFORME/PASS, MEDIUM fail-open fixed. User hand edit of settings.json pending (TODO:92). - 2026-11-12: /doc 3.0.0 on chore/docs-3.0.0: ed67c52 MIGRATION 'Upgrading to 3.0.0' (7 breaking items), CHANGELOG gitflow deny 14 entries, Fixed bullet dead 2.0.0 deny forms, [2.0.0] Linux residual fixed in place, '(98 checks)' dropped. Next: /release-candidate. - 2026-11-12: /reconcile before 3.0.0: 75 SHAs + 31 merges verified, nothing blocks the cut; TODO restructured (9 stale boxes fixed, shipped BDR-125 phase blocks to History, residuals grouped, post-3.0.0 items: empty push-guard removed-ok at next bump, D3 MEDIUM /bugfix). CHANGELOG internal headings -> [internal X.Y.Z]. +- 2026-11-12: /memory-feedback gate 1 (prompt 26/20): 28 records, 75 labels, 5 spot-checked (0 flip), 7 misses on 5 asks. Precision 57 % under the 64 % target, idle 44 %, tool channel empty. EVAL-058 + 4 TODO lines on chore/memory-feedback-gate1. diff --git a/.claude/tasks/TODO.md b/.claude/tasks/TODO.md index 62a21ad..f5cbb3c 100644 --- a/.claude/tasks/TODO.md +++ b/.claude/tasks/TODO.md @@ -29,6 +29,11 @@ Shared rules for every phase: - [ ] BLK-044: trace origin of `oom_score_adj=-1000` on claude processes; set `OOMScoreAdjust` on the `cc` scope so a runaway claude dies first. - [ ] safety-residuals gate LOWs (2026-11-06, bugfix/safety-residuals): `rtk read -n server.key` / `rtk read id_rsa` at repo root pass (`.key`/`.pem`/`id_*` rtk denies all need `*/`; pre-existing, bench before adding); session-start `memory.sh check` and digest unbounded when `timeout` is absent (stock macOS); pre-commit allowlist fails open if `git diff --cached` errors (pre-existing, Gitea backstop); `lib/memory.sh` `_mem_path_safe` line > 80 chars; `claude-stub.sh` interpolates the mktemp log path into the generated stub (test-only). - Proof: recall precision on ≥20 real prompts and ≥10 real errors, target ≥ the 14/22 baseline; digest after a real compaction; redaction test on the ledger; `/context` delta; captured-failure demo: an error from session N recalled in session N+1. + - Gate 1 result (2026-11-12, EVAL-058): prompt precision 57 % (n=75), under the 64 % target; fresh 78 %, idle 44 %, keyword 55 %; tool gate 0 records. + - [ ] 4d gate 1 (2026-11-12, EVAL-058): idle trigger at 44 % precision: tighten (skip prompts under N words, or only after a real idle gap) or drop; decide before the next gate + - [ ] 4d gate 1: cap recall at 2 entries (3rd pick 35 %) + - [ ] 4d gate 1: tool channel logged 0 records in 4 days: check the PostToolUse error match actually fires (live error in a session, then memory-feedback) + - [ ] 4d gate 1: misses 4/7 no-overlap: BM25 over title+lesson cannot reach them; option = index body too, or accept ### BDR-125 residuals (after 3.0.0) Open children of the shipped phases (History 2026-11-12), text verbatim.