From 26f70e1885d3a4b988bcd6f8862ab6a50bc0f2fb Mon Sep 17 00:00:00 2001 From: bchanot Date: Fri, 6 Nov 2026 21:19:16 +0100 Subject: [PATCH] docs(spec): memory feedback 4d design --- .../2026-11-06-memory-feedback-4d-1825.md | 25 +++ .../2026-11-06-memory-feedback-design.md | 189 ++++++++++++++++++ 2 files changed, 214 insertions(+) create mode 100644 .claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md create mode 100644 docs/superpowers/specs/2026-11-06-memory-feedback-design.md diff --git a/.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md b/.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md new file mode 100644 index 0000000..c0943c2 --- /dev/null +++ b/.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md @@ -0,0 +1,25 @@ +# CONTRACT — memory-feedback-4d +- date: 2026-11-06 | flow: ship-feature | branch: feature/memory-feedback (to start off develop c10d859) +- status: active (seed, STEP 0e; enriched at STEP 3) +- plan: .claude/tasks/plans/2026-11-05-memory-feedback-4d-1834.md (r1 draft, superseded by the STEP 2 plan) + +## REQUEST (verbatim — IMMUTABLE) +yes start it + +(context, the request as received: "Phase 4d memory feedback and promotion (BDR-125 queue). Plan r1 draft, not challenged: .claude/tasks/plans/2026-11-05-memory-feedback-4d-1834.md (4 open choices). TODO line '4d feedback and promotion': order 4d.1 log fields `cands` + `ms` → 4d.2 weekly reader (log × transcripts, labels.tsv) → 4d.3 gate (≥20 prompt + ≥10 tool injections, or 5 weeks) with thresholds fixed now → 4d.4 /prune-memory + BDR-100 promotion, conditional.") + +## CLARIFICATIONS + +## ACCEPTANCE CRITERIA +1. The feedback loop can say, from data, whether recall injections are useful (precision per channel and trigger), what they miss, and which entries are noisy or promotion candidates; no tuning is built before the gate. + (judged by the verifier against the enriched criteria below) +2. No prompt or reply text, and no secret, enters the plugin store or any committed file (LRN-124, LRN-261). + (judged by the verifier from the diff) +3. The memory mod stays fail-open; records written before any schema change still load. + (judged by the verifier; kit tests named at STEP 3) + +## FILE SCOPE +- mods/memory/hooks/register.ts, mods/memory/hooks/register.test.ts +- .claude/tasks/plans/2026-11-04-memory-recall-spike/recall-bench.ts (export only, if reused) +- a new reader + labels under .claude/tasks/plans/ (names fixed at STEP 2) +- skills/prune-memory/SKILL.md, skills/prune-memory/tests/run-deterministic.sh, one threshold lib script (4d.4, conditional) diff --git a/docs/superpowers/specs/2026-11-06-memory-feedback-design.md b/docs/superpowers/specs/2026-11-06-memory-feedback-design.md new file mode 100644 index 0000000..f4af8b7 --- /dev/null +++ b/docs/superpowers/specs/2026-11-06-memory-feedback-design.md @@ -0,0 +1,189 @@ +# Memory feedback (phase 4d): design + +Date: 2026-11-06. Queue: BDR-125 phase 4d. Host: `mods/memory` (BDR-137). +Supersedes the open choices of `.claude/tasks/plans/2026-11-05-memory-feedback-4d-1834.md` (r1). +Contract: `.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md`. + +## Goal + +Say from real data whether 4c recall helps: precision per channel and per +trigger, what it misses, which entries are noisy, which deserve promotion to +a mechanism (BDR-100). Thresholds and gates are fixed now, before the data. +Nothing is tuned before a gate is reached; each threshold action is its own +/feat or /hotfix carrying its measurement. + +In scope: 4d.1 log fields, 4d.2 reader + `/memory-feedback` skill + +session-start reminder, 4d.3 thresholds + gate verdict. Out of scope: 4d.4 +(/prune-memory step D, promotion mechanics), built only if a gate shows a +noisy entry or a promotion candidate (own TODO line). + +## User decisions (2026-11-06) + +| Choice | Decision | +|---|---| +| Cadence vs 7-day transcript retention | session-start reminder, user runs `/memory-feedback`; retention stays 7 days (BDR-057) | +| Gate population | records carrying `cands` (4d.1 on), every repo; `test` label excluded from shares | +| Labels location | local `~/.claude/memory-feedback/` (0600), never in git | +| Log fields | `cands` + `ms` | +| Gate shape | two independent gates, prompt and tool | +| Thresholds | r1 table unchanged | +| Approach | TS reader in `mods/memory/feedback/` + skill `/memory-feedback` | +| Label values | `relevant`, `irrelevant`, `test` (+ `missed` rows) | +| FEEDBACK.md | local, regenerated each run; recorded in the repo as an EVAL entry (and a BDR when an action is decided) at each gate | + +## 1. Log fields (4d.1, `mods/memory/hooks/register.ts`) + +- `LogRecord` gains `cands?: number` (BM25 candidates after the seen + filter, 0-30) on every prompt-channel record, and `ms?: number` (rerank + wall time, integer ms) on every record where a rerank call ran, timeout + included. Absent when no rerank ran (no candidate, paused). +- `isLogRecord` accepts both as optional non-negative integers; a record + without them stays valid (LG5 precedent). No other field, no text. +- Size: about 20 bytes per field; 200 records stay under the 64 KiB cap. +- `none` with `cands` = 0 (nothing to offer) is now distinct from `none` + with `cands` > 0 (haiku declined). + +## 2. Reader (`mods/memory/feedback/reader.ts`, bun) + +Imports the mod's own functions (`isLogRecord`, key helpers) rather than +porting them (LRN-261). Transcript loading lives in +`mods/memory/feedback/transcripts.ts`. + +### Inputs + +- The store file `~/.claude/plugins/store/memory_skills-dir-*.json`, every + `l:` key, all roots. Engine-internal path: a missing or unparsable + file fails loudly. +- Transcripts `~/.claude/projects//.jsonl`, slug = the + root with every non-alphanumeric character replaced by `-`. +- `~/.claude/memory-feedback/state.json`: last run, labelled record keys + (`root|session|at|channel`), tool-gate extension flag (LRN-027). + +### Population + +Records with a `cands` field (prompt channel) or written after the first +such record (tool channel), not yet labelled. The gate clock starts at the +first record carrying `cands`. + +### Join + +- Prompt, `injected`: the `attachment` record with `hook_additional_context` + and `hookName` `prompt.submit` nearest after `at`, within 10 s, same + session. Prompt = the composer user record it follows; reply = the + assistant text blocks after it up to the next composer prompt, deduped by + `message.id` (EVAL-037). +- Prompt, `none` / `timeout` / `refused` / `paused`: the last composer + prompt of the session before `at`. +- Tool: the `tool.call` attachment carrying `toolUseID`; the failing Bash + command and the next 3 Bash results of the session (retry succeeded or not). +- No transcript, no attachment: the record is `lost`, counted in + FEEDBACK.md, never dropped silently (LRN-133). +- `cited`: the entry ID appears in the reply text (lower bound). + +### Commands + +- `worksheet `: writes `/worksheet.md` and `/items.json` + (mode 0600). `` is the session scratchpad. These are the only files + that hold prompt or reply text (prompt cut at 600 chars, reply at 800). +- `apply `: reads `/labels-draft.tsv`, refuses an unknown label, + an unknown item, or an ID not in that item (non-zero rc, nothing + written); appends to `labels.tsv`, updates `state.json`, regenerates + `FEEDBACK.md`. Writes go through a temp file and rename. + +### labels.tsv (`~/.claude/memory-feedback/`, 0600) + +One row per injected entry, plus `missed` rows: `at, root name, session, +channel, trigger, outcome, cands, ms, pick (1-3), id, cited, label, cause`. +`label` in `relevant | irrelevant | test | missed`; `cause` only on +`missed`: `no-overlap | rank-over-30 | declined | not-triggered`. IDs and +enums only, no text. + +### FEEDBACK.md (`~/.claude/memory-feedback/`) + +Counts and shares only, no repo names, no IDs except in the noisy and +promotion lists: n per channel and trigger, lost count, every metric of +section 4, each gate's verdict. + +## 3. Skill `/memory-feedback` and reminder + +Skill steps (main loop; the labeller is Claude, the user spot-checks): + +1. `worksheet` into the scratchpad. +2. Label each injected entry `relevant` or `irrelevant`; proof, bench and + recall-testing sessions get `test`. +3. Spot check: 5 labelled items at random shown to the user, who confirms + or flips each (LRN-168: a label is a claim). +4. Misses: on up to 5 triggered asks, search the registries + (`memory.sh index` + bodies) for relevant entries not injected; add + `missed` rows with a cause. +5. `apply`; print each gate's verdict. A reached gate turns its threshold + actions into TODO proposals, approved one by one; the verdict is written + as an EVAL entry (and a BDR when an action is decided) through + `memory.sh`. + +Reminder (`hooks/session-start.sh`, every repo): one banner line +`memory feedback: last run d ago, transcripts expire at 7 → /memory-feedback` +when `state.json` is 5 days old or more (or absent while a store file +exists) and the store file is newer than `state.json`. Two `stat` calls, no +JSON parse, silent on any error. + +New skill: a `routing.json` row and the citers census entry. + +## 4. Thresholds and gates (`mods/memory/feedback/thresholds.ts`) + +One exported table, read by the reader and by a future /prune-memory step D +(BDR-100: thresholds in one file). + +| Signal | Threshold | Action | +|---|---|---| +| prompt precision (relevant / labelled injected entries) | ≥ 64 % (4c baseline 14/22) | keep | +| same, below 64 % | worst trigger kind < 50 % | tighten or drop that trigger (idle first) | +| precision of the 3rd pick | < 40 % | cap at 2 entries | +| timeout + paused share (asks with `cands` > 0) | > 25 % | revisit `RERANK_MS` or skip the rerank on short lists | +| `ms` p90 | < 2500 | lower `RERANK_MS` | +| tool precision | < 50 % | require 2 matched 4-grams, or exclude the entry | +| missed in the sampled asks | ≥ 1 ask in 3 | diagnose by cause; no fix built ahead | +| entry injected ≥ 3 times, 0 relevant | any | noisy: /prune-memory rewrites it | +| entry relevant ≥ 3 times in ≥ 2 sessions | any | promotion candidate (BDR-100) | + +Gates (`gateVerdict`, pure): + +- Prompt: reached at 20 labelled injected asks (non-`test`) or 35 days + after the clock start. +- Tool: reached at 10 labelled tool injections or 35 days. Below 10 at the + cap: `insufficient sample n=`, extended once by 35 days (flag in + `state.json`), then closed with no tuning. +- Each gate decides alone; the prompt gate never waits for the tool gate. + +## 5. Errors + +- Reader: fail loud (message + non-zero rc) on a missing or malformed store + file, a malformed transcript line is skipped and counted, an invalid + draft refuses the whole apply. +- Reminder: bounded, silent on any error, never breaks the banner. +- Mod: unchanged fail-open contract; log writes keep `.catch`. + +## 6. Tests + +- `register.test.ts` (4d.1): `cands` written on every prompt outcome, `ms` + written when a rerank ran (timeout included) and absent otherwise, an + old-shape record loads, `none` split by `cands`. Run through + `make test suite=lib/tests/mods.test.sh`. +- `mods/memory/feedback/reader.test.ts` (bun), synthetic store and + transcripts, no real text: each join case (injected, none, tool, lost), + `message.id` dedupe, `cited`, apply refusals (nothing written), file + modes, `gateVerdict` at each threshold boundary, the tool extension. +- Reminder: session-start suite cases with forced mtimes (shown, hidden, + absent state, unreadable state). +- Live proof: one real `/memory-feedback` run on today's log (pre-4d.1 + records serve as practice, labelled `test` where they are proofs). + +## Constraints honoured + +BDR-125 (no Stop hook, no runtime cited flag), BDR-137 (store caps, +interactive-only prompt proof), BDR-100 (one threshold file, mechanism not +lesson), BDR-129 (registry writes through memory.sh), BDR-057 (retention +stays 7 days), LRN-124/LRN-261 (no text in store or git, import not port), +LRN-027 (state file), LRN-241 (`test` label), LRN-212 (mod edits are live: +kit tests before the edit lands), LRN-073 (fictional IDs in skill +examples), EVAL-037 (message.id dedupe).