docs(spec): memory feedback 4d design

This commit is contained in:
bchanot committed 2026-11-06 21:19:16 +01:00
1 parent c10d8590dd
commit 26f70e1885
2 files changed
+214

No files matched your search

@@ -0,0 +1,25 @@
# CONTRACT — memory-feedback-4d
- date: 2026-11-06 | flow: ship-feature | branch: feature/memory-feedback (to start off develop c10d859)
- status: active (seed, STEP 0e; enriched at STEP 3)
- plan: .claude/tasks/plans/2026-11-05-memory-feedback-4d-1834.md (r1 draft, superseded by the STEP 2 plan)
## REQUEST (verbatim — IMMUTABLE)
yes start it
(context, the request as received: "Phase 4d memory feedback and promotion (BDR-125 queue). Plan r1 draft, not challenged: .claude/tasks/plans/2026-11-05-memory-feedback-4d-1834.md (4 open choices). TODO line '4d feedback and promotion': order 4d.1 log fields `cands` + `ms` → 4d.2 weekly reader (log × transcripts, labels.tsv) → 4d.3 gate (≥20 prompt + ≥10 tool injections, or 5 weeks) with thresholds fixed now → 4d.4 /prune-memory + BDR-100 promotion, conditional.")
## CLARIFICATIONS
## ACCEPTANCE CRITERIA
1. The feedback loop can say, from data, whether recall injections are useful (precision per channel and trigger), what they miss, and which entries are noisy or promotion candidates; no tuning is built before the gate.
(judged by the verifier against the enriched criteria below)
2. No prompt or reply text, and no secret, enters the plugin store or any committed file (LRN-124, LRN-261).
(judged by the verifier from the diff)
3. The memory mod stays fail-open; records written before any schema change still load.
(judged by the verifier; kit tests named at STEP 3)
## FILE SCOPE
- mods/memory/hooks/register.ts, mods/memory/hooks/register.test.ts
- .claude/tasks/plans/2026-11-04-memory-recall-spike/recall-bench.ts (export only, if reused)
- a new reader + labels under .claude/tasks/plans/ (names fixed at STEP 2)
- skills/prune-memory/SKILL.md, skills/prune-memory/tests/run-deterministic.sh, one threshold lib script (4d.4, conditional)
@@ -0,0 +1,189 @@
# Memory feedback (phase 4d): design
Date: 2026-11-06. Queue: BDR-125 phase 4d. Host: `mods/memory` (BDR-137).
Supersedes the open choices of `.claude/tasks/plans/2026-11-05-memory-feedback-4d-1834.md` (r1).
Contract: `.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md`.
## Goal
Say from real data whether 4c recall helps: precision per channel and per
trigger, what it misses, which entries are noisy, which deserve promotion to
a mechanism (BDR-100). Thresholds and gates are fixed now, before the data.
Nothing is tuned before a gate is reached; each threshold action is its own
/feat or /hotfix carrying its measurement.
In scope: 4d.1 log fields, 4d.2 reader + `/memory-feedback` skill +
session-start reminder, 4d.3 thresholds + gate verdict. Out of scope: 4d.4
(/prune-memory step D, promotion mechanics), built only if a gate shows a
noisy entry or a promotion candidate (own TODO line).
## User decisions (2026-11-06)
| Choice | Decision |
|---|---|
| Cadence vs 7-day transcript retention | session-start reminder, user runs `/memory-feedback`; retention stays 7 days (BDR-057) |
| Gate population | records carrying `cands` (4d.1 on), every repo; `test` label excluded from shares |
| Labels location | local `~/.claude/memory-feedback/` (0600), never in git |
| Log fields | `cands` + `ms` |
| Gate shape | two independent gates, prompt and tool |
| Thresholds | r1 table unchanged |
| Approach | TS reader in `mods/memory/feedback/` + skill `/memory-feedback` |
| Label values | `relevant`, `irrelevant`, `test` (+ `missed` rows) |
| FEEDBACK.md | local, regenerated each run; recorded in the repo as an EVAL entry (and a BDR when an action is decided) at each gate |
## 1. Log fields (4d.1, `mods/memory/hooks/register.ts`)
- `LogRecord` gains `cands?: number` (BM25 candidates after the seen
filter, 0-30) on every prompt-channel record, and `ms?: number` (rerank
wall time, integer ms) on every record where a rerank call ran, timeout
included. Absent when no rerank ran (no candidate, paused).
- `isLogRecord` accepts both as optional non-negative integers; a record
without them stays valid (LG5 precedent). No other field, no text.
- Size: about 20 bytes per field; 200 records stay under the 64 KiB cap.
- `none` with `cands` = 0 (nothing to offer) is now distinct from `none`
with `cands` > 0 (haiku declined).
## 2. Reader (`mods/memory/feedback/reader.ts`, bun)
Imports the mod's own functions (`isLogRecord`, key helpers) rather than
porting them (LRN-261). Transcript loading lives in
`mods/memory/feedback/transcripts.ts`.
### Inputs
- The store file `~/.claude/plugins/store/memory_skills-dir-*.json`, every
`l:<root>` key, all roots. Engine-internal path: a missing or unparsable
file fails loudly.
- Transcripts `~/.claude/projects/<slug(root)>/<session>.jsonl`, slug = the
root with every non-alphanumeric character replaced by `-`.
- `~/.claude/memory-feedback/state.json`: last run, labelled record keys
(`root|session|at|channel`), tool-gate extension flag (LRN-027).
### Population
Records with a `cands` field (prompt channel) or written after the first
such record (tool channel), not yet labelled. The gate clock starts at the
first record carrying `cands`.
### Join
- Prompt, `injected`: the `attachment` record with `hook_additional_context`
and `hookName` `prompt.submit` nearest after `at`, within 10 s, same
session. Prompt = the composer user record it follows; reply = the
assistant text blocks after it up to the next composer prompt, deduped by
`message.id` (EVAL-037).
- Prompt, `none` / `timeout` / `refused` / `paused`: the last composer
prompt of the session before `at`.
- Tool: the `tool.call` attachment carrying `toolUseID`; the failing Bash
command and the next 3 Bash results of the session (retry succeeded or not).
- No transcript, no attachment: the record is `lost`, counted in
FEEDBACK.md, never dropped silently (LRN-133).
- `cited`: the entry ID appears in the reply text (lower bound).
### Commands
- `worksheet <dir>`: writes `<dir>/worksheet.md` and `<dir>/items.json`
(mode 0600). `<dir>` is the session scratchpad. These are the only files
that hold prompt or reply text (prompt cut at 600 chars, reply at 800).
- `apply <dir>`: reads `<dir>/labels-draft.tsv`, refuses an unknown label,
an unknown item, or an ID not in that item (non-zero rc, nothing
written); appends to `labels.tsv`, updates `state.json`, regenerates
`FEEDBACK.md`. Writes go through a temp file and rename.
### labels.tsv (`~/.claude/memory-feedback/`, 0600)
One row per injected entry, plus `missed` rows: `at, root name, session,
channel, trigger, outcome, cands, ms, pick (1-3), id, cited, label, cause`.
`label` in `relevant | irrelevant | test | missed`; `cause` only on
`missed`: `no-overlap | rank-over-30 | declined | not-triggered`. IDs and
enums only, no text.
### FEEDBACK.md (`~/.claude/memory-feedback/`)
Counts and shares only, no repo names, no IDs except in the noisy and
promotion lists: n per channel and trigger, lost count, every metric of
section 4, each gate's verdict.
## 3. Skill `/memory-feedback` and reminder
Skill steps (main loop; the labeller is Claude, the user spot-checks):
1. `worksheet` into the scratchpad.
2. Label each injected entry `relevant` or `irrelevant`; proof, bench and
recall-testing sessions get `test`.
3. Spot check: 5 labelled items at random shown to the user, who confirms
or flips each (LRN-168: a label is a claim).
4. Misses: on up to 5 triggered asks, search the registries
(`memory.sh index` + bodies) for relevant entries not injected; add
`missed` rows with a cause.
5. `apply`; print each gate's verdict. A reached gate turns its threshold
actions into TODO proposals, approved one by one; the verdict is written
as an EVAL entry (and a BDR when an action is decided) through
`memory.sh`.
Reminder (`hooks/session-start.sh`, every repo): one banner line
`memory feedback: last run <n> d ago, transcripts expire at 7 → /memory-feedback`
when `state.json` is 5 days old or more (or absent while a store file
exists) and the store file is newer than `state.json`. Two `stat` calls, no
JSON parse, silent on any error.
New skill: a `routing.json` row and the citers census entry.
## 4. Thresholds and gates (`mods/memory/feedback/thresholds.ts`)
One exported table, read by the reader and by a future /prune-memory step D
(BDR-100: thresholds in one file).
| Signal | Threshold | Action |
|---|---|---|
| prompt precision (relevant / labelled injected entries) | ≥ 64 % (4c baseline 14/22) | keep |
| same, below 64 % | worst trigger kind < 50 % | tighten or drop that trigger (idle first) |
| precision of the 3rd pick | < 40 % | cap at 2 entries |
| timeout + paused share (asks with `cands` > 0) | > 25 % | revisit `RERANK_MS` or skip the rerank on short lists |
| `ms` p90 | < 2500 | lower `RERANK_MS` |
| tool precision | < 50 % | require 2 matched 4-grams, or exclude the entry |
| missed in the sampled asks | ≥ 1 ask in 3 | diagnose by cause; no fix built ahead |
| entry injected ≥ 3 times, 0 relevant | any | noisy: /prune-memory rewrites it |
| entry relevant ≥ 3 times in ≥ 2 sessions | any | promotion candidate (BDR-100) |
Gates (`gateVerdict`, pure):
- Prompt: reached at 20 labelled injected asks (non-`test`) or 35 days
after the clock start.
- Tool: reached at 10 labelled tool injections or 35 days. Below 10 at the
cap: `insufficient sample n=<n>`, extended once by 35 days (flag in
`state.json`), then closed with no tuning.
- Each gate decides alone; the prompt gate never waits for the tool gate.
## 5. Errors
- Reader: fail loud (message + non-zero rc) on a missing or malformed store
file, a malformed transcript line is skipped and counted, an invalid
draft refuses the whole apply.
- Reminder: bounded, silent on any error, never breaks the banner.
- Mod: unchanged fail-open contract; log writes keep `.catch`.
## 6. Tests
- `register.test.ts` (4d.1): `cands` written on every prompt outcome, `ms`
written when a rerank ran (timeout included) and absent otherwise, an
old-shape record loads, `none` split by `cands`. Run through
`make test suite=lib/tests/mods.test.sh`.
- `mods/memory/feedback/reader.test.ts` (bun), synthetic store and
transcripts, no real text: each join case (injected, none, tool, lost),
`message.id` dedupe, `cited`, apply refusals (nothing written), file
modes, `gateVerdict` at each threshold boundary, the tool extension.
- Reminder: session-start suite cases with forced mtimes (shown, hidden,
absent state, unreadable state).
- Live proof: one real `/memory-feedback` run on today's log (pre-4d.1
records serve as practice, labelled `test` where they are proofs).
## Constraints honoured
BDR-125 (no Stop hook, no runtime cited flag), BDR-137 (store caps,
interactive-only prompt proof), BDR-100 (one threshold file, mechanism not
lesson), BDR-129 (registry writes through memory.sh), BDR-057 (retention
stays 7 days), LRN-124/LRN-261 (no text in store or git, import not port),
LRN-027 (state file), LRN-241 (`test` label), LRN-212 (mod edits are live:
kit tests before the edit lands), LRN-073 (fictional IDs in skill
examples), EVAL-037 (message.id dedupe).