forked from bchanot/claude
docs(spec): phase 4c memory recall design, contract and spike
This commit is contained in:
1 parent
06d2576bc0
commit
fb6cbf1e34
3 files changed
+233
No files matched your search
@@ -0,0 +1,48 @@
|
||||
# CONTRACT — memory-recall (phase 4c)
|
||||
- date: 2026-11-04 | flow: ship-feature | branch: feature/memory-recall (off develop 06d2576, created at STEP 4)
|
||||
- status: active
|
||||
|
||||
## REQUEST (verbatim — IMMUTABLE)
|
||||
Phase 4c of the context mod queue (BDR-125), TODO.md "Phase 4b-4d, memory v2 capture, read, feedback" — the 4c read side: two channels only: intent recall at `prompt.submit` for classifier labels design/analysis or the first coding prompt after idle (BM25 top-30 over title + lesson + body → haiku keeps ≤3, bounded 3 s, fail open) and tool-error 4-gram match (PostToolUse; a classic hook is acceptable if the mod ledger is not needed). Session digest (open BLK via the status parser, actionable TODO, last journal line) in `hooks/session-start.sh` on `source: startup|compact|clear`; a SubagentStart twin if sub-agents need it. Injection = ID + title + lesson line; superseded never injected; budget ~1.5k chars per turn; once per context. Deferred: Read path triggers, Bash command triggers, "critical → deny once", Tier 1, cross-repo index, `memory_search` tool. Plus the 4b follow-ups queued for 4c: earlier-session milestones in `memory_pending`, common-phrase 4-gram collisions (every ENOENT reads known LRN-056), the lexical superseded reader `shortcut:`. Host: mods/memory (BDR-137). 4d stays deferred until an injection log exists.
|
||||
|
||||
(user brief 2026-11-02, step 3: "Phases 4b à 4d"; launched after the 4b merge, user "merge it" 2026-11-04. Session constraints of that brief hold: default model opus; verifier, security-auditor and plan-challengers pinned on opus; never commit a `/model` or UI write of settings.json (LRN-253); plan text through Write/Edit (BLK-039); destructive steps on the live tree in the main loop after the user's go, `mv` over `rm` (BLK-040).)
|
||||
|
||||
## CLARIFICATIONS
|
||||
Pass A (gaps found by the STEP 0d read-before, put to the user at the STEP 1 brainstorm; answers land `[gated]`):
|
||||
- The classifier label is private to model-router (`st.turnMain`, never stored or published) and the classifier never runs on prompts its keyword rules already map (`plan|design|architecture…` → plan, `why|explain|review|audit…` → reflect), nor on slash commands: a trigger "classifier labels design/analysis" read literally misses exactly the design/analysis prompts that say so.
|
||||
- No uniform "lesson line" exists in the registries (BDR `**Decision**` median 643 chars, max 1989; LRN `Apply` / `Future application` / `Pattern`; BLK `Solution` / `Real cause`; EVAL `Action`): three BDR hits overflow the 1.5k budget unless the line is cut.
|
||||
- Superseded readers diverge (memory.sh parser vs the 4b lexical reader: BDR-031 live for the lexical one, EVAL-041 falsely superseded for the shell one).
|
||||
- Engine types 2.1.316 expose `$.state` and `$.process.run` (4b recorded "no `$.state`" from the 2.1.277 types); runtime availability unverified for both; mod `session.start` has no `source`, `classic.SessionStart` does.
|
||||
- The 4-gram collision: `no such file or` / `such file or directory` belong to LRN-056 only, so every ENOENT reads known LRN-056 (document frequency does not fix it).
|
||||
- Proof needs ≥ 20 real prompts and ≥ 10 real errors labelled for relevance; precision bar = the 14/22 haiku-rerank baseline (BDR-125 amendment 2, TODO.md:114).
|
||||
|
||||
Brainstorm answers `[gated 2026-11-04]` (user: "continue" to recommendations 0-11, read as yes):
|
||||
0. Spike first (done: `.claude/tasks/plans/2026-11-04-memory-recall-spike/REPORT.md`): `$.process.run` memory.sh index 128 ms; `classic.SessionStart` fires in a mod with source startup/resume/compact and its `additionalContext` reaches the model; `tool.call` error-answer `context` reaches the model; haiku rerank median ≈ 2 s, tail 5.7-7.3 s, ≈ 4k input tokens.
|
||||
1. Intent-recall trigger without the router classifier: composer prompt, not mid-turn, not a slash command, ≥ 12 chars, AND (first prompt of a context [startup, /clear, compaction] OR first prompt after ≥ 30 min idle OR a design/analysis keyword match, the router's two default rule lists copied with their source commit).
|
||||
2. Corpus = live BDR + LRN + BLK (EVAL excluded). Lesson line: BDR `Decision`; LRN `Apply` > `Future application` > `Pattern`; BLK `Solution` > `Real cause`; first sentence cut at 280 chars. Injection line = `ID, title (≤ 100), lesson`; ≤ 3 entries; hard cap 1500 chars.
|
||||
3. Rerank: BM25 top-30 over title + lesson + body; haiku `claude-haiku-5-5`, effort low, 64 max tokens, gets the prompt + 30 candidates as ID + title + lesson, answers IDs validated against the candidates; own 3 s race (LRN-218); failure or timeout → nothing injected (no BM25-only fallback).
|
||||
4. Superseded reader = lib/memory.sh (single reader, BDR-129) through `$.process.run(['bash', memory.sh, '-C', root, 'index'])` (spike: works, 128 ms); superseded and deprecated excluded; the 4b lexical `shortcut:` reader is replaced by it for the error index too (BDR-031 becomes superseded as memory.sh says).
|
||||
5. Tool-error channel in the existing Bash hook (the ledger lives in the mod): a captured failure matching a live BLK/LRN entry attaches `memory: known <ID>, title, lesson` to the tool result context; an errno stop-phrase list (no such file or directory, permission denied, command not found, …) drops grams wholly inside a stop phrase at index and match time; main loop and sub-agents; noise measured with the 14-day replay before shipping.
|
||||
6. Once per context: injected IDs kept in the closure per session and agent; reset at `session.end` and on `classic.SessionStart` compact/clear; the 1500-char per-turn budget covers both channels.
|
||||
7. Digest in `hooks/session-start.sh`: reads the hook JSON on stdin without blocking under `</dev/null`; sources startup, clear, compact (not resume); open BLK (upstream counted apart), first 3 unchecked items of TODO `### Actionable`, last journal line; ≤ 160 chars per line, ≤ 800 per block; silent without registries or without the section. No SubagentStart twin.
|
||||
8. `memory_pending` lists earlier-session milestones after the current session's, marked `(earlier session)`.
|
||||
9. Off switch: `"enabled": false` cuts everything; a new key `"recall": false` cuts the injections only (capture and tool stay).
|
||||
10. Injection log in scope: last 200 injections per project `{at, session, channel, ids}` in the store, not displayed (4d reads it).
|
||||
11. Proof: a bun harness importing the mod's own functions (no port), haiku through `claude -p`; 20 real prompts from 14 days of transcripts and the error replay; the controller labels relevance, the user checks the labels; bar ≥ 14/22; oracle written after the measurement; live: headless error half, interactive recall + digest after a real /compact by the user.
|
||||
|
||||
## ACCEPTANCE CRITERIA (seed, enriched at STEP 3 `[gated]`)
|
||||
1. Intent recall: on the qualifying prompts (trigger decided at the brainstorm), the mod ranks live registry entries by BM25 over title + lesson + body, asks a small model to keep ≤ 3, bounded in time, and attaches them as context (ID + title + lesson line); any failure or timeout attaches nothing and passes the prompt through unchanged.
|
||||
2. Tool-error match: a captured tool failure whose error line shares a 4-gram with a live BLK/LRN entry attaches that entry (ID + title + lesson line) as context; the common-phrase collision (ENOENT → LRN-056) no longer fires.
|
||||
3. Superseded entries are never injected; the superseded reader is decided at the brainstorm and its verdicts agree with lib/memory.sh on this repo's registries (an independent oracle, LRN-237).
|
||||
4. Budget: at most ~1.5k chars of injected recall per turn; an entry is injected at most once per context (reset on /clear and compaction).
|
||||
5. Session digest: `hooks/session-start.sh` prints open BLK (status parser), actionable TODO and the last journal line on `source` startup|compact|clear, bounded in size; runs unchanged with `</dev/null` and in projects without registries or TODO.
|
||||
6. `memory_pending` lists earlier-session milestones (marked `(earlier session)`) after the current session's.
|
||||
7. Every hook fails open; the `~/.claude/memory.json` off switch covers the read side; `claude plugin validate --strict`, `claude plugin test` (mods suite) and `make test` are green.
|
||||
8. Proof: recall precision measured on ≥ 20 real prompts and tool-error matching on ≥ 10 real errors, at or above the 14/22 baseline; a digest observed after a real compaction; the injection noise measured before shipping (LRN-261); the measured oracle written only after the measurement (EVAL-056).
|
||||
|
||||
## FILE SCOPE (seed, final at STEP 3 `[gated]`)
|
||||
- mods/memory/hooks/register.ts, mods/memory/hooks/register.test.ts (host, BDR-137); mods/memory/.claude-plugin/plugin.json (version)
|
||||
- hooks/session-start.sh and the suites that run it (lib/tests/memory.test.sh, push-guard.test.sh, graphify-gate.test.sh) if its output changes
|
||||
- lib/memory.sh only if a read helper is needed (no write path change)
|
||||
- README.md, USAGE.md, CHANGELOG.md, ARCHITECTURE.md, .claude/tasks/TODO.md
|
||||
- .claude/tasks/plans/ (spike and measurement notes), docs/superpowers/{specs,plans} (transient), this contract
|
||||
@@ -0,0 +1,27 @@
|
||||
# probe4c report (CLI 2.1.318, claude -p haiku, --setting-sources local, --plugin-dir)
|
||||
|
||||
Throwaway mod `$CLAUDE_JOB_DIR/tmp/probe4c/mod` (log only in the job tmp; deleted with the job).
|
||||
Recommendation 0 of the 4c brainstorm (contract `.claude/tasks/contracts/2026-11-04-memory-recall-1859.md`).
|
||||
|
||||
| probe | result |
|
||||
|---|---|
|
||||
| `$.process.run(['bash', ~/.claude/lib/memory.sh, '-C', <repo>, 'index'], {timeoutMs})` in a hook | works: exit 0, 508 lines (4 registries, `| ID | Date | Title | Status |`), 128 ms |
|
||||
| `classic.SessionStart` in a mod | fires with `source` startup; `--continue "/compact"` fires `resume` then `compact`; keys: session_id, transcript_path, cwd, source, model, agent_id, agent_type, effort, … |
|
||||
| `classic.SessionStart` returning `additionalContext` | reaches the model (it quoted `PROBE4C-SESSIONSTART-CONTEXT source=startup`) |
|
||||
| `tool.call` Bash error answer `{...out, context: [...]}` | reaches the model (it quoted `PROBE4C-TOOL-CONTEXT` as tool-result context); the error answer has no `context` key before the hook adds one |
|
||||
| `$.model.complete({model: 'claude-haiku-5-5', prompt, maxTokens: 64, effort})` | see below |
|
||||
|
||||
haiku rerank latency (prompt = a design question + 30 candidates; same session, sequential):
|
||||
|
||||
| variant | prompt chars | input tokens | ms (n=3, + 2 earlier full/low runs: 2494, 7332) | output tokens |
|
||||
|---|---|---|---|---|
|
||||
| effort low, ~350 chars per candidate | 10 865 | 3 965 | 1725, 2743, 1790 | 225-258 |
|
||||
| no effort, same prompt | 10 865 | 3 965 | 2063, 5713, 1938 | 224-302 |
|
||||
| effort low, ~150 chars per candidate | 4 865 | 2 045 | 2921, 2060, 2755 | 326-394 |
|
||||
|
||||
- Median ≈ 2 s; tail 5.7-7.3 s (2 of 11 calls over 5 s, 3 of 11 over 2.7 s). A 3 s bound drops roughly one call in five on this sample.
|
||||
- `maxTokens: 64` does not cap output tokens (220-400 reported): the model thinks; the request type has no thinking switch. A shorter candidate list does not cut latency.
|
||||
- Answers were valid candidate IDs every time.
|
||||
- Cost per recall ≈ 4k input + 0.3k output haiku tokens.
|
||||
|
||||
Consequences for the design: decision 4 (superseded reader) = memory.sh through `$.process.run` (single reader, 128 ms, no port); the compact/clear ledger reset hooks `classic.SessionStart`; the 3 s bound is the user's number, its timeout share is measured by the proof harness on the 20 real prompts and put to the gate.
|
||||
@@ -0,0 +1,158 @@
|
||||
# Phase 4c memory recall — design (2026-11-04)
|
||||
|
||||
BDR-125 phase 4c, host `mods/memory` (BDR-137). Contract
|
||||
`.claude/tasks/contracts/2026-11-04-memory-recall-1859.md` (brainstorm answers
|
||||
0-11 `[gated 2026-11-04]`). Engine facts:
|
||||
`.claude/tasks/plans/2026-11-04-memory-recall-spike/REPORT.md` (CLI 2.1.318).
|
||||
Branch `feature/memory-recall`.
|
||||
|
||||
## Goal
|
||||
The registries hold the lessons; nothing brings them back when they matter.
|
||||
4c brings back the right entries at three moments, without noise: when a
|
||||
design or analysis prompt starts work, when a known error comes back, and at
|
||||
the start of every context. Writing is untouched (4b capture, memory.sh).
|
||||
Success: precision of the injected entries at or above the 14/22 baseline
|
||||
(64 %), the error channel's noise measured before shipping, a digest seen
|
||||
after a real compaction.
|
||||
|
||||
## Unit 1 — shared reader (register.ts)
|
||||
- Status: `lib/memory.sh -C <root> index` through `$.process.run(['bash',
|
||||
<HOME>/.claude/lib/memory.sh, '-C', root, 'index'], {timeoutMs: 2000})`,
|
||||
raced by an own 2.5 s timer; rows `| ID | Date | Title | Status |`; an ID is
|
||||
closed for recall when its status word is `superseded` or `deprecated`
|
||||
(memory.sh is the single reader, BDR-129; spike: 128 ms). The script path is
|
||||
`$HOME/.claude/lib/memory.sh` (the installed lib, the same one the skills
|
||||
call). A failed or timed-out read → the status map is unavailable for that
|
||||
call: recall injects nothing, the error channel injects nothing, and
|
||||
`memory_pending` answers as today with an `index: status reader unavailable`
|
||||
line and every failure `NEW`.
|
||||
- Entries: the registry files `decisions.md`, `learnings.md`, `blockers.md`
|
||||
read with `$.fs.read` (stat first, 4 MiB cap per file, as 4b `loadIndex`);
|
||||
the 4b `entriesOf` heading regex extends to `BDR|LRN|BLK`; each entry keeps
|
||||
`{id, kind, title, lesson, body}`.
|
||||
- Lesson line: BDR the `Decision` field; LRN `Apply`, else `Future
|
||||
application`, else `Pattern`; BLK `Solution`, else `Real cause` (label match
|
||||
case-insensitive, bold or plain, `- **Label**:` / `- Label:`); its first
|
||||
sentence, cut at 280 chars with `…`; an entry without one has the empty
|
||||
lesson (title only).
|
||||
- The 4b lexical `isSuperseded` and its `shortcut:` comment are removed: the
|
||||
error index (BLK + LRN spans, as 4b) takes its live set from the same status
|
||||
map.
|
||||
|
||||
## Unit 2 — intent recall (`prompt.submit`)
|
||||
- Trigger: origin `composer`, `turnId` absent, not a slash command (text not
|
||||
starting `/`), ≥ 12 code points, and one of: the first such prompt of the
|
||||
context (set at `session.start` and on `classic.SessionStart` source
|
||||
`clear` or `compact`, consumed by the first qualifying prompt); the first
|
||||
after ≥ 30 min without a composer prompt in this session; a match of the
|
||||
design or analysis keyword patterns, copied from model-router's default
|
||||
prompt rules (`mods/model-router/hooks/register.ts`, commit 9bbd1d5):
|
||||
`plan|planifie|planning|brainstorm|architecture|con[cç]ois|design` and
|
||||
`pourquoi|why|explique|explain|analyse|analyze|comprendre|understand|review|audit`,
|
||||
same word boundaries.
|
||||
- Rank: BM25 (k1 1.2, b 0.75) over title + lesson + body of the live
|
||||
BDR/LRN/BLK entries, tokens = lowercase letter/digit runs of ≥ 2 chars minus
|
||||
a short FR+EN stop-word list; the query is the prompt's first 2000 chars,
|
||||
redacted (4b `redact`). Entries already injected in this context are
|
||||
removed before ranking. Top 30 with a score > 0.
|
||||
- Rerank: `$.model.complete({model: 'claude-haiku-5-5', effort: 'low',
|
||||
maxTokens: 64, timeoutMs: 3000, prompt})`, raced by an own 3 s timer
|
||||
(LRN-218). The prompt: the redacted query, the candidates one per line as
|
||||
`ID | title | lesson`, and the instruction to answer at most 3 IDs, comma
|
||||
separated, or `none`. The answer is parsed for `(BDR|LRN|BLK)-\d+`, kept
|
||||
only when the ID is a candidate, at most 3, order kept.
|
||||
- Inject: one context block `memory recall:` then one line per entry
|
||||
`<ID> <title ≤ 100> — <lesson>`; the block ≤ 1500 chars (an entry that
|
||||
would overflow is dropped). No candidate, a `none`, a timeout, a refusal or
|
||||
any error → nothing attached, the prompt passes unchanged.
|
||||
- The 4b milestone note on the same prompt is attached first; recall follows.
|
||||
|
||||
## Unit 3 — tool-error recall (Bash `tool.call`)
|
||||
- After the 4b capture (same failure rule, same redacted line): the failure's
|
||||
key is matched against the error index (live BLK/LRN, Unit 1); a match whose
|
||||
ID was not injected in this context attaches `memory: known <ID> <title ≤
|
||||
100> — <lesson>` to the tool answer's `context` (the answer keeps its shape,
|
||||
LRN-205; spike: the model sees it).
|
||||
- Common phrases: before gramming, both the entry spans and the failure key
|
||||
replace each errno phrase by the placeholder `<errno>`; a gram made only of
|
||||
placeholders is skipped (LRN-258, `<errno>` joins `#`, `<path>`, `<hex>`).
|
||||
Phrases: no such file or directory, permission denied, command not found,
|
||||
is a directory, not a directory, file exists, directory not empty,
|
||||
operation not permitted, no space left on device, connection refused,
|
||||
connection timed out, broken pipe, resource temporarily unavailable,
|
||||
invalid argument, too many open files, read-only file system, device or
|
||||
resource busy, input/output error.
|
||||
- Main loop and sub-agents (the context key carries the agent id).
|
||||
|
||||
## Unit 4 — ledger, budget, log
|
||||
- Once per context: `st.seen: Map<contextKey, Set<ID>>`, key = session id +
|
||||
(`agentId` or `main`); cleared at `session.end`, and the session's keys on
|
||||
`classic.SessionStart` source `clear` or `compact`.
|
||||
- Budget: the main loop has 1500 chars per turn, shared by both channels,
|
||||
reset by each composer prompt; a sub-agent (error channel only) has 1500
|
||||
chars for its whole context.
|
||||
- Log: each injection appends `{at, session, agent, channel: 'prompt' |
|
||||
'tool', ids}` to the project's store value (`log`, last 200 kept, 4b write
|
||||
chain and size caps); never displayed; 4d reads it.
|
||||
|
||||
## Unit 5 — session digest (`hooks/session-start.sh`)
|
||||
- Reads the hook JSON from stdin only when stdin is not a terminal, with
|
||||
`timeout 1`; `source` taken with jq when present, else empty. Prints the
|
||||
digest for `startup`, `clear`, `compact`, and for an empty source (a manual
|
||||
run); never for `resume`.
|
||||
- Only in a project whose git toplevel (else cwd) has `.claude/memory/`:
|
||||
`memory digest:` then up to three kinds of lines:
|
||||
open BLK from `reconcile_blk_open` with status word `open` (`BLK-n title`,
|
||||
≤ 160 chars each, at most 3, then `+N more`), the upstream ones counted
|
||||
(`+N upstream`); the first 3 unchecked `- [ ]` items under
|
||||
`### Actionable` in `.claude/tasks/TODO.md` (≤ 160 chars each); the last
|
||||
`- ` line of `.claude/memory/journal.md` (≤ 160 chars). Block ≤ 800 chars.
|
||||
Any missing file or section → that line is omitted; nothing to show → no
|
||||
block. Fail silent: an error in the digest never changes the banner or the
|
||||
exit code.
|
||||
- No SubagentStart twin.
|
||||
|
||||
## Unit 6 — `memory_pending`
|
||||
Milestones of the earlier stored sessions follow the current session's,
|
||||
marked `(earlier session)`, same 3-session window; the answer order stays
|
||||
header, `index:`, milestones, failures.
|
||||
|
||||
## Unit 7 — off switch
|
||||
`~/.claude/memory.json`: `"enabled": false` cuts everything (4b);
|
||||
`"recall": false` cuts Units 2 and 3 injections only (capture, the note, the
|
||||
tool and the log of past injections stay). Read like `enabled` (lazily per
|
||||
session, a non-boolean logs one line and counts as absent).
|
||||
|
||||
## Docs
|
||||
README / USAGE (fr) memory sections gain recall, the error channel, the
|
||||
digest, `"recall": false`, the residuals; CHANGELOG `[Unreleased]`; TODO (4c
|
||||
ticked at merge, 4d stays deferred with the log now in place).
|
||||
|
||||
## Proof
|
||||
- Kit tests: reader (status map, lesson per kind, failure → fail open),
|
||||
trigger (first prompt, idle, keyword, slash and mid-turn excluded), BM25
|
||||
order on fixtures, rerank parse (foreign IDs dropped, `none`, timeout, never
|
||||
settles), budget and once-per-context (reset on compact/clear and
|
||||
session.end), error channel (known match attaches, ENOENT does not, sub-agent
|
||||
key), earlier milestones, `recall: false`, log cap; regex costs timed on
|
||||
64 KiB (LRN-259).
|
||||
- Shell suite: the digest on fixture projects (sources, missing files, caps,
|
||||
`</dev/null`), existing session-start callers unchanged.
|
||||
- Measured before shipping (LRN-261, EVAL-056): a bun harness under
|
||||
`.claude/tasks/plans/2026-11-04-memory-recall-spike/` imports the mod's own
|
||||
functions (no port) and calls haiku through `claude -p --model haiku`:
|
||||
(a) 20 real composer prompts from 14 days of this repo's transcripts that
|
||||
meet the trigger: injected IDs per prompt, timeout share at 3 s; the
|
||||
controller labels each injected entry relevant or not, the user checks the
|
||||
labels; bar: relevant share ≥ 64 % (14/22). (b) the 14-day Bash failure
|
||||
replay: injections per session and the noise share on the injected ones,
|
||||
ENOENT lines included. Counts and shares only in the committed notes.
|
||||
- Live: headless `-p` for the error channel (a failing command matching a
|
||||
fixture BLK attaches the line); interactive by the user for recall (a design
|
||||
prompt in a fresh session), and the digest after a real `/compact`.
|
||||
|
||||
## Out of scope
|
||||
Read path triggers, Bash command triggers, critical deny-once, Tier 1,
|
||||
cross-repo index, `memory_search` tool, reading the router's classifier
|
||||
label, a SubagentStart twin, injection citation tracking (4d), EVAL in the
|
||||
recall corpus.
|
||||
Reference in new issue
Block a user