From 26f70e1885d3a4b988bcd6f8862ab6a50bc0f2fb Mon Sep 17 00:00:00 2001 From: bchanot Date: Fri, 6 Nov 2026 21:19:16 +0100 Subject: [PATCH 01/19] docs(spec): memory feedback 4d design --- .../2026-11-06-memory-feedback-4d-1825.md | 25 +++ .../2026-11-06-memory-feedback-design.md | 189 ++++++++++++++++++ 2 files changed, 214 insertions(+) create mode 100644 .claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md create mode 100644 docs/superpowers/specs/2026-11-06-memory-feedback-design.md diff --git a/.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md b/.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md new file mode 100644 index 0000000..c0943c2 --- /dev/null +++ b/.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md @@ -0,0 +1,25 @@ +# CONTRACT — memory-feedback-4d +- date: 2026-11-06 | flow: ship-feature | branch: feature/memory-feedback (to start off develop c10d859) +- status: active (seed, STEP 0e; enriched at STEP 3) +- plan: .claude/tasks/plans/2026-11-05-memory-feedback-4d-1834.md (r1 draft, superseded by the STEP 2 plan) + +## REQUEST (verbatim — IMMUTABLE) +yes start it + +(context, the request as received: "Phase 4d memory feedback and promotion (BDR-125 queue). Plan r1 draft, not challenged: .claude/tasks/plans/2026-11-05-memory-feedback-4d-1834.md (4 open choices). TODO line '4d feedback and promotion': order 4d.1 log fields `cands` + `ms` → 4d.2 weekly reader (log × transcripts, labels.tsv) → 4d.3 gate (≥20 prompt + ≥10 tool injections, or 5 weeks) with thresholds fixed now → 4d.4 /prune-memory + BDR-100 promotion, conditional.") + +## CLARIFICATIONS + +## ACCEPTANCE CRITERIA +1. The feedback loop can say, from data, whether recall injections are useful (precision per channel and trigger), what they miss, and which entries are noisy or promotion candidates; no tuning is built before the gate. + (judged by the verifier against the enriched criteria below) +2. No prompt or reply text, and no secret, enters the plugin store or any committed file (LRN-124, LRN-261). + (judged by the verifier from the diff) +3. The memory mod stays fail-open; records written before any schema change still load. + (judged by the verifier; kit tests named at STEP 3) + +## FILE SCOPE +- mods/memory/hooks/register.ts, mods/memory/hooks/register.test.ts +- .claude/tasks/plans/2026-11-04-memory-recall-spike/recall-bench.ts (export only, if reused) +- a new reader + labels under .claude/tasks/plans/ (names fixed at STEP 2) +- skills/prune-memory/SKILL.md, skills/prune-memory/tests/run-deterministic.sh, one threshold lib script (4d.4, conditional) diff --git a/docs/superpowers/specs/2026-11-06-memory-feedback-design.md b/docs/superpowers/specs/2026-11-06-memory-feedback-design.md new file mode 100644 index 0000000..f4af8b7 --- /dev/null +++ b/docs/superpowers/specs/2026-11-06-memory-feedback-design.md @@ -0,0 +1,189 @@ +# Memory feedback (phase 4d): design + +Date: 2026-11-06. Queue: BDR-125 phase 4d. Host: `mods/memory` (BDR-137). +Supersedes the open choices of `.claude/tasks/plans/2026-11-05-memory-feedback-4d-1834.md` (r1). +Contract: `.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md`. + +## Goal + +Say from real data whether 4c recall helps: precision per channel and per +trigger, what it misses, which entries are noisy, which deserve promotion to +a mechanism (BDR-100). Thresholds and gates are fixed now, before the data. +Nothing is tuned before a gate is reached; each threshold action is its own +/feat or /hotfix carrying its measurement. + +In scope: 4d.1 log fields, 4d.2 reader + `/memory-feedback` skill + +session-start reminder, 4d.3 thresholds + gate verdict. Out of scope: 4d.4 +(/prune-memory step D, promotion mechanics), built only if a gate shows a +noisy entry or a promotion candidate (own TODO line). + +## User decisions (2026-11-06) + +| Choice | Decision | +|---|---| +| Cadence vs 7-day transcript retention | session-start reminder, user runs `/memory-feedback`; retention stays 7 days (BDR-057) | +| Gate population | records carrying `cands` (4d.1 on), every repo; `test` label excluded from shares | +| Labels location | local `~/.claude/memory-feedback/` (0600), never in git | +| Log fields | `cands` + `ms` | +| Gate shape | two independent gates, prompt and tool | +| Thresholds | r1 table unchanged | +| Approach | TS reader in `mods/memory/feedback/` + skill `/memory-feedback` | +| Label values | `relevant`, `irrelevant`, `test` (+ `missed` rows) | +| FEEDBACK.md | local, regenerated each run; recorded in the repo as an EVAL entry (and a BDR when an action is decided) at each gate | + +## 1. Log fields (4d.1, `mods/memory/hooks/register.ts`) + +- `LogRecord` gains `cands?: number` (BM25 candidates after the seen + filter, 0-30) on every prompt-channel record, and `ms?: number` (rerank + wall time, integer ms) on every record where a rerank call ran, timeout + included. Absent when no rerank ran (no candidate, paused). +- `isLogRecord` accepts both as optional non-negative integers; a record + without them stays valid (LG5 precedent). No other field, no text. +- Size: about 20 bytes per field; 200 records stay under the 64 KiB cap. +- `none` with `cands` = 0 (nothing to offer) is now distinct from `none` + with `cands` > 0 (haiku declined). + +## 2. Reader (`mods/memory/feedback/reader.ts`, bun) + +Imports the mod's own functions (`isLogRecord`, key helpers) rather than +porting them (LRN-261). Transcript loading lives in +`mods/memory/feedback/transcripts.ts`. + +### Inputs + +- The store file `~/.claude/plugins/store/memory_skills-dir-*.json`, every + `l:` key, all roots. Engine-internal path: a missing or unparsable + file fails loudly. +- Transcripts `~/.claude/projects//.jsonl`, slug = the + root with every non-alphanumeric character replaced by `-`. +- `~/.claude/memory-feedback/state.json`: last run, labelled record keys + (`root|session|at|channel`), tool-gate extension flag (LRN-027). + +### Population + +Records with a `cands` field (prompt channel) or written after the first +such record (tool channel), not yet labelled. The gate clock starts at the +first record carrying `cands`. + +### Join + +- Prompt, `injected`: the `attachment` record with `hook_additional_context` + and `hookName` `prompt.submit` nearest after `at`, within 10 s, same + session. Prompt = the composer user record it follows; reply = the + assistant text blocks after it up to the next composer prompt, deduped by + `message.id` (EVAL-037). +- Prompt, `none` / `timeout` / `refused` / `paused`: the last composer + prompt of the session before `at`. +- Tool: the `tool.call` attachment carrying `toolUseID`; the failing Bash + command and the next 3 Bash results of the session (retry succeeded or not). +- No transcript, no attachment: the record is `lost`, counted in + FEEDBACK.md, never dropped silently (LRN-133). +- `cited`: the entry ID appears in the reply text (lower bound). + +### Commands + +- `worksheet `: writes `/worksheet.md` and `/items.json` + (mode 0600). `` is the session scratchpad. These are the only files + that hold prompt or reply text (prompt cut at 600 chars, reply at 800). +- `apply `: reads `/labels-draft.tsv`, refuses an unknown label, + an unknown item, or an ID not in that item (non-zero rc, nothing + written); appends to `labels.tsv`, updates `state.json`, regenerates + `FEEDBACK.md`. Writes go through a temp file and rename. + +### labels.tsv (`~/.claude/memory-feedback/`, 0600) + +One row per injected entry, plus `missed` rows: `at, root name, session, +channel, trigger, outcome, cands, ms, pick (1-3), id, cited, label, cause`. +`label` in `relevant | irrelevant | test | missed`; `cause` only on +`missed`: `no-overlap | rank-over-30 | declined | not-triggered`. IDs and +enums only, no text. + +### FEEDBACK.md (`~/.claude/memory-feedback/`) + +Counts and shares only, no repo names, no IDs except in the noisy and +promotion lists: n per channel and trigger, lost count, every metric of +section 4, each gate's verdict. + +## 3. Skill `/memory-feedback` and reminder + +Skill steps (main loop; the labeller is Claude, the user spot-checks): + +1. `worksheet` into the scratchpad. +2. Label each injected entry `relevant` or `irrelevant`; proof, bench and + recall-testing sessions get `test`. +3. Spot check: 5 labelled items at random shown to the user, who confirms + or flips each (LRN-168: a label is a claim). +4. Misses: on up to 5 triggered asks, search the registries + (`memory.sh index` + bodies) for relevant entries not injected; add + `missed` rows with a cause. +5. `apply`; print each gate's verdict. A reached gate turns its threshold + actions into TODO proposals, approved one by one; the verdict is written + as an EVAL entry (and a BDR when an action is decided) through + `memory.sh`. + +Reminder (`hooks/session-start.sh`, every repo): one banner line +`memory feedback: last run d ago, transcripts expire at 7 → /memory-feedback` +when `state.json` is 5 days old or more (or absent while a store file +exists) and the store file is newer than `state.json`. Two `stat` calls, no +JSON parse, silent on any error. + +New skill: a `routing.json` row and the citers census entry. + +## 4. Thresholds and gates (`mods/memory/feedback/thresholds.ts`) + +One exported table, read by the reader and by a future /prune-memory step D +(BDR-100: thresholds in one file). + +| Signal | Threshold | Action | +|---|---|---| +| prompt precision (relevant / labelled injected entries) | ≥ 64 % (4c baseline 14/22) | keep | +| same, below 64 % | worst trigger kind < 50 % | tighten or drop that trigger (idle first) | +| precision of the 3rd pick | < 40 % | cap at 2 entries | +| timeout + paused share (asks with `cands` > 0) | > 25 % | revisit `RERANK_MS` or skip the rerank on short lists | +| `ms` p90 | < 2500 | lower `RERANK_MS` | +| tool precision | < 50 % | require 2 matched 4-grams, or exclude the entry | +| missed in the sampled asks | ≥ 1 ask in 3 | diagnose by cause; no fix built ahead | +| entry injected ≥ 3 times, 0 relevant | any | noisy: /prune-memory rewrites it | +| entry relevant ≥ 3 times in ≥ 2 sessions | any | promotion candidate (BDR-100) | + +Gates (`gateVerdict`, pure): + +- Prompt: reached at 20 labelled injected asks (non-`test`) or 35 days + after the clock start. +- Tool: reached at 10 labelled tool injections or 35 days. Below 10 at the + cap: `insufficient sample n=`, extended once by 35 days (flag in + `state.json`), then closed with no tuning. +- Each gate decides alone; the prompt gate never waits for the tool gate. + +## 5. Errors + +- Reader: fail loud (message + non-zero rc) on a missing or malformed store + file, a malformed transcript line is skipped and counted, an invalid + draft refuses the whole apply. +- Reminder: bounded, silent on any error, never breaks the banner. +- Mod: unchanged fail-open contract; log writes keep `.catch`. + +## 6. Tests + +- `register.test.ts` (4d.1): `cands` written on every prompt outcome, `ms` + written when a rerank ran (timeout included) and absent otherwise, an + old-shape record loads, `none` split by `cands`. Run through + `make test suite=lib/tests/mods.test.sh`. +- `mods/memory/feedback/reader.test.ts` (bun), synthetic store and + transcripts, no real text: each join case (injected, none, tool, lost), + `message.id` dedupe, `cited`, apply refusals (nothing written), file + modes, `gateVerdict` at each threshold boundary, the tool extension. +- Reminder: session-start suite cases with forced mtimes (shown, hidden, + absent state, unreadable state). +- Live proof: one real `/memory-feedback` run on today's log (pre-4d.1 + records serve as practice, labelled `test` where they are proofs). + +## Constraints honoured + +BDR-125 (no Stop hook, no runtime cited flag), BDR-137 (store caps, +interactive-only prompt proof), BDR-100 (one threshold file, mechanism not +lesson), BDR-129 (registry writes through memory.sh), BDR-057 (retention +stays 7 days), LRN-124/LRN-261 (no text in store or git, import not port), +LRN-027 (state file), LRN-241 (`test` label), LRN-212 (mod edits are live: +kit tests before the edit lands), LRN-073 (fictional IDs in skill +examples), EVAL-037 (message.id dedupe). From 8253a92cbce4c4fbaa27f8c02a551fb569003091 Mon Sep 17 00:00:00 2001 From: bchanot Date: Fri, 6 Nov 2026 21:33:03 +0100 Subject: [PATCH 02/19] docs(plan): memory feedback 4d implementation plan --- .../plans/2026-11-06-memory-feedback.md | 1989 +++++++++++++++++ 1 file changed, 1989 insertions(+) create mode 100644 docs/superpowers/plans/2026-11-06-memory-feedback.md diff --git a/docs/superpowers/plans/2026-11-06-memory-feedback.md b/docs/superpowers/plans/2026-11-06-memory-feedback.md new file mode 100644 index 0000000..310c8e3 --- /dev/null +++ b/docs/superpowers/plans/2026-11-06-memory-feedback.md @@ -0,0 +1,1989 @@ +# Memory Feedback (phase 4d) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Measure from real data whether 4c memory recall helps, through two +optional log fields, a bun reader that joins the log to transcripts into +local labels, a `/memory-feedback` skill, a session-start reminder, and a +fixed-threshold gate. + +**Architecture:** The mod (`mods/memory/hooks/register.ts`) logs `cands` +and `ms` on prompt records. A bun reader in `lib/memory-feedback/` (four +focused modules: thresholds, labels, transcripts, metrics, plus the CLI) +imports the mod's own log validator, joins records to transcript +attachments, writes a text worksheet only to the scratchpad, and keeps IDs +and enums in `~/.claude/memory-feedback/` (0600). A shell lib prints the +reminder line in the banner. The skill drives labelling. + +**Tech Stack:** TypeScript on bun (`bun test`, `node:fs`), Claude Code mod +kit (`claude-code/testing`), bash 3.2-safe shell, jq-free banner check. + +**Spec:** `docs/superpowers/specs/2026-11-06-memory-feedback-design.md` +**Contract:** `.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md` + +## Global Constraints + +- No prompt or reply text in the plugin store, `labels.tsv`, `state.json`, + `FEEDBACK.md` or any committed file; text only in `/worksheet.md` and + `/items.json` where `` is the session scratchpad (LRN-124, + LRN-261). +- `~/.claude/memory-feedback/` created 0700, every file in it 0600, writes + through a temp file and a rename. +- Our code: at most 25 logic lines per function, 80 columns, 5 params, no + global mutable state. +- No new dependency: bun built-ins and `node:*` only. +- Shell is bash 3.2-safe (no `mapfile`, `declare -A`, `local -n`). +- Every suite that runs the hook, the mod kit or bun runs contained: + `systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" [r.outcome, r.cands, r.ms]) +} + +/** The single ask of a fresh world answered by `model`. */ +async function oneAsk($: Api, on: On, model?: Model) { + const { w } = await bootRecall($, on) + if (model !== undefined) w.model = model + await start($) + await say($, GO) + return (await askMeta(w))[0] ?? [] +} + +test('LF1: an injected ask logs cands and ms', async ($, on) => { + const [outcome, cands, ms] = await oneAsk($, on) + expect(outcome).toBe('injected') + expect(typeof cands === 'number' && cands >= 1).toBe(true) + expect(ms).toBe(0) // the mock clock does not move during an answer +}) + +test('LF1b: a declined ask logs cands and ms', async ($, on) => { + const [outcome, cands, ms] = await oneAsk($, on, { text: 'none' }) + expect(outcome).toBe('none') + expect(typeof cands === 'number' && cands >= 1).toBe(true) + expect(ms).toBe(0) +}) + +test('LF1c: an unanswered ask logs cands and ms', async ($, on) => { + const [outcome, cands, ms] = await oneAsk($, on, 'unanswered') + expect(outcome).toBe('timeout') + expect(typeof cands === 'number' && cands >= 1).toBe(true) + expect(ms).toBe(0) +}) + +test('LF2: no candidate and a pause log no ms', async ($, on) => { + const { w } = await bootRecall($, on) + w.proc = { stdout: rowsOut([['BLK-001', 'x'], ['LRN-002', 'y']]) + .replace(/open/g, 'superseded') } + await start($) + await say($, GO) + expect(await askMeta(w)).toEqual([['none', 0, undefined]]) +}) + +test('LF3: a paused ask logs cands without ms', async ($, on) => { + const { w } = await bootRecall($, on) + w.model = 'unanswered' + await start($) + for (let i = 0; i < 4; i++) await say($, DESIGN) + const last = (await askMeta(w)).at(-1) + expect(last?.[0]).toBe('paused') + expect((last?.[1] as number) >= 1).toBe(true) + expect(last?.[2]).toBeUndefined() +}) + +test('LF4: cands and ms are optional counts', async ($, on) => { + const good = { ...REC(2), cands: 4, ms: 812 } + const neg = { ...REC(3), cands: -1 } + const frac = { ...REC(4), ms: 1.5 } + const seed = { [`l:${ROOT}`]: [REC(1), good, neg, frac] } + const w = await boot($, on, { seed }) + expect(await loadLog(storeApi(w), ROOT)).toEqual([REC(1), good]) + expect(isLogRecord(REC(1))).toBe(true) +}) +``` + +- [ ] **Step 2: Run the suite, see the four new tests fail** + +Run: `systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" make test suite=lib/tests/mods.test.sh ` key per project (about 28 KiB at 200 records), +// swept by `sweepLogs` against the engine's 4 MiB store cap, oldest key +// first; upgrade path: one global log key, or a smaller per-root cap. +export const LOG_PREFIX = 'l:' +const MAX_LOG = 200 +const MAX_LOG_BYTES = 65536 + +/** Absent, or a non-negative integer. */ +const isCount = (v: unknown) => + v === undefined || (Number.isInteger(v) && (v as number) >= 0) + +/** A well-formed log record; the 4d reader imports it (LRN-261). */ +export function isLogRecord(v: unknown): v is LogRecord { + const r = v as Partial | null + return typeof r === 'object' && r !== null && + typeof r.at === 'number' && typeof r.session === 'string' && + typeof r.agent === 'string' && + (r.channel === 'prompt' || r.channel === 'tool') && + Array.isArray(r.ids) && r.ids.every(i => typeof i === 'string') && + (r.trigger === undefined || TRIGGERS.includes(r.trigger)) && + (r.outcome === undefined || OUTCOMES.includes(r.outcome)) && + isCount(r.cands) && isCount(r.ms) +} +``` + +Change `Picked` and `pickRecall` (lines ~1704-1740): + +```ts +type Picked = { block: string; ids: string[]; outcome: Outcome; ms?: number } +``` + +```ts +/** Asks haiku among the candidates and claims its picks. */ +async function pickRecall( + $: Api, + st: State, + key: string, + text: string, + cands: Entry[], +): Promise { + const now = await $.clock.now() + if (now < st.rerank.until) return nothing('paused') + const answer = await askRerank($, st, text, cands, now) + const ms = (await $.clock.now()) - now + if (answer === undefined) return { ...nothing('timeout'), ms } + const picked = picksOf(answer, cands) + if (picked.length === 0) return { ...nothing('none'), ms } + const taken = claimPicks(st, key, picked) + const block = recallBlock( + picked.filter(e => taken.includes(e.id)).map(recallLine), RECALL_CAP) + if (block === '') return { ...nothing('refused'), ms } + return { block, ids: taken, outcome: 'injected', ms } +} +``` + +In `recallFor`, replace the `record` literal: + +```ts + const record = { + at: await $.clock.now(), session: ask.at.id, agent: 'main', + ids: got.ids, trigger: ask.trigger, outcome: got.outcome, + cands: cands.length, ...(got.ms === undefined ? {} : { ms: got.ms }), + } +``` + +- [ ] **Step 4: Run the suite, see it pass** + +Run: `systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" make test suite=lib/tests/mods.test.sh = {}): Item => ({ + n, root: 'r', session: `s${n}`, at: 1000 + n, channel: 'prompt', + trigger: 'fresh', outcome: 'injected', cands: 4, ms: 900, + ids: ['BDR-001', 'LRN-002'], cited: ['BDR-001'], lost: false, ...over, +}) +const ITEMS = [item(1), item(2, { ids: [], outcome: 'none' })] +const tmp = () => fs.mkdtempSync(path.join(os.tmpdir(), 'mf-')) + +test('L1: a row survives the TSV round trip', () => { + const r: Row = { at: 5, root: 'r', session: 's', channel: 'tool', + trigger: 'tool', outcome: 'injected', cands: undefined, ms: undefined, + pick: 1, id: 'BLK-003', cited: true, label: 'relevant', cause: '' } + expect(fromLine(toLine(r))).toEqual(r) + expect(fromLine('at\troot')).toBeUndefined() +}) + +test('L2: a complete draft parses', () => { + const d = parseDraft([ + '1\tBDR-001\trelevant', '1\tLRN-002\tirrelevant', + '2\tBDR-009\tmissed\tno-overlap', '1\t-\tchecked', '# note', + ].join('\n'), ITEMS) + expect(Array.isArray(d) && d.length).toBe(4) +}) + +test('L3: every bad draft is refused with a reason', () => { + const bad = [ + '1\tBDR-001\tmaybe', // unknown label + '9\tBDR-001\trelevant', // unknown item + '1\tBLK-777\trelevant\n1\tBDR-001\trelevant\n1\tLRN-002\trelevant', + '1\tBDR-001\trelevant', // LRN-002 unlabelled + '1\tBDR-001\trelevant\n1\tBDR-001\trelevant\n1\tLRN-002\trelevant', + '1\tBDR-001\trelevant\n1\tLRN-002\trelevant\n2\tBDR-001\tmissed\tbad', + '1\tBDR-001\trelevant\n1\tLRN-002\tmissed\tdeclined', + ] + for (const text of bad) expect(typeof parseDraft(text, ITEMS)).toBe('string') +}) + +test('L4: test - marks a whole item, no entry label needed', () => { + expect(Array.isArray(parseDraft('1\t-\ttest', ITEMS))).toBe(true) +}) + +test('L5: a lost item takes no label', () => { + const lost = [item(1, { lost: true })] + expect(typeof parseDraft('1\tBDR-001\trelevant', lost)).toBe('string') + expect(parseDraft('', lost)).toEqual([]) +}) + +test('L6: rowsOf writes an ask row then the entry rows', () => { + const d = parseDraft('1\tBDR-001\trelevant\n1\tLRN-002\tirrelevant', + [item(1)]) + if (typeof d === 'string') throw new Error(d) + const rows = rowsOf(item(1), d) + expect(rows.map(r => [r.label, r.id, r.pick, r.cited])).toEqual([ + ['ask', '', undefined, false], + ['relevant', 'BDR-001', 1, true], + ['irrelevant', 'LRN-002', 2, false], + ]) + expect(rowsOf(item(3, { lost: true }), [])[0]?.label).toBe('lost') +}) + +test('L7: files are 0600 in a 0700 dir, state defaults', () => { + const dir = path.join(tmp(), 'mf') + writeRows(path.join(dir, 'labels.tsv'), rowsOf(item(1), [])) + expect(fs.statSync(dir).mode & 0o777).toBe(0o700) + expect(fs.statSync(path.join(dir, 'labels.tsv')).mode & 0o777) + .toBe(0o600) + expect(readRows(path.join(dir, 'labels.tsv'))).toHaveLength(1) + expect(readState(path.join(dir, 'state.json'))) + .toEqual({ lastRun: 0, toolExtended: false }) + writeState(path.join(dir, 'state.json'), { lastRun: 9, toolExtended: true }) + expect(readState(path.join(dir, 'state.json'))) + .toEqual({ lastRun: 9, toolExtended: true }) +}) +``` + +- [ ] **Step 2: Run, see them fail** + +Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test labels.test.ts + `${r.session}|${r.at}|${r.channel}` + +const num = (v: number | undefined) => v === undefined ? '' : String(v) +const opt = (s: string | undefined) => + s === undefined || s === '' ? undefined : Number(s) + +export function toLine(r: Row): string { + return [r.at, r.root, r.session, r.channel, r.trigger, r.outcome, + num(r.cands), num(r.ms), num(r.pick), r.id, r.cited ? '1' : '0', + r.label, r.cause].join('\t') +} + +/** A labels.tsv line as a row; the header or a short line: undefined. */ +export function fromLine(line: string): Row | undefined { + const f = line.split('\t') + if (f.length !== COLS.length || f[0] === 'at') return undefined + return { + at: Number(f[0]), root: f[1] ?? '', session: f[2] ?? '', + channel: f[3] as Channel, trigger: f[4] ?? '', outcome: f[5] ?? '', + cands: opt(f[6]), ms: opt(f[7]), pick: opt(f[8]), id: f[9] ?? '', + cited: f[10] === '1', label: f[11] as Label, cause: f[12] ?? '', + } +} + +/** Writes through a 0600 temp file and a rename; the dir is made 0700. */ +export function writeSafe(file: string, text: string): void { + fs.mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 }) + const tmp = `${file}.${process.pid}.tmp` + fs.writeFileSync(tmp, text, { mode: 0o600 }) + fs.renameSync(tmp, file) +} + +export function readRows(file: string): Row[] { + if (!fs.existsSync(file)) return [] + return fs.readFileSync(file, 'utf8').split('\n').map(fromLine) + .filter((r): r is Row => r !== undefined) +} + +export function writeRows(file: string, rows: Row[]): void { + writeSafe(file, [COLS.join('\t'), ...rows.map(toLine)].join('\n') + '\n') +} + +/** state.json; absent or unreadable gives a first-run state. */ +export function readState(file: string): State { + try { + const s = JSON.parse(fs.readFileSync(file, 'utf8')) + return { lastRun: Number(s.lastRun) || 0, + toolExtended: s.toolExtended === true } + } catch { + return { lastRun: 0, toolExtended: false } + } +} + +export const writeState = (file: string, s: State) => + writeSafe(file, JSON.stringify(s) + '\n') + +/** One draft line `itemidlabel[cause]`, or why not. */ +function parseLine(line: string): Draft | string { + const f = line.split('\t') + if (f.length < 3 || f.length > 4) return `bad field count: ${line}` + const item = Number(f[0]) + const label = f[2] ?? '' + if (!Number.isInteger(item)) return `bad item: ${f[0]}` + if (!DRAFT_LABELS.includes(label)) return `bad label: ${label}` + return { item, id: f[1] ?? '', label: label as Label, cause: f[3] ?? '' } +} + +/** Why `d` does not fit item `it`; '' when it fits. */ +function fitError(d: Draft, it: Item | undefined): string { + if (it === undefined || it.lost) return `unknown or lost item ${d.item}` + const plain = d.cause === '' + if (d.id === '-') { + return plain && (d.label === 'checked' || d.label === 'test') + ? '' : `item ${d.item}: only checked or test take id -` + } + if (ENTRY_LABELS.includes(d.label)) { + return plain && it.ids.includes(d.id) + ? '' : `item ${d.item}: ${d.id} was not injected there` + } + const ok = d.label === 'missed' && it.channel === 'prompt' && + ID_RE.test(d.id) && !it.ids.includes(d.id) && CAUSES.includes(d.cause) + return ok ? '' : `item ${d.item}: bad ${d.label} row ${d.id} ${d.cause}` +} + +/** Every injected entry of every joined item labelled exactly once. */ +function completeError(drafts: Draft[], items: Item[]): string { + const whole = new Set(drafts.filter(d => d.id === '-' && + d.label === 'test').map(d => d.item)) + const got = drafts.filter(d => d.id !== '-' && + ENTRY_LABELS.includes(d.label)).map(d => `${d.item}:${d.id}`) + if (new Set(got).size !== got.length) return 'an entry is labelled twice' + for (const it of items) { + if (it.lost || whole.has(it.n)) continue + const miss = it.ids.find(id => !got.includes(`${it.n}:${id}`)) + if (miss !== undefined) return `item ${it.n}: ${miss} unlabelled` + } + return '' +} + +/** The validated draft, or the first problem found. */ +export function parseDraft(text: string, items: Item[]): Draft[] | string { + const byN = new Map(items.map(i => [i.n, i])) + const drafts: Draft[] = [] + for (const line of text.split('\n')) { + if (line.trim() === '' || line.startsWith('#')) continue + const d = parseLine(line) + if (typeof d === 'string') return d + const err = fitError(d, byN.get(d.item)) + if (err !== '') return err + drafts.push(d) + } + const err = completeError(drafts, items) + return err === '' ? drafts : err +} + +const base = (it: Item) => ({ + at: it.at, root: it.root, session: it.session, channel: it.channel, + trigger: it.trigger, outcome: it.outcome, cands: it.cands, ms: it.ms, +}) + +/** The labels.tsv rows of one item: its ask row, then its draft rows. */ +export function rowsOf(it: Item, drafts: Draft[]): Row[] { + const head: Row = { ...base(it), id: '', cited: false, + label: it.lost ? 'lost' : 'ask', cause: '' } + const mine = drafts.filter(d => d.item === it.n).map((d): Row => ({ + ...base(it), id: d.id === '-' ? '' : d.id, + pick: it.ids.includes(d.id) ? it.ids.indexOf(d.id) + 1 : undefined, + cited: it.cited.includes(d.id), label: d.label, cause: d.cause, + })) + return [head, ...mine] +} +``` + +- [ ] **Step 4: Run, see them pass** + +Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test labels.test.ts & { label: Label }): Row { + clock += 1 + return { at: clock, root: 'r', session: 's1', channel: 'prompt', + trigger: 'fresh', outcome: 'injected', cands: 3, ms: 900, + id: 'BDR-001', cited: false, cause: '', ...over } +} +/** One injected ask with entries labelled `labels`, picks 1..n. */ +function ask(labels: Label[], over: Partial = {}): Row[] { + const at = (clock += 10) + const head = row({ ...over, at, id: '', label: 'ask' }) + return [head, ...labels.map((label, i) => row({ ...over, at, label, + id: `LRN-00${i + 1}`, pick: i + 1 }))] +} + +test('M1: precision and p90', () => { + const r = ask(['relevant', 'irrelevant', 'relevant', 'relevant']) + expect(precision(r)).toBe(0.75) + expect(precision([])).toBeUndefined() + expect(p90([5, 1, 9, 3, 7, 2, 8, 4, 6, 10])).toBe(9) + expect(p90([])).toBeUndefined() +}) + +test('M2: a test row excludes its whole item', () => { + const rows = [...ask(['relevant']), ...ask(['irrelevant', 'test'])] + const m = metricsOf(rows) + expect(m.precision).toBe(1) + expect(m.promptN).toBe(1) +}) + +test('M3: noisy at 3 hits, promotion at 3 in 2 sessions', () => { + const noisy = [1, 2, 3].flatMap(() => ask(['irrelevant'])) + expect(metricsOf(noisy).noisy).toEqual(['LRN-001']) + expect(metricsOf(noisy.slice(0, 4)).noisy).toEqual([]) + const one = [1, 2, 3].flatMap(() => ask(['relevant'])) + expect(metricsOf(one).promote).toEqual([]) + const two = [...one, ...ask(['relevant'], { session: 's2' })] + expect(metricsOf(two).promote).toEqual(['LRN-001']) +}) + +test('M4: late share counts asks with candidates only', () => { + const rows = [ + row({ id: '', label: 'ask', outcome: 'timeout' }), + row({ id: '', label: 'ask', outcome: 'paused' }), + row({ id: '', label: 'ask', outcome: 'none', cands: 0 }), + ...ask(['relevant']), + ] + expect(metricsOf(rows).late).toBeCloseTo(2 / 3) +}) + +/** Metrics with `n` prompt asks and `t` tool injections, clock at 0. */ +function m(n: number, t: number): Metrics { + const p = Array.from({ length: n }, () => ask(['relevant'])).flat() + const tl = Array.from({ length: t }, () => + ask(['relevant'], { channel: 'tool', trigger: 'tool', cands: undefined })) + return { ...metricsOf([...p, ...tl]), start: 0 } +} +const CAP = GATE_DAYS * DAY_MS + +test('M5: the prompt gate at 20 asks or at the cap', () => { + expect(gateVerdict(m(19, 0), 1, false).prompt).toBe('open') + expect(gateVerdict(m(20, 0), 1, false).prompt).toBe('reached') + expect(gateVerdict(m(3, 0), CAP, false).prompt).toBe('reached') + const none = { ...metricsOf([]), start: undefined } + expect(gateVerdict(none, CAP * 9, false)) + .toEqual({ prompt: 'open', tool: 'open', extendTool: false }) +}) + +test('M6: the tool gate extends once, then closes', () => { + expect(gateVerdict(m(0, 10), 1, false).tool).toBe('reached') + expect(gateVerdict(m(0, 9), CAP - 1, false).tool).toBe('open') + expect(gateVerdict(m(0, 9), CAP, false)) + .toMatchObject({ tool: 'extended', extendTool: true }) + expect(gateVerdict(m(0, 9), 2 * CAP - 1, true)) + .toMatchObject({ tool: 'extended', extendTool: false }) + expect(gateVerdict(m(0, 9), 2 * CAP, true).tool).toBe('insufficient') +}) + +test('M7: actions only once a gate is reached', () => { + const idle = Array.from({ length: 20 }, (_, i) => + ask([i % 3 === 0 ? 'relevant' : 'irrelevant'], { trigger: 'idle' })) + .flat() // 7/20 relevant: under 64 % overall and under 50 % for idle + const mm = { ...metricsOf(idle), start: 0 } + expect(actionsOf(mm, gateVerdict(mm, 1, false))) + .toContain('prompt precision under 64 %: tighten or drop the idle trigger') + const open = { ...mm, promptN: 2 } + expect(actionsOf(open, gateVerdict(open, 1, false))).toEqual([]) +}) + +test('M8: rerank latency actions', () => { + const slow = Array.from({ length: 20 }, () => + ask(['relevant'], { ms: 2400 })).flat() + const mm = { ...metricsOf(slow), start: 0 } + expect(actionsOf(mm, gateVerdict(mm, 1, false))) + .toContain('rerank p90 under 2500 ms: lower RERANK_MS') +}) + +test('M9: FEEDBACK.md has no repo name and no prompt text', () => { + const rows = ask(['relevant'], { root: 'secret-client-repo' }) + const text = renderFeedback(metricsOf(rows), + gateVerdict(metricsOf(rows), 1, false), 0) + expect(text).not.toContain('secret-client-repo') + expect(text).toContain('Prompt gate: open (1/20)') +}) +``` + +- [ ] **Step 3: Run, see them fail** + +Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test metrics.test.ts ; thirdPick: Rate + late: Rate; msP90: Rate; tool: Rate; missShare: Rate; sampled: number + noisy: string[]; promote: string[] +} + +const share = (k: number, n: number): Rate => n === 0 ? undefined : k / n +const isEntry = (r: Row) => + r.label === 'relevant' || r.label === 'irrelevant' +const under = (v: Rate, t: number) => v !== undefined && v < t +const pct = (v: Rate) => v === undefined ? 'n/a' : `${Math.round(v * 100)} %` +const day = (ms: number) => new Date(ms).toISOString().slice(0, 10) + +/** Rows of items with no `test` row anywhere (LRN-241). */ +function liveRows(rows: Row[]): Row[] { + const test = new Set(rows.filter(r => r.label === 'test').map(keyOf)) + return rows.filter(r => !test.has(keyOf(r))) +} + +/** relevant / (relevant + irrelevant); undefined with no entry. */ +export function precision(rows: Row[]): Rate { + const e = rows.filter(isEntry) + return share(e.filter(r => r.label === 'relevant').length, e.length) +} + +/** The 90th percentile, nearest rank; undefined with no value. */ +export function p90(values: number[]): Rate { + if (values.length === 0) return undefined + const s = [...values].sort((a, b) => a - b) + return s[Math.ceil(0.9 * s.length) - 1] +} + +function byTrigger(entries: Row[]): Record { + const out: Record = {} + for (const k of ['fresh', 'idle', 'keyword']) { + out[k] = precision(entries.filter(r => r.trigger === k)) + } + return out +} + +/** timeout + paused share among asks that had candidates. */ +function lateShare(asks: Row[]): Rate { + const had = asks.filter(r => (r.cands ?? 0) > 0) + const late = had.filter(r => + r.outcome === 'timeout' || r.outcome === 'paused') + return share(late.length, had.length) +} + +/** Share of sampled asks (a missed or checked row) with a miss. */ +function misses(rows: Row[]): { rate: Rate; sampled: number } { + const sampled = new Set(rows + .filter(r => r.label === 'missed' || r.label === 'checked').map(keyOf)) + const missed = new Set(rows.filter(r => r.label === 'missed').map(keyOf)) + return { rate: share(missed.size, sampled.size), sampled: sampled.size } +} + +/** IDs injected noisyHits times or more and never relevant. */ +function noisy(entries: Row[]): string[] { + return [...new Set(entries.map(r => r.id))].filter(id => { + const mine = entries.filter(r => r.id === id) + return mine.length >= T.noisyHits && + mine.every(r => r.label !== 'relevant') + }).sort() +} + +/** IDs relevant promoteHits times in promoteSessions sessions or more. */ +function promote(entries: Row[]): string[] { + const rel = entries.filter(r => r.label === 'relevant') + return [...new Set(rel.map(r => r.id))].filter(id => { + const mine = rel.filter(r => r.id === id) + return mine.length >= T.promoteHits && + new Set(mine.map(r => r.session)).size >= T.promoteSessions + }).sort() +} + +/** Distinct labelled items of a channel. */ +const itemsOf = (entries: Row[], ch: string) => + new Set(entries.filter(r => r.channel === ch).map(keyOf)).size + +/** Every 4d signal; the clock starts at the first row carrying cands. */ +export function metricsOf(all: Row[]): Metrics { + const rows = liveRows(all) + const entries = rows.filter(isEntry) + const prompt = entries.filter(r => r.channel === 'prompt') + const asks = rows.filter(r => r.channel === 'prompt' && r.label === 'ask') + const starts = all.filter(r => r.cands !== undefined).map(r => r.at) + const miss = misses(rows) + return { + start: starts.length === 0 ? undefined : Math.min(...starts), + promptN: itemsOf(entries, 'prompt'), toolN: itemsOf(entries, 'tool'), + lost: rows.filter(r => r.label === 'lost').length, + precision: precision(prompt), byTrigger: byTrigger(prompt), + thirdPick: precision(prompt.filter(r => r.pick === 3)), + late: lateShare(asks), msP90: p90(asks.flatMap(r => r.ms ?? [])), + tool: precision(entries.filter(r => r.channel === 'tool')), + missShare: miss.rate, sampled: miss.sampled, + noisy: noisy(entries), promote: promote(entries), + } +} + +function toolGate(n: number, now: number, cap: number, ext: boolean): + { tool: ToolGate; extendTool: boolean } { + if (n >= TOOL_GATE_N) return { tool: 'reached', extendTool: false } + if (now < cap) return { tool: 'open', extendTool: false } + if (!ext) return { tool: 'extended', extendTool: true } + if (now < cap + GATE_DAYS * DAY_MS) { + return { tool: 'extended', extendTool: false } + } + return { tool: 'insufficient', extendTool: false } +} + +/** Each channel's gate; the prompt gate never waits for the tool gate. */ +export function gateVerdict(m: Metrics, now: number, ext: boolean): Verdict { + if (m.start === undefined) { + return { prompt: 'open', tool: 'open', extendTool: false } + } + const cap = m.start + GATE_DAYS * DAY_MS + const prompt: Gate = + m.promptN >= PROMPT_GATE_N || now >= cap ? 'reached' : 'open' + return { prompt, ...toolGate(m.toolN, now, cap, ext) } +} + +/** The worst trigger under the floor; ties go idle, keyword, fresh. */ +function worstTrigger(by: Record): string | undefined { + const order = ['idle', 'keyword', 'fresh'] + const low = order.filter(k => under(by[k], T.triggerFloor)) + low.sort((a, b) => (by[a] ?? 0) - (by[b] ?? 0)) + return low[0] +} + +function promptActions(m: Metrics): string[] { + const out: string[] = [] + const worst = worstTrigger(m.byTrigger) + if (under(m.precision, T.promptKeep) && worst !== undefined) { + out.push(`prompt precision under ${pct(T.promptKeep)}: ` + + `tighten or drop the ${worst} trigger`) + } + if (under(m.thirdPick, T.thirdPick)) { + out.push(`3rd pick under ${pct(T.thirdPick)}: cap recall at 2 entries`) + } + if ((m.late ?? 0) > T.lateShare) { + out.push(`timeout + paused over ${pct(T.lateShare)}: ` + + 'revisit RERANK_MS or skip the rerank on short lists') + } + if (under(m.msP90, T.msP90)) { + out.push(`rerank p90 under ${T.msP90} ms: lower RERANK_MS`) + } + if ((m.missShare ?? 0) >= T.missShare) { + out.push('misses in 1 ask in 3 or more: diagnose by cause') + } + return out +} + +/** The threshold actions, once at least one gate is reached. */ +export function actionsOf(m: Metrics, v: Verdict): string[] { + if (v.prompt !== 'reached' && v.tool !== 'reached') return [] + const out = v.prompt === 'reached' ? promptActions(m) : [] + if (v.tool === 'reached' && under(m.tool, T.toolKeep)) { + out.push(`tool precision under ${pct(T.toolKeep)}: ` + + 'require 2 matched 4-grams, or exclude the entry') + } + out.push(...m.noisy.map(id => `noisy: /prune-memory rewrites ${id}`)) + out.push(...m.promote.map(id => `promotion candidate (BDR-100): ${id}`)) + return out +} + +function toolLine(m: Metrics, v: Verdict): string { + const n = `${m.toolN}/${TOOL_GATE_N}` + return v.tool === 'insufficient' + ? `Tool gate: insufficient sample n=${m.toolN}, closed` + : `Tool gate: ${v.tool} (${n})` +} + +/** FEEDBACK.md: counts and shares, IDs only in the last three lists. */ +export function renderFeedback(m: Metrics, v: Verdict, now: number) { + const acts = actionsOf(m, v) + return [ + `# Memory feedback (${day(now)})`, '', + `Gate clock start: ${m.start === undefined ? 'not started' : day(m.start)}`, + `Prompt gate: ${v.prompt} (${m.promptN}/${PROMPT_GATE_N})`, + toolLine(m, v), `Lost records: ${m.lost}`, '', + '| Signal | Value |', '|---|---|', + `| prompt precision | ${pct(m.precision)} |`, + ...Object.entries(m.byTrigger).map(([k, x]) => + `| precision, ${k} | ${pct(x)} |`), + `| 3rd pick precision | ${pct(m.thirdPick)} |`, + `| timeout + paused | ${pct(m.late)} |`, + `| rerank ms p90 | ${m.msP90 ?? 'n/a'} |`, + `| tool precision | ${pct(m.tool)} |`, + `| asks with a miss | ${pct(m.missShare)} of ${m.sampled} |`, '', + `Noisy: ${m.noisy.join(', ') || 'none'}`, + `Promotion candidates: ${m.promote.join(', ') || 'none'}`, '', + 'Actions:', ...(acts.length ? acts.map(a => `- ${a}`) : ['- none']), + ].join('\n') + '\n' +} +``` + +- [ ] **Step 5: Run, see them pass** + +Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test labels.test.ts metrics.test.ts new Date(T0 + ms).toISOString() +const user = (ms: number, text: string): Rec => ({ type: 'user', + timestamp: iso(ms), origin: { kind: 'human' }, message: { content: text } }) +const said = (ms: number, id: string, text: string): Rec => ({ + type: 'assistant', timestamp: iso(ms), + message: { id, content: [{ type: 'text', text }] } }) +const hook = (ms: number, name: string, lines: string[], use?: string) => ({ + type: 'attachment', timestamp: iso(ms), attachment: { + type: 'hook_additional_context', hookName: name, content: lines, + ...(use ? { toolUseID: use } : {}) } }) +const rec = (over: Partial): LogRecord => ({ at: T0, session: 's', + agent: 'main', channel: 'prompt', ids: ['BDR-001'], trigger: 'fresh', + outcome: 'injected', ...over }) + +test('X1: slug, files and a torn line', () => { + expect(slugOf('/home/a.b/c_d')).toBe('-home-a-b-c-d') + const p = fs.mkdtempSync(path.join(os.tmpdir(), 'mf-')) + const dir = path.join(p, slugOf('/r')) + fs.mkdirSync(path.join(dir, 's', 'subagents'), { recursive: true }) + fs.writeFileSync(path.join(dir, 's.jsonl'), '{"a":1}\n{"torn') + fs.writeFileSync(path.join(dir, 's', 'subagents', 'x.jsonl'), '{"b":2}\n') + const files = sessionFiles(p, '/r', 's') + expect(files.map(f => path.basename(f))).toEqual(['s.jsonl', 'x.jsonl']) + expect(records(files[0]!)).toEqual([{ a: 1 }]) + expect(sessionFiles(p, '/r', 'gone')).toEqual([]) +}) + +test('X2: composer prompts only', () => { + expect(composerText(user(0, 'fix the gate'))).toBe('fix the gate') + expect(composerText(user(0, '/close'))).toBeUndefined() + expect(composerText(user(0, 'x'))).toBeUndefined() + expect(composerText({ ...user(0, 'x'), origin: { kind: 'sdk' } })) + .toBeUndefined() + expect(composerText({ ...user(0, 'x'), isSidechain: true })) + .toBeUndefined() +}) + +test('X3: an injected ask joins prompt, block and deduped reply', () => { + const recs = [ + user(-100, 'design the gate'), + hook(800, 'prompt.submit', ['memory: note', 'memory recall:\nBDR-001 x']), + said(2000, 'm1', 'see BDR-001'), said(2000, 'm1', 'see BDR-001'), + said(2100, 'm1', 'done'), + user(9000, 'next ask'), said(9500, 'm2', 'later'), + ] + expect(joinPrompt(recs, rec({}))).toEqual({ prompt: 'design the gate', + injected: 'memory recall:\nBDR-001 x', reply: 'see BDR-001\ndone' }) +}) + +const JOIN_LATE = 10001 // one ms past JOIN_MS + +test('X4: a non-injected ask joins its prompt; no turn is lost', () => { + const recs = [user(-3000, 'plan the reader')] + expect(joinPrompt(recs, rec({ outcome: 'timeout', ids: [] }))) + .toEqual({ prompt: 'plan the reader', injected: '', reply: '' }) + expect(joinPrompt([], rec({ outcome: 'none', ids: [] }))).toBeUndefined() + const late = [user(0, 'x'), hook(JOIN_LATE, 'prompt.submit', + ['memory recall:\nBDR-001 x'])] + expect(joinPrompt(late, rec({}))).toBeUndefined() +}) + +test('X5: a tool record joins command and the next results', () => { + const recs: Rec[] = [ + { type: 'assistant', timestamp: iso(0), message: { id: 'a', content: [ + { type: 'tool_use', id: 'u1', name: 'Bash', + input: { command: 'make test' } }] } }, + { type: 'user', timestamp: iso(5), message: { content: [ + { type: 'tool_result', tool_use_id: 'u1', is_error: true }] } }, + hook(10, 'tool.call', ['memory: known BLK-003 — x'], 'u1'), + { type: 'assistant', timestamp: iso(20), message: { id: 'b', content: [ + { type: 'tool_use', id: 'u2', name: 'Bash', input: {} }] } }, + { type: 'user', timestamp: iso(25), message: { content: [ + { type: 'tool_result', tool_use_id: 'u2', is_error: false }] } }, + ] + const r = rec({ channel: 'tool', ids: ['BLK-003'], trigger: 'tool' }) + expect(joinTool(recs, r)).toEqual({ command: 'make test', + known: 'memory: known BLK-003 — x', next: [false] }) + expect(joinTool(recs, { ...r, ids: ['BLK-009'] })).toBeUndefined() +}) +``` + +- [ ] **Step 2: Run, see them fail** + +Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test transcripts.test.ts +export const JOIN_MS = 10000 // attachment written at most this after `at` +const NEAR_MS = 60000 // composer prompt at most this far from the ask +const WRAPPERS = [' root.replace(/[^A-Za-z0-9]/g, '-') + +/** The main transcript of a session, then its sub-agent ones. */ +export function sessionFiles( + projects: string, root: string, session: string, +): string[] { + const dir = path.join(projects, slugOf(root)) + const sub = path.join(dir, session, 'subagents') + const subs = fs.existsSync(sub) + ? fs.readdirSync(sub).filter(f => f.endsWith('.jsonl')).sort() + .map(f => path.join(sub, f)) + : [] + return [path.join(dir, `${session}.jsonl`), ...subs] + .filter(f => fs.existsSync(f)) +} + +/** Parsed lines of a transcript; a torn or partial line is skipped. */ +export function records(file: string): Rec[] { + const out: Rec[] = [] + for (const raw of fs.readFileSync(file, 'utf8').split('\n')) { + if (raw.trim() === '') continue + try { out.push(JSON.parse(raw)) } catch { /* torn line */ } + } + return out +} + +const tsOf = (r: Rec) => Date.parse(String(r.timestamp ?? '')) + +function blocksOf(r: Rec): Rec[] { + const c = r.message?.content + return Array.isArray(c) ? c.filter(b => b && typeof b === 'object') : [] +} + +/** A typed composer prompt (recall-bench's rule); else undefined. */ +export function composerText(r: Rec): string | undefined { + if (r.type !== 'user' || r.isSidechain === true) return undefined + if (r.isMeta === true || r.isCompactSummary === true) return undefined + const c = r.message?.content + if (typeof c !== 'string' || r.origin?.kind !== 'human') return undefined + const t = c.trim() + if (t.startsWith('/')) return undefined + return WRAPPERS.some(w => t.startsWith(w)) ? undefined : t +} + +/** The joined text of a `hook` context attachment; else undefined. */ +function hookText(r: Rec, hook: string): string | undefined { + const a = r.attachment + if (r.type !== 'attachment' || a?.type !== 'hook_additional_context') { + return undefined + } + if (a.hookName !== hook) return undefined + return Array.isArray(a.content) ? a.content.map(String).join('\n') + : String(a.content ?? '') +} + +/** The earliest `hook` attachment holding `needle` in [at, at+JOIN_MS]. */ +function attachmentAfter( + recs: Rec[], at: number, hook: string, needle: string, +): Rec | undefined { + let best: Rec | undefined + for (const r of recs) { + const t = tsOf(r) + if (!(t >= at && t <= at + JOIN_MS)) continue + if (!hookText(r, hook)?.includes(needle)) continue + if (best === undefined || t < tsOf(best)) best = r + } + return best +} + +type Prompt = { t: number; text: string } + +/** The composer prompt nearest `ts`, within NEAR_MS. */ +function promptNear(recs: Rec[], ts: number): Prompt | undefined { + let best: Prompt | undefined + for (const r of recs) { + const text = composerText(r) + const t = tsOf(r) + if (text === undefined || Math.abs(t - ts) > NEAR_MS) continue + if (best === undefined || Math.abs(t - ts) < Math.abs(best.t - ts)) { + best = { t, text } + } + } + return best +} + +/** The first composer prompt after `ts`; Infinity when none. */ +function nextPromptAt(recs: Rec[], ts: number): number { + let end = Infinity + for (const r of recs) { + const t = tsOf(r) + if (t > ts && t < end && composerText(r) !== undefined) end = t + } + return end +} + +/** Main-thread assistant text after `ts` up to the next prompt. */ +function replyAfter(recs: Rec[], ts: number): string { + const end = nextPromptAt(recs, ts) + const seen = new Set() + const parts: string[] = [] + for (const r of recs) { + const t = tsOf(r) + if (r.type !== 'assistant' || r.isSidechain === true) continue + if (!(t > ts && t < end)) continue + for (const b of blocksOf(r).filter(b => b.type === 'text')) { + const key = `${r.message?.id}:${b.text}` // EVAL-037 + if (!seen.has(key)) parts.push(String(b.text)) + seen.add(key) + } + } + return parts.join('\n') +} + +export type PromptJoin = { prompt: string; injected: string; reply: string } + +/** The turn a prompt-channel record served; undefined when lost. */ +export function joinPrompt( + recs: Rec[], rec: LogRecord, +): PromptJoin | undefined { + if (rec.outcome !== 'injected') { + const p = promptNear(recs, rec.at) + return p && { prompt: p.text, injected: '', reply: '' } + } + const att = attachmentAfter(recs, rec.at, 'prompt.submit', RECALL_HEAD) + if (att === undefined) return undefined + const t = tsOf(att) + const p = promptNear(recs, t) + const text = hookText(att, 'prompt.submit') ?? '' + return { prompt: p?.text ?? '', + injected: text.slice(text.indexOf(RECALL_HEAD)), + reply: replyAfter(recs, Math.max(t, p?.t ?? t)) } +} + +/** The command of a Bash tool_use. */ +function bashCommand(recs: Rec[], useId: string): string { + for (const b of recs.flatMap(blocksOf)) { + if (b.type === 'tool_use' && b.id === useId) { + return String(b.input?.command ?? '') + } + } + return '' +} + +/** Whether each of the next 3 Bash results after `useId` failed. */ +function nextResults(recs: Rec[], useId: string): boolean[] { + const bash = new Set() + const out: boolean[] = [] + let after = false + for (const b of recs.flatMap(blocksOf)) { + if (b.type === 'tool_use' && b.name === 'Bash') bash.add(b.id) + if (b.type !== 'tool_result' || !bash.has(b.tool_use_id)) continue + if (after && out.length < 3) out.push(b.is_error === true) + if (b.tool_use_id === useId) after = true + } + return out +} + +export type ToolJoin = { command: string; known: string; next: boolean[] } + +/** The failure a known line answered and the next 3 Bash results. */ +export function joinTool(recs: Rec[], rec: LogRecord): ToolJoin | undefined { + const needle = KNOWN_HEAD + (rec.ids[0] ?? '') + const att = attachmentAfter(recs, rec.at, 'tool.call', needle) + const useId = att?.attachment?.toolUseID + if (att === undefined || typeof useId !== 'string') return undefined + return { command: bashCommand(recs, useId), + known: hookText(att, 'tool.call') ?? '', + next: nextResults(recs, useId) } +} +``` + +- [ ] **Step 4: Run, see them pass** + +Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test ): Logged[]`; + `worksheet(dir: string, p: Paths): number`; `apply(dir: string, p: Paths, + now: number): string`; CLI `bun reader.ts worksheet|apply ` + (rc 0 ok, 1 error on stderr, 2 usage). + +- [ ] **Step 1: Write the failing tests** + +`lib/memory-feedback/reader.test.ts`: + +```ts +import { expect, test } from 'bun:test' +import * as fs from 'node:fs' +import * as os from 'node:os' +import * as path from 'node:path' +import { slugOf } from './transcripts.ts' +import { + apply, population, readLogs, storeFile, worksheet, type Paths, +} from './reader.ts' + +const T0 = Date.parse('2026-11-10T10:00:00.000Z') +const iso = (ms: number) => new Date(T0 + ms).toISOString() + +/** A fixture home: store, one transcript, an empty out dir, a work dir. */ +function fixture(store: unknown): { p: Paths; work: string } { + const home = fs.mkdtempSync(path.join(os.tmpdir(), 'mf-')) + const p = { store: path.join(home, 'store'), + projects: path.join(home, 'projects'), out: path.join(home, 'mf') } + fs.mkdirSync(p.store) + fs.writeFileSync(path.join(p.store, 'memory_skills-dir-abc.json'), + typeof store === 'string' ? store : JSON.stringify(store)) + const dir = path.join(p.projects, slugOf('/repo/x')) + fs.mkdirSync(dir, { recursive: true }) + fs.writeFileSync(path.join(dir, 's1.jsonl'), [ + { type: 'user', timestamp: iso(-50), origin: { kind: 'human' }, + message: { content: 'design the reader gate' } }, + { type: 'attachment', timestamp: iso(400), attachment: { + type: 'hook_additional_context', hookName: 'prompt.submit', + content: ['memory recall:\nBDR-001 x'] } }, + { type: 'assistant', timestamp: iso(900), + message: { id: 'm', content: [{ type: 'text', text: 'per BDR-001' }] } }, + ].map(r => JSON.stringify(r)).join('\n') + '\n') + const work = fs.mkdtempSync(path.join(os.tmpdir(), 'mf-work-')) + return { p, work } +} +const REC = (at: number, over = {}) => ({ at, session: 's1', agent: 'main', + channel: 'prompt', ids: ['BDR-001'], trigger: 'fresh', + outcome: 'injected', cands: 5, ms: 800, ...over }) +const STORE = { 'l:/repo/x': [ + REC(T0 - 9e6, { cands: undefined }), // before 4d.1: out of population + REC(T0), REC(T0 + 60000, { session: 'gone' }), // second one lost +], 'p:/repo/x': { sessions: [] } } + +test('R1: no store file, or a torn one, fails loudly', () => { + const { p } = fixture('{"l:/x": [tor') + expect(() => readLogs(storeFile(p.store))).toThrow('unreadable') + expect(() => storeFile(path.join(p.store, 'none'))).toThrow('no memory') +}) + +test('R2: population starts at the first cands record, skips labelled', () => { + const { p } = fixture(STORE) + const logs = readLogs(storeFile(p.store)) + expect(population(logs, new Set()).map(l => l.rec.at)) + .toEqual([T0, T0 + 60000]) + expect(population(logs, new Set([`s1|${T0}|prompt`]))).toHaveLength(1) + expect(population(logs.slice(0, 1), new Set())).toEqual([]) +}) + +test('R3: worksheet writes 0600 text files and a lost item', () => { + const { p, work } = fixture(STORE) + expect(worksheet(work, p)).toBe(2) + const sheet = fs.readFileSync(path.join(work, 'worksheet.md'), 'utf8') + expect(sheet).toContain('design the reader gate') + expect(sheet).toContain('LOST') + for (const f of ['worksheet.md', 'items.json']) { + expect(fs.statSync(path.join(work, f)).mode & 0o777).toBe(0o600) + } + const items = JSON.parse(fs.readFileSync( + path.join(work, 'items.json'), 'utf8')) + expect(items.map((i: { cited: string[]; lost: boolean }) => + [i.cited, i.lost])).toEqual([[['BDR-001'], false], [[], true]]) +}) + +test('R4: a bad draft refuses the whole apply, nothing written', () => { + const { p, work } = fixture(STORE) + worksheet(work, p) + fs.writeFileSync(path.join(work, 'labels-draft.tsv'), '1\tBDR-001\tmaybe') + expect(() => apply(work, p, T0)).toThrow('draft refused') + expect(fs.existsSync(p.out)).toBe(false) +}) + +test('R5: apply writes labels, state, FEEDBACK; a second apply refuses', + () => { + const { p, work } = fixture(STORE) + worksheet(work, p) + fs.writeFileSync(path.join(work, 'labels-draft.tsv'), + '1\tBDR-001\trelevant\n') + const fb = apply(work, p, T0 + 1) + expect(fb).toContain('Prompt gate: open (1/20)') + expect(fb).toContain('Lost records: 1') + expect(fb).not.toContain('design the reader gate') + const labels = fs.readFileSync(path.join(p.out, 'labels.tsv'), 'utf8') + expect(labels).not.toContain('design the reader gate') + expect(labels).not.toContain('/repo/x') + expect(JSON.parse(fs.readFileSync(path.join(p.out, 'state.json'), + 'utf8')).lastRun).toBe(T0 + 1) + expect(() => apply(work, p, T0 + 2)).toThrow('already applied') + expect(worksheet(work, p)).toBe(0) + }) + +test('R6: a first run with nothing to label records the run only', () => { + const { p, work } = fixture({ 'l:/repo/x': [REC(T0, { cands: undefined })] }) + expect(worksheet(work, p)).toBe(0) + expect(fs.readFileSync(path.join(work, 'worksheet.md'), 'utf8')) + .toContain('No unlabelled record') + expect(apply(work, p, T0)).toContain('Gate clock start: not started') + expect(fs.existsSync(path.join(p.out, 'state.json'))).toBe(true) +}) +``` + +- [ ] **Step 2: Write the suite entry `lib/tests/memory-feedback.test.sh`** + +```bash +#!/usr/bin/env bash +# lib/tests/memory-feedback.test.sh — the bun unit suite of +# lib/memory-feedback (phase 4d reader, labels, metrics, transcripts). +# Fails when bun is absent: it never passes vacuously. +set -u +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +command -v bun >/dev/null 2>&1 || { echo "FAIL bun not on PATH"; exit 1; } +cd "$ROOT/lib/memory-feedback" || exit 1 +bun test 2>&1 +``` + +Then `chmod +x lib/tests/memory-feedback.test.sh`. + +- [ ] **Step 3: Run, see the reader tests fail** + +Run: `systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bash lib/tests/memory-feedback.test.sh ` joins +// the unlabelled injection-log records to their transcripts; `apply ` +// validates the labeller's draft, appends labels.tsv and regenerates +// FEEDBACK.md. Text only ever lands in , the session scratchpad. +import * as fs from 'node:fs' +import * as os from 'node:os' +import * as path from 'node:path' +import { + LOG_PREFIX, isLogRecord, type LogRecord, +} from '../../mods/memory/hooks/register.ts' +import { + keyOf, parseDraft, readRows, readState, rowsOf, writeRows, writeSafe, + writeState, type Item, +} from './labels.ts' +import { gateVerdict, metricsOf, renderFeedback } from './metrics.ts' +import { + joinPrompt, joinTool, records, sessionFiles, type Rec, +} from './transcripts.ts' + +export type Paths = { store: string; projects: string; out: string } +export type Logged = { root: string; rec: LogRecord } +type Base = Omit +type Joined = { item: Item; text: string } + +const PROMPT_MAX = 600 +const REPLY_MAX = 800 +const DRAFT_HELP = [ + 'Write labels-draft.tsv next to this file, one TAB-separated line each:', + '`