diff --git a/docs/superpowers/plans/2026-11-06-memory-feedback.md b/docs/superpowers/plans/2026-11-06-memory-feedback.md index 310c8e3..848d806 100644 --- a/docs/superpowers/plans/2026-11-06-memory-feedback.md +++ b/docs/superpowers/plans/2026-11-06-memory-feedback.md @@ -1,4 +1,4 @@ -# Memory Feedback (phase 4d) Implementation Plan +# Memory Feedback (phase 4d) Implementation Plan — r2 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. @@ -8,33 +8,88 @@ local labels, a `/memory-feedback` skill, a session-start reminder, and a fixed-threshold gate. **Architecture:** The mod (`mods/memory/hooks/register.ts`) logs `cands` -and `ms` on prompt records. A bun reader in `lib/memory-feedback/` (four -focused modules: thresholds, labels, transcripts, metrics, plus the CLI) -imports the mod's own log validator, joins records to transcript -attachments, writes a text worksheet only to the scratchpad, and keeps IDs -and enums in `~/.claude/memory-feedback/` (0600). A shell lib prints the -reminder line in the banner. The skill drives labelling. +and `ms` on prompt records and exports its log validator. A bun reader in +`lib/memory-feedback/` (labels, metrics, transcripts, CLI) joins records to +transcript attachments, redacts and writes a text worksheet only to the +scratchpad, and keeps IDs and enums in `~/.claude/memory-feedback/` (0600, +append-only labels). A plain shell script prints the reminder line. The +skill drives labelling and records a gate verdict once, in this config repo. -**Tech Stack:** TypeScript on bun (`bun test`, `node:fs`), Claude Code mod -kit (`claude-code/testing`), bash 3.2-safe shell, jq-free banner check. +**Tech Stack:** TypeScript on bun (`bun test`, `node:fs`, `node:crypto`, +`node:child_process`), Claude Code mod kit (`claude-code/testing`), bash +3.2-safe shell. **Spec:** `docs/superpowers/specs/2026-11-06-memory-feedback-design.md` **Contract:** `.claude/tasks/contracts/2026-11-06-memory-feedback-4d-1825.md` +## r2 changes (3-lens challenge, 2026-11-06) + +correctness FATAL(7: 1 BLOCKER), robustness CONCERNS(6 MAJOR), simplicity +CONCERNS(1 MAJOR). Named changes: + +- C1 BLOCKER: a real `tool.call` attachment carries `toolUseID` = + `-context`; `joinTool` strips the suffix; fixture X5 uses it. +- C2: tool walks read `assistant`/`user` records only and dedupe + `tool_result` by id (api-request-blob replays); X5 carries a replay. +- C3: M6 helper `.flat()`. C4: `oneAsk` typed `$: Engine`. +- C5/R6: no `stat -c` (portability census): age via `find -mtime`; census + run in Task 6. +- C6/R5/R14: the skill resolves bun (`command -v bun || ~/.bun/bin/bun`), + runs it from the reader's dir, stops on any non-zero rc. +- C7: STEP 6 writes only into this config repo (`-C "$CFG"`); actions on + another repo's entries are shown, never written. +- R1: prompt, reply, command and known line pass the mod's `redact` + before the cut; a planted secret test. +- R2: noisy and promotion are keyed by repo tag + ID; the tag is + `basename#sha1(root)[0:6]` (same-basename repos stay apart). +- R3: `labels.tsv` is append-only (`appendFileSync`), backed up to + `labels.tsv.bak` first; an unreadable line fails loudly, never dropped. +- R4/C14: `state.json` records when each gate verdict was first reached; + `apply` prints `NEW: …` only on that transition; STEP 6 acts only on NEW; + the reminder stops once both gates are recorded (`"closed":true`). +- S1 (partial): the tool extension is a pure function of time (no + `toolExtended` flag); `state.json` stays for the verdict record and as + the run marker the reminder reads. +- R7: `ms` clamped to a non-negative integer. R8: transcripts cached per + session; sub-agent files loaded only for sub-agent tool records. +- R9: records younger than 30 min are left for the next run. +- R10: a root at the log cap whose oldest record is newer than the last + run is reported as a possible eviction. C10: torn transcript lines + counted in the worksheet header. +- R11/S7: the reminder is a plain script. R12: silent with + `"recall": false`. R13: the CLI refuses a dir inside a git work tree. +- R15: the skill is user-invoked. R16: a corrupt `state.json` fails + loudly; state is written before the labels. R17: `sample` subcommand + replaces `shuf`; test temp dirs removed. +- S2/C11: `test` only as ` - test`. S3: one `clockStart` rule. + S4: `cited` stays in the worksheet only. S5/C12: miss causes are what + the worksheet supports: `no-overlap` or `overlap`. S6: `worksheet` + writes a skeleton draft. C13: the empty run skips the spot check. + C15: lost rows count in the late share and the ms p90. C17: unreadable + state case. C18: jq on the newest record. C19: suites through + `make test suite=…`. C8/C9: RED expectations corrected. +- Deviations from the approved spec, surfaced at the gate: reader in + `lib/memory-feedback/` (not `mods/memory/feedback/`: `claude plugin + validate --strict` and `plugin test` must not see non-mod files); miss + causes reduced to `no-overlap | overlap`; `labels.tsv` drops `cited`; + noisy and promotion lines carry a repo tag. + ## Global Constraints - No prompt or reply text in the plugin store, `labels.tsv`, `state.json`, - `FEEDBACK.md` or any committed file; text only in `/worksheet.md` and - `/items.json` where `` is the session scratchpad (LRN-124, - LRN-261). -- `~/.claude/memory-feedback/` created 0700, every file in it 0600, writes - through a temp file and a rename. + `FEEDBACK.md` or any committed file; text only in `/worksheet.md` + and `/items.json`, `` being the session scratchpad, and only + after the mod's `redact` (LRN-124, LRN-261, BDR-057). +- `~/.claude/memory-feedback/` created 0700, every file in it 0600. + `labels.tsv` is append-only; `state.json` and `FEEDBACK.md` go through a + temp file and a rename. - Our code: at most 25 logic lines per function, 80 columns, 5 params, no global mutable state. - No new dependency: bun built-ins and `node:*` only. -- Shell is bash 3.2-safe (no `mapfile`, `declare -A`, `local -n`). -- Every suite that runs the hook, the mod kit or bun runs contained: - `systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" `, contained: + `systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" make test suite= { expect(ms).toBe(0) }) -test('LF2: no candidate and a pause log no ms', async ($, on) => { +test('LF2: no candidate logs cands 0 and no ms', async ($, on) => { const { w } = await bootRecall($, on) w.proc = { stdout: rowsOut([['BLK-001', 'x'], ['LRN-002', 'y']]) .replace(/open/g, 'superseded') } @@ -156,16 +207,16 @@ test('LF4: cands and ms are optional counts', async ($, on) => { }) ``` -- [ ] **Step 2: Run the suite, see the four new tests fail** +- [ ] **Step 2: Run the suite, see it fail** Run: `systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" make test suite=lib/tests/mods.test.sh /dev/null 2>&1 || { echo "FAIL bun not on PATH"; exit 1; } +cd "$ROOT/lib/memory-feedback" || exit 1 +bun test 2>&1 +``` + +Then `mkdir -p lib/memory-feedback && chmod +x lib/tests/memory-feedback.test.sh`. + +- [ ] **Step 2: Write the failing tests** `lib/memory-feedback/labels.test.ts`: ```ts -import { expect, test } from 'bun:test' +import { afterAll, expect, test } from 'bun:test' import * as fs from 'node:fs' import * as os from 'node:os' import * as path from 'node:path' import { - fromLine, parseDraft, readRows, readState, rowsOf, toLine, writeRows, - writeState, type Item, type Row, + appendRows, fromLine, parseDraft, readRows, readState, rowsOf, skeleton, + tagOf, toLine, writeState, type Item, type Row, } from './labels.ts' +const dirs: string[] = [] +const tmp = () => { + const d = fs.mkdtempSync(path.join(os.tmpdir(), 'mf-')) + dirs.push(d) + return d +} +afterAll(() => dirs.forEach(d => fs.rmSync(d, { recursive: true }))) + const item = (n: number, over: Partial = {}): Item => ({ - n, root: 'r', session: `s${n}`, at: 1000 + n, channel: 'prompt', + n, root: 'r#000000', session: `s${n}`, at: 1000 + n, channel: 'prompt', trigger: 'fresh', outcome: 'injected', cands: 4, ms: 900, ids: ['BDR-001', 'LRN-002'], cited: ['BDR-001'], lost: false, ...over, }) const ITEMS = [item(1), item(2, { ids: [], outcome: 'none' })] -const tmp = () => fs.mkdtempSync(path.join(os.tmpdir(), 'mf-')) +const OK = '1\tBDR-001\trelevant\n1\tLRN-002\tirrelevant' test('L1: a row survives the TSV round trip', () => { - const r: Row = { at: 5, root: 'r', session: 's', channel: 'tool', - trigger: 'tool', outcome: 'injected', cands: undefined, ms: undefined, - pick: 1, id: 'BLK-003', cited: true, label: 'relevant', cause: '' } + const r: Row = { at: 5, root: 'r#000000', session: 's', + channel: 'tool', trigger: 'tool', outcome: 'injected', + cands: undefined, ms: undefined, pick: 1, id: 'BLK-003', + label: 'relevant', cause: '' } expect(fromLine(toLine(r))).toEqual(r) expect(fromLine('at\troot')).toBeUndefined() }) test('L2: a complete draft parses', () => { - const d = parseDraft([ - '1\tBDR-001\trelevant', '1\tLRN-002\tirrelevant', - '2\tBDR-009\tmissed\tno-overlap', '1\t-\tchecked', '# note', - ].join('\n'), ITEMS) + const d = parseDraft([OK, '2\tBDR-009\tmissed\tno-overlap', + '1\t-\tchecked', '# note'].join('\n'), ITEMS) expect(Array.isArray(d) && d.length).toBe(4) }) test('L3: every bad draft is refused with a reason', () => { const bad = [ - '1\tBDR-001\tmaybe', // unknown label - '9\tBDR-001\trelevant', // unknown item - '1\tBLK-777\trelevant\n1\tBDR-001\trelevant\n1\tLRN-002\trelevant', - '1\tBDR-001\trelevant', // LRN-002 unlabelled - '1\tBDR-001\trelevant\n1\tBDR-001\trelevant\n1\tLRN-002\trelevant', - '1\tBDR-001\trelevant\n1\tLRN-002\trelevant\n2\tBDR-001\tmissed\tbad', - '1\tBDR-001\trelevant\n1\tLRN-002\tmissed\tdeclined', + '1\tBDR-001\tmaybe\n1\tLRN-002\trelevant', // unknown label + `${OK}\n9\tBDR-001\trelevant`, // unknown item + `${OK}\n1\tBLK-777\trelevant`, // not injected there + '1\tBDR-001\trelevant', // LRN-002 unlabelled + `${OK}\n1\tBDR-001\trelevant`, // labelled twice + `${OK}\n2\tBDR-001\tmissed\tbad`, // unknown cause + `${OK}\n1\tLRN-002\tmissed\toverlap`, // missed but injected + '1\tBDR-001\ttest\n1\tLRN-002\trelevant', // per-id test + '1\tBDR-001\t\n1\tLRN-002\trelevant', // blank skeleton label ] - for (const text of bad) expect(typeof parseDraft(text, ITEMS)).toBe('string') + for (const text of bad) { + expect(typeof parseDraft(text, ITEMS)).toBe('string') + } }) -test('L4: test - marks a whole item, no entry label needed', () => { +test('L4: `- test` marks a whole item, no entry label needed', () => { expect(Array.isArray(parseDraft('1\t-\ttest', ITEMS))).toBe(true) }) @@ -343,43 +426,62 @@ test('L5: a lost item takes no label', () => { }) test('L6: rowsOf writes an ask row then the entry rows', () => { - const d = parseDraft('1\tBDR-001\trelevant\n1\tLRN-002\tirrelevant', - [item(1)]) + const d = parseDraft(OK, [item(1)]) if (typeof d === 'string') throw new Error(d) - const rows = rowsOf(item(1), d) - expect(rows.map(r => [r.label, r.id, r.pick, r.cited])).toEqual([ - ['ask', '', undefined, false], - ['relevant', 'BDR-001', 1, true], - ['irrelevant', 'LRN-002', 2, false], + expect(rowsOf(item(1), d).map(r => [r.label, r.id, r.pick])).toEqual([ + ['ask', '', undefined], ['relevant', 'BDR-001', 1], + ['irrelevant', 'LRN-002', 2], ]) expect(rowsOf(item(3, { lost: true }), [])[0]?.label).toBe('lost') }) -test('L7: files are 0600 in a 0700 dir, state defaults', () => { +test('L7: append-only labels, a backup, a loud damaged line', () => { const dir = path.join(tmp(), 'mf') - writeRows(path.join(dir, 'labels.tsv'), rowsOf(item(1), [])) + const file = path.join(dir, 'labels.tsv') + appendRows(file, rowsOf(item(1), [])) + appendRows(file, rowsOf(item(2), [])) expect(fs.statSync(dir).mode & 0o777).toBe(0o700) - expect(fs.statSync(path.join(dir, 'labels.tsv')).mode & 0o777) - .toBe(0o600) - expect(readRows(path.join(dir, 'labels.tsv'))).toHaveLength(1) - expect(readState(path.join(dir, 'state.json'))) - .toEqual({ lastRun: 0, toolExtended: false }) - writeState(path.join(dir, 'state.json'), { lastRun: 9, toolExtended: true }) - expect(readState(path.join(dir, 'state.json'))) - .toEqual({ lastRun: 9, toolExtended: true }) + expect(fs.statSync(file).mode & 0o777).toBe(0o600) + expect(readRows(file)).toHaveLength(2) + expect(readRows(`${file}.bak`)).toHaveLength(1) + fs.appendFileSync(file, 'broken line\n') + expect(() => readRows(file)).toThrow('labels.tsv.bak') +}) + +test('L8: state defaults, round trip, a corrupt file throws', () => { + const f = path.join(tmp(), 'state.json') + expect(readState(f)).toEqual({ lastRun: 0, closed: false }) + writeState(f, { lastRun: 9, promptAt: 9, closed: false }) + expect(readState(f)).toEqual({ lastRun: 9, promptAt: 9, closed: false }) + fs.writeFileSync(f, '{oops') + expect(() => readState(f)).toThrow('state.json') +}) + +test('L9: repo tags keep same-named repos apart', () => { + expect(tagOf('/a/x')).toMatch(/^x#[0-9a-f]{6}$/) + expect(tagOf('/a/x')).toBe(tagOf('/a/x')) + expect(tagOf('/a/x')).not.toBe(tagOf('/b/x')) +}) + +test('L10: the skeleton draft lists each injected id once', () => { + const lines = skeleton([item(1), item(2, { ids: [] }), + item(3, { lost: true })]).split('\n').filter(l => !l.startsWith('#')) + expect(lines.filter(l => l !== '')).toEqual(['1\tBDR-001\t', '1\tLRN-002\t']) }) ``` -- [ ] **Step 2: Run, see them fail** +- [ ] **Step 3: Run, see them fail** -Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test labels.test.ts `${path.basename(root)}#` + + createHash('sha1').update(root).digest('hex').slice(0, 6) /** One record's identity: session ids are unique across repos. */ export const keyOf = (r: { session: string; at: number; channel: string }) => @@ -418,8 +528,8 @@ const opt = (s: string | undefined) => export function toLine(r: Row): string { return [r.at, r.root, r.session, r.channel, r.trigger, r.outcome, - num(r.cands), num(r.ms), num(r.pick), r.id, r.cited ? '1' : '0', - r.label, r.cause].join('\t') + num(r.cands), num(r.ms), num(r.pick), r.id, r.label, r.cause] + .join('\t') } /** A labels.tsv line as a row; the header or a short line: undefined. */ @@ -430,36 +540,52 @@ export function fromLine(line: string): Row | undefined { at: Number(f[0]), root: f[1] ?? '', session: f[2] ?? '', channel: f[3] as Channel, trigger: f[4] ?? '', outcome: f[5] ?? '', cands: opt(f[6]), ms: opt(f[7]), pick: opt(f[8]), id: f[9] ?? '', - cited: f[10] === '1', label: f[11] as Label, cause: f[12] ?? '', + label: f[10] as Label, cause: f[11] ?? '', } } +const mkdir = (file: string) => + fs.mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 }) + /** Writes through a 0600 temp file and a rename; the dir is made 0700. */ export function writeSafe(file: string, text: string): void { - fs.mkdirSync(path.dirname(file), { recursive: true, mode: 0o700 }) + mkdir(file) const tmp = `${file}.${process.pid}.tmp` fs.writeFileSync(tmp, text, { mode: 0o600 }) fs.renameSync(tmp, file) } +/** Every row; a damaged line fails loudly, it is never dropped. */ export function readRows(file: string): Row[] { if (!fs.existsSync(file)) return [] - return fs.readFileSync(file, 'utf8').split('\n').map(fromLine) - .filter((r): r is Row => r !== undefined) + const lines = fs.readFileSync(file, 'utf8').split('\n') + return lines.flatMap((line, i) => { + if (line === '' || line === HEADER.trimEnd()) return [] + const r = fromLine(line) + if (r !== undefined) return [r] + throw new Error(`labels.tsv line ${i + 1} unreadable: fix it by hand ` + + 'or restore labels.tsv.bak') + }) } -export function writeRows(file: string, rows: Row[]): void { - writeSafe(file, [COLS.join('\t'), ...rows.map(toLine)].join('\n') + '\n') +/** Appends rows; the previous file is copied to labels.tsv.bak first. */ +export function appendRows(file: string, rows: Row[]): void { + mkdir(file) + if (fs.existsSync(file)) fs.copyFileSync(file, `${file}.bak`) + else fs.writeFileSync(file, HEADER, { mode: 0o600 }) + fs.appendFileSync(file, rows.map(r => toLine(r) + '\n').join('')) } -/** state.json; absent or unreadable gives a first-run state. */ +/** state.json; absent: a first run; unparsable: a loud error. */ export function readState(file: string): State { + if (!fs.existsSync(file)) return { lastRun: 0, closed: false } try { const s = JSON.parse(fs.readFileSync(file, 'utf8')) - return { lastRun: Number(s.lastRun) || 0, - toolExtended: s.toolExtended === true } + const at = (v: unknown) => typeof v === 'number' ? v : undefined + return { lastRun: Number(s.lastRun) || 0, promptAt: at(s.promptAt), + toolAt: at(s.toolAt), closed: s.closed === true } } catch { - return { lastRun: 0, toolExtended: false } + throw new Error(`state.json unreadable (${file}): fix or delete it`) } } @@ -473,7 +599,7 @@ function parseLine(line: string): Draft | string { const item = Number(f[0]) const label = f[2] ?? '' if (!Number.isInteger(item)) return `bad item: ${f[0]}` - if (!DRAFT_LABELS.includes(label)) return `bad label: ${label}` + if (!DRAFT_LABELS.includes(label)) return `bad label: '${label}'` return { item, id: f[1] ?? '', label: label as Label, cause: f[3] ?? '' } } @@ -498,8 +624,8 @@ function fitError(d: Draft, it: Item | undefined): string { function completeError(drafts: Draft[], items: Item[]): string { const whole = new Set(drafts.filter(d => d.id === '-' && d.label === 'test').map(d => d.item)) - const got = drafts.filter(d => d.id !== '-' && - ENTRY_LABELS.includes(d.label)).map(d => `${d.item}:${d.id}`) + const got = drafts.filter(d => ENTRY_LABELS.includes(d.label)) + .map(d => `${d.item}:${d.id}`) if (new Set(got).size !== got.length) return 'an entry is labelled twice' for (const it of items) { if (it.lost || whole.has(it.n)) continue @@ -532,35 +658,43 @@ const base = (it: Item) => ({ /** The labels.tsv rows of one item: its ask row, then its draft rows. */ export function rowsOf(it: Item, drafts: Draft[]): Row[] { - const head: Row = { ...base(it), id: '', cited: false, + const head: Row = { ...base(it), id: '', label: it.lost ? 'lost' : 'ask', cause: '' } const mine = drafts.filter(d => d.item === it.n).map((d): Row => ({ ...base(it), id: d.id === '-' ? '' : d.id, pick: it.ids.includes(d.id) ? it.ids.indexOf(d.id) + 1 : undefined, - cited: it.cited.includes(d.id), label: d.label, cause: d.cause, + label: d.label, cause: d.cause, })) return [head, ...mine] } + +/** A draft to fill in: one ` ` line per injected id. */ +export function skeleton(items: Item[]): string { + const lines = items.filter(i => !i.lost) + .flatMap(i => i.ids.map(id => `${i.n}\t${id}\t`)) + return ['# fill the 3rd column: relevant or irrelevant', ...lines] + .join('\n') + '\n' +} ``` -- [ ] **Step 4: Run, see them pass** +- [ ] **Step 5: Run, see them pass** -Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test labels.test.ts & { label: Label }): Row { clock += 1 - return { at: clock, root: 'r', session: 's1', channel: 'prompt', + return { at: clock, root: 'r#000000', session: 's1', channel: 'prompt', trigger: 'fresh', outcome: 'injected', cands: 3, ms: 900, - id: 'BDR-001', cited: false, cause: '', ...over } + id: 'BDR-001', cause: '', ...over } } /** One injected ask with entries labelled `labels`, picks 1..n. */ function ask(labels: Label[], over: Partial = {}): Row[] { @@ -632,67 +766,74 @@ function ask(labels: Label[], over: Partial = {}): Row[] { id: `LRN-00${i + 1}`, pick: i + 1 }))] } -test('M1: precision and p90', () => { +test('M1: precision, p90 and the clock start rule', () => { const r = ask(['relevant', 'irrelevant', 'relevant', 'relevant']) expect(precision(r)).toBe(0.75) expect(precision([])).toBeUndefined() expect(p90([5, 1, 9, 3, 7, 2, 8, 4, 6, 10])).toBe(9) expect(p90([])).toBeUndefined() + expect(clockStart(r, [1])).toBe(r[0]?.at) + expect(clockStart([], [7, 3])).toBe(3) + expect(clockStart([], [])).toBeUndefined() }) -test('M2: a test row excludes its whole item', () => { - const rows = [...ask(['relevant']), ...ask(['irrelevant', 'test'])] +test('M2: a `- test` row excludes its whole item', () => { + const t = ask(['irrelevant']) + const rows = [...ask(['relevant']), ...t, + row({ at: t[0]!.at, id: '', label: 'test' })] const m = metricsOf(rows) expect(m.precision).toBe(1) expect(m.promptN).toBe(1) }) -test('M3: noisy at 3 hits, promotion at 3 in 2 sessions', () => { +test('M3: noisy and promotion keyed by repo and ID', () => { const noisy = [1, 2, 3].flatMap(() => ask(['irrelevant'])) - expect(metricsOf(noisy).noisy).toEqual(['LRN-001']) + expect(metricsOf(noisy).noisy).toEqual(['r#000000 LRN-001']) expect(metricsOf(noisy.slice(0, 4)).noisy).toEqual([]) + const split = [...noisy.slice(0, 4), ...ask(['irrelevant'], + { root: 'other#111111' })] + expect(metricsOf(split).noisy).toEqual([]) const one = [1, 2, 3].flatMap(() => ask(['relevant'])) expect(metricsOf(one).promote).toEqual([]) const two = [...one, ...ask(['relevant'], { session: 's2' })] - expect(metricsOf(two).promote).toEqual(['LRN-001']) + expect(metricsOf(two).promote).toEqual(['r#000000 LRN-001']) }) -test('M4: late share counts asks with candidates only', () => { +test('M4: late share counts asks with candidates, lost ones too', () => { const rows = [ row({ id: '', label: 'ask', outcome: 'timeout' }), - row({ id: '', label: 'ask', outcome: 'paused' }), + row({ id: '', label: 'lost', outcome: 'paused' }), row({ id: '', label: 'ask', outcome: 'none', cands: 0 }), ...ask(['relevant']), ] expect(metricsOf(rows).late).toBeCloseTo(2 / 3) + expect(metricsOf(rows).lost).toBe(1) }) /** Metrics with `n` prompt asks and `t` tool injections, clock at 0. */ function m(n: number, t: number): Metrics { const p = Array.from({ length: n }, () => ask(['relevant'])).flat() - const tl = Array.from({ length: t }, () => - ask(['relevant'], { channel: 'tool', trigger: 'tool', cands: undefined })) + const tl = Array.from({ length: t }, () => ask(['relevant'], + { channel: 'tool', trigger: 'tool', cands: undefined })).flat() return { ...metricsOf([...p, ...tl]), start: 0 } } const CAP = GATE_DAYS * DAY_MS test('M5: the prompt gate at 20 asks or at the cap', () => { - expect(gateVerdict(m(19, 0), 1, false).prompt).toBe('open') - expect(gateVerdict(m(20, 0), 1, false).prompt).toBe('reached') - expect(gateVerdict(m(3, 0), CAP, false).prompt).toBe('reached') + expect(gateVerdict(m(19, 0), 1).prompt).toBe('open') + expect(gateVerdict(m(20, 0), 1).prompt).toBe('reached') + expect(gateVerdict(m(3, 0), CAP).prompt).toBe('reached') const none = { ...metricsOf([]), start: undefined } - expect(gateVerdict(none, CAP * 9, false)) - .toEqual({ prompt: 'open', tool: 'open', extendTool: false }) + expect(gateVerdict(none, CAP * 9)) + .toEqual({ prompt: 'open', tool: 'open' }) }) -test('M6: the tool gate extends once, then closes', () => { - expect(gateVerdict(m(0, 10), 1, false).tool).toBe('reached') - expect(gateVerdict(m(0, 9), CAP - 1, false).tool).toBe('open') - expect(gateVerdict(m(0, 9), CAP, false)) - .toMatchObject({ tool: 'extended', extendTool: true }) - expect(gateVerdict(m(0, 9), 2 * CAP - 1, true)) - .toMatchObject({ tool: 'extended', extendTool: false }) - expect(gateVerdict(m(0, 9), 2 * CAP, true).tool).toBe('insufficient') +test('M6: the tool gate extends once by time, then closes', () => { + expect(gateVerdict(m(0, 10), 1).tool).toBe('reached') + expect(gateVerdict(m(0, 9), CAP - 1).tool).toBe('open') + expect(gateVerdict(m(0, 9), CAP).tool).toBe('extended') + expect(gateVerdict(m(0, 9), 2 * CAP - 1).tool).toBe('extended') + expect(gateVerdict(m(0, 9), 2 * CAP).tool).toBe('insufficient') }) test('M7: actions only once a gate is reached', () => { @@ -700,33 +841,33 @@ test('M7: actions only once a gate is reached', () => { ask([i % 3 === 0 ? 'relevant' : 'irrelevant'], { trigger: 'idle' })) .flat() // 7/20 relevant: under 64 % overall and under 50 % for idle const mm = { ...metricsOf(idle), start: 0 } - expect(actionsOf(mm, gateVerdict(mm, 1, false))) + expect(actionsOf(mm, gateVerdict(mm, 1))) .toContain('prompt precision under 64 %: tighten or drop the idle trigger') const open = { ...mm, promptN: 2 } - expect(actionsOf(open, gateVerdict(open, 1, false))).toEqual([]) + expect(actionsOf(open, gateVerdict(open, 1))).toEqual([]) }) -test('M8: rerank latency actions', () => { +test('M8: rerank latency action', () => { const slow = Array.from({ length: 20 }, () => ask(['relevant'], { ms: 2400 })).flat() const mm = { ...metricsOf(slow), start: 0 } - expect(actionsOf(mm, gateVerdict(mm, 1, false))) + expect(actionsOf(mm, gateVerdict(mm, 1))) .toContain('rerank p90 under 2500 ms: lower RERANK_MS') }) -test('M9: FEEDBACK.md has no repo name and no prompt text', () => { - const rows = ask(['relevant'], { root: 'secret-client-repo' }) - const text = renderFeedback(metricsOf(rows), - gateVerdict(metricsOf(rows), 1, false), 0) - expect(text).not.toContain('secret-client-repo') +test('M9: FEEDBACK.md counts, no repo tag without a noisy entry', () => { + const rows = ask(['relevant'], { root: 'client#abcdef' }) + const mm = metricsOf(rows) + const text = renderFeedback(mm, gateVerdict(mm, 1), 0) + expect(text).not.toContain('client#abcdef') expect(text).toContain('Prompt gate: open (1/20)') }) ``` - [ ] **Step 3: Run, see them fail** -Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test metrics.test.ts const under = (v: Rate, t: number) => v !== undefined && v < t const pct = (v: Rate) => v === undefined ? 'n/a' : `${Math.round(v * 100)} %` const day = (ms: number) => new Date(ms).toISOString().slice(0, 10) +const entryKey = (r: Row) => `${r.root} ${r.id}` -/** Rows of items with no `test` row anywhere (LRN-241). */ +/** The gate clock start: the first labelled row with cands, else the + * first log record with cands (one rule for reader and metrics). */ +export function clockStart(rows: Row[], logAts: number[]): Rate { + const labelled = rows.filter(r => r.cands !== undefined).map(r => r.at) + const ats = labelled.length > 0 ? labelled : logAts + return ats.length === 0 ? undefined : Math.min(...ats) +} + +/** Rows of items with no `test` row (LRN-241). */ function liveRows(rows: Row[]): Row[] { const test = new Set(rows.filter(r => r.label === 'test').map(keyOf)) return rows.filter(r => !test.has(keyOf(r))) @@ -784,7 +934,7 @@ function byTrigger(entries: Row[]): Record { return out } -/** timeout + paused share among asks that had candidates. */ +/** timeout + paused among asks (lost ones too) that had candidates. */ function lateShare(asks: Row[]): Rate { const had = asks.filter(r => (r.cands ?? 0) > 0) const late = had.filter(r => @@ -800,20 +950,20 @@ function misses(rows: Row[]): { rate: Rate; sampled: number } { return { rate: share(missed.size, sampled.size), sampled: sampled.size } } -/** IDs injected noisyHits times or more and never relevant. */ +/** ` ` injected noisyHits times or more, never relevant. */ function noisy(entries: Row[]): string[] { - return [...new Set(entries.map(r => r.id))].filter(id => { - const mine = entries.filter(r => r.id === id) + return [...new Set(entries.map(entryKey))].filter(k => { + const mine = entries.filter(r => entryKey(r) === k) return mine.length >= T.noisyHits && mine.every(r => r.label !== 'relevant') }).sort() } -/** IDs relevant promoteHits times in promoteSessions sessions or more. */ +/** ` ` relevant promoteHits times in promoteSessions. */ function promote(entries: Row[]): string[] { const rel = entries.filter(r => r.label === 'relevant') - return [...new Set(rel.map(r => r.id))].filter(id => { - const mine = rel.filter(r => r.id === id) + return [...new Set(rel.map(entryKey))].filter(k => { + const mine = rel.filter(r => entryKey(r) === k) return mine.length >= T.promoteHits && new Set(mine.map(r => r.session)).size >= T.promoteSessions }).sort() @@ -823,16 +973,16 @@ function promote(entries: Row[]): string[] { const itemsOf = (entries: Row[], ch: string) => new Set(entries.filter(r => r.channel === ch).map(keyOf)).size -/** Every 4d signal; the clock starts at the first row carrying cands. */ +/** Every 4d signal, from labels.tsv rows. */ export function metricsOf(all: Row[]): Metrics { const rows = liveRows(all) const entries = rows.filter(isEntry) const prompt = entries.filter(r => r.channel === 'prompt') - const asks = rows.filter(r => r.channel === 'prompt' && r.label === 'ask') - const starts = all.filter(r => r.cands !== undefined).map(r => r.at) + const asks = rows.filter(r => r.channel === 'prompt' && + (r.label === 'ask' || r.label === 'lost')) const miss = misses(rows) return { - start: starts.length === 0 ? undefined : Math.min(...starts), + start: clockStart(all, []), promptN: itemsOf(entries, 'prompt'), toolN: itemsOf(entries, 'tool'), lost: rows.filter(r => r.label === 'lost').length, precision: precision(prompt), byTrigger: byTrigger(prompt), @@ -844,32 +994,26 @@ export function metricsOf(all: Row[]): Metrics { } } -function toolGate(n: number, now: number, cap: number, ext: boolean): - { tool: ToolGate; extendTool: boolean } { - if (n >= TOOL_GATE_N) return { tool: 'reached', extendTool: false } - if (now < cap) return { tool: 'open', extendTool: false } - if (!ext) return { tool: 'extended', extendTool: true } - if (now < cap + GATE_DAYS * DAY_MS) { - return { tool: 'extended', extendTool: false } - } - return { tool: 'insufficient', extendTool: false } +/** The tool gate by count, else by time: one extension, then closed. */ +function toolGate(n: number, now: number, cap: number): ToolGate { + if (n >= TOOL_GATE_N) return 'reached' + if (now < cap) return 'open' + return now < cap + GATE_DAYS * DAY_MS ? 'extended' : 'insufficient' } /** Each channel's gate; the prompt gate never waits for the tool gate. */ -export function gateVerdict(m: Metrics, now: number, ext: boolean): Verdict { - if (m.start === undefined) { - return { prompt: 'open', tool: 'open', extendTool: false } - } +export function gateVerdict(m: Metrics, now: number): Verdict { + if (m.start === undefined) return { prompt: 'open', tool: 'open' } const cap = m.start + GATE_DAYS * DAY_MS const prompt: Gate = m.promptN >= PROMPT_GATE_N || now >= cap ? 'reached' : 'open' - return { prompt, ...toolGate(m.toolN, now, cap, ext) } + return { prompt, tool: toolGate(m.toolN, now, cap) } } /** The worst trigger under the floor; ties go idle, keyword, fresh. */ function worstTrigger(by: Record): string | undefined { - const order = ['idle', 'keyword', 'fresh'] - const low = order.filter(k => under(by[k], T.triggerFloor)) + const low = ['idle', 'keyword', 'fresh'] + .filter(k => under(by[k], T.triggerFloor)) low.sort((a, b) => (by[a] ?? 0) - (by[b] ?? 0)) return low[0] } @@ -905,19 +1049,18 @@ export function actionsOf(m: Metrics, v: Verdict): string[] { out.push(`tool precision under ${pct(T.toolKeep)}: ` + 'require 2 matched 4-grams, or exclude the entry') } - out.push(...m.noisy.map(id => `noisy: /prune-memory rewrites ${id}`)) - out.push(...m.promote.map(id => `promotion candidate (BDR-100): ${id}`)) + out.push(...m.noisy.map(k => `noisy: /prune-memory rewrites ${k}`)) + out.push(...m.promote.map(k => `promotion candidate (BDR-100): ${k}`)) return out } function toolLine(m: Metrics, v: Verdict): string { - const n = `${m.toolN}/${TOOL_GATE_N}` return v.tool === 'insufficient' ? `Tool gate: insufficient sample n=${m.toolN}, closed` - : `Tool gate: ${v.tool} (${n})` + : `Tool gate: ${v.tool} (${m.toolN}/${TOOL_GATE_N})` } -/** FEEDBACK.md: counts and shares, IDs only in the last three lists. */ +/** FEEDBACK.md: counts and shares; repo tags only in the last lists. */ export function renderFeedback(m: Metrics, v: Verdict, now: number) { const acts = actionsOf(m, v) return [ @@ -943,8 +1086,8 @@ export function renderFeedback(m: Metrics, v: Verdict, now: number) { - [ ] **Step 5: Run, see them pass** -Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test labels.test.ts metrics.test.ts dirs.forEach(d => fs.rmSync(d, { recursive: true }))) const T0 = Date.parse('2026-11-10T10:00:00.000Z') const iso = (ms: number) => new Date(T0 + ms).toISOString() const user = (ms: number, text: string): Rec => ({ type: 'user', @@ -1005,18 +1151,21 @@ const hook = (ms: number, name: string, lines: string[], use?: string) => ({ const rec = (over: Partial): LogRecord => ({ at: T0, session: 's', agent: 'main', channel: 'prompt', ids: ['BDR-001'], trigger: 'fresh', outcome: 'injected', ...over }) +const JOIN_LATE = 10001 // one ms past JOIN_MS -test('X1: slug, files and a torn line', () => { +test('X1: slug, files, sub-agent files on demand, torn lines counted', () => { expect(slugOf('/home/a.b/c_d')).toBe('-home-a-b-c-d') const p = fs.mkdtempSync(path.join(os.tmpdir(), 'mf-')) + dirs.push(p) const dir = path.join(p, slugOf('/r')) fs.mkdirSync(path.join(dir, 's', 'subagents'), { recursive: true }) fs.writeFileSync(path.join(dir, 's.jsonl'), '{"a":1}\n{"torn') fs.writeFileSync(path.join(dir, 's', 'subagents', 'x.jsonl'), '{"b":2}\n') - const files = sessionFiles(p, '/r', 's') - expect(files.map(f => path.basename(f))).toEqual(['s.jsonl', 'x.jsonl']) - expect(records(files[0]!)).toEqual([{ a: 1 }]) - expect(sessionFiles(p, '/r', 'gone')).toEqual([]) + const all = sessionFiles(p, '/r', 's', true) + expect(all.map(f => path.basename(f))).toEqual(['s.jsonl', 'x.jsonl']) + expect(sessionFiles(p, '/r', 's', false)).toHaveLength(1) + expect(records(all[0]!)).toEqual({ recs: [{ a: 1 }], torn: 1 }) + expect(sessionFiles(p, '/r', 'gone', true)).toEqual([]) }) test('X2: composer prompts only', () => { @@ -1041,41 +1190,44 @@ test('X3: an injected ask joins prompt, block and deduped reply', () => { injected: 'memory recall:\nBDR-001 x', reply: 'see BDR-001\ndone' }) }) -const JOIN_LATE = 10001 // one ms past JOIN_MS +test('X4: a non-injected ask joins its prompt; a missing turn is lost', + () => { + const recs = [user(-3000, 'plan the reader')] + expect(joinPrompt(recs, rec({ outcome: 'timeout', ids: [] }))) + .toEqual({ prompt: 'plan the reader', injected: '', reply: '' }) + expect(joinPrompt([], rec({ outcome: 'none', ids: [] }))) + .toBeUndefined() + const late = [user(0, 'x'), hook(JOIN_LATE, 'prompt.submit', + ['memory recall:\nBDR-001 x'])] + expect(joinPrompt(late, rec({}))).toBeUndefined() + }) -test('X4: a non-injected ask joins its prompt; no turn is lost', () => { - const recs = [user(-3000, 'plan the reader')] - expect(joinPrompt(recs, rec({ outcome: 'timeout', ids: [] }))) - .toEqual({ prompt: 'plan the reader', injected: '', reply: '' }) - expect(joinPrompt([], rec({ outcome: 'none', ids: [] }))).toBeUndefined() - const late = [user(0, 'x'), hook(JOIN_LATE, 'prompt.submit', - ['memory recall:\nBDR-001 x'])] - expect(joinPrompt(late, rec({}))).toBeUndefined() -}) - -test('X5: a tool record joins command and the next results', () => { - const recs: Rec[] = [ - { type: 'assistant', timestamp: iso(0), message: { id: 'a', content: [ - { type: 'tool_use', id: 'u1', name: 'Bash', - input: { command: 'make test' } }] } }, - { type: 'user', timestamp: iso(5), message: { content: [ - { type: 'tool_result', tool_use_id: 'u1', is_error: true }] } }, - hook(10, 'tool.call', ['memory: known BLK-003 — x'], 'u1'), - { type: 'assistant', timestamp: iso(20), message: { id: 'b', content: [ - { type: 'tool_use', id: 'u2', name: 'Bash', input: {} }] } }, - { type: 'user', timestamp: iso(25), message: { content: [ - { type: 'tool_result', tool_use_id: 'u2', is_error: false }] } }, - ] - const r = rec({ channel: 'tool', ids: ['BLK-003'], trigger: 'tool' }) - expect(joinTool(recs, r)).toEqual({ command: 'make test', - known: 'memory: known BLK-003 — x', next: [false] }) - expect(joinTool(recs, { ...r, ids: ['BLK-009'] })).toBeUndefined() -}) +test('X5: a tool record joins through the -context id, replays ignored', + () => { + const use = (ms: number, id: string, command?: string): Rec => ({ + type: 'assistant', timestamp: iso(ms), message: { id: `a${id}`, + content: [{ type: 'tool_use', id, name: 'Bash', + input: command ? { command } : {} }] } }) + const result = (ms: number, id: string, err: boolean, type = 'user') => + ({ type, timestamp: iso(ms), message: { content: [ + { type: 'tool_result', tool_use_id: id, is_error: err }] } }) + const recs: Rec[] = [ + use(0, 'u1', 'make test'), result(5, 'u1', true), + hook(10, 'tool.call', ['memory: known BLK-003 — x'], 'u1-context'), + result(15, 'u1', true, 'api-request-blob'), // a replay, not a call + use(20, 'u2'), result(25, 'u2', false), + result(30, 'u2', false), // a duplicate result of the same call + ] + const r = rec({ channel: 'tool', ids: ['BLK-003'], trigger: 'tool' }) + expect(joinTool(recs, r)).toEqual({ command: 'make test', + known: 'memory: known BLK-003 — x', next: [false] }) + expect(joinTool(recs, { ...r, ids: ['BLK-009'] })).toBeUndefined() + }) ``` - [ ] **Step 2: Run, see them fail** -Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test transcripts.test.ts +export type Loaded = { recs: Rec[]; torn: number } export const JOIN_MS = 10000 // attachment written at most this after `at` const NEAR_MS = 60000 // composer prompt at most this far from the ask const WRAPPERS = ['-context /** The transcript folder name of a project root. */ export const slugOf = (root: string) => root.replace(/[^A-Za-z0-9]/g, '-') -/** The main transcript of a session, then its sub-agent ones. */ +/** The main transcript of a session, then its sub-agent ones if asked. */ export function sessionFiles( - projects: string, root: string, session: string, + projects: string, root: string, session: string, withSub: boolean, ): string[] { const dir = path.join(projects, slugOf(root)) const sub = path.join(dir, session, 'subagents') - const subs = fs.existsSync(sub) + const subs = withSub && fs.existsSync(sub) ? fs.readdirSync(sub).filter(f => f.endsWith('.jsonl')).sort() .map(f => path.join(sub, f)) : [] @@ -1111,14 +1265,15 @@ export function sessionFiles( .filter(f => fs.existsSync(f)) } -/** Parsed lines of a transcript; a torn or partial line is skipped. */ -export function records(file: string): Rec[] { - const out: Rec[] = [] +/** Parsed lines of a transcript; torn or partial lines are counted. */ +export function records(file: string): Loaded { + const recs: Rec[] = [] + let torn = 0 for (const raw of fs.readFileSync(file, 'utf8').split('\n')) { if (raw.trim() === '') continue - try { out.push(JSON.parse(raw)) } catch { /* torn line */ } + try { recs.push(JSON.parse(raw)) } catch { torn += 1 } } - return out + return { recs, torn } } const tsOf = (r: Rec) => Date.parse(String(r.timestamp ?? '')) @@ -1228,24 +1383,28 @@ export function joinPrompt( reply: replyAfter(recs, Math.max(t, p?.t ?? t)) } } +/** Blocks of real calls and results: blob replays are skipped. */ +const callBlocks = (recs: Rec[]) => recs + .filter(r => r.type === 'assistant' || r.type === 'user') + .flatMap(blocksOf) + /** The command of a Bash tool_use. */ -function bashCommand(recs: Rec[], useId: string): string { - for (const b of recs.flatMap(blocksOf)) { - if (b.type === 'tool_use' && b.id === useId) { - return String(b.input?.command ?? '') - } - } - return '' +function bashCommand(blocks: Rec[], useId: string): string { + const b = blocks.find(x => x.type === 'tool_use' && x.id === useId) + return String(b?.input?.command ?? '') } /** Whether each of the next 3 Bash results after `useId` failed. */ -function nextResults(recs: Rec[], useId: string): boolean[] { +function nextResults(blocks: Rec[], useId: string): boolean[] { const bash = new Set() + const done = new Set() const out: boolean[] = [] let after = false - for (const b of recs.flatMap(blocksOf)) { + for (const b of blocks) { if (b.type === 'tool_use' && b.name === 'Bash') bash.add(b.id) if (b.type !== 'tool_result' || !bash.has(b.tool_use_id)) continue + if (done.has(b.tool_use_id)) continue + done.add(b.tool_use_id) if (after && out.length < 3) out.push(b.is_error === true) if (b.tool_use_id === useId) after = true } @@ -1258,18 +1417,20 @@ export type ToolJoin = { command: string; known: string; next: boolean[] } export function joinTool(recs: Rec[], rec: LogRecord): ToolJoin | undefined { const needle = KNOWN_HEAD + (rec.ids[0] ?? '') const att = attachmentAfter(recs, rec.at, 'tool.call', needle) - const useId = att?.attachment?.toolUseID - if (att === undefined || typeof useId !== 'string') return undefined - return { command: bashCommand(recs, useId), + const raw = att?.attachment?.toolUseID + if (att === undefined || typeof raw !== 'string') return undefined + const useId = raw.replace(CONTEXT_SUFFIX, '') + const blocks = callBlocks(recs) + return { command: bashCommand(blocks, useId), known: hookText(att, 'tool.call') ?? '', - next: nextResults(recs, useId) } + next: nextResults(blocks, useId) } } ``` - [ ] **Step 4: Run, see them pass** -Run: `cd lib/memory-feedback && systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bun test ): Logged[]`; - `worksheet(dir: string, p: Paths): number`; `apply(dir: string, p: Paths, - now: number): string`; CLI `bun reader.ts worksheet|apply ` - (rc 0 ok, 1 error on stderr, 2 usage). +- Consumes: Task 1 `LOG_PREFIX`, `MAX_LOG`, `isLogRecord`, `redact`, + `LogRecord`; Task 2 `appendRows`, `keyOf`, `parseDraft`, `readRows`, + `readState`, `rowsOf`, `skeleton`, `tagOf`, `writeSafe`, `writeState`, + `Item`, `State`; Task 3 `clockStart`, `gateVerdict`, `metricsOf`, + `renderFeedback`, `Verdict`; Task 4 `joinPrompt`, `joinTool`, `records`, + `sessionFiles`, `Loaded`. +- Produces: `type Paths`; `type Logged`; `storeFile(dir: string): string`; + `readLogs(file: string): Logged[]`; `population(logs: Logged[], done: + Set, start: number | undefined, now: number): Logged[]`; + `worksheet(dir: string, p: Paths, now: number): number`; `apply(dir: + string, p: Paths, now: number): string`; `sample(dir: string, n: number): + string[]`; `checkDir(dir: string | undefined): string`; CLI + `bun reader.ts worksheet|apply|sample ` (rc 0 ok, 1 error on + stderr, 2 usage or refused dir). - [ ] **Step 1: Write the failing tests** `lib/memory-feedback/reader.test.ts`: ```ts -import { expect, test } from 'bun:test' +import { afterAll, expect, test } from 'bun:test' +import { spawnSync } from 'node:child_process' import * as fs from 'node:fs' import * as os from 'node:os' import * as path from 'node:path' import { slugOf } from './transcripts.ts' import { - apply, population, readLogs, storeFile, worksheet, type Paths, + apply, checkDir, population, readLogs, sample, storeFile, worksheet, + type Paths, } from './reader.ts' +const dirs: string[] = [] +afterAll(() => dirs.forEach(d => fs.rmSync(d, { recursive: true }))) +const mk = (pfx: string) => { + const d = fs.mkdtempSync(path.join(os.tmpdir(), pfx)) + dirs.push(d) + return d +} const T0 = Date.parse('2026-11-10T10:00:00.000Z') +const NOW = T0 + 2 * 3600000 // every fixture record is 2 h old const iso = (ms: number) => new Date(T0 + ms).toISOString() +// built at run time: no secret-shaped literal in the repo (BLK-041) +const PLANTED = ['API', 'TOKEN'].join('_') + '=' + 'hunter' + '2value' -/** A fixture home: store, one transcript, an empty out dir, a work dir. */ +/** A fixture home: store, one transcript, an out dir, a work dir. */ function fixture(store: unknown): { p: Paths; work: string } { - const home = fs.mkdtempSync(path.join(os.tmpdir(), 'mf-')) + const home = mk('mf-') const p = { store: path.join(home, 'store'), projects: path.join(home, 'projects'), out: path.join(home, 'mf') } fs.mkdirSync(p.store) @@ -1333,23 +1508,24 @@ function fixture(store: unknown): { p: Paths; work: string } { fs.mkdirSync(dir, { recursive: true }) fs.writeFileSync(path.join(dir, 's1.jsonl'), [ { type: 'user', timestamp: iso(-50), origin: { kind: 'human' }, - message: { content: 'design the reader gate' } }, + message: { content: `design the reader gate with ${PLANTED}` } }, { type: 'attachment', timestamp: iso(400), attachment: { type: 'hook_additional_context', hookName: 'prompt.submit', content: ['memory recall:\nBDR-001 x'] } }, { type: 'assistant', timestamp: iso(900), message: { id: 'm', content: [{ type: 'text', text: 'per BDR-001' }] } }, ].map(r => JSON.stringify(r)).join('\n') + '\n') - const work = fs.mkdtempSync(path.join(os.tmpdir(), 'mf-work-')) - return { p, work } + return { p, work: mk('mf-work-') } } const REC = (at: number, over = {}) => ({ at, session: 's1', agent: 'main', channel: 'prompt', ids: ['BDR-001'], trigger: 'fresh', outcome: 'injected', cands: 5, ms: 800, ...over }) const STORE = { 'l:/repo/x': [ - REC(T0 - 9e6, { cands: undefined }), // before 4d.1: out of population - REC(T0), REC(T0 + 60000, { session: 'gone' }), // second one lost + REC(T0 - 9e6, { cands: undefined }), // before 4d.1: out of population + REC(T0), REC(T0 + 60000, { session: 'gone' }), // the second one lost ], 'p:/repo/x': { sessions: [] } } +const draft = (work: string, text: string) => + fs.writeFileSync(path.join(work, 'labels-draft.tsv'), text) test('R1: no store file, or a torn one, fails loudly', () => { const { p } = fixture('{"l:/x": [tor') @@ -1357,125 +1533,148 @@ test('R1: no store file, or a torn one, fails loudly', () => { expect(() => storeFile(path.join(p.store, 'none'))).toThrow('no memory') }) -test('R2: population starts at the first cands record, skips labelled', () => { +test('R2: population from the clock start, unlabelled, 30 min old', () => { const { p } = fixture(STORE) const logs = readLogs(storeFile(p.store)) - expect(population(logs, new Set()).map(l => l.rec.at)) + expect(population(logs, new Set(), T0, NOW).map(l => l.rec.at)) .toEqual([T0, T0 + 60000]) - expect(population(logs, new Set([`s1|${T0}|prompt`]))).toHaveLength(1) - expect(population(logs.slice(0, 1), new Set())).toEqual([]) + expect(population(logs, new Set([`s1|${T0}|prompt`]), T0, NOW)) + .toHaveLength(1) + expect(population(logs, new Set(), T0, T0 + 600000)).toHaveLength(0) + expect(population(logs, new Set(), undefined, NOW)).toEqual([]) }) -test('R3: worksheet writes 0600 text files and a lost item', () => { - const { p, work } = fixture(STORE) - expect(worksheet(work, p)).toBe(2) - const sheet = fs.readFileSync(path.join(work, 'worksheet.md'), 'utf8') - expect(sheet).toContain('design the reader gate') - expect(sheet).toContain('LOST') - for (const f of ['worksheet.md', 'items.json']) { - expect(fs.statSync(path.join(work, f)).mode & 0o777).toBe(0o600) - } - const items = JSON.parse(fs.readFileSync( - path.join(work, 'items.json'), 'utf8')) - expect(items.map((i: { cited: string[]; lost: boolean }) => - [i.cited, i.lost])).toEqual([[['BDR-001'], false], [[], true]]) -}) +test('R3: worksheet: 0600 files, redacted text, a lost item, a skeleton', + () => { + const { p, work } = fixture(STORE) + expect(worksheet(work, p, NOW)).toBe(2) + const sheet = fs.readFileSync(path.join(work, 'worksheet.md'), 'utf8') + expect(sheet).toContain('design the reader gate') + expect(sheet).not.toContain('hunter2value') + expect(sheet).toContain('LOST') + for (const f of ['worksheet.md', 'items.json', 'labels-draft.tsv']) { + expect(fs.statSync(path.join(work, f)).mode & 0o777).toBe(0o600) + } + expect(fs.readFileSync(path.join(work, 'labels-draft.tsv'), 'utf8')) + .toContain('1\tBDR-001\t') + const items = JSON.parse(fs.readFileSync( + path.join(work, 'items.json'), 'utf8')) + expect(items.map((i: { cited: string[]; lost: boolean }) => + [i.cited, i.lost])).toEqual([[['BDR-001'], false], [[], true]]) + }) test('R4: a bad draft refuses the whole apply, nothing written', () => { const { p, work } = fixture(STORE) - worksheet(work, p) - fs.writeFileSync(path.join(work, 'labels-draft.tsv'), '1\tBDR-001\tmaybe') - expect(() => apply(work, p, T0)).toThrow('draft refused') + worksheet(work, p, NOW) + draft(work, '1\tBDR-001\tmaybe') + expect(() => apply(work, p, NOW)).toThrow('draft refused') expect(fs.existsSync(p.out)).toBe(false) }) test('R5: apply writes labels, state, FEEDBACK; a second apply refuses', () => { const { p, work } = fixture(STORE) - worksheet(work, p) - fs.writeFileSync(path.join(work, 'labels-draft.tsv'), - '1\tBDR-001\trelevant\n') - const fb = apply(work, p, T0 + 1) + worksheet(work, p, NOW) + draft(work, '1\tBDR-001\trelevant\n') + const fb = apply(work, p, NOW) expect(fb).toContain('Prompt gate: open (1/20)') expect(fb).toContain('Lost records: 1') expect(fb).not.toContain('design the reader gate') const labels = fs.readFileSync(path.join(p.out, 'labels.tsv'), 'utf8') - expect(labels).not.toContain('design the reader gate') + expect(labels).not.toContain('design') expect(labels).not.toContain('/repo/x') + expect(labels).toMatch(/\tx#[0-9a-f]{6}\t/) expect(JSON.parse(fs.readFileSync(path.join(p.out, 'state.json'), - 'utf8')).lastRun).toBe(T0 + 1) - expect(() => apply(work, p, T0 + 2)).toThrow('already applied') - expect(worksheet(work, p)).toBe(0) + 'utf8')).lastRun).toBe(NOW) + expect(() => apply(work, p, NOW + 1)).toThrow('already applied') + expect(worksheet(work, p, NOW + 2)).toBe(0) }) test('R6: a first run with nothing to label records the run only', () => { const { p, work } = fixture({ 'l:/repo/x': [REC(T0, { cands: undefined })] }) - expect(worksheet(work, p)).toBe(0) + expect(worksheet(work, p, NOW)).toBe(0) expect(fs.readFileSync(path.join(work, 'worksheet.md'), 'utf8')) .toContain('No unlabelled record') - expect(apply(work, p, T0)).toContain('Gate clock start: not started') + expect(apply(work, p, NOW)).toContain('Gate clock start: not started') expect(fs.existsSync(path.join(p.out, 'state.json'))).toBe(true) }) + +test('R7: a gate verdict is NEW once, then recorded', () => { + const { p, work } = fixture(STORE) + const late = T0 + 36 * 86400000 // past the 5-week cap + worksheet(work, p, late) + draft(work, '1\tBDR-001\trelevant\n') + expect(apply(work, p, late)).toContain('NEW: prompt gate reached') + worksheet(work, p, late + 1) + draft(work, '') + const again = apply(work, p, late + 1) + expect(again).toContain('Prompt gate: reached') + expect(again).not.toContain('NEW:') +}) + +test('R8: the CLI refuses a dir inside a git work tree', () => { + const repo = mk('mf-repo-') + spawnSync('git', ['-C', repo, 'init', '-q']) + expect(checkDir(repo)).toContain('git work tree') + expect(checkDir('relative/dir')).not.toBe('') + expect(checkDir(mk('mf-ok-'))).toBe('') +}) + +test('R9: sample picks entry lines of the draft', () => { + const { p, work } = fixture(STORE) + worksheet(work, p, NOW) + draft(work, '1\tBDR-001\trelevant\n1\t-\tchecked\n') + expect(sample(work, 5)).toEqual(['1\tBDR-001\trelevant']) +}) ``` -- [ ] **Step 2: Write the suite entry `lib/tests/memory-feedback.test.sh`** +- [ ] **Step 2: Run, see them fail** -```bash -#!/usr/bin/env bash -# lib/tests/memory-feedback.test.sh — the bun unit suite of -# lib/memory-feedback (phase 4d reader, labels, metrics, transcripts). -# Fails when bun is absent: it never passes vacuously. -set -u -ROOT="$(cd "$(dirname "$0")/../.." && pwd)" -command -v bun >/dev/null 2>&1 || { echo "FAIL bun not on PATH"; exit 1; } -cd "$ROOT/lib/memory-feedback" || exit 1 -bun test 2>&1 -``` - -Then `chmod +x lib/tests/memory-feedback.test.sh`. - -- [ ] **Step 3: Run, see the reader tests fail** - -Run: `systemd-run --user --scope -q -p TasksMax=256 env PATH="$HOME/.bun/bin:$PATH" bash lib/tests/memory-feedback.test.sh ` joins // the unlabelled injection-log records to their transcripts; `apply ` // validates the labeller's draft, appends labels.tsv and regenerates -// FEEDBACK.md. Text only ever lands in , the session scratchpad. +// FEEDBACK.md; `sample ` picks entries for the spot check. Redacted +// text only ever lands in , the session scratchpad. +import { spawnSync } from 'node:child_process' import * as fs from 'node:fs' import * as os from 'node:os' import * as path from 'node:path' import { - LOG_PREFIX, isLogRecord, type LogRecord, + LOG_PREFIX, MAX_LOG, isLogRecord, redact, type LogRecord, } from '../../mods/memory/hooks/register.ts' import { - keyOf, parseDraft, readRows, readState, rowsOf, writeRows, writeSafe, - writeState, type Item, + appendRows, keyOf, parseDraft, readRows, readState, rowsOf, skeleton, + tagOf, writeSafe, writeState, type Item, type State, } from './labels.ts' -import { gateVerdict, metricsOf, renderFeedback } from './metrics.ts' import { - joinPrompt, joinTool, records, sessionFiles, type Rec, + clockStart, gateVerdict, metricsOf, renderFeedback, type Verdict, +} from './metrics.ts' +import { + joinPrompt, joinTool, records, sessionFiles, type Loaded, } from './transcripts.ts' export type Paths = { store: string; projects: string; out: string } export type Logged = { root: string; rec: LogRecord } type Base = Omit type Joined = { item: Item; text: string } +type Load = (l: Logged) => Loaded +const AGE_FLOOR_MS = 1800000 // a younger record may still be mid-turn const PROMPT_MAX = 600 const REPLY_MAX = 800 const DRAFT_HELP = [ - 'Write labels-draft.tsv next to this file, one TAB-separated line each:', - '`