Merge feature/advisor-usage into develop

This commit is contained in:
bchanot committed 2026-11-19 18:11:30 +01:00
commit 6e455361db
14 files changed
+715 -9

No files matched your search

+3
View File
@@ -141,3 +141,6 @@
<!-- /feat merge-reuse-clause 2026-11-19 (af846d6): executor report -->
- Run 1 feater: first full `make test` exited `make: *** [Makefile:36: test] Error 1`, suite unknown (only the tail captured); rerun green, GATE 0 + verifier make test green, no stray process after. Possibly a flaky suite or overlap with the scratch-clone `make test` running at the same time (the scratch run's only red was run-reconcile T6c = darwin-skill path absent in a fresh clone, environment). Next red: keep the full log, name the suite. Severity: low. Discovered while: /feat merge-reuse-clause STEP 3.
<!-- /feat advisor-usage 2026-11-19 (c6f2dad): GATE 0 first run -->
- lib/tests/mods.test.sh: `FAIL model-router: plugin test timed out after 120s` in a full make test at load avg ~30; the suite alone took 121 s (CLI_TIMEOUT=120 at :12) and passed. Probable cause of the run 1 feater's unexplained red (entry above). Fix direction: raise CLI_TIMEOUT for the plugin test or cut its runtime. Severity: low. Discovered while: /feat advisor-usage GATE 0.
+1
View File
@@ -785,3 +785,4 @@ rules:
## 2026-11-19
- /bugfix degraded-tool-line 9814720: 3 security readers read DEGRADED off TOOL line (BLK-051), BUGS-FOUND:134 FIXED, TODO:13 done, 2 pre-existing minors logged; CHANGELOG Fixed entry.
- /feat x2 merge-reuse (af846d6, 87566e2, feature/advisor-merge-reuse): 6 gate-flow end lines restate effort-shift item 3 merge-reuse clause, loops-light 97; TODO:14 done, 3 follow-ups logged; run 1 GATE 0 ERROR = contract without EVIDENCE: pending (LRN-173 again).
- Run C /feat advisor-usage (c6f2dad + retention fix, feature/advisor-usage): FEEDBACK.md advisor usage table per caller/model, reconciled vs probe (156 calls, 21.7M); found general-purpose sub-agent advisor leak (30 calls, 7.1M) → TODO; LRN-301 (call = served iteration), LRN-302 (scratch develop ref); mods.test.sh timeout under load → BUGS-FOUND.
+12
View File
@@ -2162,3 +2162,15 @@ Rule: when editing a doctrine file under structure locks, grep the test's lock s
**Context**: advisor run B3 (2b7aeb8, 51f343f). feat/bugfix skip end-of-run advisor call when security PASS "(not DEGRADED)". security-auditor prints `SECURITY — VERDICT: PASS` on DEGRADED scan; DEGRADED only on `TOOL:` line (agents/security-auditor.md OUTPUT), while lib/verify-secure-loop.md teaches reading DEGRADED off VERDICT line → reader checks wrong line, wrong skip. 3 lenses + re-challenge missed it; opus STEP 6 caught it.
**Pattern**: rule keyed on gate verdict must name the line its qualifier prints on. Agent OUTPUT format = source of truth, not the doctrine paraphrasing it.
**Apply**: before writing a condition on an agent's verdict, grep that agent's OUTPUT block for where each qualifier prints; state the line at the site.
## LRN-301 — Advisor call = served advisor_message iteration, not request block; reconcile usage table vs independent probe (2026-11-19)
- **Pattern**: advisor call = served `advisor_message` iteration in `message.usage.iterations[]`, not `server_tool_use` block named `advisor`. Block without iteration = request never run (sent beside client tool_use, re-issued next message) or `advisor_tool_result_error`. Count iterations, dedupe by `message.id`. Usage table built from transcripts: reconcile row by row against independent probe (other language, same window, same instant) before calling it right; gates re-running same code prove nothing about real numbers.
- **Context**: 2026-11-19 Run C (/memory-feedback advisor usage, c6f2dad). Real transcripts: 147 blocks vs 141 iterations (5 unrun, 1 error). Python probe vs bun table: 8 rows identical, 156 calls 21.7M input tok. First table exposed leak: general-purpose sub-agents 30 calls / 7.1M (16 after 6ef3069 rule), doc-syncer 4.
- **Apply**: any advisor cost/usage count: iterations not blocks; corrects LRN-295 call definition. Report generator on real data: independent probe + timing before FEAT COMPLETE.
- **Reference**: lib/memory-feedback/advisor.ts, LRN-295, BDR-150, TODO advisor doctrine leak
## LRN-302 — Scratch clone -b lacks local develop: create ref before simulating develop-based oracles (2026-11-19)
- **Pattern**: `git clone -q -b <branch> . <scratch>` scratch copy has no local `develop` (only `origin/develop`): oracles naming `develop` (`git diff --quiet develop -- …`, floor-guard) fail `fatal: bad revision 'develop'` on scratch, pass in real repo. Before simulating contract CHECKs on clone: `git -C <scratch> branch develop origin/develop`.
- **Context**: 2026-11-19 merge-reuse run 2 simulation (LRN-296 method): criterion 3 SKIP-INTACT failed only on scratch clone; fixed by creating ref.
- **Apply**: every LRN-296 scratch simulation from a non-develop branch: create local develop ref first, else false red.
- **Reference**: LRN-296, .claude/tasks/contracts/2026-11-19-merge-reuse-feat-bugfix-1450.md
+3 -1
View File
@@ -15,7 +15,9 @@ Done history before 2026-10-08 removed 2026-10-23 (git keeps it: `git log -p --
- [ ] lib/effort-shift.md § Advisor item 3 reuse clause has no fallback: end-of-run answer absent (advisor absent/error) or out of view at the merge signal → candidate "answer absent or out of view → the merge proposal runs the call" (source, not the sites); the merge happens at the gitflow finish gate (skills/gitflow/SKILL.md:68-89, 0 advisor mentions) where the rule is not loaded (merge-reuse challengers, low)
- [ ] item 3 "names any code commit made after it" has no anchor: ADVISOR field records no HEAD SHA, commits made outside the session are invisible; candidate: record the short SHA with the answer (visible format change, user decision) (merge-reuse robustness challenger, low)
- [ ] end-of-run sites restate the call but not "one end-of-run call per skill run": a fix that goes back through the gates re-enters the end step and may call again (merge-reuse robustness challenger, low)
- [ ] Run C /feat: advisor usage section in /memory-feedback (main loop vs sub-agents, model served, input tokens; dedupe usage rows by message.id, model from the `advisor_message` iteration)
- [x] DONE c6f2dad + 63f5fa4 (docs 7d09837) (/feat feature/advisor-usage; user 2026-11-19: FEEDBACK.md section, per agentType, all projects no repo, since last run, retention + drift lines). Was: Run C /feat: advisor usage section in /memory-feedback (main loop vs sub-agents, model served, input tokens; dedupe usage rows by message.id, model from the `advisor_message` iteration)
- [ ] Advisor doctrine leak: general-purpose sub-agents called the advisor 30× (7.1M input tok, the largest line of the first advisor usage table; 16 on 2026-11-18/19, after 6ef3069 2026-11-17 12:59 limited sub-agents to plan-challenger/verifier/security-auditor/analyzer), doc-syncer 4× (2026-11-16/17). Likely SDD implementers/reviewers dispatched as general-purpose: find why the "executors and every other sub-agent do not" rule does not reach them (brief, agent prompt or tool listing) (Run C first table, medium)
- [ ] lib/tests/mods.test.sh model-router plugin test runs ~121 s alone vs CLI_TIMEOUT=120 (load avg 26-31 on 2026-11-19): flaky red in make test, probable cause of run 1 feater's unexplained red; raise the bound or speed the test (Run C GATE 0, low)
- [x] Stale citers after A: README/USAGE/CHANGELOG patched, BUGS-FOUND residual superseded + LOW logged, BDR-150 status, LRN-295 (LRN-294 kept: its lesson holds)
### BDR-125 close-out run (user 2026-11-12): no phase 0/0b/4a/1/5/2/4b/4c/4d line left in TODO or BUGS-FOUND
@@ -0,0 +1,49 @@
# CONTRACT — advisor-usage (Run C)
- date: 2026-11-19 | flow: feat | branch: feature/advisor-usage
- status: active
## REQUEST (verbatim — IMMUTABLE)
Merge et corrige le bug et fais les todo
Scope of this run (main loop): TODO:18 Run C — advisor usage section in /memory-feedback (main loop vs sub-agents, model served, input tokens; dedupe usage rows by message.id, model from the `advisor_message` iteration).
## CLARIFICATIONS
Q: placement / A: a "## Advisor usage (<from> → <to>)" section appended to FEEDBACK.md by `reader.ts apply`, printed with the gates, even at 0 records [gated 2026-11-19]
Q: sub-agent breakdown / A: per agent type from subagents/agent-<id>.meta.json `agentType`; unknown → "sub-agent (unknown)"; main thread = "main loop" [gated 2026-11-19]
Q: scope / A: every project under ~/.claude/projects summed, no repo name or tag in the table [gated 2026-11-19]
Q: window / A: since the last /memory-feedback run (state.json lastRun) to now; first run (lastRun 0) = everything on disk [gated 2026-11-19]
Q: retention note / A: when the window exceeds 7 days, "Transcripts kept 7 d: calls before <date> may be missing." under the heading [gated 2026-11-19]
Q: shape drift / A: 0 calls but usage rows mention advisor_message → "N transcript lines mention advisor_message, none parsed: check the transcript shape." in place of the plain zero [gated 2026-11-19]
Q: challenge (3 lenses) changes: call = served advisor iteration (block without iteration not counted), per-dir skip, dated Unavailable heading, no 6th local in commit(), compact 999950 → 1.0M / A: main loop, internal [2026-11-19]
Q: GATE 0 run 1 criterion 1 oracle broken (bun prints file names only on failure) / A: main loop, oracle corrected to run advisor.test.ts directly; criterion 4 red = mods.test.sh model-router plugin test timeout at load avg ~30 (121 s alone vs CLI_TIMEOUT 120), unrelated to the diff [2026-11-19]
Q: retention note fires every weekly run (window just over 7 d, cut day = start day) / A: print it only when the cut day is later than the heading's start day [gated 2026-11-19]
Q: privacy wording / A: skills/memory-feedback/SKILL.md:12 "keeps IDs, enums and repo tags only" → also names advisor call counts [gated 2026-11-19]
## ACCEPTANCE CRITERIA
1. The memory-feedback bun suite is green and advisor.test.ts runs at least 17 passing tests.
CHECK: cd /home/bchanot/Documents/claude/lib/memory-feedback && PATH=$HOME/.bun/bin:$PATH bun test advisor.test.ts 2>&1 | grep -qE '^ *(1[7-9]|[2-9][0-9]) pass' && cd /home/bchanot/Documents/claude && PATH=$HOME/.bun/bin:$PATH bash lib/tests/memory-feedback.test.sh 2>&1 | grep -qE '^ *0 fail' && echo MF-GREEN
EXPECT: MF-GREEN
EVIDENCE: MET exit=0 marker-found :: MF-GREEN
2. On real transcripts, advisorUsage + renderAdvisor read this machine's advisor calls: the section heading, a "main loop" row, a Fable-served row and the total row appear (total arithmetic: advisor.test.ts).
CHECK: cd /home/bchanot/Documents/claude/lib/memory-feedback && PATH=$HOME/.bun/bin:$PATH bun -e 'import {advisorUsage,renderAdvisor} from "./advisor.ts"; const p=require("os").homedir()+"/.claude/projects"; const t=advisorUsage(p,0,Date.now()); const u=t.uses; const s=renderAdvisor(t,0,Date.now()); const ok=s.startsWith("## Advisor usage (all on disk → ")&&u.some(x=>x.caller==="main loop")&&u.some(x=>x.model.startsWith("claude-fable"))&&s.includes("| total |"); console.log(ok?"ADV-SMOKE":"bad")'
EXPECT: ADV-SMOKE
EVIDENCE: MET exit=0 marker-found :: ADV-SMOKE
3. `reader.ts apply` writes the section into FEEDBACK.md (reader.test.ts locks it) and the skill's STEP 5 tells the user to show it.
CHECK: cd /home/bchanot/Documents/claude && grep -qF '## Advisor usage' lib/memory-feedback/reader.test.ts && grep -qF '`## Advisor usage`' skills/memory-feedback/SKILL.md && echo WIRED
EXPECT: WIRED
EVIDENCE: MET exit=0 marker-found :: WIRED
4. Full make test green (code-limits and the citers census included).
CHECK: cd /home/bchanot/Documents/claude && PATH=$HOME/.bun/bin:$PATH make test </dev/null 2>&1 | tail -1
EXPECT: all suites green
EVIDENCE: MET exit=0 marker-found :: all suites green
5. Manual: the table carries no repo name, slug, path or prompt text (callers are agent types or the two fixed labels, models are id-shaped); rows are deduped by message.id; an unreadable transcript is skipped, a failure of the whole section never fails `apply`; new code within the global limits (25 logic lines/function, 80 cols, 5 params, 5 locals).
6. The retention note prints only when its cut day is later than the heading's start day (advisor.test.ts test 13 covers a 7-day-plus-hours window with no note), and SKILL.md's data line names the advisor counts.
CHECK: cd /home/bchanot/Documents/claude/lib/memory-feedback && PATH=$HOME/.bun/bin:$PATH bun -e 'import {renderAdvisor} from "./advisor.ts"; const D=86400000, to=Date.UTC(2026,10,19,17); const t={uses:[],unparsed:0,files:1}; const wk=renderAdvisor(t,to-7*D-5*3600000,to), two=renderAdvisor(t,to-14*D,to); console.log(!wk.includes("Transcripts kept")&&two.includes("calls before 2026-11-12 may be missing.")?"RET-OK":"bad")' | grep -qF RET-OK && grep -qF 'advisor call counts' /home/bchanot/Documents/claude/skills/memory-feedback/SKILL.md && echo RET-OK
EXPECT: RET-OK
EVIDENCE: MET exit=0 marker-found :: RET-OK
## FILE SCOPE
lib/memory-feedback/advisor.ts (new), lib/memory-feedback/advisor.test.ts (new), lib/memory-feedback/reader.ts, lib/memory-feedback/reader.test.ts, skills/memory-feedback/SKILL.md
@@ -0,0 +1,178 @@
# Feat plan: advisor usage section in /memory-feedback (Run C, TODO:18)
## Why
LRN-295 measured advisor use by hand (per caller, model, uncached input) and named Run C to
automate it. User choices 2026-11-19 [gated]: a section of FEEDBACK.md written by `reader.ts
apply`; sub-agents per agent type; all projects summed, no repo in the table; window = since
the last run; a retention note when the window exceeds 7 days; a shape-drift line in place of
a false zero.
## Disposition
- honors LRN-295 (the method), with one correction: a call = one served advisor iteration, i.e.
one `message.usage.iterations[]` entry with `type == "advisor_message"` (model from that entry's
`model`, tokens = its `input_tokens`). A `server_tool_use` block named `advisor` with no
iteration is NOT a call: on real transcripts 147 blocks vs 141 iterations; the 6 gaps are
requests that never ran (re-issued next message) or an `advisor_tool_result_error`. Usage rows
repeat once per content block → dedupe by `message.id`.
- probe 2026-11-19 over all 1095 transcripts (136 calls then): 0 advisor iterations with
cache_read/cache_creation tokens, so `input_tokens` is the whole input; rows sharing an id
always carry identical advisor iterations, so first-seen is exact; no `isSidechain` row in a
main transcript; no advisor row outside the two path shapes; every meta `agentType` today is a
global or built-in agent name (general-purpose, plan-challenger, verifier, …).
- honors LRN-267: only `type == "assistant"` records count (blob replays and tool results never
do); sub-agent transcripts = `<session>/subagents/agent-<id>.jsonl`, meta beside it
(`agent-<id>.meta.json`, field `agentType`).
- honors BDR-140: FEEDBACK.md stays local (~/.claude/memory-feedback, never git); the section
adds no repo name, slug, path or prompt text: callers are validated agent type strings or two
fixed labels, models validated id strings. A project-scoped agent's own name would appear
verbatim (none exists today).
- honors BDR-100: thresholds.ts untouched; no new gate.
- global code limits (our code): ≤ 25 logic lines/function, ≤ 80 cols, ≤ 5 params, ≤ 5 locals
(a destructured name counts as a local). Unexported helpers are allowed and expected; the
exported names below are final.
## Files
1. lib/memory-feedback/advisor.ts (new)
2. lib/memory-feedback/advisor.test.ts (new)
3. lib/memory-feedback/reader.ts — `commit()` appends the section
4. lib/memory-feedback/reader.test.ts — the apply test asserts the section
5. skills/memory-feedback/SKILL.md — STEP 5 shows it
## advisor.ts — exported API (names final)
```ts
// lib/memory-feedback/advisor.ts: advisor usage read from Claude Code
// transcripts (Run C, LRN-295): served calls and input tokens per caller
// and model, deduped by message.id, for the FEEDBACK.md section.
export type Use = { caller: string; model: string; calls: number; input: number }
export type Tally = { uses: Use[]; unparsed: number; files: number }
export type Source = { file: string; caller: string }
export const MAIN = 'main loop'
export const UNKNOWN_AGENT = 'sub-agent (unknown)'
export const RETENTION_DAYS = 7
export function transcriptsOf(projects: string, from: number): Source[]
export function callerOf(jsonl: string): string
export function advisorUsage(projects: string, from: number, to: number): Tally
export function compact(n: number): string
export function renderAdvisor(t: Tally, from: number, to: number): string
export function advisorSection(projects: string, from: number, to: number): string
```
Suggested private split (names free): a directory lister that never throws, a per-file line
reader that returns the counted records, a tally step for one record, the sort comparator.
Behaviour:
- `transcriptsOf`: every directory read is wrapped (`readdirSync`/`lstatSync` in a try; an
unreadable or vanished directory or file is skipped, never thrown). `projects` missing → `[]`.
For each entry of `projects` that is a real directory (`lstat`, symlinks skipped): every
regular `*.jsonl` file directly inside → caller `MAIN`; for each real sub-directory `<session>`,
its `subagents` entry when it is a real directory (lstat), every regular `agent-*.jsonl` inside
→ caller `callerOf(file)`. A file whose `mtimeMs <= from` is skipped (a small saving: it can
hold no row in the window; the substring pre-filter below is what keeps the scan cheap).
- `callerOf`: lstats the sibling `<name without .jsonl>.meta.json` first: not a regular file
(symlink, FIFO, dir) or larger than 64 KiB → `UNKNOWN_AGENT`; else reads it; its `agentType` when a string
matching `/^[A-Za-z0-9_.:-]{1,64}$/`, else `UNKNOWN_AGENT` (missing file, unparsable JSON,
non-object, bad type: never throws).
- `advisorUsage`: one `seen: Set<string>` of message ids across all files, one tally
`Map<string, Use>` keyed `caller + '\t' + model`, one `unparsed` counter, `files` = the number
of sources `transcriptsOf` returned. Per source file
(read in a try: an unreadable file is skipped): split lines; a line is parsed only when it
contains `advisor_message` (pre-filter); JSON errors and non-objects skipped. A record is
considered when `type === 'assistant'`, `Date.parse(timestamp)` is in `(from, to]`, and
`message.id` is a non-empty string not yet in `seen` (then added). For each element of
`message.usage.iterations` (array, else none) with `type === 'advisor_message'`: model = its
`model` when a string matching `/^[A-Za-z0-9._:-]{1,80}$/`, else `'unknown'`; calls += 1;
input += `input_tokens` when a non-negative safe integer, else 0. A considered record whose
`JSON.stringify(message.usage ?? null)` contains `advisor_message` but which yielded no call
→ `unparsed += 1` (shape drift; text that merely mentions the word is not in `usage`).
`uses` sorted: `MAIN` first, then calls descending, then caller, then model ascending.
- `compact(n)`: `n < 1000` → `String(n)`; `n < 999950` → `(n / 1e3).toFixed(1) + 'k'`; else
`(n / 1e6).toFixed(1) + 'M'` (so `999950` prints `1.0M`, never `1000.0k`).
- `renderAdvisor(t, from, to)`: lines, joined with `\n`, ending with `\n`:
1. `## Advisor usage (<from> → <to>)`, `<from>` = `all on disk` when `from === 0`, else the
UTC day `YYYY-MM-DD` (`new Date(ms).toISOString().slice(0, 10)`, as metrics.ts `day`);
`<to>` = that day of `to`.
2. blank line.
3. only when `from > 0` and `to - from > RETENTION_DAYS * 86400000`: the line
`Transcripts kept 7 d: calls before <day(to - 7 d)> may be missing.` then a blank line.
4. body: when `t.uses` is empty: `t.files === 0` → `No transcript modified since <from>
under the projects dir: check the path.` (`<from>` as in the heading); else `t.unparsed > 0`
→ the drift line `<t.unparsed> transcript lines mention advisor_message, none parsed: check
the transcript shape.`; else `No advisor call in the window.`. Otherwise the table,
followed (blank line between) by the same drift line when `t.unparsed > 0`:
```
| caller | model | calls | input tok |
|---|---|---|---|
| main loop | claude-fable-5-2 | 14 | 3.4M |
| total | | 41 | 5.1M |
```
(`input tok` via `compact`; the total row sums calls and input).
- `advisorSection(projects, from, to)`: first normalises `from` (outside the try, pure
arithmetic): a finite number in `[0, to]`, else `0` (state.json is hand-editable: `1e999`,
`1e17` or `NaN` must not reach `toISOString`). Then `renderAdvisor(advisorUsage(projects, f,
to), f, to)` in a try; on any throw returns `## Advisor usage (<from> → <to>)` (same dates), a blank
line and `Unavailable: transcripts unreadable.` (fixed text: an error message could carry a
path), ending with `\n`. Never throws.
## reader.ts
`commit()` gains no local (it has 5: `fresh, m, v, next, feedback`): the section call reads the
old `lastRun` inline, before any write:
```ts
const feedback = renderFeedback(m, v, now) + '\n' +
advisorSection(p.projects, readState(stateOf(p)).lastRun, now)
```
(import `advisorSection` from `./advisor.ts`). The order stays as today: the scan runs while
`feedback` is built, before `appendRows`/`writeState`/`writeSafe`; a throw inside is caught by
`advisorSection`; a kill during the scan leaves nothing written, so a rerun is clean. The
section goes into FEEDBACK.md and the printed output (NEW lines still follow it).
## SKILL.md STEP 5
The sentence "It prints FEEDBACK.md: show the gate lines (`Post-fix prompt asks: k/20` included:
actions wait for 20 post-fix asks) and the Actions list to the user." becomes "It prints
FEEDBACK.md: show the gate lines (`Post-fix prompt asks: k/20` included: actions wait for 20
post-fix asks), the Actions list and the `## Advisor usage` table (advisor calls since the last
run per caller and served model) to the user." Wrapped ≤ 80 cols like the file, with the
backticked span `` `## Advisor usage` `` kept whole on one line (contract CHECK 3 greps it).
## Tests
advisor.test.ts (bun:test, fixtures in `fs.mkdtempSync(path.join(os.tmpdir(), 'mf-adv-'))`,
removed in afterAll, like reader.test.ts):
1. main transcript: two assistant rows sharing one `message.id`, one advisor iteration
(model claude-fable-5-2, input_tokens 1500) → one `main loop` row, calls 1, input 1500.
2. a row outside the window (timestamp ≤ from) and one after `to` → not counted.
3. a `user` record and a torn line, both containing `advisor_message` → ignored.
4. one message with two advisor iterations → calls 2.
5. iteration without `model` → model `unknown`.
6. sub-agent `agent-a1.jsonl` + meta `{"agentType":"verifier"}` → caller `verifier`;
`agent-a2.jsonl` without meta and `agent-a3.jsonl` with `agentType: "../x"` →
`sub-agent (unknown)`.
7. a transcript whose mtime ≤ from (`fs.utimesSync`) → skipped even with in-window rows.
8. missing projects dir → `{ uses: [], unparsed: 0, files: 0 }`; `advisorSection` on it →
heading + the `No transcript modified since … check the path.` line; a projects dir with one
in-window transcript holding no advisor row → `No advisor call in the window.`
9. render: `from 0` → `all on disk`; `compact` 999 → `999`, 1500 → `1.5k`, 999950 → `1.0M`,
2500000 → `2.5M`; total row sums calls and input; sort order (main loop first, then calls
desc).
10. the rendered section never contains the fixture's project slug or its path.
11. a message with a `server_tool_use` block named `advisor` and no `advisor_message` iteration
(and no usage mention) → 0 calls, `unparsed` 0.
12. shape drift: an in-window assistant record whose `usage` holds `advisor_message` in a shape
the reader does not parse (e.g. `usage.advisor_message: {...}` with no `iterations`) →
`unparsed` 1, and the render prints the `none parsed` line instead of the plain zero.
13. retention: `from = to - 10 d` → the `Transcripts kept 7 d: calls before <day> may be
missing.` line; `from = to - 3 d` and `from = 0` → no such line.
14. an unreadable sub-directory (`chmod 000` on a fixture dir, mode restored in a try/finally
inside the test itself; skip the assertion when running as root) → still returns the other
files' calls, no throw.
15. `advisorSection` with `from = Infinity`, `from = 1e17` and `from = NaN` → heading `all on
disk`, never throws.
16. drift under a table: one parsed call plus one drift record → the table, then the drift line.
17. `callerOf` with a meta that is a symlink (to a valid meta elsewhere) → `sub-agent (unknown)`.
reader.test.ts: the existing apply test (or one new apply test on the same fixture) asserts the
output and FEEDBACK.md contain `## Advisor usage`.
## Edge cases
- `from` = state.lastRun = 0 on a first run → everything still on disk, heading `all on disk`.
- Huge files (largest 63 MB, 1.1 GB total): substring pre-filter before `JSON.parse`.
- A message id seen in two files (never seen in the probe) counts once.
RISK: low — read-only reporting added after the existing FEEDBACK body; the labelling path and
its write order are unchanged.
+1
View File
@@ -10,6 +10,7 @@ The BDR-125 context queue is delivered: phases 0, 0b, 4a, 1, 5, 2 and 4b-4d ship
### Added
- **/prune-memory feedback step (D)**: reads this repo's `Noisy:` and `Promotion candidates:` lines from `~/.claude/memory-feedback/FEEDBACK.md` (written by `/memory-feedback`). A noisy entry gets an inline rewrite of its title and lesson sentence, narrowed to its concrete trigger, committed as `chore(memory): noisy-rewrite <ID>`. A promotion candidate gets a mechanism proposal (rule line, hook, guard, test); on accept, a `<current status word>, promoted to <mechanism>` status line is recorded (a closed entry stays closed) and the mechanism is built in a separate `/feat`. Rows already rewritten or promoted are dropped. Cutoffs stay in `lib/memory-feedback/thresholds.ts`.
- **/memory-feedback advisor usage**: each run appends `## Advisor usage (<from> → <to>)` to `~/.claude/memory-feedback/FEEDBACK.md`: advisor calls served since the last run (first run: every transcript on disk), per caller (`main loop`, or the sub-agent type from its `.meta.json`, `sub-agent (unknown)` when absent or malformed) and model, with input tokens, deduped by `message.id`, all projects summed with no repo name. When the window starts before the 7-day transcript retention, a line says older calls may be missing. No transcript found, or messages that mention `advisor_message` but no longer parse, print a line in place of a false zero; a read failure prints `Unavailable: transcripts unreadable.`. `/memory-feedback` STEP 5 shows the table. New module `lib/memory-feedback/advisor.ts`; tests `lib/memory-feedback/advisor.test.ts`.
### Changed
- **memory recall**: a prompt after 30 minutes idle now needs 8 content words, up from 5 (a context's first prompt or a design/analysis prompt still needs 5), and the haiku rerank keeps at most 2 entries, down from 3. Measured on the first feedback gate: idle precision 44 %, third pick 35 %.
+2 -2
View File
@@ -186,7 +186,7 @@ Limits: a main-loop switch re-reads cold the part of the context the target mode
- Digest: `hooks/session-start.sh` prints `memory digest:` on `startup`, `clear` and `compact` (never `resume`) in a project that has `.claude/memory/`: up to 3 open BLK from `memory.sh index blocker` (the rest counted, upstream ones counted; `- open: none` when none is open, `- open: unknown (memory.sh index failed)` when the read fails, times out or returns a `| BLK-` row it cannot parse), up to 3 unchecked TODO items (from a `## Actionable` or `### Actionable` section, else the file's first 3 unchecked top-level items; items inside HTML comments ignored), the last journal line, 800 characters at most. When the cap bites, TODO lines drop first and a `- todo: +N not shown, read .claude/tasks/TODO.md` line counts them; nothing else drops. Without `timeout` and `jq` it prints only from a terminal.
- Recall switch: `{"enabled": true, "recall": false}` in `~/.claude/memory.json` cuts the injections (recall and known-error line) and keeps capture. Every ask is logged under the `l:<root>` store key (last 200 records, 64 KiB), never shown: when, session, agent, channel, the IDs injected, the `trigger` (`fresh`, `idle`, `keyword`, `tool`) and the `outcome` (`injected`, `none`, `timeout`, `refused`, `paused`; an ask that injects nothing is logged with no IDs); a prompt ask also logs `cands` (BM25 candidates after the seen filter) and `ms` (rerank wall time, when a rerank ran). `/memory-feedback` reads it.
- Recall residuals: measured on 14 days of this repo's transcripts, precision is 54 to 61 % against a 64 % bar, so recall ships on and switchable, and the log decides. The router's classifier label is not used. EVAL entries are not recalled. Each failing command that matches an entry spawns one `memory.sh index`. The haiku bound drops the recalls that answer slower than 5 seconds.
- Feedback: `/memory-feedback` (phase 4d) labels each injected entry from its transcript, with a 5-label spot check by you, and reports precision, misses, noisy entries and promotion candidates against `lib/memory-feedback/thresholds.ts`. Two gates: prompt at 20 labelled asks or 35 days, tool at 10 injections (one 35-day extension, then closed). Precision, trigger precision and the miss share count only rows after the gate-1 fix; their actions wait for 20 post-fix asks (`Post-fix prompt asks: k/20` in FEEDBACK.md, pre-fix entries excluded). Data stays local in `~/.claude/memory-feedback/` (IDs, enums and repo tags only, never in git). The session banner prints `memory feedback: … → /memory-feedback` when a run is due, and goes silent once the prompt gate, the tool gate and the post-fix sample are recorded, or with `"recall": false`.
- Feedback: `/memory-feedback` (phase 4d) labels each injected entry from its transcript, with a 5-label spot check by you, and reports precision, misses, noisy entries and promotion candidates against `lib/memory-feedback/thresholds.ts`. Each run also appends `## Advisor usage (<from> → <to>)` to FEEDBACK.md: advisor calls served since the last run (first run: every transcript on disk), per caller (main loop or sub-agent type) and model, with input tokens, all projects summed with no repo name. When the window starts before the 7-day transcript retention, a line warns that older calls may be missing. Two gates: prompt at 20 labelled asks or 35 days, tool at 10 injections (one 35-day extension, then closed). Precision, trigger precision and the miss share count only rows after the gate-1 fix; their actions wait for 20 post-fix asks (`Post-fix prompt asks: k/20` in FEEDBACK.md, pre-fix entries excluded). Data stays local in `~/.claude/memory-feedback/` (IDs, enums, repo tags and advisor call counts only, never in git). The session banner prints `memory feedback: … → /memory-feedback` when a run is due, and goes silent once the prompt gate, the tool gate and the post-fix sample are recorded, or with `"recall": false`.
- Off switch: `{"enabled": false}` in `~/.claude/memory.json`, read at the first event of a session, so it applies at the next session or after `/reload-plugins`. Off, the tool answers `memory capture is off (~/.claude/memory.json)`. A switch file that cannot be read, or whose `enabled` is not a boolean, keeps capture on and logs one line.
- Residuals: cross-process writes are unverified (two concurrent sessions: last writer wins). Non-Bash failures are not captured. `-b` counts as curl's cookie flag only when `curl` opens its command segment, after optional env assignments, `sudo`, `rtk` or `$(`: `time curl -b …` keeps its cookies. 40-hex Gitea tokens are kept by design: they look like commit SHAs. A session root that moves with `/cd` or a worktree move hides earlier captures of the same repo from /capitalize. A bare `gitflow.sh finish` on a chore branch counts as a milestone. Measured at 83 % noise before the error-shaped filter over 14 days of this repo's transcripts; the filter keeps every useful sample and cuts captures by two thirds (`.claude/tasks/plans/2026-11-03-memory-capture-spike/NOISE.md`).
@@ -270,7 +270,7 @@ a different package, ships its own conflicting `graphify` bin) — see
| `/audit-delta` | Recurring audit of changes since last run (norms, bugs, dead code, security) |
| `/capitalize` | Flush uncapitalized context + reconcile TODO before /clear or /compact (`--ritual` adds the end-of-session reflection) |
| `/prune-memory` | Curate and compress the .claude/memory/ registries; rewrite noisy recall entries and propose promotions from `/memory-feedback` |
| `/memory-feedback` | Label the memory mod's recall injections relevant or irrelevant from the transcripts, spot-check 5 labels with you, report precision and the phase 4d gates in ~/.claude/memory-feedback/FEEDBACK.md. Run it weekly: transcripts live 7 days |
| `/memory-feedback` | Label the memory mod's recall injections relevant or irrelevant from the transcripts, spot-check 5 labels with you, report precision, the phase 4d gates and advisor usage in ~/.claude/memory-feedback/FEEDBACK.md. Run it weekly: transcripts live 7 days |
| `/reconcile` | Confront declared status (TODO, registries) against real git/fs state — surface stale items |
| `/pdf-translate` | Translate a PDF to another language, output as HTML (via Vision) |
| `/explain-diff` | Explain a diff, commit, branch or PR as one self-contained HTML page (background, intuition, code walkthrough, 5-question quiz), saved outside the repo |
+2 -2
View File
@@ -161,7 +161,7 @@ Tu veux...
| `/audit-delta` | Audit récurrent du delta depuis le dernier run | Axes : conformité / bugs / dead code / sécurité |
| `/capitalize` | Avant /clear ou /compact | Flush contexte non capitalisé + réconcilie .claude/tasks/TODO.md |
| `/prune-memory` | Registres trop longs / bruyants | Curation : merge, superseded, compression ; réécriture des entrées bruyantes et propositions de promotion en mécanisme (règle, hook, garde, test) lues dans `~/.claude/memory-feedback/FEEDBACK.md` |
| `/memory-feedback` | Chaque semaine, ou quand la bannière le rappelle | Étiquette les injections du rappel mémoire depuis les transcripts, contrôle de 5 étiquettes avec toi, précision et gates de la phase 4d dans `~/.claude/memory-feedback/FEEDBACK.md` |
| `/memory-feedback` | Chaque semaine, ou quand la bannière le rappelle | Étiquette les injections du rappel mémoire depuis les transcripts, contrôle de 5 étiquettes avec toi, précision, gates de la phase 4d et usage de l'advisor dans `~/.claude/memory-feedback/FEEDBACK.md` |
| `/reconcile` | Connaître l'état réel du travail ouvert (TODO/registres douteux) | Confronte statut déclaré vs git/fs réel |
| `/close` | Fin de session | Alias de /capitalize --ritual — dedup + TODO + réflexion 3 questions |
| `/harden` | Audit sécurité web (SSL, CSP, HSTS) | Projet web avec config HTTP |
@@ -220,7 +220,7 @@ Le mod `mods/memory/` note les échecs Bash et les jalons de la session pour que
- Digest : `hooks/session-start.sh` affiche `memory digest:` sur `startup`, `clear` et `compact` (jamais `resume`) dans un projet qui a `.claude/memory/` : jusqu'à 3 BLK ouverts issus de `memory.sh index blocker` (le reste compté, les « upstream » comptés ; `- open: none` quand aucun n'est ouvert, `- open: unknown (memory.sh index failed)` quand la lecture échoue, dépasse son délai ou renvoie une ligne `| BLK-` illisible), jusqu'à 3 items non cochés du TODO (section `## Actionable` ou `### Actionable`, sinon les 3 premiers items de premier niveau du fichier ; items en commentaire HTML ignorés), la dernière ligne du journal, 800 caractères au plus. Quand le plafond mord, les lignes TODO tombent d'abord et une ligne `- todo: +N not shown, read .claude/tasks/TODO.md` les compte ; rien d'autre ne tombe. Sans `timeout` ni `jq`, il ne s'affiche que depuis un terminal.
- Interrupteur du rappel : `{"enabled": true, "recall": false}` dans `~/.claude/memory.json` coupe les injections (rappel et ligne d'erreur connue) et garde la capture. Chaque demande est journalisée sous la clé de store `l:<racine>` (200 enregistrements, 64 Kio), jamais affichée : date, session, agent, canal, IDs injectés, `trigger` (`fresh`, `idle`, `keyword`, `tool`) et `outcome` (`injected`, `none`, `timeout`, `refused`, `paused` ; une demande qui n'injecte rien est journalisée sans ID) ; une demande sur prompt note aussi `cands` (candidats BM25 après le filtre des entrées déjà vues) et `ms` (durée du rerank, quand il a tourné). `/memory-feedback` la lit.
- Limites du rappel : mesurée sur 14 jours de transcripts de ce dépôt, la précision est de 54 à 61 % pour une barre à 64 %, donc le rappel est livré actif et coupable, et le journal tranchera. L'étiquette du classifieur du routeur n'est pas utilisée. Les entrées EVAL ne sont pas rappelées. Chaque commande en échec qui correspond à une entrée lance un `memory.sh index`. La borne haiku écarte les rappels qui répondent après 5 secondes.
- Retour d'usage : `/memory-feedback` (phase 4d) étiquette chaque entrée injectée depuis son transcript, avec un contrôle de 5 étiquettes par toi, et donne la précision, les manqués, les entrées bruyantes et les candidates à promotion face aux seuils de `lib/memory-feedback/thresholds.ts`. Deux gates : prompt à 20 demandes étiquetées ou 35 jours, outil à 10 injections (une prolongation de 35 jours, puis fermé). La précision, la précision par trigger et la part de manqués ne comptent que les lignes postérieures au correctif du gate 1 ; leurs actions attendent 20 demandes post-correctif (`Post-fix prompt asks: k/20` dans FEEDBACK.md, entrées antérieures exclues). Les données restent locales dans `~/.claude/memory-feedback/` (IDs, enums et étiquettes de dépôt seulement, jamais dans git). La bannière de session affiche `memory feedback: … → /memory-feedback` quand un passage est dû, et se tait une fois le gate prompt, le gate outil et l'échantillon post-correctif enregistrés, ou avec `"recall": false`.
- Retour d'usage : `/memory-feedback` (phase 4d) étiquette chaque entrée injectée depuis son transcript, avec un contrôle de 5 étiquettes par toi, et donne la précision, les manqués, les entrées bruyantes et les candidates à promotion face aux seuils de `lib/memory-feedback/thresholds.ts`. Chaque passage ajoute aussi `## Advisor usage (<from> → <to>)` à FEEDBACK.md : les appels à l'advisor servis depuis le passage précédent (premier passage : tous les transcripts présents), par appelant (boucle principale ou type de sous-agent) et par modèle, avec les tokens d'entrée, tous projets confondus, sans nom de dépôt. Quand la fenêtre commence avant la rétention de 7 jours des transcripts, une ligne prévient que des appels plus anciens peuvent manquer. Deux gates : prompt à 20 demandes étiquetées ou 35 jours, outil à 10 injections (une prolongation de 35 jours, puis fermé). La précision, la précision par trigger et la part de manqués ne comptent que les lignes postérieures au correctif du gate 1 ; leurs actions attendent 20 demandes post-correctif (`Post-fix prompt asks: k/20` dans FEEDBACK.md, entrées antérieures exclues). Les données restent locales dans `~/.claude/memory-feedback/` (IDs, enums, étiquettes de dépôt et compteurs d'appels advisor seulement, jamais dans git). La bannière de session affiche `memory feedback: … → /memory-feedback` quand un passage est dû, et se tait une fois le gate prompt, le gate outil et l'échantillon post-correctif enregistrés, ou avec `"recall": false`.
- Interrupteur : `{"enabled": false}` dans `~/.claude/memory.json`, lu au premier événement d'une session : effet à la session suivante ou après `/reload-plugins`. Coupé, l'outil répond `memory capture is off (~/.claude/memory.json)`. Un fichier illisible, ou dont `enabled` n'est pas un booléen, laisse la capture active et journalise une ligne.
- Limites connues : les écritures entre processus ne sont pas vérifiées (deux sessions simultanées : la dernière écrit). Les échecs hors Bash ne sont pas captés. `-b` ne vaut drapeau de cookies de curl que si `curl` ouvre son segment de commande, après d'éventuelles affectations de variables, `sudo`, `rtk` ou `$(` : `time curl -b …` garde ses cookies. Un jeton de 40 hexadécimaux nu est gardé à dessein : il ressemble à un SHA de commit (expurgé seulement comme utilisateur d'URL). Une racine de session qui bouge avec `/cd` ou un changement de worktree cache à /capitalize les captures précédentes du même dépôt. Un `gitflow.sh finish` nu sur une branche chore compte comme un jalon. Bruit mesuré à 83 % avant le filtre de forme d'erreur sur 14 jours de transcripts de ce dépôt ; le filtre garde tous les échantillons utiles et réduit les captures des deux tiers (`.claude/tasks/plans/2026-11-03-memory-capture-spike/NOISE.md`).
+242
View File
@@ -0,0 +1,242 @@
import { afterAll, expect, test } from 'bun:test'
import * as fs from 'node:fs'
import * as os from 'node:os'
import * as path from 'node:path'
import {
MAIN, UNKNOWN_AGENT, advisorSection, advisorUsage, callerOf, compact,
renderAdvisor, type Tally,
} from './advisor.ts'
const dirs: string[] = []
afterAll(() => dirs.forEach(d => fs.rmSync(d, { recursive: true })))
const mk = () => {
const d = fs.mkdtempSync(path.join(os.tmpdir(), 'mf-adv-'))
dirs.push(d)
return d
}
const DAY = 86400000
const FROM = Date.parse('2026-01-01T00:00:00Z')
const TO = FROM + 10 * DAY
const SLUG = '-home-secret-slug'
const iso = (ms: number) => new Date(FROM + ms).toISOString()
const adv = (input: number, model = 'claude-fable-5-2') =>
({ type: 'advisor_message', model, input_tokens: input })
const row = (ms: number, id: string, its: object[], type = 'assistant') =>
({ type, timestamp: iso(ms), message: { id, usage: { iterations: its } } })
const lines = (recs: object[]) =>
recs.map(r => JSON.stringify(r)).join('\n') + '\n'
/** A projects dir with one project dir; returns both paths. */
function fixture() {
const projects = mk()
const proj = path.join(projects, SLUG)
fs.mkdirSync(proj)
return { projects, proj }
}
const put = (file: string, recs: object[]) =>
fs.writeFileSync(file, lines(recs))
const subDir = (proj: string, session = 's1') => {
const d = path.join(proj, session, 'subagents')
fs.mkdirSync(d, { recursive: true })
return d
}
const only = (t: Tally) => t.uses.map(u =>
[u.caller, u.model, u.calls, u.input])
test('1: one main row, duplicate rows deduped by message.id', () => {
const { projects, proj } = fixture()
put(path.join(proj, 's1.jsonl'), [
row(10, 'm1', [adv(1500)]), row(10, 'm1', [adv(1500)])])
expect(only(advisorUsage(projects, FROM, TO)))
.toEqual([[MAIN, 'claude-fable-5-2', 1, 1500]])
})
test('2: rows outside (from, to] are not counted', () => {
const { projects, proj } = fixture()
put(path.join(proj, 's1.jsonl'), [row(0, 'a', [adv(1)]),
row(-5, 'b', [adv(1)]), row(11 * DAY, 'c', [adv(1)]),
row(1000, 'd', [adv(7)])])
expect(only(advisorUsage(projects, FROM, TO))[0]?.[3]).toBe(7)
})
test('3: a user record and a torn line are ignored', () => {
const { projects, proj } = fixture()
fs.writeFileSync(path.join(proj, 's1.jsonl'),
lines([row(10, 'u', [adv(5)], 'user')]) + '{"torn advisor_message\n')
const t = advisorUsage(projects, FROM, TO)
expect(t.uses).toEqual([])
expect(t.unparsed).toBe(0)
})
test('4: two advisor iterations in one message are two calls', () => {
const { projects, proj } = fixture()
put(path.join(proj, 's1.jsonl'), [row(10, 'm', [adv(1), adv(2)])])
expect(only(advisorUsage(projects, FROM, TO)))
.toEqual([[MAIN, 'claude-fable-5-2', 2, 3]])
})
test('5: an iteration without model is `unknown`', () => {
const { projects, proj } = fixture()
const bare = { type: 'advisor_message', input_tokens: 4 }
put(path.join(proj, 's1.jsonl'), [row(10, 'm', [bare])])
expect(advisorUsage(projects, FROM, TO).uses[0]?.model).toBe('unknown')
})
/** One sub-agent transcript per entry; its meta file only when given. */
function putAgents(sub: string, metas: Record<string, string | undefined>) {
for (const [id, meta] of Object.entries(metas)) {
put(path.join(sub, `agent-${id}.jsonl`), [row(10, id, [adv(1)])])
if (meta) fs.writeFileSync(path.join(sub, `agent-${id}.meta.json`), meta)
}
}
test('6: sub-agent callers come from the meta agentType', () => {
const { projects, proj } = fixture()
putAgents(subDir(proj), {
a1: '{"agentType":"verifier"}', a2: undefined,
a3: '{"agentType":"../x"}' })
const t = advisorUsage(projects, FROM, TO)
expect(t.uses.map(u => [u.caller, u.calls]).sort())
.toEqual([['verifier', 1], [UNKNOWN_AGENT, 2]].sort())
})
test('7: a transcript with mtime <= from is skipped', () => {
const { projects, proj } = fixture()
const f = path.join(proj, 's1.jsonl')
put(f, [row(10, 'm', [adv(1)])])
fs.utimesSync(f, new Date(FROM - 1000), new Date(FROM - 1000))
const t = advisorUsage(projects, FROM, TO)
expect(t.uses).toEqual([])
expect(t.files).toBe(0)
})
test('8: missing projects dir, and a window with no advisor row', () => {
const none = path.join(mk(), 'absent')
expect(advisorUsage(none, FROM, TO))
.toEqual({ uses: [], unparsed: 0, files: 0 })
const s = advisorSection(none, FROM, TO)
expect(s).toContain('## Advisor usage (2026-01-01 → 2026-01-11)')
expect(s).toContain('No transcript modified since 2026-01-01')
expect(s).toContain('check the path.')
const { projects, proj } = fixture()
put(path.join(proj, 's1.jsonl'), [row(10, 'm', [])])
expect(advisorSection(projects, FROM, TO))
.toContain('No advisor call in the window.')
})
test('9: render: all on disk, compact, total row, sort order', () => {
expect([999, 1500, 999950, 2500000].map(compact))
.toEqual(['999', '1.5k', '1.0M', '2.5M'])
const uses = [
{ caller: 'verifier', model: 'm', calls: 9, input: 1000 },
{ caller: MAIN, model: 'm', calls: 2, input: 1500 },
{ caller: 'planner', model: 'm', calls: 3, input: 500 }]
const t = { uses: [...uses].sort((a, b) =>
Number(b.caller === MAIN) - Number(a.caller === MAIN) ||
b.calls - a.calls), unparsed: 0, files: 3 }
const s = renderAdvisor(t, 0, TO)
expect(s.startsWith('## Advisor usage (all on disk → 2026-01-11)\n'))
.toBe(true)
expect(s).toContain('| total | | 14 | 3.0k |')
expect(s.indexOf('| main loop')).toBeLessThan(s.indexOf('| verifier'))
expect(s.indexOf('| verifier')).toBeLessThan(s.indexOf('| planner'))
expect(s.endsWith('\n')).toBe(true)
})
test('9b: advisorUsage sorts main loop first, then calls desc', () => {
const { projects, proj } = fixture()
put(path.join(proj, 's1.jsonl'), [row(10, 'm', [adv(1)])])
const sub = subDir(proj)
put(path.join(sub, 'agent-a.jsonl'),
[row(10, 'x', [adv(1)]), row(20, 'y', [adv(1)])])
fs.writeFileSync(path.join(sub, 'agent-a.meta.json'),
'{"agentType":"verifier"}')
expect(advisorUsage(projects, FROM, TO).uses.map(u => u.caller))
.toEqual([MAIN, 'verifier'])
})
test('10: the section holds no project slug or path', () => {
const { projects, proj } = fixture()
put(path.join(proj, 's1.jsonl'), [row(10, 'm', [adv(1)])])
const s = advisorSection(projects, FROM, TO)
expect(s).toContain('| main loop |')
expect(s).not.toContain('secret-slug')
expect(s).not.toContain(projects)
})
test('11: an advisor block with no iteration is not a call', () => {
const { projects, proj } = fixture()
const blockOnly = { type: 'assistant', timestamp: iso(10), message: {
id: 'm', content: [{ type: 'server_tool_use', name: 'advisor' }],
usage: { iterations: [] } } }
fs.writeFileSync(path.join(proj, 's1.jsonl'), lines([blockOnly]) +
'{"type":"assistant","note":"advisor_message"}\n')
const t = advisorUsage(projects, FROM, TO)
expect(t.uses).toEqual([])
expect(t.unparsed).toBe(0)
})
const drift = { type: 'assistant', timestamp: iso(10), message: {
id: 'd', usage: { advisor_message: { input_tokens: 1 } } } }
test('12: shape drift is counted and replaces the plain zero', () => {
const { projects, proj } = fixture()
put(path.join(proj, 's1.jsonl'), [drift])
const t = advisorUsage(projects, FROM, TO)
expect(t.unparsed).toBe(1)
const s = renderAdvisor(t, FROM, TO)
expect(s).toContain('1 transcript lines mention advisor_message, none ' +
'parsed: check the transcript shape.')
expect(s).not.toContain('No advisor call')
})
test('13: the retention note appears past 7 calendar days only', () => {
const t: Tally = { uses: [], unparsed: 0, files: 1 }
expect(renderAdvisor(t, TO - 10 * DAY, TO)).toContain(
'Transcripts kept 7 d: calls before 2026-01-04 may be missing.')
expect(renderAdvisor(t, TO - 3 * DAY, TO)).not.toContain('Transcripts kept')
expect(renderAdvisor(t, 0, TO)).not.toContain('Transcripts kept')
const to = TO + 17 * 3600000 // a weekly run lands mid-day, not at 00:00
expect(renderAdvisor(t, to - 7 * DAY - 5 * 3600000, to))
.not.toContain('Transcripts kept')
})
test('14: an unreadable sub-directory is skipped, the rest counts', () => {
const { projects, proj } = fixture()
put(path.join(proj, 's1.jsonl'), [row(10, 'm', [adv(1)])])
const sub = subDir(proj, 's2')
put(path.join(sub, 'agent-a.jsonl'), [row(10, 'z', [adv(1)])])
const locked = path.join(proj, 's2')
fs.chmodSync(locked, 0o000)
try {
const t = advisorUsage(projects, FROM, TO)
if (process.getuid?.() !== 0) expect(only(t))
.toEqual([[MAIN, 'claude-fable-5-2', 1, 1]])
} finally {
fs.chmodSync(locked, 0o700)
}
})
test('15: a hand-edited from never throws', () => {
const none = path.join(mk(), 'absent')
for (const bad of [Infinity, 1e17, NaN]) {
expect(advisorSection(none, bad, TO)).toContain('(all on disk → ')
}
})
test('16: drift under a table prints the table then the drift line', () => {
const { projects, proj } = fixture()
put(path.join(proj, 's1.jsonl'), [row(10, 'm', [adv(1)]), drift])
const s = advisorSection(projects, FROM, TO)
expect(s.indexOf('| total |')).toBeGreaterThan(0)
expect(s.indexOf('none parsed')).toBeGreaterThan(s.indexOf('| total |'))
})
test('17: a symlinked meta file is not trusted', () => {
const { proj } = fixture()
const sub = subDir(proj)
const real = path.join(mk(), 'meta.json')
fs.writeFileSync(real, '{"agentType":"verifier"}')
fs.symlinkSync(real, path.join(sub, 'agent-a.meta.json'))
expect(callerOf(path.join(sub, 'agent-a.jsonl'))).toBe(UNKNOWN_AGENT)
})
+211
View File
@@ -0,0 +1,211 @@
// lib/memory-feedback/advisor.ts: advisor usage read from Claude Code
// transcripts (Run C, LRN-295): served calls and input tokens per caller
// and model, deduped by message.id, for the FEEDBACK.md section.
import * as fs from 'node:fs'
import * as path from 'node:path'
export type Use = {
caller: string; model: string; calls: number; input: number
}
export type Tally = { uses: Use[]; unparsed: number; files: number }
export type Source = { file: string; caller: string }
type Obj = Record<string, unknown>
type Acc = { seen: Set<string>; uses: Map<string, Use>; unparsed: number }
export const MAIN = 'main loop'
export const UNKNOWN_AGENT = 'sub-agent (unknown)'
export const RETENTION_DAYS = 7
const DAY_MS = 86400000
const META_MAX = 65536
const MARK = 'advisor_message'
const AGENT_RE = /^[A-Za-z0-9_.:-]{1,64}$/
const MODEL_RE = /^[A-Za-z0-9._:-]{1,80}$/
const UNAVAILABLE = 'Unavailable: transcripts unreadable.'
const day = (ms: number) => new Date(ms).toISOString().slice(0, 10)
const since = (from: number) => from === 0 ? 'all on disk' : day(from)
const heading = (from: number, to: number) =>
`## Advisor usage (${since(from)} → ${day(to)})`
const obj = (v: unknown): Obj | undefined =>
typeof v === 'object' && v !== null && !Array.isArray(v)
? v as Obj : undefined
function lstatOf(p: string): fs.Stats | undefined {
try {
return fs.lstatSync(p)
} catch {
return undefined
}
}
/** Directory entries; [] when unreadable or gone, never throws. */
function listDir(dir: string): string[] {
try {
return fs.readdirSync(dir)
} catch {
return []
}
}
const isRealDir = (p: string) => lstatOf(p)?.isDirectory() === true
/** Regular `<prefix>*.jsonl` files of `dir` newer than `from`. */
function jsonlIn(
dir: string, prefix: string, from: number, callerOf: (f: string) => string,
): Source[] {
return listDir(dir)
.filter(n => n.startsWith(prefix) && n.endsWith('.jsonl'))
.map(n => path.join(dir, n))
.flatMap(file => {
const st = lstatOf(file)
return st?.isFile() && st.mtimeMs > from
? [{ file, caller: callerOf(file) }] : []
})
}
/** Every transcript under `projects` modified after `from`, with its
* caller: main threads and `<session>/subagents/agent-*.jsonl`. */
export function transcriptsOf(projects: string, from: number): Source[] {
return listDir(projects).map(n => path.join(projects, n))
.filter(isRealDir).flatMap(proj => {
const subs = listDir(proj).map(n => path.join(proj, n))
.filter(isRealDir).map(s => path.join(s, 'subagents'))
.filter(isRealDir)
return [...jsonlIn(proj, '', from, () => MAIN),
...subs.flatMap(s => jsonlIn(s, 'agent-', from, callerOf))]
})
}
/** The agent type of a sub-agent transcript, from its sibling meta file;
* UNKNOWN_AGENT for anything missing, odd or oversized. */
export function callerOf(jsonl: string): string {
const meta = jsonl.replace(/\.jsonl$/, '') + '.meta.json'
const st = lstatOf(meta)
if (!st?.isFile() || st.size > META_MAX) return UNKNOWN_AGENT
try {
const type = obj(JSON.parse(fs.readFileSync(meta, 'utf8')))?.agentType
return typeof type === 'string' && AGENT_RE.test(type)
? type : UNKNOWN_AGENT
} catch {
return UNKNOWN_AGENT
}
}
/** The assistant messages of a file that mention advisor_message and
* fall in (from, to]; [] for an unreadable file. */
function windowed(file: string, from: number, to: number): Obj[] {
let text: string
try {
text = fs.readFileSync(file, 'utf8')
} catch {
return []
}
return text.split('\n').filter(l => l.includes(MARK)).flatMap(line => {
try {
const rec = obj(JSON.parse(line))
const at = Date.parse(String(rec?.timestamp))
const msg = obj(rec?.message)
return rec?.type === 'assistant' && msg !== undefined &&
at > from && at <= to ? [msg] : []
} catch {
return []
}
})
}
function bump(uses: Map<string, Use>, caller: string, it: Obj): void {
const model = typeof it.model === 'string' && MODEL_RE.test(it.model)
? it.model : 'unknown'
const key = `${caller}\t${model}`
const use = uses.get(key) ?? { caller, model, calls: 0, input: 0 }
use.calls += 1
use.input += Number.isSafeInteger(it.input_tokens) &&
(it.input_tokens as number) >= 0 ? it.input_tokens as number : 0
uses.set(key, use)
}
/** Counts one message once: its served advisor iterations, or a drift. */
function tallyOne(msg: Obj, caller: string, acc: Acc): void {
const id = msg.id
if (typeof id !== 'string' || id === '' || acc.seen.has(id)) return
acc.seen.add(id)
const its = obj(msg.usage)?.iterations
const calls = (Array.isArray(its) ? its : []).map(obj)
.filter((i): i is Obj => i?.type === MARK)
calls.forEach(it => bump(acc.uses, caller, it))
if (calls.length === 0 &&
JSON.stringify(msg.usage ?? null).includes(MARK)) acc.unparsed += 1
}
const byUse = (a: Use, b: Use) =>
Number(b.caller === MAIN) - Number(a.caller === MAIN) ||
b.calls - a.calls || a.caller.localeCompare(b.caller) ||
a.model.localeCompare(b.model)
/** Served advisor calls and input tokens in (from, to], per caller and
* model, over every transcript under `projects`. */
export function advisorUsage(
projects: string, from: number, to: number,
): Tally {
const sources = transcriptsOf(projects, from)
const acc: Acc = { seen: new Set(), uses: new Map(), unparsed: 0 }
for (const s of sources) {
windowed(s.file, from, to).forEach(m => tallyOne(m, s.caller, acc))
}
return { uses: [...acc.uses.values()].sort(byUse),
unparsed: acc.unparsed, files: sources.length }
}
/** 999 → `999`, 1500 → `1.5k`, 2500000 → `2.5M`. */
export function compact(n: number): string {
if (n < 1000) return String(n)
return n < 999950 ? `${(n / 1e3).toFixed(1)}k` : `${(n / 1e6).toFixed(1)}M`
}
function table(uses: Use[]): string {
const sum = (f: (u: Use) => number) => uses.reduce((a, u) => a + f(u), 0)
const rows = uses.map(u =>
`| ${u.caller} | ${u.model} | ${u.calls} | ${compact(u.input)} |`)
return ['| caller | model | calls | input tok |', '|---|---|---|---|',
...rows, `| total | | ${sum(u => u.calls)} | ` +
`${compact(sum(u => u.input))} |`].join('\n')
}
function bodyOf(t: Tally, from: number): string {
const drift = `${t.unparsed} transcript lines mention ${MARK}, none ` +
'parsed: check the transcript shape.'
if (t.uses.length > 0) {
return t.unparsed > 0 ? `${table(t.uses)}\n\n${drift}` : table(t.uses)
}
if (t.files === 0) {
return `No transcript modified since ${since(from)} under the ` +
'projects dir: check the path.'
}
return t.unparsed > 0 ? drift : 'No advisor call in the window.'
}
/** The `## Advisor usage` markdown section, ending with a newline. */
export function renderAdvisor(t: Tally, from: number, to: number): string {
const blocks = [heading(from, to)]
const cut = day(to - RETENTION_DAYS * DAY_MS)
if (from > 0 && cut > day(from)) {
blocks.push(`Transcripts kept ${RETENTION_DAYS} d: calls before ` +
`${cut} may be missing.`)
}
blocks.push(bodyOf(t, from))
return blocks.join('\n\n') + '\n'
}
/** The section for FEEDBACK.md; a fixed Unavailable text on any failure
* (an error message could carry a path). Never throws. */
export function advisorSection(
projects: string, from: number, to: number,
): string {
const f = Number.isFinite(from) && from >= 0 && from <= to ? from : 0
try {
return renderAdvisor(advisorUsage(projects, f, to), f, to)
} catch {
return `${heading(f, to)}\n\n${UNAVAILABLE}\n`
}
}
+3
View File
@@ -109,6 +109,9 @@ test('R5: apply writes labels, state, FEEDBACK; a second apply refuses',
const fb = apply(work, p, NOW)
expect(fb).toContain('Prompt gate: open (1/20)')
expect(fb).toContain('Lost records: 1')
expect(fb).toContain('## Advisor usage')
expect(fs.readFileSync(path.join(p.out, 'FEEDBACK.md'), 'utf8'))
.toContain('## Advisor usage')
expect(fb).not.toContain('design the reader gate')
const labels = fs.readFileSync(path.join(p.out, 'labels.tsv'), 'utf8')
expect(labels).not.toContain('design')
+3 -1
View File
@@ -16,6 +16,7 @@ import {
tagOf, writeSafe, writeState, type Draft, type Item, type Row,
type State,
} from './labels.ts'
import { advisorSection } from './advisor.ts'
import { PROMPT_GATE_N } from './thresholds.ts'
import {
clockStart, gateVerdict, metricsOf, renderFeedback, type Metrics,
@@ -301,7 +302,8 @@ function commit(
assertFresh(items, labelsOf(p))
const { fresh, m, v } = projected(items, drafts, p, now)
const next = nextState(readState(stateOf(p)), m, v, now)
const feedback = renderFeedback(m, v, now)
const feedback = renderFeedback(m, v, now) + '\n' +
advisorSection(p.projects, readState(stateOf(p)).lastRun, now)
if (fresh.length > 0) appendRows(labelsOf(p), fresh)
writeState(stateOf(p), next.state)
writeSafe(path.join(p.out, 'FEEDBACK.md'), feedback)
+5 -3
View File
@@ -8,8 +8,8 @@ description: Run only when the user asks for it ("memory feedback", "label recal
Phase 4d (BDR-125). The memory mod logs every recall ask; transcripts live
7 days, so this runs weekly. Claude labels, the user spot-checks. Redacted
text stays in the scratchpad; `~/.claude/memory-feedback/` keeps IDs, enums
and repo tags only, never in git.
text stays in the scratchpad; `~/.claude/memory-feedback/` keeps IDs, enums,
repo tags and advisor call counts only, never in git.
## STEP 1 — WORKSHEET
@@ -73,7 +73,9 @@ MFD="$(cd -P "$HOME/.claude/lib" && pwd)/memory-feedback"
rc 1 `draft refused: <reason>` → fix the draft, rerun. It prints
FEEDBACK.md: show the gate lines (`Post-fix prompt asks: k/20` included:
actions wait for 20 post-fix asks) and the Actions list to the user.
actions wait for 20 post-fix asks), the Actions list and the
`## Advisor usage` table (advisor calls since the last run per caller and
served model) to the user.
## STEP 6 — A GATE REACHED FOR THE FIRST TIME