chore(memory): BDR-137 + LRN-258..261 + BLK-041 + EVAL-057 — ship-feature phase 4b memory capture

This commit is contained in:
bchanot committed 2026-11-04 15:26:01 +01:00
1 parent 181bcfc7f2
commit 08e36dcc0b
5 files changed
+44

No files matched your search

+6
View File
@@ -358,3 +358,9 @@ rules:
- **Real cause**: destructive-shaped text in one call or one brief; the classifier judges the dispatch prompt, the rules judge each command.
- **Solution**: split build steps (implementer) from state steps (main loop, after the user's explicit approval); one `rm` of literal names per call; `mv` to the scratch dir instead of `rmdir`/`rm -r`; never retry a refused command through a wrapper. Cost: three extra user gates, no data loss.
- **Status**: resolved 2026-11-02.
## BLK-041 — gitleaks 8.16 pre-commit ignores .gitleaks.toml allowlists: secret-shaped fixtures built at runtime [resolved] (2026-11-04)
- **Friction**: 4b spec quoted the fake AWS key and test fixtures held `AKIA…`-shaped literals; the pre-commit hook refused: `gitleaks` found leaks although `.gitleaks.toml` `[[allowlists]]` named them (gitleaks 8.16).
- **Real cause**: the installed gitleaks does not honour the allowlist block under the hook's invocation (rules compiled in; allowlist silently ignored).
- **Solution**: secret-shaped fixtures built at runtime by concatenation (`'AKIA' + 'I'.repeat(16)`, `'eyJ' + 'a'.repeat(20) + …`), expected strings stay literal (LRN-237); spec and NOISE.md quote shapes by reference, never a literal.
- **Status**: resolved 2026-11-03.
+8
View File
@@ -1487,3 +1487,11 @@ Supersedes [[BDR-030]].
- **Alternatives rejected**: `$.ui.ask` mod gate, phrase-list gate (challenges 2026-10-25); 16 routing rows for the guard (run slot drops mid-run, would open first-use + ask-tier dialogs); `git -C * merge *` (asks on any commit text with " merge ", sub-agent ask refused); quoted finish rule (matcher ignores quote in pattern); `gitflow_finish*`, `bash -s*` (bench RAN); `Bash(/*)` (kills `*/rtk` twins); per-spawn guard.
- **Residuals**: Bash write forms outside guard; latch reset on register.ts reload; sourced `gitflow_finish`, `git -C <dir> merge`; `rtk read -n .env` flag-first (security MEDIUM, TODO); code-limits TMPDIR state dir ownership unchecked; silent in this repo (code-origin foreign).
- **Gates**: plan r1 correctness FATAL(8) robustness FATAL(5) → r2 → confirmation CONCERNS(4) → r3; user interactive ask check; SDD 11 tasks; final opus review With fixes → fix wave; GATE 0 ABANDONED(2) user-accepted; verifier ECARTS(6) → ECARTS(2) human-gated; security PASS (1 MEDIUM, 3 LOW). Refs [[BDR-125]], [[BDR-068]], [[BDR-123]], [[BLK-018]], [[LRN-256]], [[LRN-257]].
## BDR-137 — Phase 4b memory capture mod: error-shaped Bash failures redacted into a per-project store, 4-gram index per call, one note per milestone, memory_pending (2026-11-04)
- **Context**: BDR-125 phase 4b, `feature/memory-capture` (f1d4f9d..HEAD), contract `.claude/tasks/contracts/2026-11-03-memory-capture-0100.md`, spec `docs/superpowers/specs/2026-11-03-memory-capture-design.md`, plan r3, spike `.claude/tasks/plans/2026-11-03-memory-capture-spike/REPORT.md`. Capture = weak link: 5/7 real errors of 2026-10-23 no entry.
- **Decision**: mod `mods/memory` (`memory@skills-dir`). (1) Failure = Bash `isError` text `Exit code N`; distinctive line = last line after rtk-trailer strip, must hold a letter AND an error word (`ERRORISH`, answer 2-amended after 83 % measured noise); permission/guard/tool_use texts never captured; sub-agent failures marked. (2) Redact BEFORE store: `KEY=value` (TOKEN|API_KEY|SECRET|PASSWORD|PASSWD, digits ok, quote-aware), 7 guardrails shapes, JWT first, Bearer, URL creds, `-u`/`--password`/`-b`(curl)/`--cookie`, `Authorization: Basic|Bearer|token`, vendor prefixes (sk-ant-, sk-proj-, github_pat_, glpat-, AIza, npm_), `--token|--api-key|--secret`, API-key/cookie headers, JSON/YAML password|token|secret|api[_-]?key keys, 40-hex URL user (answers 6-amended, 6-bis). (3) Store `$.store` key `p:<root>` → 3 sessions × 50 failures, 2 KiB/entry, milestones 20, red 10, 1 MiB sweep at session creation, one closure write chain. (4) Index per call from BLK/LRN backticked spans ≥ 12 (cap 300), word 4-grams, all-placeholder grams skipped (4-amended), superseded excluded (lexical reader, `shortcut:`), first ID wins. (5) Milestones main loop only: feat|fix commit (`-m` and heredoc forms), `gitflow.sh finish` ≠ `finish chore`, `make test` red→green; one `memory: <kind> <label> — n new failure mode(s)` note at next composer prompt, `st.pending` in closure, never per turn, never Stop, never sub-agent. (6) Tool `memory_pending` deferred (`ToolSearch` once), served `mcp__memory__memory_pending`, answer header / `index:` / milestones / NEW / known / earlier sessions, 20 000-char cap; consumers capitalize (`**Captured first.**`), close (alias), feat, bugfix, ship-feature, hotfix (NEW current-session non-sub-agent only), commit-change (both commit-changer dispatches, `MEMCAP_EOF_<random>` quoted heredoc body only); init-project excluded (7-amended). Off switch `~/.claude/memory.json` `{"enabled": false}` read lazily per session; unreadable or non-boolean → on + one log line.
- **Why**: engine facts (spike): refusals come back `isError` with the reason in text, no `deny` field; `$.store` per plugin, machine-wide, 4 MiB; no `$.state`; `-p` prompts origin `sdk` (note proven interactive only). Noise on the shipped rule 83 % (NOISE.md), filter keeps every useful sample, 81 → 27 captures.
- **Alternatives rejected**: classic hooks (no store, no tool); gitleaks as redaction engine (rules compiled in, allowlist ignored); per-session store (lost at /clear); index cache (stale registries); option B no sub-agent capture (loses reviewer findings); option C ship as is (70 % noise per call).
- **Residuals**: cross-process writes last-writer-wins; non-Bash failures; lexical superseded reader; 40-hex Gitea tokens kept (SHA-shaped); `user:p@ss@host` partial; `key: value` pair rule over-redacts prose (`token: expected`); session root moves with /cd or a worktree; bare finish on a chore branch counts; common-phrase 4-gram collisions (every ENOENT → known LRN-056 here) → 4c.
- **Gates**: plan r1 correctness FATAL(9) → r2, confirmation CONCERNS(5) → r3; SDD Tasks 1-6 + 8 (sonnet, 7 fix rounds), 7 and 9 controller; headless proof PASS; final opus review Approved with fixes (2 Important, 13 minors, 2 rounds); verifier ECARTS(3) → ECARTS(1) (interactive /clear half, user); security PASS ×3 (coverage rounds). 36+ commits.
+7
View File
@@ -440,3 +440,10 @@ Dogfood: 3 blind lenses attacked the v1 plan for the plan-challenge feature itse
- **Result**: plan r1 FATAL(8)+FATAL(5): bench script broken (pipe ate stdin), quoted finish form escaped the gate — both caught before any code. SDD 11 tasks, 1 fix round (Task 3). Final review: 3 Important (macOS T13, symlink whole-file report, guard undocumented) → fix wave. Verifier 1: ECARTS(6), 3 human-gated criteria written from the plan/spec rather than the contract; verifier 2: ECARTS(2) (post-merge user step, USAGE scope). Security PASS, 1 MEDIUM (`rtk read -n .env`).
- **Anomalies**: criterion 11 encoded a hypothesis (M3q ASKED) as an oracle before the bench measured it → abandoned; contract criteria 1/5/7 drifted from gated plan/spec decisions (r3 deviations not mirrored into the contract) → 3 human round-trips at the end.
- **Action**: write a measured oracle only after the measurement; when a plan revision deviates from a contract criterion, amend the criterion `[gated]` at the plan gate, not at GATE 1.
## EVAL-057 — 4b run: task reviews caught 5 Important, final opus review 2 more + 13 minors, fix rounds regressed twice, one controller ruling wrong [keep] (2026-11-04)
- **Checked**: 4b /ship-feature run quality on an 8-task mod branch (`feature/memory-capture`).
- **Method**: defects counted per stage: plan challenge, per-task sonnet reviews + scoped re-reviews, final opus review, verifier ×2 + re-judge, security gate ×3.
- **Result**: plan r1 FATAL(9) (note unprovable in `-p`) before code. Task reviews caught 5 Important (quadratic STATUS regex, heredoc commit form never matched, unguarded `fs.exists`, whole-command milestone regexes, fs.exists missing/unreadable). Final opus review still found 2 Important + 13 minors (all-placeholder grams on real registries, redaction gaps, `> \d+` letterless, success path reading the store per call). Security gate PASS ×3 but each coverage round found new shapes (cookie tail from the fix itself, `-b`, JSON affixes). Fix rounds introduced 2 regressions (quadratic JWT, cookie 400-cap), both caught by scoped re-reviews.
- **Anomalies**: controller ruling "letterless rule subsumed by ERRORISH" was wrong (`> \d+` alternative) and cost one revert; "1:1 port" claim in noise.py drifted as the TS rule grew; the verifier's second pass refused `gates.sh run` (it rewrites the contract) and re-ran the CHECKs read-only instead.
- **Action**: a ruling that removes a check lists the counterexample search over the alternatives first; reviewers measure regex cost by running; keep a scoped re-review after every fix round, a round of its own can regress; a port of a live rule carries the commit it was ported from.
+3
View File
@@ -689,3 +689,6 @@ rules:
## 2026-11-03
- Phase 3 closed, not built (user /context 2026-11-03: full 40k, dev 38.4k, web 40k; connector instr have a native lever, jit-subtractable doctrine ~1-2k). dev System tools 11.5k vs 7.6k full: tool-search threshold suspected, queued. Phase 2 merged 29624d7. Next: phase 4b capture on feature/memory-capture.
## 2026-11-04
- 2026-11-04 — 4b memory capture built on feature/memory-capture (mod, consumers, docs, noise 83 % → errorish filter; GATE 0 MET, verifier ECARTS(1) interactive half, security PASS ×3); interactive note check + merge pending user; 4c next.
+20
View File
@@ -1931,3 +1931,23 @@ Rule: when editing a doctrine file under structure locks, grep the test's lock s
- **Pattern**: headless permission bench (`claude -p --setting-sources local --settings <file>`): (1) `printf … | cd "$4" && claude -p` runs the cd in a pipe subshell and claude reads the caller's stdin (the cases file) → `cd` on its own line, every call `</dev/null`; (2) controls carry `allow Bash(*)` + `defaultMode: auto` (+ `skipAutoPermissionPrompt`), else default mode asks everything; (3) parse the stream-json `system/init` `permissionMode` per row, reject rows not `auto`; (4) verdict = DENIED ("has been denied") / ASKED ("requested permissions") / else RAN (a non-zero command exit is still RAN); (5) a rule is approved only when every row naming it hits its want and one row expects ASKED/DENIED; (6) append-only results, ids selectable, full run in background.
- **Context**: phase 2 plan r1 bench broken (correctness BLOCKER), fixed r2/r3; run 2026-11-02, 41 cases + controls, all `auto`.
- **Apply**: reuse `.claude/tasks/plans/2026-11-02-phase2-bench/run.sh` + `bench.sh` for any future rule (docker mount spellings, `rtk read * .env*` twins).
## LRN-258 — All-placeholder n-grams in a normalized error index match everything: skip grams with no content word (2026-11-04)
- **Pattern**: normalization turns values into placeholders (`#`, `<path>`, `<hex>`); a word 4-gram made only of placeholders (`# # # #`: timestamps, IP:port, version quads) indexes a registry ID and every such failure line reads `known`. Measured on this repo: `# # # #` → LRN-074, so `connect ECONNREFUSED 127.0.0.1:5432` and `Error at 2026-11-04 12:00:00.123` read known.
- **Context**: 4b final review I1 (`mods/memory/hooks/register.ts` `grams`), fixed c81a7a2 at index and match time, test IDX9. Open sibling: common-phrase grams (`no such file or directory` → LRN-056) for 4c.
- **Apply**: any n-gram index over normalized text drops grams with no content word; test with an IP span and a timestamp line against the real registries, not fixtures only.
## LRN-259 — Regex over free text: measure on 64 KiB repeats, lookbehind the start token, never \s* under /m (2026-11-04)
- **Pattern**: three quadratic or costly regex shapes landed in one branch, each caught by a reviewer who RAN it: STATUS `^\s*-\s*` under /m (3.1 s on 80k newlines: `\s*` crosses newlines → `[ \t]*`); JWT `eyJ…\.eyJ…` rescanning at every `eyJ` (1.58 s on 64 KiB → prefix `(?<![A-Za-z0-9_-])`, 2 ms); URL scheme `[a-z][a-z0-9+.-]{0,30}:\/\/` backtracking at every letter (7 ms, linear but 8× the rest → `(?<![a-z0-9+.-])`, 1 ms). A fix round introduced two of them.
- **Context**: 4b Task 4 review, final re-review, security re-scan; LRN-219 sets the 64 KiB cap, this is the method: `'x'.repeat(65536)` per suspicious rule in a bun one-liner, 64 KiB → 256 KiB must grow ×4.
- **Apply**: a TIME1-style test per regex family; a reviewer measures regex cost by running, never by reading; a lookbehind on the start token is the cheap cure for "every position is a candidate start".
## LRN-260 — Headless mod proof: claude -p haiku with allowedTools naming the mcp tool and ToolSearch, four scripted steps (2026-11-04)
- **Pattern**: headless proof of a mod with a tool: scratch repo `mktemp -d` on a feature branch, `GITFLOW_NO_PUSH=1` exported in the creating command, `claude -p "<4 steps>" --model haiku --output-format stream-json --verbose --allowedTools 'Bash(ls *)' 'Bash(git commit *)' 'mcp__memory__memory_pending' 'ToolSearch'`; prompt "exactly these four steps, one tool call each, no retry, reply with the tool result verbatim" → haiku complied first try, rc 0, $0.011; `stream-json` holds the `tool_use` records to assert the call sequence.
- **Context**: 4b Task 9 (`$CLAUDE_JOB_DIR/tmp/proof9/run.sh`); the note half cannot be proven in `-p` (prompt origin `sdk`, no composer) → interactive check by the user; the run leaves a `p:<tmp>` store key, harmless.
- **Apply**: split any mod proof into a headless capture half and an interactive UI half; name the mcp tool and `ToolSearch` in `--allowedTools`, else the tool is "not granted".
## LRN-261 — Noise measured before shipping a capture rule: python port, 14-day replay, 30 labelled samples, alternatives on the whole corpus (2026-11-04)
- **Pattern**: before shipping a rule that fires on tool output, measure its noise: sonnet ports the rule to Python (`noise.py`, stdlib, read-only, `--seed`), replays 14 days of `~/.claude/projects/<repo>/*.jsonl` + `*/subagents/*.jsonl`, prints 30 redacted samples; the controller labels useful/noise. Result 4b: 83 % noise, concentrated in sub-agents (23/25) vs main loop (2/5); candidate tightenings (`probe`, `errorish`) measured on the WHOLE corpus (81 → 60 / 27) before the gate; the user picked one, amending a gated answer.
- **Context**: 4b Task 7, LRN-047 lineage; NOISE.md holds counts and shares only (contract check forbids `excerpt:`/`command:` lines); samples stay in the job tmp.
- **Apply**: a capture or injection rule ships with its measured noise share and the numbers of the alternatives; a port claims "1:1" only while the TS rule is frozen (ours drifted twice).