chore(bench): phase 4b capture noise measured on 14 days of transcripts

This commit is contained in:
bchanot committed 2026-11-04 12:48:54 +01:00
1 parent 76013d6812
commit ced6cb96f8
2 files changed
+363

No files matched your search

@@ -0,0 +1,90 @@
# Phase 4b capture noise — 14 days of transcripts (2026-11-04)
Method (plan Task 7, LRN-047): `noise.py` (stdlib, read-only) replays every
Bash `tool_use`/`tool_result` pair of this repo's sessions modified in the
last 14 days through a 1:1 port of the mod's capture rule (`failureLine`,
`normalize`, `redact`), main-loop and sub-agent transcripts separately
(a sub-agent file is attributed to its parent session). Run:
`python3 -I noise.py --seed 42 --samples 30`. Nothing captured is written
here: counts, labels and shares only.
## Counts (default rule, as shipped on the branch)
| scope | files | captured before dedupe | after dedupe per session |
|---|---|---|---|
| main loop | 25 | 28 | 23 |
| sub-agents | 320 | 66 | 58 |
| total | 345 | 94 | 81 |
- 17 of 25 sessions captured nothing; the rest: 2 to 26 entries each.
- Letterless lines dropped: 39 (a `0` from `grep -c`, a bare `#`): the rule
removes about a third of the raw candidates on its own.
- Distinctive lines taken from an rtk recall trailer: 1 without stripping,
0 with it (the trailer strip works).
## Human labels on 30 random samples (seed 42)
A sample is **useful** when its line names a failure mode a future session
could meet again and a registry entry would help; **noise** otherwise (a
probe's expected non-zero exit, a `grep` with no match, a `diff`, a last
stdout line that is not an error at all).
| scope | samples | useful | noise |
|---|---|---|---|
| main loop | 5 | 3 | 2 |
| sub-agents | 25 | 2 | 23 |
| total | 30 | 5 (17 %) | 25 (83 %) |
Useful kinds: a shell `unexpected EOF while looking for matching` on a
nested heredoc (twice, BLK-039 territory), two shellcheck findings
(SC2016, SC2059), a line-width check (`N added lines > 80`).
Noise kinds: a command chain whose last segment is `grep`/`diff`/`test`
exiting 1 with the previous command's stdout as last line (8); probe
output such as `rc=0`, `file=`, `== before`, an ID, a size (9); a
`No such file or directory` from `ls`/`cat`/`sed` on a guessed path during
exploration (6); listing or status lines (2).
The noise sits almost entirely in sub-agents (explorers and reviewers
probing with `ls`, `grep -c`, `diff`). The main loop's captures are mostly
real.
## Tightenings measured (whole corpus, applied before dedupe)
| rule | main before/after | sub before/after | sample useful kept | sample noise kept |
|---|---|---|---|---|
| none (shipped) | 28 / 23 | 66 / 58 | 5 / 5 | 25 |
| `probe` | 23 / 18 | 48 / 40 | 5 / 5 | 15 |
| `errorish` | 14 / 11 | 24 / 16 | 5 / 5 | 7 |
| both | 14 / 11 | 22 / 14 | 5 / 5 | 7 |
- `probe`: drop an exit 1 whose last segment starts with `grep`, `egrep`,
`fgrep`, `rg`, `test`, `diff`, `cmp`, `[`, `[[`.
- `errorish`: keep only a distinctive line matching, case-insensitive,
`error|fatal|cannot|can't|no such|not found|denied|failed|failure|`
`unexpected|invalid|exception|traceback|panic|usage:|SC\d{4}|> \d+|assert`.
- The 7 noise samples `errorish` keeps: 5 `No such file or directory`
probes (4 of them sub-agents), 1 `non-fatal` banner (matches `fatal`),
1 `cannot access` on a scratch path.
## Gate input (plan Task 7 step 3: noise share > 50 % → propose)
Noise share on the shipped rule: 83 % of samples, 76 % of sub-agent
captures by volume. Above the 50 % line. Proposal for the merge gate,
the user decides (it amends gated answer 2, the capture rule):
- **A (recommended):** add the `errorish` filter to `failureLine`: a
distinctive line that holds no error-shaped word is no line (same
treatment as a letterless line). Keeps every useful sample, cuts the
answer from 81 to 27 entries over 14 days, costs one bounded regex on a
≤ 300-char line. Risk: a real failure whose last line has no keyword is
missed (none seen in the sample; the word list can grow in 4c).
- **B:** A plus no capture in sub-agents (gated answer 2 says main loop
and sub-agents): 81 → 11 entries, loses the 2 useful sub-agent samples
(shellcheck and line-width findings a reviewer hit). Not recommended:
the sub-agent mark and the hotfix exclusion already keep them out of the
way.
- **C:** ship as is; the CAPITALIZE consumers judge each NEW line. Costs
attention on ~70 % noise per call.
`probe` alone is not worth its complexity (10 of 25 noise samples).
@@ -0,0 +1,273 @@
"""Offline noise measurement of the memory mod capture rule.
1:1 port of failureLine / normalize / redact of
mods/memory/hooks/register.ts. Read-only, stdout only.
Run: python3 -I noise.py [--dir D] [--days 14] [--samples 30] [--seed N]
"""
import argparse
import json
import os
import random
import re
import time
from collections import namedtuple
DEFAULT_DIR = os.path.expanduser(
"~/.claude/projects/-home-bchanot-Documents-claude")
MAX_REGEX_INPUT = 65536
MAX_LINE = 300
MAX_COMMAND = 300
LINE_WINDOW = 1000
A = re.ASCII # JS \w \d \b are ASCII-only
# Secret-shaped sources are assembled from parts on purpose.
PEM = "-----"
PEM_KIND = "(?:[A-Z]+ )?PRIVATE KEY"
REDACTIONS = [
(re.compile(r"\b([A-Za-z_]{0,60}(?:TOKEN|API_KEY|SECRET|PASSWORD|"
r"PASSWD)[A-Za-z_]{0,60})=\S+", re.I | A), r"\1=<redacted>"),
(re.compile("sk_" + "live_[0-9A-Za-z]{10,}"), "<redacted:stripe>"),
(re.compile("AK" + "IA[0-9A-Z]{16}"), "<redacted:aws>"),
(re.compile("gh[pousr]" + "_[A-Za-z0-9]{30,}"), "<redacted:github>"),
(re.compile("xox" + "[abprs]-[A-Za-z0-9-]{10,}"), "<redacted:slack>"),
(re.compile(r"SG\.[A-Za-z0-9_-]{16,}\.[A-Za-z0-9_-]{16,}"),
"<redacted:sendgrid>"),
(re.compile("sb_" + "secret_[A-Za-z0-9_-]{10,}"), "<redacted:supabase>"),
(re.compile(f"{PEM}BEGIN {PEM_KIND}{PEM}.*?(?:{PEM}END {PEM_KIND}{PEM}"
r"|\Z)", re.S), "<redacted:private-key>"),
(re.compile(r"\bBearer\s+[A-Za-z0-9._~+/=-]{8,}", A),
"Bearer <redacted>"),
(re.compile(r"([a-z][a-z0-9+.-]{0,30}://)[^\s/@:]+:[^\s/@]+@", re.I),
r"\1<redacted>@"),
]
HEX_RUN = re.compile(r"\b[0-9a-f]{7,}\b", A)
PATH = re.compile(r"(?:\.{0,2}/)?[\w.-]{1,200}(?:/[\w.-]{1,200}){1,30}", A)
EXIT_CODE = re.compile(r"^Exit code \d+", A)
RTK_TRAILER = re.compile(
r"\s*\[(?:\+\d+ hidden|full output): rtk recall [0-9a-f]+\]\Z")
Entry = namedtuple("Entry", "key line command agent code tool")
Pair = namedtuple("Pair", "command text is_error")
def redact(text):
out = text[:MAX_REGEX_INPUT]
for pattern, mask in REDACTIONS:
out = pattern.sub(mask, out)
return out
def normalize(line):
out = HEX_RUN.sub("<hex>", line.lower())
out = PATH.sub("<path>", out)
out = re.sub(r"\d+", "#", out, flags=A)
return re.sub(r"\s+", " ", out).strip()
def has_letter(line):
letters = re.sub(r"<path>|<hex>|#", "", normalize(line))
return any(c.isalpha() for c in letters)
def clean_lines(text, strip_trailer):
lines = []
for raw in text.split("\n"):
line = raw[-LINE_WINDOW:]
if strip_trailer:
line = RTK_TRAILER.sub("", line)
line = line.strip()
if line != "":
lines.append(line)
return lines
def count_trailer_pick(text, stats):
"""Counts a pick that, without stripping, would be a trailer line."""
raw = clean_lines(text, False)
if len(raw) >= 2 and RTK_TRAILER.search(raw[-1]) and has_letter(raw[-1]):
stats["rtk_would_pick"] += 1
def failure_line(text, stats):
"""The distinctive line, or None; stats counts the two drops."""
if not EXIT_CODE.match(text):
return None
count_trailer_pick(text, stats)
lines = clean_lines(text, True)
if len(lines) < 2:
return None
line = lines[-1][:MAX_LINE]
if not has_letter(line):
stats["letterless"] += 1
return None
return line
PROBES = {"grep", "egrep", "fgrep", "rg", "test", "diff", "cmp", "[", "[["}
SEGMENT_SPLIT = re.compile(r";|&&|\|\||\||\n")
ERRORISH = re.compile(
r"error|fatal|cannot|can't|no such|not found|denied|failed|failure|"
r"unexpected|invalid|exception|traceback|panic|usage:|SC\d{4}|> \d+|"
r"assert", re.I)
TIGHTEN_RULES = ("probe", "errorish")
def last_word(command):
"""First word of the last segment, env prefixes and bin dir dropped."""
parts = [p.strip() for p in SEGMENT_SPLIT.split(command) if p.strip()]
words = parts[-1].split() if parts else []
while words and re.match(r"[A-Za-z_]\w*=", words[0]):
words.pop(0)
if not words:
return ""
return re.sub(r"^/(?:usr/)?bin/", "", words[0])
def survives(entry, rules):
if "probe" in rules and entry.code == 1 and entry.tool in PROBES:
return False
return "errorish" not in rules or ERRORISH.search(entry.line) is not None
def build_entry(pair, agent, stats):
red = redact(pair.text[-MAX_REGEX_INPUT:])
line = failure_line(red, stats)
if line is None:
return None
full = redact(pair.command[:MAX_REGEX_INPUT])
code = int(EXIT_CODE.match(red).group(0).split()[2])
return Entry(normalize(line), line, full[:MAX_COMMAND], agent, code,
last_word(full))
def result_text(content):
if isinstance(content, str):
return content
if isinstance(content, list):
return "".join(b.get("text", "") for b in content
if isinstance(b, dict) and b.get("type") == "text")
return ""
def blocks_of(record):
content = (record.get("message") or {}).get("content")
if not isinstance(content, list):
return []
return [b for b in content if isinstance(b, dict)]
def pairs_of(path):
"""Bash tool_use paired with its tool_result by tool_use_id."""
calls, pairs = {}, []
with open(path, encoding="utf-8", errors="replace") as fh:
for raw in fh:
try:
record = json.loads(raw)
except ValueError:
continue
for b in blocks_of(record):
if b.get("type") == "tool_use" and b.get("name") == "Bash":
cmd = (b.get("input") or {}).get("command")
calls[b.get("id")] = str(cmd or "")
elif b.get("type") == "tool_result":
cid = b.get("tool_use_id")
if cid in calls:
pairs.append(Pair(calls[cid],
result_text(b.get("content")),
b.get("is_error") is True))
return pairs
def recent(path, cutoff):
return os.path.getmtime(path) >= cutoff
def find_files(root, days):
"""(session id, is_agent, path) of main and sub-agent transcripts."""
cutoff = time.time() - days * 86400
found = []
for name in sorted(os.listdir(root)):
full = os.path.join(root, name)
if name.endswith(".jsonl") and recent(full, cutoff):
found.append((name[:-6], False, full))
sub = os.path.join(full, "subagents")
if os.path.isdir(sub):
for f in sorted(os.listdir(sub)):
p = os.path.join(sub, f)
if f.endswith(".jsonl") and recent(p, cutoff):
found.append((name, True, p))
return found
def new_session():
return {"before": {False: 0, True: 0}, "after": {False: 0, True: 0},
"seen": set(), "kept": []}
def measure(files, stats, rules=()):
"""Per session counts; dedupe by key pooled over main and sub-agents."""
sessions = {}
for sid, agent, path in sorted(files, key=lambda f: f[1]):
s = sessions.setdefault(sid, new_session())
for pair in pairs_of(path):
if not pair.is_error:
continue
entry = build_entry(pair, agent, stats)
if entry is None or not survives(entry, rules):
continue
s["before"][agent] += 1
if entry.key not in s["seen"]:
s["seen"].add(entry.key)
s["after"][agent] += 1
s["kept"].append(entry)
return sessions
def print_counts(sessions, stats):
tot = {k: {False: 0, True: 0} for k in ("before", "after")}
print("session main(before/after) sub(before/after)")
for sid, s in sorted(sessions.items()):
print(f"{sid[:8]} {s['before'][False]:>6}/{s['after'][False]:<6}"
f" {s['before'][True]:>6}/{s['after'][True]:<6}")
for k in tot:
for agent in (False, True):
tot[k][agent] += s[k][agent]
for agent, label in ((False, "main loop"), (True, "sub-agents")):
print(f"TOTAL {label}: before dedupe {tot['before'][agent]}, "
f"after dedupe {tot['after'][agent]}")
print(f"rtk-trailer picks without stripping: {stats['rtk_would_pick']}")
print("rtk-trailer picks after stripping: 0 (trailer removed per line)")
print(f"letterless dropped: {stats['letterless']}")
def print_samples(sessions, count, seed):
kept = [e for s in sessions.values() for e in s["kept"]]
rng = random.Random(seed)
picked = rng.sample(kept, min(count, len(kept)))
print(f"\n{len(picked)} samples of {len(kept)} captured entries")
for e in picked:
cmd = redact(e.command)[:200].replace("\n", " ")
who = "sub" if e.agent else "main"
print(f"- [{who}] key: {redact(e.key)}\n line: {redact(e.line)}"
f"\n cmd: {cmd}")
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--dir", default=DEFAULT_DIR)
ap.add_argument("--days", type=int, default=14)
ap.add_argument("--samples", type=int, default=30)
ap.add_argument("--seed", type=int, default=None)
ap.add_argument("--tighten", action="append", default=[],
choices=TIGHTEN_RULES)
args = ap.parse_args()
stats = {"letterless": 0, "rtk_would_pick": 0}
files = find_files(args.dir, args.days)
print(f"files: {len(files)} ({sum(1 for f in files if f[1])} sub-agent)")
sessions = measure(files, stats, args.tighten)
print_counts(sessions, stats)
print_samples(sessions, args.samples, args.seed)
if __name__ == "__main__":
main()