chore(bench): phase 4b capture noise measured on 14 days of transcripts
This commit is contained in:
1 parent
76013d6812
commit
ced6cb96f8
2 files changed
+363
No files matched your search
@@ -0,0 +1,90 @@
|
||||
# Phase 4b capture noise — 14 days of transcripts (2026-11-04)
|
||||
|
||||
Method (plan Task 7, LRN-047): `noise.py` (stdlib, read-only) replays every
|
||||
Bash `tool_use`/`tool_result` pair of this repo's sessions modified in the
|
||||
last 14 days through a 1:1 port of the mod's capture rule (`failureLine`,
|
||||
`normalize`, `redact`), main-loop and sub-agent transcripts separately
|
||||
(a sub-agent file is attributed to its parent session). Run:
|
||||
`python3 -I noise.py --seed 42 --samples 30`. Nothing captured is written
|
||||
here: counts, labels and shares only.
|
||||
|
||||
## Counts (default rule, as shipped on the branch)
|
||||
|
||||
| scope | files | captured before dedupe | after dedupe per session |
|
||||
|---|---|---|---|
|
||||
| main loop | 25 | 28 | 23 |
|
||||
| sub-agents | 320 | 66 | 58 |
|
||||
| total | 345 | 94 | 81 |
|
||||
|
||||
- 17 of 25 sessions captured nothing; the rest: 2 to 26 entries each.
|
||||
- Letterless lines dropped: 39 (a `0` from `grep -c`, a bare `#`): the rule
|
||||
removes about a third of the raw candidates on its own.
|
||||
- Distinctive lines taken from an rtk recall trailer: 1 without stripping,
|
||||
0 with it (the trailer strip works).
|
||||
|
||||
## Human labels on 30 random samples (seed 42)
|
||||
|
||||
A sample is **useful** when its line names a failure mode a future session
|
||||
could meet again and a registry entry would help; **noise** otherwise (a
|
||||
probe's expected non-zero exit, a `grep` with no match, a `diff`, a last
|
||||
stdout line that is not an error at all).
|
||||
|
||||
| scope | samples | useful | noise |
|
||||
|---|---|---|---|
|
||||
| main loop | 5 | 3 | 2 |
|
||||
| sub-agents | 25 | 2 | 23 |
|
||||
| total | 30 | 5 (17 %) | 25 (83 %) |
|
||||
|
||||
Useful kinds: a shell `unexpected EOF while looking for matching` on a
|
||||
nested heredoc (twice, BLK-039 territory), two shellcheck findings
|
||||
(SC2016, SC2059), a line-width check (`N added lines > 80`).
|
||||
|
||||
Noise kinds: a command chain whose last segment is `grep`/`diff`/`test`
|
||||
exiting 1 with the previous command's stdout as last line (8); probe
|
||||
output such as `rc=0`, `file=`, `== before`, an ID, a size (9); a
|
||||
`No such file or directory` from `ls`/`cat`/`sed` on a guessed path during
|
||||
exploration (6); listing or status lines (2).
|
||||
|
||||
The noise sits almost entirely in sub-agents (explorers and reviewers
|
||||
probing with `ls`, `grep -c`, `diff`). The main loop's captures are mostly
|
||||
real.
|
||||
|
||||
## Tightenings measured (whole corpus, applied before dedupe)
|
||||
|
||||
| rule | main before/after | sub before/after | sample useful kept | sample noise kept |
|
||||
|---|---|---|---|---|
|
||||
| none (shipped) | 28 / 23 | 66 / 58 | 5 / 5 | 25 |
|
||||
| `probe` | 23 / 18 | 48 / 40 | 5 / 5 | 15 |
|
||||
| `errorish` | 14 / 11 | 24 / 16 | 5 / 5 | 7 |
|
||||
| both | 14 / 11 | 22 / 14 | 5 / 5 | 7 |
|
||||
|
||||
- `probe`: drop an exit 1 whose last segment starts with `grep`, `egrep`,
|
||||
`fgrep`, `rg`, `test`, `diff`, `cmp`, `[`, `[[`.
|
||||
- `errorish`: keep only a distinctive line matching, case-insensitive,
|
||||
`error|fatal|cannot|can't|no such|not found|denied|failed|failure|`
|
||||
`unexpected|invalid|exception|traceback|panic|usage:|SC\d{4}|> \d+|assert`.
|
||||
- The 7 noise samples `errorish` keeps: 5 `No such file or directory`
|
||||
probes (4 of them sub-agents), 1 `non-fatal` banner (matches `fatal`),
|
||||
1 `cannot access` on a scratch path.
|
||||
|
||||
## Gate input (plan Task 7 step 3: noise share > 50 % → propose)
|
||||
|
||||
Noise share on the shipped rule: 83 % of samples, 76 % of sub-agent
|
||||
captures by volume. Above the 50 % line. Proposal for the merge gate,
|
||||
the user decides (it amends gated answer 2, the capture rule):
|
||||
|
||||
- **A (recommended):** add the `errorish` filter to `failureLine`: a
|
||||
distinctive line that holds no error-shaped word is no line (same
|
||||
treatment as a letterless line). Keeps every useful sample, cuts the
|
||||
answer from 81 to 27 entries over 14 days, costs one bounded regex on a
|
||||
≤ 300-char line. Risk: a real failure whose last line has no keyword is
|
||||
missed (none seen in the sample; the word list can grow in 4c).
|
||||
- **B:** A plus no capture in sub-agents (gated answer 2 says main loop
|
||||
and sub-agents): 81 → 11 entries, loses the 2 useful sub-agent samples
|
||||
(shellcheck and line-width findings a reviewer hit). Not recommended:
|
||||
the sub-agent mark and the hotfix exclusion already keep them out of the
|
||||
way.
|
||||
- **C:** ship as is; the CAPITALIZE consumers judge each NEW line. Costs
|
||||
attention on ~70 % noise per call.
|
||||
|
||||
`probe` alone is not worth its complexity (10 of 25 noise samples).
|
||||
@@ -0,0 +1,273 @@
|
||||
"""Offline noise measurement of the memory mod capture rule.
|
||||
|
||||
1:1 port of failureLine / normalize / redact of
|
||||
mods/memory/hooks/register.ts. Read-only, stdout only.
|
||||
Run: python3 -I noise.py [--dir D] [--days 14] [--samples 30] [--seed N]
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import time
|
||||
from collections import namedtuple
|
||||
|
||||
DEFAULT_DIR = os.path.expanduser(
|
||||
"~/.claude/projects/-home-bchanot-Documents-claude")
|
||||
MAX_REGEX_INPUT = 65536
|
||||
MAX_LINE = 300
|
||||
MAX_COMMAND = 300
|
||||
LINE_WINDOW = 1000
|
||||
A = re.ASCII # JS \w \d \b are ASCII-only
|
||||
|
||||
# Secret-shaped sources are assembled from parts on purpose.
|
||||
PEM = "-----"
|
||||
PEM_KIND = "(?:[A-Z]+ )?PRIVATE KEY"
|
||||
REDACTIONS = [
|
||||
(re.compile(r"\b([A-Za-z_]{0,60}(?:TOKEN|API_KEY|SECRET|PASSWORD|"
|
||||
r"PASSWD)[A-Za-z_]{0,60})=\S+", re.I | A), r"\1=<redacted>"),
|
||||
(re.compile("sk_" + "live_[0-9A-Za-z]{10,}"), "<redacted:stripe>"),
|
||||
(re.compile("AK" + "IA[0-9A-Z]{16}"), "<redacted:aws>"),
|
||||
(re.compile("gh[pousr]" + "_[A-Za-z0-9]{30,}"), "<redacted:github>"),
|
||||
(re.compile("xox" + "[abprs]-[A-Za-z0-9-]{10,}"), "<redacted:slack>"),
|
||||
(re.compile(r"SG\.[A-Za-z0-9_-]{16,}\.[A-Za-z0-9_-]{16,}"),
|
||||
"<redacted:sendgrid>"),
|
||||
(re.compile("sb_" + "secret_[A-Za-z0-9_-]{10,}"), "<redacted:supabase>"),
|
||||
(re.compile(f"{PEM}BEGIN {PEM_KIND}{PEM}.*?(?:{PEM}END {PEM_KIND}{PEM}"
|
||||
r"|\Z)", re.S), "<redacted:private-key>"),
|
||||
(re.compile(r"\bBearer\s+[A-Za-z0-9._~+/=-]{8,}", A),
|
||||
"Bearer <redacted>"),
|
||||
(re.compile(r"([a-z][a-z0-9+.-]{0,30}://)[^\s/@:]+:[^\s/@]+@", re.I),
|
||||
r"\1<redacted>@"),
|
||||
]
|
||||
HEX_RUN = re.compile(r"\b[0-9a-f]{7,}\b", A)
|
||||
PATH = re.compile(r"(?:\.{0,2}/)?[\w.-]{1,200}(?:/[\w.-]{1,200}){1,30}", A)
|
||||
EXIT_CODE = re.compile(r"^Exit code \d+", A)
|
||||
RTK_TRAILER = re.compile(
|
||||
r"\s*\[(?:\+\d+ hidden|full output): rtk recall [0-9a-f]+\]\Z")
|
||||
|
||||
Entry = namedtuple("Entry", "key line command agent code tool")
|
||||
Pair = namedtuple("Pair", "command text is_error")
|
||||
|
||||
|
||||
def redact(text):
|
||||
out = text[:MAX_REGEX_INPUT]
|
||||
for pattern, mask in REDACTIONS:
|
||||
out = pattern.sub(mask, out)
|
||||
return out
|
||||
|
||||
|
||||
def normalize(line):
|
||||
out = HEX_RUN.sub("<hex>", line.lower())
|
||||
out = PATH.sub("<path>", out)
|
||||
out = re.sub(r"\d+", "#", out, flags=A)
|
||||
return re.sub(r"\s+", " ", out).strip()
|
||||
|
||||
|
||||
def has_letter(line):
|
||||
letters = re.sub(r"<path>|<hex>|#", "", normalize(line))
|
||||
return any(c.isalpha() for c in letters)
|
||||
|
||||
|
||||
def clean_lines(text, strip_trailer):
|
||||
lines = []
|
||||
for raw in text.split("\n"):
|
||||
line = raw[-LINE_WINDOW:]
|
||||
if strip_trailer:
|
||||
line = RTK_TRAILER.sub("", line)
|
||||
line = line.strip()
|
||||
if line != "":
|
||||
lines.append(line)
|
||||
return lines
|
||||
|
||||
|
||||
def count_trailer_pick(text, stats):
|
||||
"""Counts a pick that, without stripping, would be a trailer line."""
|
||||
raw = clean_lines(text, False)
|
||||
if len(raw) >= 2 and RTK_TRAILER.search(raw[-1]) and has_letter(raw[-1]):
|
||||
stats["rtk_would_pick"] += 1
|
||||
|
||||
|
||||
def failure_line(text, stats):
|
||||
"""The distinctive line, or None; stats counts the two drops."""
|
||||
if not EXIT_CODE.match(text):
|
||||
return None
|
||||
count_trailer_pick(text, stats)
|
||||
lines = clean_lines(text, True)
|
||||
if len(lines) < 2:
|
||||
return None
|
||||
line = lines[-1][:MAX_LINE]
|
||||
if not has_letter(line):
|
||||
stats["letterless"] += 1
|
||||
return None
|
||||
return line
|
||||
|
||||
|
||||
PROBES = {"grep", "egrep", "fgrep", "rg", "test", "diff", "cmp", "[", "[["}
|
||||
SEGMENT_SPLIT = re.compile(r";|&&|\|\||\||\n")
|
||||
ERRORISH = re.compile(
|
||||
r"error|fatal|cannot|can't|no such|not found|denied|failed|failure|"
|
||||
r"unexpected|invalid|exception|traceback|panic|usage:|SC\d{4}|> \d+|"
|
||||
r"assert", re.I)
|
||||
TIGHTEN_RULES = ("probe", "errorish")
|
||||
|
||||
|
||||
def last_word(command):
|
||||
"""First word of the last segment, env prefixes and bin dir dropped."""
|
||||
parts = [p.strip() for p in SEGMENT_SPLIT.split(command) if p.strip()]
|
||||
words = parts[-1].split() if parts else []
|
||||
while words and re.match(r"[A-Za-z_]\w*=", words[0]):
|
||||
words.pop(0)
|
||||
if not words:
|
||||
return ""
|
||||
return re.sub(r"^/(?:usr/)?bin/", "", words[0])
|
||||
|
||||
|
||||
def survives(entry, rules):
|
||||
if "probe" in rules and entry.code == 1 and entry.tool in PROBES:
|
||||
return False
|
||||
return "errorish" not in rules or ERRORISH.search(entry.line) is not None
|
||||
|
||||
|
||||
def build_entry(pair, agent, stats):
|
||||
red = redact(pair.text[-MAX_REGEX_INPUT:])
|
||||
line = failure_line(red, stats)
|
||||
if line is None:
|
||||
return None
|
||||
full = redact(pair.command[:MAX_REGEX_INPUT])
|
||||
code = int(EXIT_CODE.match(red).group(0).split()[2])
|
||||
return Entry(normalize(line), line, full[:MAX_COMMAND], agent, code,
|
||||
last_word(full))
|
||||
|
||||
|
||||
def result_text(content):
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
if isinstance(content, list):
|
||||
return "".join(b.get("text", "") for b in content
|
||||
if isinstance(b, dict) and b.get("type") == "text")
|
||||
return ""
|
||||
|
||||
|
||||
def blocks_of(record):
|
||||
content = (record.get("message") or {}).get("content")
|
||||
if not isinstance(content, list):
|
||||
return []
|
||||
return [b for b in content if isinstance(b, dict)]
|
||||
|
||||
|
||||
def pairs_of(path):
|
||||
"""Bash tool_use paired with its tool_result by tool_use_id."""
|
||||
calls, pairs = {}, []
|
||||
with open(path, encoding="utf-8", errors="replace") as fh:
|
||||
for raw in fh:
|
||||
try:
|
||||
record = json.loads(raw)
|
||||
except ValueError:
|
||||
continue
|
||||
for b in blocks_of(record):
|
||||
if b.get("type") == "tool_use" and b.get("name") == "Bash":
|
||||
cmd = (b.get("input") or {}).get("command")
|
||||
calls[b.get("id")] = str(cmd or "")
|
||||
elif b.get("type") == "tool_result":
|
||||
cid = b.get("tool_use_id")
|
||||
if cid in calls:
|
||||
pairs.append(Pair(calls[cid],
|
||||
result_text(b.get("content")),
|
||||
b.get("is_error") is True))
|
||||
return pairs
|
||||
|
||||
|
||||
def recent(path, cutoff):
|
||||
return os.path.getmtime(path) >= cutoff
|
||||
|
||||
|
||||
def find_files(root, days):
|
||||
"""(session id, is_agent, path) of main and sub-agent transcripts."""
|
||||
cutoff = time.time() - days * 86400
|
||||
found = []
|
||||
for name in sorted(os.listdir(root)):
|
||||
full = os.path.join(root, name)
|
||||
if name.endswith(".jsonl") and recent(full, cutoff):
|
||||
found.append((name[:-6], False, full))
|
||||
sub = os.path.join(full, "subagents")
|
||||
if os.path.isdir(sub):
|
||||
for f in sorted(os.listdir(sub)):
|
||||
p = os.path.join(sub, f)
|
||||
if f.endswith(".jsonl") and recent(p, cutoff):
|
||||
found.append((name, True, p))
|
||||
return found
|
||||
|
||||
|
||||
def new_session():
|
||||
return {"before": {False: 0, True: 0}, "after": {False: 0, True: 0},
|
||||
"seen": set(), "kept": []}
|
||||
|
||||
|
||||
def measure(files, stats, rules=()):
|
||||
"""Per session counts; dedupe by key pooled over main and sub-agents."""
|
||||
sessions = {}
|
||||
for sid, agent, path in sorted(files, key=lambda f: f[1]):
|
||||
s = sessions.setdefault(sid, new_session())
|
||||
for pair in pairs_of(path):
|
||||
if not pair.is_error:
|
||||
continue
|
||||
entry = build_entry(pair, agent, stats)
|
||||
if entry is None or not survives(entry, rules):
|
||||
continue
|
||||
s["before"][agent] += 1
|
||||
if entry.key not in s["seen"]:
|
||||
s["seen"].add(entry.key)
|
||||
s["after"][agent] += 1
|
||||
s["kept"].append(entry)
|
||||
return sessions
|
||||
|
||||
|
||||
def print_counts(sessions, stats):
|
||||
tot = {k: {False: 0, True: 0} for k in ("before", "after")}
|
||||
print("session main(before/after) sub(before/after)")
|
||||
for sid, s in sorted(sessions.items()):
|
||||
print(f"{sid[:8]} {s['before'][False]:>6}/{s['after'][False]:<6}"
|
||||
f" {s['before'][True]:>6}/{s['after'][True]:<6}")
|
||||
for k in tot:
|
||||
for agent in (False, True):
|
||||
tot[k][agent] += s[k][agent]
|
||||
for agent, label in ((False, "main loop"), (True, "sub-agents")):
|
||||
print(f"TOTAL {label}: before dedupe {tot['before'][agent]}, "
|
||||
f"after dedupe {tot['after'][agent]}")
|
||||
print(f"rtk-trailer picks without stripping: {stats['rtk_would_pick']}")
|
||||
print("rtk-trailer picks after stripping: 0 (trailer removed per line)")
|
||||
print(f"letterless dropped: {stats['letterless']}")
|
||||
|
||||
|
||||
def print_samples(sessions, count, seed):
|
||||
kept = [e for s in sessions.values() for e in s["kept"]]
|
||||
rng = random.Random(seed)
|
||||
picked = rng.sample(kept, min(count, len(kept)))
|
||||
print(f"\n{len(picked)} samples of {len(kept)} captured entries")
|
||||
for e in picked:
|
||||
cmd = redact(e.command)[:200].replace("\n", " ")
|
||||
who = "sub" if e.agent else "main"
|
||||
print(f"- [{who}] key: {redact(e.key)}\n line: {redact(e.line)}"
|
||||
f"\n cmd: {cmd}")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--dir", default=DEFAULT_DIR)
|
||||
ap.add_argument("--days", type=int, default=14)
|
||||
ap.add_argument("--samples", type=int, default=30)
|
||||
ap.add_argument("--seed", type=int, default=None)
|
||||
ap.add_argument("--tighten", action="append", default=[],
|
||||
choices=TIGHTEN_RULES)
|
||||
args = ap.parse_args()
|
||||
stats = {"letterless": 0, "rtk_would_pick": 0}
|
||||
files = find_files(args.dir, args.days)
|
||||
print(f"files: {len(files)} ({sum(1 for f in files if f[1])} sub-agent)")
|
||||
sessions = measure(files, stats, args.tighten)
|
||||
print_counts(sessions, stats)
|
||||
print_samples(sessions, args.samples, args.seed)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in new issue
Block a user