From d8b9a97980c91a3bd88a96b1926b931baa3f4646 Mon Sep 17 00:00:00 2001 From: bchanot Date: Mon, 2 Nov 2026 18:32:24 +0100 Subject: [PATCH] chore(bench): phase 2 rule-shape bench (cases, isolated auto-mode runner) --- .../plans/2026-11-02-phase2-bench/README.md | 36 ++++++++++++++++ .../plans/2026-11-02-phase2-bench/bench.sh | 31 ++++++++++++++ .../plans/2026-11-02-phase2-bench/cases.tsv | 41 +++++++++++++++++++ .../2026-11-02-phase2-bench/long-brief.txt | 15 +++++++ .../plans/2026-11-02-phase2-bench/results.tsv | 2 + .../plans/2026-11-02-phase2-bench/run.sh | 28 +++++++++++++ 6 files changed, 153 insertions(+) create mode 100644 .claude/tasks/plans/2026-11-02-phase2-bench/README.md create mode 100644 .claude/tasks/plans/2026-11-02-phase2-bench/bench.sh create mode 100644 .claude/tasks/plans/2026-11-02-phase2-bench/cases.tsv create mode 100644 .claude/tasks/plans/2026-11-02-phase2-bench/long-brief.txt create mode 100644 .claude/tasks/plans/2026-11-02-phase2-bench/results.tsv create mode 100644 .claude/tasks/plans/2026-11-02-phase2-bench/run.sh diff --git a/.claude/tasks/plans/2026-11-02-phase2-bench/README.md b/.claude/tasks/plans/2026-11-02-phase2-bench/README.md new file mode 100644 index 0000000..1c8d15a --- /dev/null +++ b/.claude/tasks/plans/2026-11-02-phase2-bench/README.md @@ -0,0 +1,36 @@ +# Phase 2 rule-shape bench + +Proves which permission-rule shapes (ask and deny) hold on Claude Code +`2.1.316`, headless, before any is added to `settings.json`. + +## Method + +`bench.sh [id…]` reads `cases.tsv` (`id kind rule cmd want`, tab-separated, +`\n` in `cmd` is a newline) and runs each row in a fresh temp dir (git repo +with a `.env` and `sub/.env`), through `run.sh`: `claude -p --model haiku +--tools Bash` with only the bench settings (`--setting-sources local`), asked +to run the command verbatim. Rows with want ASKED or DENIED also run a +control: same command, no rule. `results.tsv` is append-only, the last row per +id wins. Row format: `want`, then `label verdict sent final permissionMode`. + +Settings shape (auto mode, rule present): + + {"permissions": {"allow": ["Bash(*)"], "defaultMode": "auto", + "": [""]}, + "skipAutoPermissionPrompt": true} + +A row whose `permissionMode` (from the stream-json `system init` event) is +not `auto` is invalid. `run.sh