forked from bchanot/claude
Merge feature/routing-audit-phase into develop
# Conflicts: # .claude/memory/journal.md # .claude/memory/learnings.md # .claude/tasks/TODO.md
This commit is contained in:
commit
b961ce6b98
18 files changed
+259
-39
No files matched your search
@@ -0,0 +1,64 @@
|
||||
# PLAN r3 — routing-audit-phase (feat A, BLK-032 routing policy)
|
||||
|
||||
r2 after the 2026-10-21 challenge round 1 (outage floor, test fixture, citers, no stash). r3 after round 2: single about string, no `confirmed.phases.audit` (first-use dialog = tripwire for main), unprobed-spawn residual + its test, explicit branch step, oracles 1/3/10/11 tightened, tier_floor flip test.
|
||||
|
||||
Contract: `.claude/tasks/contracts/2026-10-21-routing-audit-phase-1506.md`
|
||||
|
||||
## Why
|
||||
User policy 2026-10-19 (BLK-032): Fable only for orchestrator reflection/plans and the important audits (security, functional); other judges opus; executors sonnet. Today `verifier` routes to `verify` (work tier = sonnet xhigh) and `security-auditor` to `judge` (big tier = opus xhigh). Both are the merge-blocking gates of verify-secure-loop.md (GATE 1 contract verdict, GATE 2 security). A new best-tier phase `audit` carries them; `judge` keeps plan-challenger, analyzer, Plan, plugin-advisor, seo/geo-analyzer on opus.
|
||||
|
||||
## Router facts that bind the plan
|
||||
- Rows are phase names; a phase names a tier; a tier is an ordered alias list, first alias not down is used (`best` = fable > opus > sonnet). An agent gets its row's model at spawn, upward only within the tier; explicit `model=` at a call site wins (BDR-115, BDR-077). No dispatch of verifier / security-auditor carries an explicit `model=` (grep 2026-10-21: tour, onboard, audit-delta dispatch them bare).
|
||||
- `lib/tests/effort-routing.test.sh` locks: every routing.json phase equals the code copy in `register.ts` DEFAULT_CONFIG (`code_phase`); every rowed agent's frontmatter `model:` equals its phase's tier head and `effort:` equals the phase effort, unless a `changed` entry explains the drift (WARN). Neither verifier nor security-auditor has a `changed` entry, so the frontmatter MUST move with the row or the census FAILs.
|
||||
- `lib/tests/model-routing.test.sh:23-24` greps the literal old pins.
|
||||
- `register.test.ts` builds every world from `baseRouting()` (11 phases, :65-78). `logMissingPhases` (register.ts:862-869) diffs `DEFAULT_CONFIG.phases` against the file, so a DEFAULT_CONFIG phase absent from the fixture is logged in every test: :1592-1601 expects exactly `lacks phases reflect`, and `/route pending` counts `st.cfg.phases` (:2555 `pending confirmation: 19`, :2567 `17`). The fixture gains `audit` and the two counts become 20 and 18.
|
||||
- `inTier` (register.ts:3163-3174) writes a model only when it is the tier head or ranks above it; otherwise "tier … down, frontmatter model kept" and the frontmatter runs (test :1278-1291). A best-tier row with a `fable` frontmatter therefore has NO fallback: every Fable credit outage (BLK-030/031) would spawn both gates on a dead model. Decision (user 2026-10-21): the gates keep `model: opus` as the outage floor; the router lifts them to Fable when it is up; Fable down or router off = opus. The census gets a named carve-out: a best-tier agent row's model floor is the tier's SECOND alias.
|
||||
- No stash. `~/.claude/settings.json` and `~/.claude/skills/model-router` are symlinks into the repo: stashing settings.json puts every new session back on Fable 1M (the BLK-032 condition) and stashing routing.json re-arms the router's dialogs. The dirty copies carry over the branch start unchanged (`git diff develop..HEAD` is empty for both files). They are committed FIRST on the feature branch as one dedicated commit (the router's own dialog writes: `deploy` apply→verify + `changed` entry, `release-executor` Keep; the session model `opus` = BLK-032 policy point 1), before the executor runs. Feat B then carries CLAUDE.global.md, the BDR and the docs only.
|
||||
- `confirmed.agents` records first-use dialog answers; an unconfirmed row re-opens the dialog at first use. This feat is the user's decision → set `audit` by hand for both.
|
||||
|
||||
## Branch (main loop, before the executor)
|
||||
1. `bash ~/.claude/lib/gitflow.sh start feature routing-audit-phase` from the current `bugfix/guardrails-residuals` (not merged): the lib branches off develop; the dirty `routing.json` / `settings.json` carry over (develop and HEAD hold the same blobs).
|
||||
2. One dedicated commit of those two files: `chore(model-router): dialog decisions (deploy verify, release-executor) and session model opus` — the router's own writes plus BLK-032 policy point 1.
|
||||
3. Dispatch the executor on the clean tree.
|
||||
4. Memory IDs for this run's capitalize start above the highest on BOTH develop and the unmerged `bugfix/guardrails-residuals` (8555bff: BLK-033, LRN-222/223): next BDR-123, LRN-224, BLK-034. The TODO line for this run goes under the 2026-10-19 routing bullet, away from the bugfix hunk at TODO.md:5-6.
|
||||
|
||||
## Checklist (dispatch-ready)
|
||||
[ ] mods/model-router/routing.json — phases: insert after `judge` (line 23-27):
|
||||
"audit": { "tier": "best", "effort": "xhigh", "about": "the fresh gates: contract verdict and security scan; dispatched agents, not a main-loop phase" }
|
||||
(exact string, 93 chars; `about` is shown in the first-use dialog only, register.ts:154, so this text is what main's first `route(phase="audit")` would print). Reword `judge.about` → "independent judgment of work: challenges, analyses and reviews"; `verify.about` → "checking a result against its spec: tests and suites". Rows: `agents.verifier` "verify" → "audit"; `agents.security-auditor` "judge" → "audit". `confirmed.agents.verifier` → "audit", `confirmed.agents.security-auditor` → "audit" (the two spawns the user decided here). Do NOT write `confirmed.phases.audit`: an unconfirmed phase keeps the first-use dialog (`confirmPhase` register.ts:2580-2585, suppressed by `isDecided` :2081 once confirmed) as the tripwire against a main-loop `route(phase="audit")`; `/route pending` listing `phases:audit` is the price. Nothing else changes: the `changed` block keeps the committed `deploy` entry, no entry is forged for the hand-moved rows.
|
||||
[ ] mods/model-router/hooks/register.ts — DEFAULT_CONFIG.phases (line ~280): insert `audit: { tier: 'best', effort: 'xhigh' },` right after the `judge` line. Same indentation, trailing comma. No other code change (inTier untouched).
|
||||
[ ] mods/model-router/hooks/register.test.ts — `baseRouting()` phases: insert `audit: { tier: 'best', effort: 'xhigh' },` after `judge` (:70-71); `pending confirmation: 19` → `20` (:2555), `17` → `18` (:2567). Leave the fixture's `verifier: 'verify'` row (:89) as is, other tests use it. Add two tests next to :1278-1291, built with `bootRun($, on, FABLE, newWorld({ edit: f => { (f.agents as Rec)['security-auditor'] = 'audit' } }))` (bootRun's 4th param, :1035-1041) and bare `spawnOf('security-auditor')` (no `model` field: an explicit model skips routing, :3199):
|
||||
(a) `test('audit row: fable down, no write, frontmatter runs', …)`: `switchTo($, FABLE, SONNET, 'auto')`, spawn once, expect `rig.specs` `[undefined]` and the one log line `'model-router: security-auditor tier best down, frontmatter model kept'` (recordLogs filter on `frontmatter model kept`).
|
||||
(b) `test('audit row: first spawn dies on fable, the retry lands on the floor', …)`: spawn, `await failWith($, 'rate_limit', 'a1')`, spawn again, expect `rig.specs` `[FABLE, undefined]` (same shape as :886-891); no log assertion in (b). This is the path verify-secure-loop's single fresh retry now depends on.
|
||||
[ ] agents/verifier.md — frontmatter `model: sonnet` → `model: opus` (effort stays xhigh). Body untouched. Rationale in the census comment, not here.
|
||||
[ ] agents/security-auditor.md — frontmatter stays `model: opus`, `effort: xhigh`: no edit.
|
||||
[ ] lib/tests/effort-routing.test.sh — add `tier_floor <file> <tier>` right after `tier_head`: second alias when `$2` = best, else `tier_head`. ONE comment above it (the only place citing BLK-030 in the file): `# a best-tier row floors on its 2nd alias: inTier keeps the frontmatter when the head is down (Fable credits, BLK-030); a fable floor would spawn the gates on a dead model`. `field_of` calls `tier_floor "$REG" "$tier"` for the model field. Section (c) header becomes `# ── (c) tracked agents with a row: tier floor == model:, effort == effort: ─` (no BLK cite). Flip test: add `best: ['fable', 'opus', 'sonnet'],` to the fixture's `reg.ts` tiers block (:88-92) and two checks: `[ "$(tier_floor "$FIX/reg.ts" best)" = opus ] && ok || ko "flip: best floors on its 2nd alias"` and `[ "$(tier_floor "$FIX/reg.ts" big)" = opus ] && ok || ko "flip: big floors on its head"`.
|
||||
[ ] lib/tests/model-routing.test.sh:23-24 — `has "agents/verifier.md" 'model: opus' # audit row (best tier, opus = outage floor; BLK-032 policy 2026-10-21)` and `has "agents/security-auditor.md" 'model: opus' # audit row since 2026-10-21 (was judge since 2026-10-11)`. Keep the column alignment of the block.
|
||||
[ ] agents/plan-challenger.md:112-113 — replace "(Contrast the verifier on its `verify` row and the executors on their `implement` row, both sonnet, oracle-anchored to a contract.)" with "(Contrast the executors on their `implement` row, sonnet, oracle-anchored to a contract; the verifier and security-auditor sit on the `audit` row, Fable with an opus floor.)". Reflow the paragraph to 80 columns.
|
||||
[ ] lib/challenge-plan.md:50 — "(The executor gates stay sonnet.)" → "(The executors stay sonnet; the two merge gates sit on the `audit` row, Fable with an opus floor.)". Reflow to 80 columns.
|
||||
[ ] skills/tour/SKILL.md:83 — "security-auditor (sonnet frontmatter)" → "security-auditor (`audit` row: Fable, opus floor)". Reflow if the line exceeds 80 columns.
|
||||
[ ] README.md:86-87 — verifier row: `| verifier | Fable (`audit` row: best/xhigh, opus frontmatter floor) | fresh gate (≤3×/loop) |`; security-auditor row: `| security-auditor | Fable (`audit` row: best/xhigh, opus frontmatter floor) | fresh gate (≤3×/loop) |`. Also README.md:111-113 phase list: insert `` `audit` best/xhigh `` after `` `judge` big/xhigh ``; :114-116 "kept equal to the rows by the census" → "kept equal to the rows by the census `lib/tests/effort-routing.test.sh` (a best-tier row floors on the tier's second alias, opus, since Fable is credit-gated; a row changed through the first-use dialog passes with a `WARN floor drift` line"; :135 best-tier row list → "(`plan`, `reflect`, `orchestrate`, `escalate`; `audit` is an agent row, never typed)". Every citer edit above carries the token `` `audit` row `` (oracle 11).
|
||||
[ ] run: the four suites first (`make test suite=lib/tests/effort-routing.test.sh`, `model-routing`, `mods`, `doctrine-citers`), then the full `make test` (plan-challenger.test.sh locks strings in plan-challenger.md / challenge-plan.md; contract-verifier and gates suites read verifier.md); then the contract oracles.
|
||||
|
||||
## Edge cases
|
||||
- Fable down (credits, BLK-030/031): `inTier` writes nothing, the `opus` frontmatter runs: the gates degrade to opus, one log line `tier best down, frontmatter model kept`. Router off (`/route off`, `enabled:false`, unreadable routing.json): opus too. Locked by the new register.test.ts case and the census carve-out.
|
||||
- `audit` appears in the main loop's `route` tool enum as a bare name (register.ts:2912-2916; `about` is never in the tool text). A main-loop `route(phase="audit")` opens the first-use phase dialog (phase left unconfirmed), which prints the about: that dialog is the tripwire. Same capability as `plan`; no orchestrator skill calls it; feat B's ceiling rule covers inline work on Fable.
|
||||
- Unprobed spawn path (TODO.md:56 residual, unchanged): the router learns Fable is down from a failure, never from a probe before a spawn. In an outage window the FIRST gate spawn dies on Fable (429), `StopFailure` marks it down, the fresh retry lands on the opus floor. verify-secure-loop's single retry covers /feat, /bugfix, /ship-feature, /init-project, /onboard, /guardrails, /a11y; tour Phase A, onboard and audit-delta dispatch the auditor with no retry rule: one dead dispatch per outage window there. Accepted, recorded in TODO; locked by test (b).
|
||||
- `onAutoSwitch` (register.ts:3662-3674) charges a sub-agent's engine fallback to main's last plan (TODO 2026-10-13 residual): a gate falling off Fable while main's last plan is sonnet marks sonnet down 15 min; with main's last plan on opus (the new session model) the early return at :3670 marks nothing, so an engine-side fallback would never mark Fable down and every gate spawn would pay the 429 again. Which event the engine emits for a sub-agent credit 429 is not established (BLK-030 saw StopFailure on main). Accepted, no code change here; test (b) locks the StopFailure path only.
|
||||
- `audit` duplicates `plan`'s (best, xhigh) pair: the Change dialog maps a pair to the first phase that fits in file order (`targetPhase` :2311-2317), so a gate row Changed away and back to "fable + xhigh" lands on `plan`; `audit` is reachable by hand edit only. Accepted.
|
||||
- `register.ts` validates a phase's tier against `tiers` keys: `best` exists. A phase named like an alias would be rejected: `audit` is not an alias.
|
||||
- `model-gate.md` (`names a fable or opus id | proceed`) is about the MAIN loop, not the gates: unaffected.
|
||||
- `lib/challenge-plan.md:46` says challengers are on the judge row (opus): still true, no edit.
|
||||
- The `verify` phase keeps one consumer once the dialog writes are committed first (skill row `deploy` = verify); no `route(phase="verify")` caller exists in skills/agents/lib. Only its `about` changes.
|
||||
- The dirty `routing.json` / `settings.json` are committed first on the feature branch (see Router facts); the executor starts from a clean tree and edits the committed routing.json, which already holds the `deploy` row and `changed` entry.
|
||||
|
||||
## Tests
|
||||
- `register.test.ts`: fixture + counts + the new outage-floor case. `effort-routing.test.sh`: best-tier floor carve-out. `model-routing.test.sh`: literal pins. `mods.test.sh` runs `claude plugin test` (CLI present on this machine). `doctrine-citers.test.sh`: citer census.
|
||||
|
||||
## Disposition (STEP 0.6, decisions-first)
|
||||
- BDR-066 [in force, partially superseded by this feat]: "verifier + security-auditor STAY sonnet" → overturned by user policy 2026-10-19; feat B writes the superseding BDR. Executors stay sonnet (honored).
|
||||
- BDR-077 [in force]: fail-safe pin rule (a forgotten override over-tiers, never downgrades judgment); no call site overrides the gates → honored: the frontmatter floor (opus) never sits below today's security-auditor pin, and the verifier floor rises sonnet → opus.
|
||||
- BDR-115 [in force]: rows in routing.json are the route, frontmatter = off-state floor, "an agent moves UP from its frontmatter alias, never below" (inTier) → honored: opus floor, router lifts to Fable; `confirmed` set because the user decided here, no `changed` entry forged.
|
||||
- BLK-030/031 [resolved, residual TODO.md:56 "spawn path not probed"]: the opus floor gives the retry a landing spot once the first failure marks Fable down; it does not save the first spawn of an outage window (Edge cases). An explicit `model:"fable"` pin would have neither.
|
||||
- LRN-203 [in force]: hooks write full ids from `config.models`; `fable` alias maps to `claude-fable-5-1` → honored, no id typed.
|
||||
- BLK-030/031 [resolved]: Fable credit outages → the `best` tier falls back; gates still run (edge case above).
|
||||
Reference in new issue
Block a user