forked from bchanot/claude
9.5 KiB
9.5 KiB
CONTRACT — routing-audit-phase (feat A of the routing policy, BLK-032)
REQUEST (verbatim, feat A scope after the 2026-10-21 scope split)
model-router judge level split: security-auditor and verifier gates → Fable, plan-challenger / analyzer / code review → opus; executors stay sonnet. Citers census: skills/agents/lib rows that name the judge level must be patched in the same commit (make test runs the census). Feat B (settings model, CLAUDE.global.md ceiling rule, BDR) is a separate run.
CLARIFICATIONS
- [gated 2026-10-21] New phase name:
audit(user pick overgate), tierbest, effortxhigh. - [gated 2026-10-21, amended after the challenge] Scope split: feat A = router + agent floors + census + citers, preceded on the feature branch by one dedicated commit of the pre-existing routing.json (router dialog writes) and settings.json (session model
opus) diffs; feat B = CLAUDE.global.md ceiling rule, BDR, docs. - Internal:
confirmed.agentsfor verifier and security-auditor set toauditby hand (this feat is the user's confirmation; no dialog re-fire). Theaboutlines ofjudgeandverifydrop the words that now describeaudit. - [gated 2026-10-21] Outage floor: the gates keep
model: opusin frontmatter (inTier never writes below the tier head; a fable floor would spawn on a dead Fable during a credit outage). The census gets a named carve-out: a best-tier agent row floors on the tier's second alias. - [gated 2026-10-21] Scope: 11 files in one /feat run (user call over /ship-feature after the challenge).
- Internal: no stash. The pre-existing diffs on
mods/model-router/routing.json(router dialog writes) andsettings.json(session modelopus) are committed first on the feature branch as one dedicated commit, before the executor runs. - Internal:
confirmed.phases.auditis NOT written: the first-use dialog stays as the tripwire against a main-looproute(phase="audit"). - Internal: feature branch
feature/routing-audit-phasestarted from the lib off develop before any commit.
ACCEPTANCE CRITERIA
- routing.json has the
auditphase: tier best, effort xhigh, the exact about "the fresh gates: contract verdict and security scan; dispatched agents, not a main-loop phase", placed right afterjudge. CHECK: python3 -I -c "import json;d=json.load(open('mods/model-router/routing.json'));p=d['phases'];k=list(p);a=p['audit'];assert a['tier']=='best' and a['effort']=='xhigh' and a['about']=='the fresh gates: contract verdict and security scan; dispatched agents, not a main-loop phase' and k[k.index('judge')+1]=='audit';print('AUDIT-PHASE ok')" EXPECT: AUDIT-PHASE ok EVIDENCE: MET exit=0 marker-found :: AUDIT-PHASE ok - register.ts DEFAULT_CONFIG phases carry the same
auditline, right afterjudge. CHECK: grep -n -A1 "^ judge: { tier: 'big', effort: 'xhigh' }," mods/model-router/hooks/register.ts | grep -c "audit: { tier: 'best', effort: 'xhigh' }," EXPECT: 1 EVIDENCE: MET exit=0 marker-found :: 1 - routing.json rows
agents.verifierandagents.security-auditorareaudit, so are theirconfirmed.agentsentries;confirmed.phaseshas NOauditkey (first-use dialog kept for main). CHECK: python3 -I -c "import json;d=json.load(open('mods/model-router/routing.json'));assert all(t[n]=='audit' for t in (d['agents'],d['confirmed']['agents']) for n in ('verifier','security-auditor'));assert 'audit' not in d['confirmed']['phases'];print('ROWS ok')" EXPECT: ROWS ok EVIDENCE: MET exit=0 marker-found :: ROWS ok agents/verifier.mdandagents/security-auditor.mdfrontmatter readmodel: opusandeffort: xhigh(opus = outage floor of the best tier). CHECK: for f in agents/verifier.md agents/security-auditor.md; do awk 'NR==1&&/^---$/{p=1;next} p&&/^---$/{exit} p' "$f" | grep -cE '^(model: opus|effort: xhigh)$'; done | tr '\n' ' ' EXPECT: 2 2 EVIDENCE: MET exit=0 marker-found :: 2 2- The
judgeandverifyaboutlines no longer claim audits or security review: judge = "independent judgment of work: challenges, analyses and reviews", verify = "checking a result against its spec: tests and suites". CHECK: grep -c '"about": "independent judgment of work: challenges, analyses and reviews"|"about": "checking a result against its spec: tests and suites"' mods/model-router/routing.json EXPECT: 2 EVIDENCE: MET exit=0 marker-found :: 2 lib/tests/model-routing.test.shlocks the new floors (agents/verifier.mdmodel: opus,agents/security-auditor.mdmodel: opus, both commented as the audit row) and no longer locksverifier.mdon sonnet. CHECK: a=$(grep -c "has "agents/verifier.md" *'model: opus'.*audit|has "agents/security-auditor.md" *'model: opus'.*audit" lib/tests/model-routing.test.sh); b=$(grep -c "verifier.md" *'model: sonnet'" lib/tests/model-routing.test.sh); echo "$a/$b" EXPECT: 2/0 EVIDENCE: MET exit=0 marker-found :: 2/0- Census suites green:
make test suite=lib/tests/effort-routing.test.shandmake test suite=lib/tests/model-routing.test.shexit 0. CHECK: make test suite=lib/tests/effort-routing.test.sh >/dev/null 2>&1 && make test suite=lib/tests/model-routing.test.sh >/dev/null 2>&1 && echo CENSUS-GREEN EXPECT: CENSUS-GREEN EVIDENCE: MET exit=0 marker-found :: CENSUS-GREEN - Mod suite green and not skipped:
make test suite=lib/tests/mods.test.shexits 0 and reportsplugin test passedfor model-router (the CLI is present on this machine). CHECK: make test suite=lib/tests/mods.test.sh 2>&1 | grep -c 'model-router: plugin test passed' EXPECT: 1 EVIDENCE: MET exit=0 marker-found :: 1 register.test.tscarries theauditphase inbaseRouting(), thepending confirmationcounts 20 and 18, and two audit-row cases: fable already down → no write + logtier best down, frontmatter model kept; first spawn on fable thenrate_limit→ the retry writes nothing ([FABLE, undefined]). CHECK: f=mods/model-router/hooks/register.test.ts; a=$(grep -c "^ audit: { tier: 'best', effort: 'xhigh' }," $f); b=$(grep -c "pending confirmation: 20'|pending confirmation: 18'" $f); c=$(grep -c "tier best down, frontmatter model kept" $f); d=$(grep -c "toEqual([FABLE, undefined])" $f); echo "$a/$b/$([ "$c" -ge 1 ] && echo ok)/$d" EXPECT: 1/2/ok/1 EVIDENCE: MET exit=0 marker-found :: 1/2/ok/1lib/tests/effort-routing.test.shnames the best-tier carve-out: atier_floorhelper returning the second alias forbest, used by the model check, a comment naming BLK-030, and a flip test of the helper on the fixture (best → opus, big → opus). CHECK: f=lib/tests/effort-routing.test.sh; a=$(grep -c '^tier_floor()' $f); b=$(grep -c 'tier_floor "$REG" "$tier"' $f); c=$(grep -c 'BLK-030' $f); d=$(grep -c 'tier_floor "$FIX/reg.ts" best|tier_floor "$FIX/reg.ts" big' $f); echo "$a/$b/$([ "$c" -ge 1 ] && echo ok)/$d" EXPECT: 1/1/ok/2 EVIDENCE: MET exit=0 marker-found :: 1/1/ok/2- The four citers no longer claim sonnet gates:
agents/plan-challenger.md,lib/challenge-plan.md,skills/tour/SKILL.md,README.mdeach carry the token`audit` row(whitespace-normalised, survives a reflow), the old sentences are gone, and README's phase list names`audit` best/xhigh(checked by the verifier, no oracle). CHECK: a=0; for f in agents/plan-challenger.md lib/challenge-plan.md skills/tour/SKILL.md README.md; do tr '\n' ' ' < "$f" | tr -s ' ' | grep -q 'auditrow' && a=$((a+1)); done; b=$(cat agents/plan-challenger.md lib/challenge-plan.md skills/tour/SKILL.md README.md | tr '\n' ' ' | tr -s ' ' | grep -o 'both sonnet, oracle-anchored|The executor gates stay sonnet|security-auditor (sonnet frontmatter)|| verifier | sonnet (pinned)|| security-auditor | opus (judge row' | wc -l); echo "$a/$b" EXPECT: 4/0 EVIDENCE: MET exit=0 marker-found :: 4/0 - Doctrine citers census green:
make test suite=lib/tests/doctrine-citers.test.shexits 0. CHECK: make test suite=lib/tests/doctrine-citers.test.sh >/dev/null 2>&1 && echo CITERS-GREEN EXPECT: CITERS-GREEN EVIDENCE: MET exit=0 marker-found :: CITERS-GREEN lib/challenge-plan.mdstill states the challengers run on thejudgerow (opus): no edit needed, the statement stays true. CHECK: grep -c 'routed to thejudgerow (opus)' lib/challenge-plan.md EXPECT: 1 EVIDENCE: MET exit=0 marker-found :: 1- No new lint or test suppression in the diff (floor-guard clean). CHECK: git diff -U0 -- . ':!.claude' | grep -E '^+' | grep -cE 'shellcheck disable|noqa|eslint-disable|@ts-ignore|@ts-expect-error' | sed 's/^/n=/' EXPECT: n=0 EVIDENCE: MET exit=0 marker-found :: n=0
- The diff touches only the FILE SCOPE:
git status --shortlists nothing outside it (the dialog-write and session-model diffs are committed before the executor runs). CHECK: git status --short -- . ':!.claude' | awk '{print $2}' | grep -vcE '^(mods/model-router/routing.json|mods/model-router/hooks/register.ts|mods/model-router/hooks/register.test.ts|agents/verifier.md|agents/security-auditor.md|lib/tests/model-routing.test.sh|lib/tests/effort-routing.test.sh|agents/plan-challenger.md|lib/challenge-plan.md|skills/tour/SKILL.md|README.md)$' | sed 's/^/n=/' EXPECT: n=0 EVIDENCE: MET exit=0 marker-found :: n=0
FILE SCOPE
- mods/model-router/routing.json
- mods/model-router/hooks/register.ts
- mods/model-router/hooks/register.test.ts
- agents/verifier.md
- agents/security-auditor.md (no edit expected: frontmatter already opus/xhigh)
- lib/tests/model-routing.test.sh
- lib/tests/effort-routing.test.sh
- agents/plan-challenger.md
- lib/challenge-plan.md
- skills/tour/SKILL.md
- README.md