chore(tasks): model-router W1-C plan r2 + contract amendments after the FATAL round

This commit is contained in:
bchanot
2026-10-09 12:55:14 +02:00
parent 2d8cd6bf4c
commit 140c16a67f
2 changed files with 175 additions and 7 deletions
@@ -13,25 +13,29 @@ Q: main-loop model moves / A: UPGRADE (phase tier ranks above the current model)
Q: no `/route` typed by the user / A: phases come from (1) prompt rules at turn start (keyword → phase as the turn's DEFAULT route, source 'prompt', overridable; `ultrathink` stays a FLOOR), (2) the model's own `route` calls and the skills table, (3) derived: an Agent dispatch from main pushes `orchestrate` and the previous route is restored when the last live agent ends, (4) optional classifier (`classifier: false` by default) that asks the engine's small model for a phase label when no rule matched. [user: "ne pas lancer les /route moi-même"]
Q: default prompt rules / A: floor: `\bultrathink\b` → escalate. Defaults: `\b(plan|planifie|planning|brainstorm|architecture|con[cç]ois|design)\b` → plan; `\b(pourquoi|why|explique|explain|analyse|analyze|comprendre|understand|review|audit)\b` → reflect. No default rule lowers a turn (no `mechanical` rule): lowering is explicit (route tool, skills). [orchestrator — conservative defaults, user-editable in the override]
Q: typed `/effort-<l>` / A: unchanged (floor + default of the turn, B1).
Q (r2): breaker signal / A: `classic.StopFailure` error kinds `rate_limit | overloaded | billing_error | model_not_found` (+ the engine's own fallback detected at the step), NOT `turn.complete` reasons (a context-limit or network error is not unavailability); backoff 15 → 300 min; `/clear` keeps the marks (account-wide), `/route reload` and a user `/model` clear them. [orchestrator — challenge r1]
Q (r2): classifier / A: deferred to wave 2 (dead code by default, no test can reach it, no timeout on the call). [orchestrator — challenge r1]
Q (r2): upgrade cost / A: an upgrade is skipped above `upgradeMaxTokens` (default 200000 context tokens): one cold read of the whole context per switch into a model. [orchestrator — LRN-204]
Q (r2): superseded clauses / A: floor AC4 (`turnMain` sources gain 'derived' and 'prompt'), W1-A AC6 (the switch gates downgrades only), BDR-115 (6) (window guard on every switch). [orchestrator]
## ACCEPTANCE CRITERIA
1. Suite green: `claude plugin test` passes with at least 40 `test(` calls; `claude plugin validate` passes with no warning; no line over 80 chars; no `any` type.
CHECK: cd mods/model-router && out=$(claude plugin test . 2>&1); rc=$?; echo "$out" | tail -n 3; [ $rc -eq 0 ] && [ "$(grep -cE '^\s*test\(' hooks/register.test.ts)" -ge 40 ] && v=$(claude plugin validate . 2>&1) && echo "$v" | grep -q 'Validation passed' && ! echo "$v" | grep -qi 'warning' && ! grep -nE '.{81,}' hooks/register.ts hooks/register.test.ts && ! grep -nE ':\s*any\b|<any>|as any\b' hooks/register.ts && echo TIERS-SUITE-OK
1. Suite green: `claude plugin test` passes with at least 43 `test(` calls; `claude plugin validate` passes with no warning; no line over 80 chars; no `any` type.
CHECK: cd mods/model-router && out=$(claude plugin test . 2>&1); rc=$?; echo "$out" | tail -n 3; [ $rc -eq 0 ] && [ "$(grep -cE '^\s*test\(' hooks/register.test.ts)" -ge 43 ] && v=$(claude plugin validate . 2>&1) && echo "$v" | grep -q 'Validation passed' && ! echo "$v" | grep -qi 'warning' && ! grep -nE '.{81,}' hooks/register.ts hooks/register.test.ts && ! grep -nE ':\s*any\b|<any>|as any\b' hooks/register.ts && echo TIERS-SUITE-OK
EXPECT: TIERS-SUITE-OK
EVIDENCE: pending
2. Type-check clean against this build's declarations.
CHECK: T=/Users/b.chanot/Documents/claude/mods/model-router/.claude-plugin/types; W=$(mktemp -d) && printf '{"compilerOptions":{"target":"es2023","lib":["es2023"],"types":[],"module":"esnext","moduleResolution":"bundler","strict":true,"noUncheckedIndexedAccess":true,"noEmit":true,"skipLibCheck":true,"jsx":"react","jsxFactory":"h","jsxFragmentFactory":"Fragment"},"include":["%s/claude-code/index.d.ts","%s/claude-code-tools/index.d.ts","%s/hooks"]}' "$T" "$T" "$PWD/mods/model-router" > "$W/tsconfig.json" && (cd "$W" && npx --yes -p typescript@5 tsc -p tsconfig.json) && echo TSC-OK
EXPECT: TSC-OK
EVIDENCE: pending
3. Tiers in the config: `tiers` (best/big/work/cheap), `fallback`, `cooldownMinutes`, `mainUpgrade`, `classifier` exist in DEFAULT_CONFIG; every default phase names a tier, none a bare model; `/route show` prints the resolved model of each phase and a `down:` line.
CHECK: cd mods/model-router/hooks && grep -q "tiers:" register.ts && grep -q "fallback:" register.ts && grep -q "cooldownMinutes" register.ts && grep -q "mainUpgrade" register.ts && grep -q "classifier" register.ts && [ "$(awk '/^const DEFAULT_CONFIG/,/^}/' register.ts | grep -cE "tier: '(best|big|work|cheap)'")" -ge 10 ] && ! awk '/^const DEFAULT_CONFIG/,/^}/' register.ts | grep -qE "^\s+[a-z]+: \{ model: '" && grep -q "down:" register.ts && echo TIERS-CONFIG-OK
3. Tiers in the config: `tiers` (best/big/work/cheap), `fallback`, `cooldownMinutes`, `mainUpgrade`, `upgradeMaxTokens` exist in DEFAULT_CONFIG; every default phase names a tier, none a bare model; `/route show` prints the resolved model of each phase and a `down:` line; no `classifier` (deferred to wave 2); no `Loop.model` / `spawnModel` / `loop.model` identifier (W1-A AC8).
CHECK: cd mods/model-router/hooks && grep -q "tiers:" register.ts && grep -q "fallback:" register.ts && grep -q "cooldownMinutes" register.ts && grep -q "mainUpgrade" register.ts && grep -q "upgradeMaxTokens" register.ts && ! grep -q "classifier" register.ts && [ "$(awk '/^const DEFAULT_CONFIG/,/^}/' register.ts | grep -cE "tier: '(best|big|work|cheap)'")" -ge 10 ] && ! awk '/^const DEFAULT_CONFIG/,/^}/' register.ts | grep -qE "model: '(haiku|sonnet|opus|fable)'" && ! grep -qE "loop\.model|explicitModel|spawnModel" register.ts && grep -q "down:" register.ts && echo TIERS-CONFIG-OK
EXPECT: TIERS-CONFIG-OK
EVIDENCE: pending
4. Tests prove (names contain the quoted word): `tier` — a `plan` route on a session model `claude-haiku-4-5-20251001` makes the main step run on `claude-fable-5-1` at xhigh (upgrade, default on); `downgrade` — a `mechanical` route on a fable session leaves the model unchanged while `mainModelSwitch` is off; `fallback` — after a main `turn.complete` with `reason: 'error'` on fable, the next main step runs on `claude-opus-5-5` with its effort unchanged, and after `/route reload` fable is used again; `breaker` — a `classic.PostModelSwitch` with `source: 'auto'` from fable marks fable down and `/route show` lists it; `spawn` — `Explore` spawns on `claude-opus-5-5` while sonnet is down; `derived` — an Agent tool call from main sets `orchestrate` and the previous route (`plan`) is back after the last agent's `turn.complete`; `default rule` — a prompt "planifie la migration" sets the `plan` route as the turn default and a later `route` tool call overrides it; `floor` tests from B1 still pass.
CHECK: cd mods/model-router/hooks && for w in tier downgrade fallback breaker spawn derived "default rule"; do grep -qE "test\('[^']*$w" register.test.ts || { echo "missing test: $w"; exit 1; }; done && echo TIERS-TESTS-OK
4. Tests prove (names contain the quoted word; plan r2 R14 lists them): `tier` — a `plan` route on a session model `claude-haiku-4-5-20251001` makes the main step run on `claude-fable-5-1` at xhigh (upgrade, default on); `downgrade` — a `mechanical` route on a fable session leaves the model unchanged while `mainModelSwitch` is off; `fallback` — with a `plan` route, a `classic.StopFailure` `rate_limit` on main after a fable step makes the next main step run on `claude-opus-5-5` at xhigh, and after `/route reload` fable is used again; `breaker` — an `invalid_request` failure never marks a model down, backoff expiry restores it, a `/model` command (`PostModelSwitch` source `command`) clears it; `engine fallback` — a step arriving on a model other than `$.session.model()` is never upgraded back and the session model is marked down; `unknown` — a session model absent from the table is never switched; `spawn` — `Explore` spawns on `claude-opus-5-5` while sonnet is down (agent StopFailure with `agent_id`); `derived` — a main Agent call sets `orchestrate` and the previous `plan` route is back when the spawned agent ends; a route declared after the dispatch is not overwritten by the pop; `default rule` — "planifie la migration" sets `plan` as the turn default and a later route call overrides it; a typed `/analyze …` and a `/effort-low pourquoi …` prompt get no default rule; `per axis` — a model-less sticky never hides a turn route's tier; `floor` tests from B1 still pass.
CHECK: cd mods/model-router/hooks && for w in tier downgrade fallback breaker "engine fallback" unknown spawn derived "default rule" "per axis"; do grep -qE "test\('[^']*$w" register.test.ts || { echo "missing test: $w"; exit 1; }; done && [ "$(grep -cE "test\('[^']*(breaker|derived|default rule)" register.test.ts)" -ge 7 ] && echo TIERS-TESTS-OK
EXPECT: TIERS-TESTS-OK
EVIDENCE: pending
5. Judged by reading: ONE resolver (`resolveModel`) turns a tier name, an alias or a full id into the first AVAILABLE full id (tier → list → skip down → alias → id; a bare alias or id passes through even when down, since explicit means explicit); rank = position in `fallback`; the main-loop decision is: route model wanted → upgrade allowed by `mainUpgrade`, downgrade gated by `mainModelSwitch` + window, same rank → keep; no route and current model down → next available in `fallback`; the breaker is fed only by `turn.complete` `reason` in `error|refusal` (main: the last main step's model; agent: the loop's spawn model) and by `classic.PostModelSwitch` `source: 'auto'` (from_model), never by an aborted turn; the derived `orchestrate` push/pop never overrides a route the model declared after the dispatch; prompt default rules write `turnMain` (source 'prompt'), floor rules write `turnFloor`; the classifier runs only when `classifier` is true and no rule matched, through `$.model.classify`, labels limited to the phase names plus `other`, any failure → no route; explicit Agent params still win; agent model fixed at spawn; every B1/1-A criterion still holds; no function over 25 logic lines; truthful texts name the resolved model and say "fallback" when the breaker chose it.
5. Judged by reading (plan r2 R1-R15 are binding): ONE resolver turns a tier name, an alias or a full id into an AVAILABLE canonical id (tier → list → skip down → id; an exhausted tier → the global chain; a bare alias or id passes through even when down); ids are canonical everywhere (`[1m]` stripped for comparison and carried on the replacement, alias → id, two-way prefix); ONE decision `decideMain(cur, wanted, ctx)` in the binding order (off → unknown cur: no switch → cur down: wanted or next available, windowOk → same alias: keep → better: `mainUpgrade` and `upgradeMaxTokens` → cheaper: `mainModelSwitch` and windowOk), used by `mainPlan` AND by every text; the engine's own fallback is respected (a step arriving off the session model marks the session model down and is never upgraded back); the breaker is fed only by `classic.StopFailure` errors `rate_limit | overloaded | billing_error | model_not_found` (main → the last main plan's model, agent → `agentModels`) and by the engine-fallback detection; `turn.complete` reasons and `PostModelSwitch` `auto` never mark (auto is logged); a user `/model` (`PostModelSwitch` command|picker|sdk) clears the target's mark; backoff 15 → 30 → 60 → 120 → 300 min per id, `model_not_found` until reload; the breaker survives `/clear` and is cleared by `/route reload` before the config read; the derived `orchestrate` push/pop tracks this turn's spawns and never overwrites a route the model declared after the dispatch; default prompt rules (two passes, absent `mode` = floor, `iu` flags, Unicode guards) write `turnMain` (source 'prompt') and are skipped for a leading `/`, for a prompt carrying a floor or a typed slash, and mid-turn; floor rules write `turnFloor`; no classifier; explicit Agent params still win; agent model fixed at spawn; every B1/1-A criterion still holds EXCEPT the three clauses R15 names (turnMain sources, the switch clause, the window-guard scope); no function over 25 logic lines; truthful texts come from `decideMain` and say `upgrade`, `fallback`, `switch off` or `unchanged`.
## FILE SCOPE
mods/model-router/hooks/register.ts · mods/model-router/hooks/register.test.ts
@@ -186,3 +186,167 @@ required fields; `prompt.submit`). Engine effort `high` in steps.
cold-cache step).
- Deferred: repo agents' frontmatter pins cannot fall back (the mod does not
see them in wave 1) → wave 2 moves them into the table with tiers.
## r2 — challenge round (3 lenses, all FATAL: 4 BLOCKER, 20 MAJOR): BINDING, overrides every section above where they conflict
R1. ONE phase field for the fallback-aware choice: `Route = { tier?: string;
model?: string; effort?: Level }`. Default phases use `tier:` only (plan,
reflect, orchestrate, escalate → best; judge → big; implement, write,
verify, explore → work; mechanical → cheap). `acceptPhase` refuses a route
carrying both `tier` and `model`, and refuses a `tiers` key that collides
with a `models` alias. The earlier "model: 'best'" drafts and the "no new
tier field" sentence are VOID. `/route model=<alias|id>` keeps writing
`model`; the route tool schema is unchanged.
R2. Ids: `canonical(st, id)` = strip a trailing `[1m]`, then alias → table id,
then two-way prefix match against the table ids (`id.startsWith(tableId)
|| tableId.startsWith(id)`), else the id itself. `aliasOf(st, id)` and
`modelRank(st, id)` (= index of the alias in `fallback`, `undefined` when
unknown) work on canonical ids. The existing effort `rank` keeps its name.
Breaker keys, `agentModels` values and comparisons are canonical. When the
current main model carries `[1m]`, a resolved replacement carries `[1m]`
too (the long-context tier is a property of the session, not of the
alias); log the first time it happens (unverified live: see Verify).
R3. Model axis per slot: `routeModelName(route) = route.tier ?? route.model`;
the main model axis is the FIRST defined `routeModelName` across
userMain, turnMain, turnFloor (per-axis, like B1's effort). `resolveName`
turns that name into an available id: a `tiers` key → first alias of the
list not down → `models` id; a tier whose every alias is down →
`nextAvailable(st, cur)` (global chain) → may be `undefined` (keep cur);
an alias or full id → canonical id, never skipped (explicit means explicit).
R4. Main decision `decideMain(st, cur, wanted, ctx)` → `{ model, why }`, used
by `mainPlan` AND by every text (texts pass `cur = canonical(await
$.session.model())`); order is BINDING:
1. `st.off` → cur.
2. `rankCur = modelRank(cur)`; UNKNOWN cur (not in the table) → cur, log
once per session (`model-router: <id> unknown to the models table; no
model switch`), the breaker still applies at step 3 if it is down.
3. cur DOWN → `wanted` if defined and not down, else `nextAvailable(cur)`;
apply `windowOk`; if nothing fits → cur (why `fallback`).
4. `wanted` undefined or `aliasOf(wanted) === aliasOf(cur)` → cur.
5. `modelRank(wanted) < rankCur` (better) → `cfg.mainUpgrade &&
ctx.tokens <= cfg.upgradeMaxTokens` ? wanted (why `upgrade`) : cur
(why `upgrade skipped: context <n> tokens over <max>` or `switch off`).
6. cheaper → `cfg.mainModelSwitch && windowOk` ? wanted (why `downgrade`)
: cur (why `switch off`).
New config scalar `upgradeMaxTokens` (default 200000): an upgrade pays a
cold read of the whole context on the new model (LRN-204); above the
threshold it is skipped and logged once per turn. `ctx.tokens` comes from
`$.session.usage()` read once per main step (fail → treat as 0).
R5. Engine fallback respected: at every main step `sess = canonical(await
$.session.model())`; when `canonical(e.model) !== sess`, the engine is on
a fallback → `markDown(sess, 'engine fallback')` and `cur = e.model` (the
router never upgrades back to the model the engine just left).
R6. Breaker inputs (replace the r1 list): (a) `classic.StopFailure` with
`error` ∈ rate_limit | overloaded | billing_error | model_not_found →
`markDown(target)` where target = `st.agentModels.get(e.agent_id)` when
`e.agent_id` is set, else `st.lastPlan?.model`; other errors (context
limit = invalid_request, server_error, auth, max_output_tokens…) → nothing;
(b) R5's engine-fallback detection; (c) `classic.PostModelSwitch`: source
`command | picker | sdk` → `st.down.delete(canonical(to_model))` and reset
its strikes (the user's explicit `/model` wins); source `auto` → LOG only
(`requested_model`, from, to), never a mark (unverified semantics).
`turn.complete` `reason` is NOT a breaker input any more (context-limit
and network errors are not availability); refusal → nothing.
Backoff per canonical id: strikes 1, 2, 3… → 15, 30, 60, 120, 300 min
(cap); `model_not_found` → until `/route reload`. `markDown` logs ALWAYS:
`model-router: <id> unavailable (<reason>) until <HH:MM>; routing falls
back`. Inert while `st.off`.
Lifecycle: `/route reload` clears `down` and strikes BEFORE loading the
config (whatever the read result); `session.end` (/clear) KEEPS `down`,
strikes and `agentModels` (availability is account-wide); expired
entries are pruned at the start of any hook that reads them, with `now`
read ONLY when `st.down.size > 0` (`$.clock.now()`), passed explicitly to
the helpers (no clock read in sync text functions: they receive the
pruned map).
R7. Agent models: `st.agentModels: Map<agentId, canonicalId>` set at spawn
from `started.model` (canonicalized; an alias answered by a hook above is
mapped through the table); deleted with the loop. No `Loop.model`,
`spawnModel` or `loop.model` identifier anywhere (W1-A AC8 grep).
`spawnRoute` resolves `route.tier ?? route.model` through `resolveName`
(skips down aliases); explicit `e.model` still wins even when down.
Deferred (noted): agents without a table row and no explicit model follow
`parentModel`; forks always inherit; neither falls back in wave 1.
R8. Derived orchestrate (D1) made exact: state `pushed: { prev: Routed | null;
spawnIds: Set<string> } | null`. In the main Agent `tool.call` hook:
before `next`, if `st.turnMain?.source` is not 'model' or 'skill' and
`st.pushed` is null → `st.pushed = { prev: st.turnMain, spawnIds: new Set() }`
and `st.turnMain = { phase: 'orchestrate', route: phases.orchestrate,
source: 'derived' }`; after `next` resolves: the spawned `agentId` (from
`st.spawnByCall: Map<tool_use_id, agentId>` filled at `agent.spawn`) is
added to `pushed.spawnIds`; if NO agent was registered for this
`tool_use_id` (foreground run already finished, or denied) → nothing to
wait for from this call. Pop rule: when `pushed.spawnIds` is empty after
the Agent call returned, or when the LAST id of `pushed.spawnIds` ends
(`turn.complete` with that agentId, deleted from the set), and
`st.turnMain?.source === 'derived'` → `st.turnMain = pushed.prev`,
`st.pushed = null`. A route/skill write in between (source model/skill)
replaces turnMain; the pop then only clears `pushed`. `endMainTurn`
clears `pushed` and `spawnByCall`. In `turn.complete` for an agent, delete
the loop and the maps FIRST, inside `safely`, before any other work.
R9. Prompt default rules (D2) made safe: rules scanned in two passes (floor
rules, then default rules), each pass first match; absent `mode` →
'floor' (B1 override files keep their meaning). Default rules are SKIPPED
when the trimmed text starts with `/` (slash commands and skills route
themselves), when the same prompt carries a floor match or sets
`typedSlash` (the user's explicit level wins), or when typed mid-turn.
Patterns compile with flags `iu` and the defaults use Unicode-aware
guards instead of `\b`: `(?<![\p{L}\p{N}-])(plan|planifie|planning|
brainstorm|architecture|con[cç]ois|design)(?![\p{L}\p{N}-])` and the
reflect list likewise; the validator requires the pattern to compile
with `iu`. A default-rule route is written to `turnMain` (source
'prompt'); it never lowers (no cheap/work default rule shipped).
R10. Classifier (D3) DEFERRED to wave 2: no `classifier` key, no code.
R11. Texts: `routedText`, `effortBridge`/`mainNote`, `slashEffort`, `show`,
`statusLine`, the route tool description and `mainOnHaiku` derive their
MODEL words from `decideMain` with `cur = canonical(await
$.session.model())` (hooks are async; `show` becomes async — the
command hook awaits it); they print the decided id and `why`
(`upgrade`, `fallback`, `switch off`, `unchanged`). The tool description
says: "the main loop moves UP to a phase's tier by itself, DOWN only with
the switch on; a sub-agent's model is fixed at spawn". `show` prints:
`upgrade: on|off`, `switch (downgrade): on|off`, `down: <id> until <HH:MM>
(<reason>) …| none`, each phase as `name=<tier or model>→<resolved id>/<effort>`.
Existing test 3f (`/route model=sonnet` shows `claude-sonnet-5-5`) is
adapted: on the kit's session model the line reads `asked claude-sonnet-5-5,
keeps <cur> (switch off)`; the alias→id resolution is asserted on the
`asked` part.
R12. `st.lastPlan: Plan | null` replaces `lastMain` and `lastMainModel`; the
spinner text is derived at render; `endMainTurn` resets it.
R13. Config validation additions: `tiers` values non-empty arrays of alias
keys (bad entries dropped, logged), `fallback` deduplicated non-empty
alias list (else default, logged), `cooldownMinutes` and
`upgradeMaxTokens` positive integers, `mode` ∈ floor|default, a log at
load when `tiers.best[0] !== fallback[0]` (rank comes from `fallback`
alone). `mainModelSwitch` documented as DOWNGRADE-only in the Config
comment.
R14. Tests (≥ 43 total, names carry the contract words): keep all 30; add:
`tier` (plan on a haiku session → fable xhigh, with mock.clock installed
where the breaker is touched), `downgrade` (mechanical on fable keeps
fable, switch off), `fallback` (plan route + `$.classic.StopFailure({
error: 'rate_limit', … })` on main after a fable step → next step
`claude-opus-5-5` at xhigh; `/route reload` → fable again), `breaker`
×3 (an aborted/`invalid_request` failure never marks down; backoff expiry
via `mock.clock` advance restores fable; `/model` command
`PostModelSwitch source: 'command'` clears a down model), `engine fallback`
(`$.session.model` mocked/answered as fable while the step arrives on
opus → no upgrade back, fable marked down), `unknown` (cur
`claude-zz-9` never switches), `spawn` (Explore → opus while sonnet is
down through an agent StopFailure with `agent_id`), `derived` ×2 (push on
dispatch, pop when the spawned agent ends → plan back; a route call after
the dispatch is NOT overwritten by the pop), `default rule` ×3 (planifie
→ plan then a route call overrides; `/analyze …` typed → no rule;
`/effort-low pourquoi …` → no default rule, floor low), `per axis`
(`/route effort=low` sticky + turn `plan` tier → model axis = best).
Read `mock.clock` and how `$.session.model` is answered in the kit
(a bottom `on('session.model', …)` hook) before writing them.
R15. Disposition, superseded clauses named: floor contract AC4 "`turnMain`
only ever holds 'model' or 'skill' sources" → now also 'derived' and
'prompt'; W1-A AC6 "main-loop model changes happen only when
`mainModelSwitch` is true" → true for DOWNGRADES only; upgrades follow
`mainUpgrade` + `upgradeMaxTokens`, and the breaker/engine-fallback path
moves off a dead model unconditionally; BDR-115 (6) window guard → applied
to every switch (up, down, fallback) through `windowOk`. The tiers
contract AC5 reads "every B1/1-A criterion still holds EXCEPT the three
clauses above".
R16. Live verification after reload (orchestrator, not the executor): the
`[1m]` carry-over on a fallback id, `PostModelSwitch` `source: 'auto'`
semantics, `StopFailure` reaching the mod with `agent_id`.