diff --git a/.claude/tasks/contracts/2026-10-09-model-router-tiers-1237.md b/.claude/tasks/contracts/2026-10-09-model-router-tiers-1237.md index 57eb85d..62754b7 100644 --- a/.claude/tasks/contracts/2026-10-09-model-router-tiers-1237.md +++ b/.claude/tasks/contracts/2026-10-09-model-router-tiers-1237.md @@ -13,25 +13,29 @@ Q: main-loop model moves / A: UPGRADE (phase tier ranks above the current model) Q: no `/route` typed by the user / A: phases come from (1) prompt rules at turn start (keyword → phase as the turn's DEFAULT route, source 'prompt', overridable; `ultrathink` stays a FLOOR), (2) the model's own `route` calls and the skills table, (3) derived: an Agent dispatch from main pushes `orchestrate` and the previous route is restored when the last live agent ends, (4) optional classifier (`classifier: false` by default) that asks the engine's small model for a phase label when no rule matched. [user: "ne pas lancer les /route moi-même"] Q: default prompt rules / A: floor: `\bultrathink\b` → escalate. Defaults: `\b(plan|planifie|planning|brainstorm|architecture|con[cç]ois|design)\b` → plan; `\b(pourquoi|why|explique|explain|analyse|analyze|comprendre|understand|review|audit)\b` → reflect. No default rule lowers a turn (no `mechanical` rule): lowering is explicit (route tool, skills). [orchestrator — conservative defaults, user-editable in the override] Q: typed `/effort-` / A: unchanged (floor + default of the turn, B1). +Q (r2): breaker signal / A: `classic.StopFailure` error kinds `rate_limit | overloaded | billing_error | model_not_found` (+ the engine's own fallback detected at the step), NOT `turn.complete` reasons (a context-limit or network error is not unavailability); backoff 15 → 300 min; `/clear` keeps the marks (account-wide), `/route reload` and a user `/model` clear them. [orchestrator — challenge r1] +Q (r2): classifier / A: deferred to wave 2 (dead code by default, no test can reach it, no timeout on the call). [orchestrator — challenge r1] +Q (r2): upgrade cost / A: an upgrade is skipped above `upgradeMaxTokens` (default 200000 context tokens): one cold read of the whole context per switch into a model. [orchestrator — LRN-204] +Q (r2): superseded clauses / A: floor AC4 (`turnMain` sources gain 'derived' and 'prompt'), W1-A AC6 (the switch gates downgrades only), BDR-115 (6) (window guard on every switch). [orchestrator] ## ACCEPTANCE CRITERIA -1. Suite green: `claude plugin test` passes with at least 40 `test(` calls; `claude plugin validate` passes with no warning; no line over 80 chars; no `any` type. - CHECK: cd mods/model-router && out=$(claude plugin test . 2>&1); rc=$?; echo "$out" | tail -n 3; [ $rc -eq 0 ] && [ "$(grep -cE '^\s*test\(' hooks/register.test.ts)" -ge 40 ] && v=$(claude plugin validate . 2>&1) && echo "$v" | grep -q 'Validation passed' && ! echo "$v" | grep -qi 'warning' && ! grep -nE '.{81,}' hooks/register.ts hooks/register.test.ts && ! grep -nE ':\s*any\b||as any\b' hooks/register.ts && echo TIERS-SUITE-OK +1. Suite green: `claude plugin test` passes with at least 43 `test(` calls; `claude plugin validate` passes with no warning; no line over 80 chars; no `any` type. + CHECK: cd mods/model-router && out=$(claude plugin test . 2>&1); rc=$?; echo "$out" | tail -n 3; [ $rc -eq 0 ] && [ "$(grep -cE '^\s*test\(' hooks/register.test.ts)" -ge 43 ] && v=$(claude plugin validate . 2>&1) && echo "$v" | grep -q 'Validation passed' && ! echo "$v" | grep -qi 'warning' && ! grep -nE '.{81,}' hooks/register.ts hooks/register.test.ts && ! grep -nE ':\s*any\b||as any\b' hooks/register.ts && echo TIERS-SUITE-OK EXPECT: TIERS-SUITE-OK EVIDENCE: pending 2. Type-check clean against this build's declarations. CHECK: T=/Users/b.chanot/Documents/claude/mods/model-router/.claude-plugin/types; W=$(mktemp -d) && printf '{"compilerOptions":{"target":"es2023","lib":["es2023"],"types":[],"module":"esnext","moduleResolution":"bundler","strict":true,"noUncheckedIndexedAccess":true,"noEmit":true,"skipLibCheck":true,"jsx":"react","jsxFactory":"h","jsxFragmentFactory":"Fragment"},"include":["%s/claude-code/index.d.ts","%s/claude-code-tools/index.d.ts","%s/hooks"]}' "$T" "$T" "$PWD/mods/model-router" > "$W/tsconfig.json" && (cd "$W" && npx --yes -p typescript@5 tsc -p tsconfig.json) && echo TSC-OK EXPECT: TSC-OK EVIDENCE: pending -3. Tiers in the config: `tiers` (best/big/work/cheap), `fallback`, `cooldownMinutes`, `mainUpgrade`, `classifier` exist in DEFAULT_CONFIG; every default phase names a tier, none a bare model; `/route show` prints the resolved model of each phase and a `down:` line. - CHECK: cd mods/model-router/hooks && grep -q "tiers:" register.ts && grep -q "fallback:" register.ts && grep -q "cooldownMinutes" register.ts && grep -q "mainUpgrade" register.ts && grep -q "classifier" register.ts && [ "$(awk '/^const DEFAULT_CONFIG/,/^}/' register.ts | grep -cE "tier: '(best|big|work|cheap)'")" -ge 10 ] && ! awk '/^const DEFAULT_CONFIG/,/^}/' register.ts | grep -qE "^\s+[a-z]+: \{ model: '" && grep -q "down:" register.ts && echo TIERS-CONFIG-OK +3. Tiers in the config: `tiers` (best/big/work/cheap), `fallback`, `cooldownMinutes`, `mainUpgrade`, `upgradeMaxTokens` exist in DEFAULT_CONFIG; every default phase names a tier, none a bare model; `/route show` prints the resolved model of each phase and a `down:` line; no `classifier` (deferred to wave 2); no `Loop.model` / `spawnModel` / `loop.model` identifier (W1-A AC8). + CHECK: cd mods/model-router/hooks && grep -q "tiers:" register.ts && grep -q "fallback:" register.ts && grep -q "cooldownMinutes" register.ts && grep -q "mainUpgrade" register.ts && grep -q "upgradeMaxTokens" register.ts && ! grep -q "classifier" register.ts && [ "$(awk '/^const DEFAULT_CONFIG/,/^}/' register.ts | grep -cE "tier: '(best|big|work|cheap)'")" -ge 10 ] && ! awk '/^const DEFAULT_CONFIG/,/^}/' register.ts | grep -qE "model: '(haiku|sonnet|opus|fable)'" && ! grep -qE "loop\.model|explicitModel|spawnModel" register.ts && grep -q "down:" register.ts && echo TIERS-CONFIG-OK EXPECT: TIERS-CONFIG-OK EVIDENCE: pending -4. Tests prove (names contain the quoted word): `tier` — a `plan` route on a session model `claude-haiku-4-5-20251001` makes the main step run on `claude-fable-5-1` at xhigh (upgrade, default on); `downgrade` — a `mechanical` route on a fable session leaves the model unchanged while `mainModelSwitch` is off; `fallback` — after a main `turn.complete` with `reason: 'error'` on fable, the next main step runs on `claude-opus-5-5` with its effort unchanged, and after `/route reload` fable is used again; `breaker` — a `classic.PostModelSwitch` with `source: 'auto'` from fable marks fable down and `/route show` lists it; `spawn` — `Explore` spawns on `claude-opus-5-5` while sonnet is down; `derived` — an Agent tool call from main sets `orchestrate` and the previous route (`plan`) is back after the last agent's `turn.complete`; `default rule` — a prompt "planifie la migration" sets the `plan` route as the turn default and a later `route` tool call overrides it; `floor` tests from B1 still pass. - CHECK: cd mods/model-router/hooks && for w in tier downgrade fallback breaker spawn derived "default rule"; do grep -qE "test\('[^']*$w" register.test.ts || { echo "missing test: $w"; exit 1; }; done && echo TIERS-TESTS-OK +4. Tests prove (names contain the quoted word; plan r2 R14 lists them): `tier` — a `plan` route on a session model `claude-haiku-4-5-20251001` makes the main step run on `claude-fable-5-1` at xhigh (upgrade, default on); `downgrade` — a `mechanical` route on a fable session leaves the model unchanged while `mainModelSwitch` is off; `fallback` — with a `plan` route, a `classic.StopFailure` `rate_limit` on main after a fable step makes the next main step run on `claude-opus-5-5` at xhigh, and after `/route reload` fable is used again; `breaker` — an `invalid_request` failure never marks a model down, backoff expiry restores it, a `/model` command (`PostModelSwitch` source `command`) clears it; `engine fallback` — a step arriving on a model other than `$.session.model()` is never upgraded back and the session model is marked down; `unknown` — a session model absent from the table is never switched; `spawn` — `Explore` spawns on `claude-opus-5-5` while sonnet is down (agent StopFailure with `agent_id`); `derived` — a main Agent call sets `orchestrate` and the previous `plan` route is back when the spawned agent ends; a route declared after the dispatch is not overwritten by the pop; `default rule` — "planifie la migration" sets `plan` as the turn default and a later route call overrides it; a typed `/analyze …` and a `/effort-low pourquoi …` prompt get no default rule; `per axis` — a model-less sticky never hides a turn route's tier; `floor` tests from B1 still pass. + CHECK: cd mods/model-router/hooks && for w in tier downgrade fallback breaker "engine fallback" unknown spawn derived "default rule" "per axis"; do grep -qE "test\('[^']*$w" register.test.ts || { echo "missing test: $w"; exit 1; }; done && [ "$(grep -cE "test\('[^']*(breaker|derived|default rule)" register.test.ts)" -ge 7 ] && echo TIERS-TESTS-OK EXPECT: TIERS-TESTS-OK EVIDENCE: pending -5. Judged by reading: ONE resolver (`resolveModel`) turns a tier name, an alias or a full id into the first AVAILABLE full id (tier → list → skip down → alias → id; a bare alias or id passes through even when down, since explicit means explicit); rank = position in `fallback`; the main-loop decision is: route model wanted → upgrade allowed by `mainUpgrade`, downgrade gated by `mainModelSwitch` + window, same rank → keep; no route and current model down → next available in `fallback`; the breaker is fed only by `turn.complete` `reason` in `error|refusal` (main: the last main step's model; agent: the loop's spawn model) and by `classic.PostModelSwitch` `source: 'auto'` (from_model), never by an aborted turn; the derived `orchestrate` push/pop never overrides a route the model declared after the dispatch; prompt default rules write `turnMain` (source 'prompt'), floor rules write `turnFloor`; the classifier runs only when `classifier` is true and no rule matched, through `$.model.classify`, labels limited to the phase names plus `other`, any failure → no route; explicit Agent params still win; agent model fixed at spawn; every B1/1-A criterion still holds; no function over 25 logic lines; truthful texts name the resolved model and say "fallback" when the breaker chose it. +5. Judged by reading (plan r2 R1-R15 are binding): ONE resolver turns a tier name, an alias or a full id into an AVAILABLE canonical id (tier → list → skip down → id; an exhausted tier → the global chain; a bare alias or id passes through even when down); ids are canonical everywhere (`[1m]` stripped for comparison and carried on the replacement, alias → id, two-way prefix); ONE decision `decideMain(cur, wanted, ctx)` in the binding order (off → unknown cur: no switch → cur down: wanted or next available, windowOk → same alias: keep → better: `mainUpgrade` and `upgradeMaxTokens` → cheaper: `mainModelSwitch` and windowOk), used by `mainPlan` AND by every text; the engine's own fallback is respected (a step arriving off the session model marks the session model down and is never upgraded back); the breaker is fed only by `classic.StopFailure` errors `rate_limit | overloaded | billing_error | model_not_found` (main → the last main plan's model, agent → `agentModels`) and by the engine-fallback detection; `turn.complete` reasons and `PostModelSwitch` `auto` never mark (auto is logged); a user `/model` (`PostModelSwitch` command|picker|sdk) clears the target's mark; backoff 15 → 30 → 60 → 120 → 300 min per id, `model_not_found` until reload; the breaker survives `/clear` and is cleared by `/route reload` before the config read; the derived `orchestrate` push/pop tracks this turn's spawns and never overwrites a route the model declared after the dispatch; default prompt rules (two passes, absent `mode` = floor, `iu` flags, Unicode guards) write `turnMain` (source 'prompt') and are skipped for a leading `/`, for a prompt carrying a floor or a typed slash, and mid-turn; floor rules write `turnFloor`; no classifier; explicit Agent params still win; agent model fixed at spawn; every B1/1-A criterion still holds EXCEPT the three clauses R15 names (turnMain sources, the switch clause, the window-guard scope); no function over 25 logic lines; truthful texts come from `decideMain` and say `upgrade`, `fallback`, `switch off` or `unchanged`. ## FILE SCOPE mods/model-router/hooks/register.ts · mods/model-router/hooks/register.test.ts diff --git a/.claude/tasks/plans/2026-10-09-model-router-tiers-1237.md b/.claude/tasks/plans/2026-10-09-model-router-tiers-1237.md index 639d822..2e309c1 100644 --- a/.claude/tasks/plans/2026-10-09-model-router-tiers-1237.md +++ b/.claude/tasks/plans/2026-10-09-model-router-tiers-1237.md @@ -186,3 +186,167 @@ required fields; `prompt.submit`). Engine effort `high` in steps. cold-cache step). - Deferred: repo agents' frontmatter pins cannot fall back (the mod does not see them in wave 1) → wave 2 moves them into the table with tiers. + +## r2 — challenge round (3 lenses, all FATAL: 4 BLOCKER, 20 MAJOR): BINDING, overrides every section above where they conflict +R1. ONE phase field for the fallback-aware choice: `Route = { tier?: string; + model?: string; effort?: Level }`. Default phases use `tier:` only (plan, + reflect, orchestrate, escalate → best; judge → big; implement, write, + verify, explore → work; mechanical → cheap). `acceptPhase` refuses a route + carrying both `tier` and `model`, and refuses a `tiers` key that collides + with a `models` alias. The earlier "model: 'best'" drafts and the "no new + tier field" sentence are VOID. `/route model=` keeps writing + `model`; the route tool schema is unchanged. +R2. Ids: `canonical(st, id)` = strip a trailing `[1m]`, then alias → table id, + then two-way prefix match against the table ids (`id.startsWith(tableId) + || tableId.startsWith(id)`), else the id itself. `aliasOf(st, id)` and + `modelRank(st, id)` (= index of the alias in `fallback`, `undefined` when + unknown) work on canonical ids. The existing effort `rank` keeps its name. + Breaker keys, `agentModels` values and comparisons are canonical. When the + current main model carries `[1m]`, a resolved replacement carries `[1m]` + too (the long-context tier is a property of the session, not of the + alias); log the first time it happens (unverified live: see Verify). +R3. Model axis per slot: `routeModelName(route) = route.tier ?? route.model`; + the main model axis is the FIRST defined `routeModelName` across + userMain, turnMain, turnFloor (per-axis, like B1's effort). `resolveName` + turns that name into an available id: a `tiers` key → first alias of the + list not down → `models` id; a tier whose every alias is down → + `nextAvailable(st, cur)` (global chain) → may be `undefined` (keep cur); + an alias or full id → canonical id, never skipped (explicit means explicit). +R4. Main decision `decideMain(st, cur, wanted, ctx)` → `{ model, why }`, used + by `mainPlan` AND by every text (texts pass `cur = canonical(await + $.session.model())`); order is BINDING: + 1. `st.off` → cur. + 2. `rankCur = modelRank(cur)`; UNKNOWN cur (not in the table) → cur, log + once per session (`model-router: unknown to the models table; no + model switch`), the breaker still applies at step 3 if it is down. + 3. cur DOWN → `wanted` if defined and not down, else `nextAvailable(cur)`; + apply `windowOk`; if nothing fits → cur (why `fallback`). + 4. `wanted` undefined or `aliasOf(wanted) === aliasOf(cur)` → cur. + 5. `modelRank(wanted) < rankCur` (better) → `cfg.mainUpgrade && + ctx.tokens <= cfg.upgradeMaxTokens` ? wanted (why `upgrade`) : cur + (why `upgrade skipped: context tokens over ` or `switch off`). + 6. cheaper → `cfg.mainModelSwitch && windowOk` ? wanted (why `downgrade`) + : cur (why `switch off`). + New config scalar `upgradeMaxTokens` (default 200000): an upgrade pays a + cold read of the whole context on the new model (LRN-204); above the + threshold it is skipped and logged once per turn. `ctx.tokens` comes from + `$.session.usage()` read once per main step (fail → treat as 0). +R5. Engine fallback respected: at every main step `sess = canonical(await + $.session.model())`; when `canonical(e.model) !== sess`, the engine is on + a fallback → `markDown(sess, 'engine fallback')` and `cur = e.model` (the + router never upgrades back to the model the engine just left). +R6. Breaker inputs (replace the r1 list): (a) `classic.StopFailure` with + `error` ∈ rate_limit | overloaded | billing_error | model_not_found → + `markDown(target)` where target = `st.agentModels.get(e.agent_id)` when + `e.agent_id` is set, else `st.lastPlan?.model`; other errors (context + limit = invalid_request, server_error, auth, max_output_tokens…) → nothing; + (b) R5's engine-fallback detection; (c) `classic.PostModelSwitch`: source + `command | picker | sdk` → `st.down.delete(canonical(to_model))` and reset + its strikes (the user's explicit `/model` wins); source `auto` → LOG only + (`requested_model`, from, to), never a mark (unverified semantics). + `turn.complete` `reason` is NOT a breaker input any more (context-limit + and network errors are not availability); refusal → nothing. + Backoff per canonical id: strikes 1, 2, 3… → 15, 30, 60, 120, 300 min + (cap); `model_not_found` → until `/route reload`. `markDown` logs ALWAYS: + `model-router: unavailable () until ; routing falls + back`. Inert while `st.off`. + Lifecycle: `/route reload` clears `down` and strikes BEFORE loading the + config (whatever the read result); `session.end` (/clear) KEEPS `down`, + strikes and `agentModels` (availability is account-wide); expired + entries are pruned at the start of any hook that reads them, with `now` + read ONLY when `st.down.size > 0` (`$.clock.now()`), passed explicitly to + the helpers (no clock read in sync text functions: they receive the + pruned map). +R7. Agent models: `st.agentModels: Map` set at spawn + from `started.model` (canonicalized; an alias answered by a hook above is + mapped through the table); deleted with the loop. No `Loop.model`, + `spawnModel` or `loop.model` identifier anywhere (W1-A AC8 grep). + `spawnRoute` resolves `route.tier ?? route.model` through `resolveName` + (skips down aliases); explicit `e.model` still wins even when down. + Deferred (noted): agents without a table row and no explicit model follow + `parentModel`; forks always inherit; neither falls back in wave 1. +R8. Derived orchestrate (D1) made exact: state `pushed: { prev: Routed | null; + spawnIds: Set } | null`. In the main Agent `tool.call` hook: + before `next`, if `st.turnMain?.source` is not 'model' or 'skill' and + `st.pushed` is null → `st.pushed = { prev: st.turnMain, spawnIds: new Set() }` + and `st.turnMain = { phase: 'orchestrate', route: phases.orchestrate, + source: 'derived' }`; after `next` resolves: the spawned `agentId` (from + `st.spawnByCall: Map` filled at `agent.spawn`) is + added to `pushed.spawnIds`; if NO agent was registered for this + `tool_use_id` (foreground run already finished, or denied) → nothing to + wait for from this call. Pop rule: when `pushed.spawnIds` is empty after + the Agent call returned, or when the LAST id of `pushed.spawnIds` ends + (`turn.complete` with that agentId, deleted from the set), and + `st.turnMain?.source === 'derived'` → `st.turnMain = pushed.prev`, + `st.pushed = null`. A route/skill write in between (source model/skill) + replaces turnMain; the pop then only clears `pushed`. `endMainTurn` + clears `pushed` and `spawnByCall`. In `turn.complete` for an agent, delete + the loop and the maps FIRST, inside `safely`, before any other work. +R9. Prompt default rules (D2) made safe: rules scanned in two passes (floor + rules, then default rules), each pass first match; absent `mode` → + 'floor' (B1 override files keep their meaning). Default rules are SKIPPED + when the trimmed text starts with `/` (slash commands and skills route + themselves), when the same prompt carries a floor match or sets + `typedSlash` (the user's explicit level wins), or when typed mid-turn. + Patterns compile with flags `iu` and the defaults use Unicode-aware + guards instead of `\b`: `(? until + () …| none`, each phase as `name=→/`. + Existing test 3f (`/route model=sonnet` shows `claude-sonnet-5-5`) is + adapted: on the kit's session model the line reads `asked claude-sonnet-5-5, + keeps (switch off)`; the alias→id resolution is asserted on the + `asked` part. +R12. `st.lastPlan: Plan | null` replaces `lastMain` and `lastMainModel`; the + spinner text is derived at render; `endMainTurn` resets it. +R13. Config validation additions: `tiers` values non-empty arrays of alias + keys (bad entries dropped, logged), `fallback` deduplicated non-empty + alias list (else default, logged), `cooldownMinutes` and + `upgradeMaxTokens` positive integers, `mode` ∈ floor|default, a log at + load when `tiers.best[0] !== fallback[0]` (rank comes from `fallback` + alone). `mainModelSwitch` documented as DOWNGRADE-only in the Config + comment. +R14. Tests (≥ 43 total, names carry the contract words): keep all 30; add: + `tier` (plan on a haiku session → fable xhigh, with mock.clock installed + where the breaker is touched), `downgrade` (mechanical on fable keeps + fable, switch off), `fallback` (plan route + `$.classic.StopFailure({ + error: 'rate_limit', … })` on main after a fable step → next step + `claude-opus-5-5` at xhigh; `/route reload` → fable again), `breaker` + ×3 (an aborted/`invalid_request` failure never marks down; backoff expiry + via `mock.clock` advance restores fable; `/model` command + `PostModelSwitch source: 'command'` clears a down model), `engine fallback` + (`$.session.model` mocked/answered as fable while the step arrives on + opus → no upgrade back, fable marked down), `unknown` (cur + `claude-zz-9` never switches), `spawn` (Explore → opus while sonnet is + down through an agent StopFailure with `agent_id`), `derived` ×2 (push on + dispatch, pop when the spawned agent ends → plan back; a route call after + the dispatch is NOT overwritten by the pop), `default rule` ×3 (planifie + → plan then a route call overrides; `/analyze …` typed → no rule; + `/effort-low pourquoi …` → no default rule, floor low), `per axis` + (`/route effort=low` sticky + turn `plan` tier → model axis = best). + Read `mock.clock` and how `$.session.model` is answered in the kit + (a bottom `on('session.model', …)` hook) before writing them. +R15. Disposition, superseded clauses named: floor contract AC4 "`turnMain` + only ever holds 'model' or 'skill' sources" → now also 'derived' and + 'prompt'; W1-A AC6 "main-loop model changes happen only when + `mainModelSwitch` is true" → true for DOWNGRADES only; upgrades follow + `mainUpgrade` + `upgradeMaxTokens`, and the breaker/engine-fallback path + moves off a dead model unconditionally; BDR-115 (6) window guard → applied + to every switch (up, down, fallback) through `windowOk`. The tiers + contract AC5 reads "every B1/1-A criterion still holds EXCEPT the three + clauses above". +R16. Live verification after reload (orchestrator, not the executor): the + `[1m]` carry-over on a fallback id, `PostModelSwitch` `source: 'auto'` + semantics, `StopFailure` reaching the mod with `agent_id`.