chore(memory): BDR-116 amendment (classifier) + EVAL-045 + journal/TODO/contract w4b — feat model-router wave 4-B

This commit is contained in:
bchanot
2026-10-11 21:07:04 +02:00
parent e54a2b5d61
commit 5389ee53e7
6 changed files with 220 additions and 3 deletions
+3 -2
View File
@@ -34,13 +34,14 @@ Plan `.claude/tasks/plans/2026-10-10-model-router-w3a-confirm-1201.md` r4, contr
- [x] W4-A transition logs + residuals (2026-10-11, 68d7ea6, contract `2026-10-11-model-router-w4a-logs-residuals-1637`, plan r3): verbose = transitions only (main line, hand-off per dispatch, agent moves, answered-by anomaly), stream lines removed, host prefix respected; residuals of W3-A/B/C closed (a)-(g). 3 lenses (BLOCKER: agent moves decided at spawn) + 1 confirmation, feater + 4 rounds, verifier at the cap on coverage (user accepted), security PASS + 2 MEDIUM fixed. Kit 284 → 358.
- [ ] W4-A residuals (LOW, accepted): `readLayer` logs `String(err)` of a JSON parse error (a file fragment can appear in the local log); `logCall`/`noteMain` ui calls on the step path sit outside `safely` (fail-open on routing, no security impact).
- [ ] push-guard hotfix: the PreToolUse hook denies a read-only command whose TEXT contains the push verb next to `git` (grep pattern, heredoc); match the command's own argv only.
- [ ] W4-B classifier (next): haiku `$.model.classify` on typed prompts → turn default route (plan/reflect/implement/apply/chat), first-use confirmation per phase, `/route classify on|off`, transition cause `classifier`.
- [x] W4-B classifier (2026-10-11, e54a2b5, contract `2026-10-11-model-router-w4b-classifier-1858`, plan r3): `$.model.classify` on plain composer prompts with no route in force → turn effort only (design→plan, analysis→reflect, coding→implement, bookkeeping→apply, chat→nothing), 3-s race, abort not counted, 3 failures pause `cooldownMinutes`, no dialog, no file write, `classify` config key, `/route classify on|off`, status + verbose line. 3 lenses (BLOCKER: unbounded await) + 1 confirmation, feater + 1 round, verifier CONFORME (2nd), security PASS. Kit 358 → 428. Deviation from the TODO line: no first-use confirmation on classified phases (noise per prompt; T3 already confirms the phase).
- [ ] W4-B residuals: `smallModelBusy` checks `cfg.models.haiku` but classify uses the engine's default small model (guard misses when they differ); abort paths judged by reading only; the 3-s tests pin the upper bound only. Live AC9 open: type a few plain prompts, read `route: classifier <phase>` (verbose line with `/route verbose on`), drop `bookkeeping` if it misfires.
- [ ] W3-A live checks still open: T1 (typed `/feat` etc.) dialog (T3 seen live 2026-10-11: orchestrate Keep); a parallel same-agent dispatch answered after > 10 s; what an unanswered dialog resolves to on the terminal (must be Later or nothing written). Record in the contract as `[gated]` when seen.
- [ ] W3-A residuals (security LOW, accepted): non-atomic cross-process write (two sessions, crash mid-write → file reads as failed, previous config kept); `host/path` of a private remote in the tracked file; a persistent write failure re-asks every use (`asked.delete` in the catch); `out.length` vs byte size at the cap; `changePatch` scope 'project' with a vanished key writes Everywhere (cwd change mid-dialog). Deferred: `/route set`, `/route confirm`, `(unconfirmed)` in show, dialog edits of phases, frontmatter auto-alignment, session-start warning when routing.json is dirty, per-machine `projects`.
- [ ] routing.json ships `confirmed` for verifier/feater (user Keeps) and phases verify/implement: every clone inherits them (by design: decisions travel). Revisit at release if unwanted.
- [ ] W2 residuals: probe 4 unobserved (above); the push-guard hook denies a read-only grep (or a heredoc) whose TEXT contains the push verb next to `git` (security-auditor + orchestrator 2026-10-10, false positives, reworded); ~27 loose "sonnet pin"/"opus pin" shorthand sites kept (true by census); README config key list omits `tiers`/`fallback`/`cooldownMinutes`/`mainUpgrade`/`upgradeMaxTokens` (doc audit item 6, pre-existing); MIGRATION.md "Upgrading to 3.0.0" at release time; `skillCalls`/`spawning` counters global; offers keyed by name.
- [ ] W3 optional: step heuristics, haiku classifier, quota-aware downgrade, A/B
- [ ] W3 optional: step heuristics, quota-aware downgrade, A/B (haiku classifier: done W4-B)
## 2026-09-30 — Higgsfield pack: CLI + skills in the install process, off by default (feature/higgsfield-pack)
Contract `.claude/tasks/contracts/2026-09-30-higgsfield-pack-1412.md`, spec + plan under
@@ -0,0 +1,32 @@
# CONTRACT — model-router-w4b-classifier
- date: 2026-10-11 | flow: feat | branch: feature/model-router-w4
- status: active
## REQUEST (verbatim — IMMUTABLE)
okm j'aimerais que tu corrige donc les residus techniques, et aussi que tu fasse cette wave optionnelle du classifieur. [earlier, same day: "est-ce quil y a une comande pour clear…"; "tu me dit le routing de tout skills et agents, mais c'est valable aussi pour la conversatio principale ? Si tu invoaue ni skill ni agebt il faut que ca route aussi on est d'accord ?" → orchestrator: the plain conversation routes only through prompt keywords, ultrathink, derived orchestrate and the route tool; the missing piece is the planned haiku classifier on each typed prompt → user chose it]
## CLARIFICATIONS
(pass A: none — the W1 plan § W3.2 and the 2026-10-11 exchange define outcome and scope)
Q: public shape (after three lenses, orchestrator default, user may veto): labels `design | analysis | coding | bookkeeping | chat` mapped to plan / reflect / implement / apply / none; the classified phase becomes the TURN DEFAULT route carrying its EFFORT ONLY (never a model move), and only when NO route is in force (no typed skill, no run slot, no route tool, no sticky `/route`, no floor, no keyword rule, not mid-turn, composer only); no first-use dialog on it (visibility: status line `route: classifier <phase>` and the transition line with `/route verbose on`); `classify` is a machine config key (default `true`, `"classify": false` in `~/.claude/model-router.json` turns it off) with a session toggle `/route classify on|off`; the call is bounded to 3 s and pauses for `cooldownMinutes` after 3 failures; one small-model call per typed prompt of ≥ 12 characters [stated 2026-10-11]
Q: the abort paths (`next.signal.aborted` first in the catch and after the await; classify's own `aborted` cause not counted) cannot be driven by the kit (no abortable dispatch, a throwing bottom hook becomes "no implementation") / A: judged by reading; the live check covers the rest [gated 2026-10-11]
## ACCEPTANCE CRITERIA
1. On a composer `prompt.submit` with `classify` on, not mid-turn, a trimmed text ≥ 12 chars not starting with `/`, no prompt rule matched and NO route in force on main (no sticky, turn, run or floor), the mod calls `$.model.classify(<first 500 + last 1500 chars>, [design, analysis, coding, bookkeeping, chat])` once, with no `model` option, and writes the turn route `{ effort }` of the mapped phase with source `classifier` (`chat` or an unknown label → nothing); it never runs for a sub-agent prompt, a non-composer origin (so never under `-p`), mid-turn, with the mod off, with `classify` off, when a rule matched, or when a route is in force. One kit test per clause (a bottom `model.classify` mock).
2. The classified route is turn-scoped (gone at `turn.complete`), carries the effort only (the main model never moves because of it: no tier), may lower the effort (coding → medium, bookkeeping → low), is replaced by any later route tool call or typed skill in the turn, and is ignored at a step whose engine effort differs from the session's baseline (a user `/effort` beats it). One kit test per clause.
3. Bounded and fail-safe: the call races a 3 s `$.clock.sleep` (timeout → no route, counts as a failure); a user interrupt (`next.signal.aborted`) neither counts nor writes; every guard is re-checked after the await (a route landing meanwhile wins); a rejection or timeout counts, three in a row pause the classifier for `cooldownMinutes` (×60000, kept across `/clear`, cleared by `/route reload`) with one toast and a status marker, a settled answer resets the counter; an unknown label neither warns nor counts; the classifier is skipped while the haiku id is marked down or the main loop runs on haiku. The prompt always proceeds. One kit test per clause.
4. No dialog and no file write: a classified route opens no first-use dialog and never writes routing.json. One kit test (no `ui.ask`, no `fs.write` on a classified prompt).
5. Config and command: `classify` is a Config key (default `true`) read from the machine override through `pickSwitch` (invalid → default + one log); `/route classify on|off` is a session toggle (composer only) that survives a rebuild like `switch`/`verbose`; `/route show` prints `classify: on|off` on the `router:` line; the argumentHint lists it. One kit test per clause.
6. Visibility: the status line reads `route: classifier <phase>` and the main transition line "(<phase>: classifier)" (with verbose on). One kit test per clause.
7. Kit suite, mods suite, validate and census green.
CHECK: cd mods/model-router && out="$(claude plugin test . 2>&1)" && printf '%s\n' "$out" | grep -qE '[0-9]+ pass' && ! printf '%s\n' "$out" | grep -qE '[1-9][0-9]* fail' && claude plugin validate . 2>&1 | grep -q 'passed' && cd ../.. && make test suite=lib/tests/mods.test.sh 2>&1 | grep -q 'all suites green' && bash lib/tests/effort-routing.test.sh >/dev/null 2>&1 && echo W4B-GREEN
EXPECT: W4B-GREEN
EVIDENCE: MET exit=0 marker-found :: W4B-GREEN
8. Docs: README (`classifier` paragraph in the model-router section: what, cost, `/route classify`), USAGE (`/route` row + one sentence in "Niveau d'effort"), CHANGELOG Unreleased Added, `lib/effort-shift.md` (≤ 2 lines).
CHECK: grep -qi 'classif' README.md && grep -qi 'classif' USAGE.md && grep -qi 'classif' CHANGELOG.md && grep -qi 'classif' lib/effort-shift.md && echo W4B-DOCS
EXPECT: W4B-DOCS
EVIDENCE: MET exit=0 marker-found :: W4B-DOCS
9. Live check (orchestrator + user, after the gates, `[gated]`): a few plain prompts typed in the terminal, each with the label it got (status line `route: classifier <phase>`, transition line with verbose on); drop `bookkeeping` if it misfires.
## FILE SCOPE
mods/model-router/hooks/register.ts, mods/model-router/hooks/register.test.ts, README.md, USAGE.md, CHANGELOG.md, lib/effort-shift.md, lib/tests/effort-routing.test.sh (only if a lock needs it)
@@ -0,0 +1,172 @@
# PLAN r3 — model-router wave 4-B: prompt classifier (2026-10-11)
r1 → r2 after simplicity CONCERNS(4), robustness FATAL(6) (BLOCKER: the
classify await is unbounded on the critical path), correctness CONCERNS(4).
r2 → r3 after the confirmation (robustness CONCERNS(4)). Precedence
r3 > r2 > r1. Contract `.claude/tasks/contracts/2026-10-11-model-router-
w4b-classifier-1858.md`. Base: feature/model-router-w4 at 86845fc.
Corrected facts: default keyword rules never run mid-turn and only write
plan/reflect (they never lower); `mainEffort` ranks sticky > turn > run >
floor > engine and `mainModel` ranks turn above floor; `runMain` holds a
typed best-tier skill's route across the run's turns; `$` calls do not
count against the hook budget (an unbounded `classify` await freezes the
prompt), while `$.clock.sleep` does; `ClassifyOptions` has `model` only;
`claude -p` prompts arrive with origin `sdk`; the breaker is fed by
`classic.StopFailure` only; `warnOnce` fires once per session; config
switches (`enabled`, `verbose`, `mainModelSwitch`) are Config keys read
through `pickSwitch` with session toggles; `confirmPhase` (T3) is
confirm-only, so a dialog on a classified phase would decide nothing.
## Decisions
- D1 no first-use dialog on classified phases (Later and Keep would apply
the same route). Visibility = the status line (`route: classifier
<phase>`) and the W4-A transition line (needs `/route verbose on`).
- D2 `classify` is a Config key (default `true`), read from the machine
override through `pickSwitch` (invalid → default + one log), with a
session toggle `/route classify on|off` next to `switch`/`verbose`,
printed on the `router:` line of `/route show`. Persistent off =
`"classify": false` in `~/.claude/model-router.json`. routing.json is
NOT touched; no new writer.
- D3 the classifier fills only an EMPTY route: it runs only when no route
is in force on main (`mainRoute(st) === null && st.turnFloor === null`:
no sticky `/route`, no turn route from a rule/skill/tool, no run slot
from a typed skill, no floor), evaluated after `routeFromPrompt`. Real
coverage, stated in the docs: a typed best-tier skill (feat, bugfix,
ship-feature…) holds its run slot across turns, so nothing is
classified until `/route clear`, `/route off`, a user `/model` or a
typed non-best skill; a non-best skill's row is turn-scoped, so its
follow-up prompts ARE classified (accepted: those rows are low/medium
and a complex follow-up may rightly rise). Never mid-turn (`e.turnId !==
undefined` → skip, as the rules). Never for a non-composer origin (so
never under `-p`). Never for a sub-agent prompt (prompt.submit is main
only; a test proves an agent's prompt is not classified).
- D4 a classified route carries the EFFORT ONLY: `turnMain = { phase,
route: { effort: cfg.phases[phase].effort }, source: 'classifier' }`.
No tier, so the main model never moves because of a guess (no cold
context re-reads on opus/sonnet sessions, no cost surprise). It may
lower the effort (coding = medium, bookkeeping = low); turn-scoped.
An explicit user `/effort` beats it: `st.engineBase` = the engine
effort of the first main step of the session (kept across /clear);
at `turn.step`, when `e.effort !== st.engineBase` (the user moved the
builtin effort) a `classifier`-sourced turn route is ignored for that
step (the engine effort applies) — "un choix explicite bat la phase
déduite" (BDR-115 amendment b).
- D5 bounded call: `label = await Promise.race([classify, $.clock.sleep(
CLASSIFY_MS = 3000, { signal: next.signal }).then(() => TIMEOUT)])`
inside try/catch: in the catch, `next.signal.aborted` is tested FIRST
(a user interrupt: no count, no write, return); a TIMEOUT counts as a
failure (D7) and gives no route; after the await every guard of D8 is
RE-CHECKED (`st.off`, the switch, no route in force, not paused) before
the write, and `next.signal.aborted` skips it (a late result never
lands in a later turn). The losing classify promise is left to settle
(the kit offers no cancel); stated.
- D6 labels are descriptive and mapped: `design` → plan, `analysis` →
reflect, `coding` → implement, `bookkeeping` → apply, `chat` → none
(lookup with `hasKey`; a mapped phase without an effort → no route).
The call passes no `model` option (the engine's small fast model; no
alias from a hook, LRN-203). The sample = the whole trimmed text when
≤ `CLASSIFY_MAX = 2000` code points, else the first `CLASSIFY_HEAD =
500` + last `CLASSIFY_TAIL = 1500` code points (cut by code points,
never a lone surrogate). A pasted tail can steer the label for one
turn (effort only, no file write): accepted and documented.
- D7 failures: a rejected call (classify's own `aborted` cause excepted)
or a TIMEOUT increments `st.classifyFails`; a settled answer resets it;
at `CLASSIFY_FAILS = 3` the classifier pauses until `clock.now() +
cooldownMinutes * 60000` (`st.classifyUntil`) with one toast
"classifier paused <n> min (3 failures)" and the status marker
`classifier: paused`; both fields are in `resetSession`'s kept set and
cleared by `/route reload` (like the breaker); the expiry check runs
BEFORE the route-in-force guard (the marker never stays stale); an
unknown label neither warns nor counts; the classifier is skipped while
`cfg.models.haiku` (a stated proxy for the engine's small model) is in
`st.down`, and when the main loop itself runs on haiku (the effort is
stripped there anyway).
- D8 guards summary (in order): `st.off` → `cfg.classify` (+ toggle) →
composer → `e.turnId === undefined` → not a slash → trimmed length ≥ 12
→ no rule matched this prompt → no route in force → not paused, model
not down → classify.
## Steps
- [ ] S1 config: `classify` key (DEFAULT true, pickSwitch), toggle,
`/route classify on|off`, show `router:` line, argumentHint.
- [ ] S2 `classifyPrompt($, st, e, next)` in the prompt.submit hook after
`routeFromPrompt` and the typed marker; `LABELS`/`LABEL_PHASE`
tables; the sample helper; the race; the failure counter and pause;
the effort-only route write with source `classifier`; `Source`
gains `'classifier'`; `slotWord` maps it to `classifier`; the
status line already prints `route: <source> <phase>`.
- [ ] S3 docs: README (classifier paragraph: what, one small-model call
per typed prompt when no route is in force, effort only, `/route
classify off` or `"classify": false` per machine, where to see it:
status line, `/route verbose on`); the `/route` subcommand lists
(README ×2, USAGE) gain `classify on|off`; the machine keys lists
(README, USAGE) gain `classify`; USAGE "Niveau d'effort" one
sentence; CHANGELOG Unreleased Added; `lib/effort-shift.md` ≤ 2
lines (the plain conversation routes by content now).
- [ ] S4 kit tests (a bottom `model.classify` mock; `$.clock` mock for
the race), one per clause: called once with the head+tail sample
and the five labels, no model option; each label → its phase effort
only (no tier on the route) and `chat`/`undefined` → none; never
resolving → no route after 3 s, prompt proceeds; rejection ×3 →
pause + toast + status marker, then no call during the cooldown,
a success resets the counter; `aborted` → no count, no warn; small
model down → not called; not called: short text, slash, sdk origin,
mid-turn (`turnId`), rule matched (keyword and ultrathink), sticky
`/route`, run slot held, floor carried, `classify` off (config,
toggle), mod off, an agent's prompt; turn-scoped (gone at
turn.complete); effort lowered to low for bookkeeping; the main
model unchanged on a sonnet session for `design`; `/route show`
prints `classify: on`; status line `route: classifier reflect`;
W4-A line "(reflect: classifier)"; invalid config value; 3 timeouts
→ pause; a dispatch abort mid-race → no count, no write; a `/route
<phase>` landing during the await → the classifier result dropped;
pause expiry at cooldown-1 ms (still paused) and +1 ms (resumes);
`/clear` keeps the pause, `/route reload` clears it; a user
`/effort` change (engine effort ≠ baseline) → the classified effort
ignored at the step; a 2000-code-point text sent whole, a longer
one head+tail with no lone surrogate; `constructor` label → none;
a phase without effort → none; main on haiku → not called; a
non-best skill's follow-up prompt classified; a plain prompt after
a finished best-tier run not classified until `/route clear`.
- Edge cases: a prompt that is only a URL or code (the classifier
decides; no special-casing); every classified prompt waits one small-
model round trip (typically under a second, 3 s at most: stated in the
README and measured in the AC9 live check); a `[1m]` session (no
effect: effort only).
- Disposition: honors BDR-115 (no alias from a hook, no model write, no
file write), BDR-116 (writers unchanged; no new dialog), LRN-203,
LRN-210, LRN-212 (live mod: the first plain prompt after the feater
saves is classified).
- Docs note for the user: a misclassification lowers one turn's effort
only; `/route <phase>` or `ultrathink` overrides it at once.
## Challenge ledger (r1 → r2)
- simp 1, corr 3/7, rob 6 (dialog decides nothing / misstates) → D1 dropped.
- simp 2, corr 11, rob 11 (flag plumbing, new writer) → D2 Config key.
- simp 3/4, corr 1/2/12, rob 3/4 (mid-turn, run slot, floor) → D3.
- rob 5 (model flips per turn) → D4 effort only.
- rob 1 BLOCKER, corr 9 (unbounded await, late write) → D5.
- simp 5, corr 4 (alias from a hook) → D6 no model option.
- corr 8, simp 8 (label words) → D6 descriptive labels mapped.
- rob 7 (head-only sample) → D6 head + tail.
- rob 2, corr 5 (failure handling, warn spent) → D7.
- corr 6, rob 9 (`-p`) → D3 never.
- corr 10 (sub-agent clause) → S4 test.
- simp 6, rob 10 (visibility) → status line + verbose note.
- simp 7 (doc lists) → S3 lists named.
## Confirmation ledger (r2 → r3)
- conf 1 (timeouts uncounted) → timeouts count toward the pause.
- conf 2 (abort lands in the counted branch) → `next.signal.aborted`
tested first in the catch.
- conf 3 (classifier overrides a user `/effort`) → `engineBase` guard at
the step: an explicit builtin effort beats a classified route.
- conf 4 (run coverage) → D3 states the real coverage; docs name it.
- conf 5 (guards not re-checked after the await) → re-checked.
- conf 6 (pause bookkeeping) → ×60000, kept across /clear, cleared on
reload, expiry checked first, boundary tests.
- conf 7 (sample, labels) → whole text ≤ 2000 cp, code-point cuts,
hasKey, no-effort phase skipped.
- conf 8 (down check) → haiku id as a stated proxy; skip on a haiku main.
- conf 9 (pasted tail) → accepted, documented.
- conf 10 (latency) → stated in the README and measured live.