chore(memory): BDR-116 amendment (classifier) + EVAL-045 + journal/TODO/contract w4b — feat model-router wave 4-B
This commit is contained in:
@@ -34,13 +34,14 @@ Plan `.claude/tasks/plans/2026-10-10-model-router-w3a-confirm-1201.md` r4, contr
|
||||
- [x] W4-A transition logs + residuals (2026-10-11, 68d7ea6, contract `2026-10-11-model-router-w4a-logs-residuals-1637`, plan r3): verbose = transitions only (main line, hand-off per dispatch, agent moves, answered-by anomaly), stream lines removed, host prefix respected; residuals of W3-A/B/C closed (a)-(g). 3 lenses (BLOCKER: agent moves decided at spawn) + 1 confirmation, feater + 4 rounds, verifier at the cap on coverage (user accepted), security PASS + 2 MEDIUM fixed. Kit 284 → 358.
|
||||
- [ ] W4-A residuals (LOW, accepted): `readLayer` logs `String(err)` of a JSON parse error (a file fragment can appear in the local log); `logCall`/`noteMain` ui calls on the step path sit outside `safely` (fail-open on routing, no security impact).
|
||||
- [ ] push-guard hotfix: the PreToolUse hook denies a read-only command whose TEXT contains the push verb next to `git` (grep pattern, heredoc); match the command's own argv only.
|
||||
- [ ] W4-B classifier (next): haiku `$.model.classify` on typed prompts → turn default route (plan/reflect/implement/apply/chat), first-use confirmation per phase, `/route classify on|off`, transition cause `classifier`.
|
||||
- [x] W4-B classifier (2026-10-11, e54a2b5, contract `2026-10-11-model-router-w4b-classifier-1858`, plan r3): `$.model.classify` on plain composer prompts with no route in force → turn effort only (design→plan, analysis→reflect, coding→implement, bookkeeping→apply, chat→nothing), 3-s race, abort not counted, 3 failures pause `cooldownMinutes`, no dialog, no file write, `classify` config key, `/route classify on|off`, status + verbose line. 3 lenses (BLOCKER: unbounded await) + 1 confirmation, feater + 1 round, verifier CONFORME (2nd), security PASS. Kit 358 → 428. Deviation from the TODO line: no first-use confirmation on classified phases (noise per prompt; T3 already confirms the phase).
|
||||
- [ ] W4-B residuals: `smallModelBusy` checks `cfg.models.haiku` but classify uses the engine's default small model (guard misses when they differ); abort paths judged by reading only; the 3-s tests pin the upper bound only. Live AC9 open: type a few plain prompts, read `route: classifier <phase>` (verbose line with `/route verbose on`), drop `bookkeeping` if it misfires.
|
||||
- [ ] W3-A live checks still open: T1 (typed `/feat` etc.) dialog (T3 seen live 2026-10-11: orchestrate Keep); a parallel same-agent dispatch answered after > 10 s; what an unanswered dialog resolves to on the terminal (must be Later or nothing written). Record in the contract as `[gated]` when seen.
|
||||
- [ ] W3-A residuals (security LOW, accepted): non-atomic cross-process write (two sessions, crash mid-write → file reads as failed, previous config kept); `host/path` of a private remote in the tracked file; a persistent write failure re-asks every use (`asked.delete` in the catch); `out.length` vs byte size at the cap; `changePatch` scope 'project' with a vanished key writes Everywhere (cwd change mid-dialog). Deferred: `/route set`, `/route confirm`, `(unconfirmed)` in show, dialog edits of phases, frontmatter auto-alignment, session-start warning when routing.json is dirty, per-machine `projects`.
|
||||
- [ ] routing.json ships `confirmed` for verifier/feater (user Keeps) and phases verify/implement: every clone inherits them (by design: decisions travel). Revisit at release if unwanted.
|
||||
|
||||
- [ ] W2 residuals: probe 4 unobserved (above); the push-guard hook denies a read-only grep (or a heredoc) whose TEXT contains the push verb next to `git` (security-auditor + orchestrator 2026-10-10, false positives, reworded); ~27 loose "sonnet pin"/"opus pin" shorthand sites kept (true by census); README config key list omits `tiers`/`fallback`/`cooldownMinutes`/`mainUpgrade`/`upgradeMaxTokens` (doc audit item 6, pre-existing); MIGRATION.md "Upgrading to 3.0.0" at release time; `skillCalls`/`spawning` counters global; offers keyed by name.
|
||||
- [ ] W3 optional: step heuristics, haiku classifier, quota-aware downgrade, A/B
|
||||
- [ ] W3 optional: step heuristics, quota-aware downgrade, A/B (haiku classifier: done W4-B)
|
||||
|
||||
## 2026-09-30 — Higgsfield pack: CLI + skills in the install process, off by default (feature/higgsfield-pack)
|
||||
Contract `.claude/tasks/contracts/2026-09-30-higgsfield-pack-1412.md`, spec + plan under
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# CONTRACT — model-router-w4b-classifier
|
||||
- date: 2026-10-11 | flow: feat | branch: feature/model-router-w4
|
||||
- status: active
|
||||
|
||||
## REQUEST (verbatim — IMMUTABLE)
|
||||
okm j'aimerais que tu corrige donc les residus techniques, et aussi que tu fasse cette wave optionnelle du classifieur. [earlier, same day: "est-ce quil y a une comande pour clear…"; "tu me dit le routing de tout skills et agents, mais c'est valable aussi pour la conversatio principale ? Si tu invoaue ni skill ni agebt il faut que ca route aussi on est d'accord ?" → orchestrator: the plain conversation routes only through prompt keywords, ultrathink, derived orchestrate and the route tool; the missing piece is the planned haiku classifier on each typed prompt → user chose it]
|
||||
|
||||
## CLARIFICATIONS
|
||||
(pass A: none — the W1 plan § W3.2 and the 2026-10-11 exchange define outcome and scope)
|
||||
Q: public shape (after three lenses, orchestrator default, user may veto): labels `design | analysis | coding | bookkeeping | chat` mapped to plan / reflect / implement / apply / none; the classified phase becomes the TURN DEFAULT route carrying its EFFORT ONLY (never a model move), and only when NO route is in force (no typed skill, no run slot, no route tool, no sticky `/route`, no floor, no keyword rule, not mid-turn, composer only); no first-use dialog on it (visibility: status line `route: classifier <phase>` and the transition line with `/route verbose on`); `classify` is a machine config key (default `true`, `"classify": false` in `~/.claude/model-router.json` turns it off) with a session toggle `/route classify on|off`; the call is bounded to 3 s and pauses for `cooldownMinutes` after 3 failures; one small-model call per typed prompt of ≥ 12 characters [stated 2026-10-11]
|
||||
|
||||
Q: the abort paths (`next.signal.aborted` first in the catch and after the await; classify's own `aborted` cause not counted) cannot be driven by the kit (no abortable dispatch, a throwing bottom hook becomes "no implementation") / A: judged by reading; the live check covers the rest [gated 2026-10-11]
|
||||
|
||||
## ACCEPTANCE CRITERIA
|
||||
1. On a composer `prompt.submit` with `classify` on, not mid-turn, a trimmed text ≥ 12 chars not starting with `/`, no prompt rule matched and NO route in force on main (no sticky, turn, run or floor), the mod calls `$.model.classify(<first 500 + last 1500 chars>, [design, analysis, coding, bookkeeping, chat])` once, with no `model` option, and writes the turn route `{ effort }` of the mapped phase with source `classifier` (`chat` or an unknown label → nothing); it never runs for a sub-agent prompt, a non-composer origin (so never under `-p`), mid-turn, with the mod off, with `classify` off, when a rule matched, or when a route is in force. One kit test per clause (a bottom `model.classify` mock).
|
||||
2. The classified route is turn-scoped (gone at `turn.complete`), carries the effort only (the main model never moves because of it: no tier), may lower the effort (coding → medium, bookkeeping → low), is replaced by any later route tool call or typed skill in the turn, and is ignored at a step whose engine effort differs from the session's baseline (a user `/effort` beats it). One kit test per clause.
|
||||
3. Bounded and fail-safe: the call races a 3 s `$.clock.sleep` (timeout → no route, counts as a failure); a user interrupt (`next.signal.aborted`) neither counts nor writes; every guard is re-checked after the await (a route landing meanwhile wins); a rejection or timeout counts, three in a row pause the classifier for `cooldownMinutes` (×60000, kept across `/clear`, cleared by `/route reload`) with one toast and a status marker, a settled answer resets the counter; an unknown label neither warns nor counts; the classifier is skipped while the haiku id is marked down or the main loop runs on haiku. The prompt always proceeds. One kit test per clause.
|
||||
4. No dialog and no file write: a classified route opens no first-use dialog and never writes routing.json. One kit test (no `ui.ask`, no `fs.write` on a classified prompt).
|
||||
5. Config and command: `classify` is a Config key (default `true`) read from the machine override through `pickSwitch` (invalid → default + one log); `/route classify on|off` is a session toggle (composer only) that survives a rebuild like `switch`/`verbose`; `/route show` prints `classify: on|off` on the `router:` line; the argumentHint lists it. One kit test per clause.
|
||||
6. Visibility: the status line reads `route: classifier <phase>` and the main transition line "(<phase>: classifier)" (with verbose on). One kit test per clause.
|
||||
7. Kit suite, mods suite, validate and census green.
|
||||
CHECK: cd mods/model-router && out="$(claude plugin test . 2>&1)" && printf '%s\n' "$out" | grep -qE '[0-9]+ pass' && ! printf '%s\n' "$out" | grep -qE '[1-9][0-9]* fail' && claude plugin validate . 2>&1 | grep -q 'passed' && cd ../.. && make test suite=lib/tests/mods.test.sh 2>&1 | grep -q 'all suites green' && bash lib/tests/effort-routing.test.sh >/dev/null 2>&1 && echo W4B-GREEN
|
||||
EXPECT: W4B-GREEN
|
||||
EVIDENCE: MET exit=0 marker-found :: W4B-GREEN
|
||||
8. Docs: README (`classifier` paragraph in the model-router section: what, cost, `/route classify`), USAGE (`/route` row + one sentence in "Niveau d'effort"), CHANGELOG Unreleased Added, `lib/effort-shift.md` (≤ 2 lines).
|
||||
CHECK: grep -qi 'classif' README.md && grep -qi 'classif' USAGE.md && grep -qi 'classif' CHANGELOG.md && grep -qi 'classif' lib/effort-shift.md && echo W4B-DOCS
|
||||
EXPECT: W4B-DOCS
|
||||
EVIDENCE: MET exit=0 marker-found :: W4B-DOCS
|
||||
9. Live check (orchestrator + user, after the gates, `[gated]`): a few plain prompts typed in the terminal, each with the label it got (status line `route: classifier <phase>`, transition line with verbose on); drop `bookkeeping` if it misfires.
|
||||
|
||||
## FILE SCOPE
|
||||
mods/model-router/hooks/register.ts, mods/model-router/hooks/register.test.ts, README.md, USAGE.md, CHANGELOG.md, lib/effort-shift.md, lib/tests/effort-routing.test.sh (only if a lock needs it)
|
||||
@@ -0,0 +1,172 @@
|
||||
# PLAN r3 — model-router wave 4-B: prompt classifier (2026-10-11)
|
||||
|
||||
r1 → r2 after simplicity CONCERNS(4), robustness FATAL(6) (BLOCKER: the
|
||||
classify await is unbounded on the critical path), correctness CONCERNS(4).
|
||||
r2 → r3 after the confirmation (robustness CONCERNS(4)). Precedence
|
||||
r3 > r2 > r1. Contract `.claude/tasks/contracts/2026-10-11-model-router-
|
||||
w4b-classifier-1858.md`. Base: feature/model-router-w4 at 86845fc.
|
||||
Corrected facts: default keyword rules never run mid-turn and only write
|
||||
plan/reflect (they never lower); `mainEffort` ranks sticky > turn > run >
|
||||
floor > engine and `mainModel` ranks turn above floor; `runMain` holds a
|
||||
typed best-tier skill's route across the run's turns; `$` calls do not
|
||||
count against the hook budget (an unbounded `classify` await freezes the
|
||||
prompt), while `$.clock.sleep` does; `ClassifyOptions` has `model` only;
|
||||
`claude -p` prompts arrive with origin `sdk`; the breaker is fed by
|
||||
`classic.StopFailure` only; `warnOnce` fires once per session; config
|
||||
switches (`enabled`, `verbose`, `mainModelSwitch`) are Config keys read
|
||||
through `pickSwitch` with session toggles; `confirmPhase` (T3) is
|
||||
confirm-only, so a dialog on a classified phase would decide nothing.
|
||||
|
||||
## Decisions
|
||||
- D1 no first-use dialog on classified phases (Later and Keep would apply
|
||||
the same route). Visibility = the status line (`route: classifier
|
||||
<phase>`) and the W4-A transition line (needs `/route verbose on`).
|
||||
- D2 `classify` is a Config key (default `true`), read from the machine
|
||||
override through `pickSwitch` (invalid → default + one log), with a
|
||||
session toggle `/route classify on|off` next to `switch`/`verbose`,
|
||||
printed on the `router:` line of `/route show`. Persistent off =
|
||||
`"classify": false` in `~/.claude/model-router.json`. routing.json is
|
||||
NOT touched; no new writer.
|
||||
- D3 the classifier fills only an EMPTY route: it runs only when no route
|
||||
is in force on main (`mainRoute(st) === null && st.turnFloor === null`:
|
||||
no sticky `/route`, no turn route from a rule/skill/tool, no run slot
|
||||
from a typed skill, no floor), evaluated after `routeFromPrompt`. Real
|
||||
coverage, stated in the docs: a typed best-tier skill (feat, bugfix,
|
||||
ship-feature…) holds its run slot across turns, so nothing is
|
||||
classified until `/route clear`, `/route off`, a user `/model` or a
|
||||
typed non-best skill; a non-best skill's row is turn-scoped, so its
|
||||
follow-up prompts ARE classified (accepted: those rows are low/medium
|
||||
and a complex follow-up may rightly rise). Never mid-turn (`e.turnId !==
|
||||
undefined` → skip, as the rules). Never for a non-composer origin (so
|
||||
never under `-p`). Never for a sub-agent prompt (prompt.submit is main
|
||||
only; a test proves an agent's prompt is not classified).
|
||||
- D4 a classified route carries the EFFORT ONLY: `turnMain = { phase,
|
||||
route: { effort: cfg.phases[phase].effort }, source: 'classifier' }`.
|
||||
No tier, so the main model never moves because of a guess (no cold
|
||||
context re-reads on opus/sonnet sessions, no cost surprise). It may
|
||||
lower the effort (coding = medium, bookkeeping = low); turn-scoped.
|
||||
An explicit user `/effort` beats it: `st.engineBase` = the engine
|
||||
effort of the first main step of the session (kept across /clear);
|
||||
at `turn.step`, when `e.effort !== st.engineBase` (the user moved the
|
||||
builtin effort) a `classifier`-sourced turn route is ignored for that
|
||||
step (the engine effort applies) — "un choix explicite bat la phase
|
||||
déduite" (BDR-115 amendment b).
|
||||
- D5 bounded call: `label = await Promise.race([classify, $.clock.sleep(
|
||||
CLASSIFY_MS = 3000, { signal: next.signal }).then(() => TIMEOUT)])`
|
||||
inside try/catch: in the catch, `next.signal.aborted` is tested FIRST
|
||||
(a user interrupt: no count, no write, return); a TIMEOUT counts as a
|
||||
failure (D7) and gives no route; after the await every guard of D8 is
|
||||
RE-CHECKED (`st.off`, the switch, no route in force, not paused) before
|
||||
the write, and `next.signal.aborted` skips it (a late result never
|
||||
lands in a later turn). The losing classify promise is left to settle
|
||||
(the kit offers no cancel); stated.
|
||||
- D6 labels are descriptive and mapped: `design` → plan, `analysis` →
|
||||
reflect, `coding` → implement, `bookkeeping` → apply, `chat` → none
|
||||
(lookup with `hasKey`; a mapped phase without an effort → no route).
|
||||
The call passes no `model` option (the engine's small fast model; no
|
||||
alias from a hook, LRN-203). The sample = the whole trimmed text when
|
||||
≤ `CLASSIFY_MAX = 2000` code points, else the first `CLASSIFY_HEAD =
|
||||
500` + last `CLASSIFY_TAIL = 1500` code points (cut by code points,
|
||||
never a lone surrogate). A pasted tail can steer the label for one
|
||||
turn (effort only, no file write): accepted and documented.
|
||||
- D7 failures: a rejected call (classify's own `aborted` cause excepted)
|
||||
or a TIMEOUT increments `st.classifyFails`; a settled answer resets it;
|
||||
at `CLASSIFY_FAILS = 3` the classifier pauses until `clock.now() +
|
||||
cooldownMinutes * 60000` (`st.classifyUntil`) with one toast
|
||||
"classifier paused <n> min (3 failures)" and the status marker
|
||||
`classifier: paused`; both fields are in `resetSession`'s kept set and
|
||||
cleared by `/route reload` (like the breaker); the expiry check runs
|
||||
BEFORE the route-in-force guard (the marker never stays stale); an
|
||||
unknown label neither warns nor counts; the classifier is skipped while
|
||||
`cfg.models.haiku` (a stated proxy for the engine's small model) is in
|
||||
`st.down`, and when the main loop itself runs on haiku (the effort is
|
||||
stripped there anyway).
|
||||
- D8 guards summary (in order): `st.off` → `cfg.classify` (+ toggle) →
|
||||
composer → `e.turnId === undefined` → not a slash → trimmed length ≥ 12
|
||||
→ no rule matched this prompt → no route in force → not paused, model
|
||||
not down → classify.
|
||||
|
||||
## Steps
|
||||
- [ ] S1 config: `classify` key (DEFAULT true, pickSwitch), toggle,
|
||||
`/route classify on|off`, show `router:` line, argumentHint.
|
||||
- [ ] S2 `classifyPrompt($, st, e, next)` in the prompt.submit hook after
|
||||
`routeFromPrompt` and the typed marker; `LABELS`/`LABEL_PHASE`
|
||||
tables; the sample helper; the race; the failure counter and pause;
|
||||
the effort-only route write with source `classifier`; `Source`
|
||||
gains `'classifier'`; `slotWord` maps it to `classifier`; the
|
||||
status line already prints `route: <source> <phase>`.
|
||||
- [ ] S3 docs: README (classifier paragraph: what, one small-model call
|
||||
per typed prompt when no route is in force, effort only, `/route
|
||||
classify off` or `"classify": false` per machine, where to see it:
|
||||
status line, `/route verbose on`); the `/route` subcommand lists
|
||||
(README ×2, USAGE) gain `classify on|off`; the machine keys lists
|
||||
(README, USAGE) gain `classify`; USAGE "Niveau d'effort" one
|
||||
sentence; CHANGELOG Unreleased Added; `lib/effort-shift.md` ≤ 2
|
||||
lines (the plain conversation routes by content now).
|
||||
- [ ] S4 kit tests (a bottom `model.classify` mock; `$.clock` mock for
|
||||
the race), one per clause: called once with the head+tail sample
|
||||
and the five labels, no model option; each label → its phase effort
|
||||
only (no tier on the route) and `chat`/`undefined` → none; never
|
||||
resolving → no route after 3 s, prompt proceeds; rejection ×3 →
|
||||
pause + toast + status marker, then no call during the cooldown,
|
||||
a success resets the counter; `aborted` → no count, no warn; small
|
||||
model down → not called; not called: short text, slash, sdk origin,
|
||||
mid-turn (`turnId`), rule matched (keyword and ultrathink), sticky
|
||||
`/route`, run slot held, floor carried, `classify` off (config,
|
||||
toggle), mod off, an agent's prompt; turn-scoped (gone at
|
||||
turn.complete); effort lowered to low for bookkeeping; the main
|
||||
model unchanged on a sonnet session for `design`; `/route show`
|
||||
prints `classify: on`; status line `route: classifier reflect`;
|
||||
W4-A line "(reflect: classifier)"; invalid config value; 3 timeouts
|
||||
→ pause; a dispatch abort mid-race → no count, no write; a `/route
|
||||
<phase>` landing during the await → the classifier result dropped;
|
||||
pause expiry at cooldown-1 ms (still paused) and +1 ms (resumes);
|
||||
`/clear` keeps the pause, `/route reload` clears it; a user
|
||||
`/effort` change (engine effort ≠ baseline) → the classified effort
|
||||
ignored at the step; a 2000-code-point text sent whole, a longer
|
||||
one head+tail with no lone surrogate; `constructor` label → none;
|
||||
a phase without effort → none; main on haiku → not called; a
|
||||
non-best skill's follow-up prompt classified; a plain prompt after
|
||||
a finished best-tier run not classified until `/route clear`.
|
||||
- Edge cases: a prompt that is only a URL or code (the classifier
|
||||
decides; no special-casing); every classified prompt waits one small-
|
||||
model round trip (typically under a second, 3 s at most: stated in the
|
||||
README and measured in the AC9 live check); a `[1m]` session (no
|
||||
effect: effort only).
|
||||
- Disposition: honors BDR-115 (no alias from a hook, no model write, no
|
||||
file write), BDR-116 (writers unchanged; no new dialog), LRN-203,
|
||||
LRN-210, LRN-212 (live mod: the first plain prompt after the feater
|
||||
saves is classified).
|
||||
- Docs note for the user: a misclassification lowers one turn's effort
|
||||
only; `/route <phase>` or `ultrathink` overrides it at once.
|
||||
|
||||
## Challenge ledger (r1 → r2)
|
||||
- simp 1, corr 3/7, rob 6 (dialog decides nothing / misstates) → D1 dropped.
|
||||
- simp 2, corr 11, rob 11 (flag plumbing, new writer) → D2 Config key.
|
||||
- simp 3/4, corr 1/2/12, rob 3/4 (mid-turn, run slot, floor) → D3.
|
||||
- rob 5 (model flips per turn) → D4 effort only.
|
||||
- rob 1 BLOCKER, corr 9 (unbounded await, late write) → D5.
|
||||
- simp 5, corr 4 (alias from a hook) → D6 no model option.
|
||||
- corr 8, simp 8 (label words) → D6 descriptive labels mapped.
|
||||
- rob 7 (head-only sample) → D6 head + tail.
|
||||
- rob 2, corr 5 (failure handling, warn spent) → D7.
|
||||
- corr 6, rob 9 (`-p`) → D3 never.
|
||||
- corr 10 (sub-agent clause) → S4 test.
|
||||
- simp 6, rob 10 (visibility) → status line + verbose note.
|
||||
- simp 7 (doc lists) → S3 lists named.
|
||||
|
||||
## Confirmation ledger (r2 → r3)
|
||||
- conf 1 (timeouts uncounted) → timeouts count toward the pause.
|
||||
- conf 2 (abort lands in the counted branch) → `next.signal.aborted`
|
||||
tested first in the catch.
|
||||
- conf 3 (classifier overrides a user `/effort`) → `engineBase` guard at
|
||||
the step: an explicit builtin effort beats a classified route.
|
||||
- conf 4 (run coverage) → D3 states the real coverage; docs name it.
|
||||
- conf 5 (guards not re-checked after the await) → re-checked.
|
||||
- conf 6 (pause bookkeeping) → ×60000, kept across /clear, cleared on
|
||||
reload, expiry checked first, boundary tests.
|
||||
- conf 7 (sample, labels) → whole text ≤ 2000 cp, code-point cuts,
|
||||
hasKey, no-effort phase skipped.
|
||||
- conf 8 (down check) → haiku id as a stated proxy; skip on a haiku main.
|
||||
- conf 9 (pasted tail) → accepted, documented.
|
||||
- conf 10 (latency) → stated in the README and measured live.
|
||||
Reference in New Issue
Block a user