diff --git a/CHANGELOG.md b/CHANGELOG.md index 74a59b1..8268b55 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,8 @@ Format follows [Keep a Changelog](https://keepachangelog.com/) and this project - **model-router first-use confirmation**: `mods/model-router/routing.json` (tracked) is the single source of the phase table and of every skill and agent row, and keeps the decisions: `confirmed`, `changed` (`from`/`to`), `projects` exceptions keyed by the origin remote reduced to `host/path` (credentials and local paths never stored), and `ask`. The first use of a rowed typed skill, a rowed agent spawn or a main-loop phase declared through the `route` tool opens a dialog with context: the skill's description (first sentence of its `SKILL.md` frontmatter) or the agent's, the phase with its `about` line (a new field on each of the 11 phases) and the model id and effort the next step really runs on. A row offers Later, Keep or Change; a main-loop phase Later or Keep. Change asks the model (fable, opus, sonnet, haiku with their tier), then the effort among those the phases of that model use (skipped when there is only one), then Everywhere or This project only; the pair maps to an existing phase (rows stay phase names, the same phase counts as Keep, a pair no phase offers is added by hand as a new phase). A skill-row change toasts the model and effort it now runs on, with the `/route switch on` hint when the main loop holds back a downgrade; a free-text answer counts as Later. One dialog at a time, never in headless (`-p`) runs or inside a sub-agent. Only a dialog answer, `/route ask on|off` or `/route forget` (which takes decisions back; a changed row returns to its shipped phase, the frontmatter floors stay yours to realign) writes the file (serialized, 64 KiB cap, never created when absent); each write asks you to commit it from the config repo. `/route pending` lists unconfirmed rows and phases. Layers: routing.json < `~/.claude/model-router.json` (its `ask` wins); a project's `.claude/model-router.json` is never read. Tests: `mods/model-router/hooks/register.test.ts`. - **Manual-push mode**: `git config gitflow.autopush false` (human-set) now stops every push the gitflow lib makes, not only the post-commit / post-merge hooks. `gitflow start` and `finish` branch, commit and merge locally and push nothing; `gitflow delete` leaves the `origin/` copy in place and prints `git push origin --delete
` for the user to run. `hooks/unpushed-guard.sh` stays silent at turn end in this mode and opens each session with one `ℹ manual push mode:` line counting the commits no remote holds across every local branch; an unparseable or unreadable `gitflow.autopush` value is treated as manual push mode too, and that line names it. `hooks/push-guard.sh` (PreToolUse, `Bash|Monitor`) refuses any `git push` Claude types while `gitflow.autopush` reads false in the session cwd or in a literal `-C`/`cd` directory the command names (global config counts outside a repo); the refusal tells the user to run it with `! git push`. It reads the mode through the same lib verb as every other reader and fails closed: an unparseable or unreadable value reads as manual, and an internal error, a missing `lib/gitflow.sh`, more than 20 distinct directory tokens in one command (capped before any token is classified), a `cd`/`-C` directory token mixing quoted and unquoted parts, or a payload jq cannot parse whose raw text looks like a push refuse the push (these pathological cases fire in auto mode too). Directory tokens are read as whole shell words, adjacent quoted segments and backslash escapes included. In manual mode it over-blocks any command where a `push` word follows a `git` token; the misses listed in its header fall to a new `autoMode.soft_deny` rule that no request in the turn clears. The session banner adds `🔒 push : manual (autopush=false) — ! git push` when the key reads false, and `🔒 push : manual (autopush bad) — ! git push` when the value is invalid. Skills read the mode through a new lib verb, `bash ~/.claude/lib/gitflow.sh push-mode`: it prints `auto`, `manual` or `invalid` (rc 0) and names an invalid value on stderr (printable characters only, 64 at most). It is the one reader a skill may call, since the `git config` read of the key is denied to Claude. Skills push nothing on their own, except the `/release-candidate` tag in auto-push mode on an explicit go. Every "on origin" or "not pushed" line they print comes from `git rev-list --count origin/
..
` read after the fact, with the complete `! git …` command when something is left for the user to push. An invalid value (anything but unset, true or false, or a read that fails) is manual push mode for every reader and is named where it is read (see Fixed). Tests: `lib/gitflow-test.sh` T11b (push-mode verb), T18m and T18q blocks, `lib/tests/unpushed-guard.test.sh` T10-T16, `lib/tests/push-guard.test.sh` (98 checks). +- **model-router prompt classifier**: a plain prompt typed in the composer (12 characters or more, not a slash command, not mid-turn, never under `-p` or in a sub-agent) that no keyword rule, skill, run slot, `/route` or floor covers is labelled by one call to the engine's small model (`design`, `analysis`, `coding`, `bookkeeping`, `chat`), bounded to 3 s. The label maps to plan, reflect, implement or apply and that phase's effort becomes the turn's default: effort only, no model move, no dialog, no file write, gone at the end of the turn, replaced by any route call or typed skill, and beaten by a `/effort` you set yourself. Three failures in a row pause it for `cooldownMinutes` (one toast, `classifier: paused` in the status line). `"classify": false` in `~/.claude/model-router.json` and `/route classify on|off` turn it off; `/route show` prints `classify:`. The status line reads `route: classifier ` and `/route verbose on` adds `(: classifier)`. + ### Changed - model-router `verbose: true` logs transitions only: one line when main changes model or effort (`model-router: main: fable/high → fable/xhigh (plan: route tool)`), one hand-off line per dispatch and one per later effort move of an agent, each with its cause; the per-step, "answered by", spawn, route-tool and skill-path lines are gone. An agent answering on a model other than the planned one is said once. A non-boolean `verbose` falls back to `false`, said once. - `security-auditor` runs on the `judge` row (opus/xhigh) by a first-use decision; its frontmatter floor is aligned (`model: opus`). diff --git a/README.md b/README.md index 24b0cbb..2f5d0cd 100644 --- a/README.md +++ b/README.md @@ -135,12 +135,13 @@ effort, never version. Transcript audit `python3 lib/effort-audit.py`. - Main loop: every request gets the route in force. A typed skill with a row routes the main loop to it; a best-tier row (`plan`, `reflect`, `orchestrate`, `escalate`) holds across turns in a run slot until `/route clear`, `/route off`, a user `/model` or a typed skill on a non-best row. A skill without a row leaves the route as it is. - Sub-agents: a routed agent gets its row's model at spawn, within its tier and only upward from its frontmatter model, and the row's effort on every step. Explicit `model` / `effort` params on the Agent call win; a project-defined agent of the same name keeps its own definition. Built-ins: Explore runs on sonnet/medium, Plan on opus/xhigh. - Levers inside a run: `ultrathink` in a prompt sets the turn's minimum effort; `/route effort=max` holds until `/route clear`. The built-in `/effort` is not a lever inside a run, rows and routes outrank it. -- `/route` (user command) shows or sets the route: `show`, `clear`, `off`, `on`, `reload`, `pending` (rows and phases not confirmed yet), `ask on|off` (first-use dialog on or off), `forget ` (takes decisions back: a changed row returns to its shipped phase; `all` and `projects` ask first), a phase name, `model= effort=`, `switch on|off`, `verbose on|off` (logs transitions only: each change of model or effort, with a few words of cause). `/route show` names the run slot when one holds (`main: run `). The model sets routes through a `route` tool. +- Prompt classifier: a plain prompt typed in the composer (12 characters or more, not a slash command) that no keyword rule, skill, run slot, `/route` or floor covers is sent to the engine's small fast model, one call, bounded to 3 seconds (the prompt waits for it, typically under a second). The label (`design`, `analysis`, `coding`, `bookkeeping`, `chat`) maps to plan, reflect, implement or apply (`chat` routes nothing), and that phase's effort becomes the turn's default. Effort only: the main model never moves because of a guess, no dialog opens and no file is written. The route is gone at the end of the turn, a later route call or typed skill replaces it, and a `/effort` you set yourself beats it. A typed best-tier skill holds its run slot across turns, so nothing is classified there until `/route clear`. Long prompts are sampled (first 500 and last 1500 code points), so pasted text at the end can steer one turn's effort. Three failures in a row pause it for `cooldownMinutes` (one toast, `classifier: paused` in the status line). Never in a sub-agent, mid-turn or under `-p`. Off per session with `/route classify off`, per machine with `"classify": false`. The status line reads `route: classifier `; `/route verbose on` adds the transition line. +- `/route` (user command) shows or sets the route: `show`, `clear`, `off`, `on`, `reload`, `pending` (rows and phases not confirmed yet), `ask on|off` (first-use dialog on or off), `forget ` (takes decisions back: a changed row returns to its shipped phase; `all` and `projects` ask first), a phase name, `model= effort=`, `switch on|off`, `classify on|off` (the prompt classifier, below), `verbose on|off` (logs transitions only: each change of model or effort, with a few words of cause). `/route show` names the run slot when one holds (`main: run `). The model sets routes through a `route` tool. - First use: the first time a rowed skill is typed, a rowed agent is spawned or a phase is declared on the main loop through the `route` tool, the mod asks once whether the route is right. The question gives context: the skill's description (first sentence of its `SKILL.md` frontmatter) or the agent's, the phase with its `about` line, and the model id and effort the next step really runs on; a declared phase also says what uses it (rows, prompt rules). A row offers Later, Keep or Change; a declared phase offers Later or Keep. Change asks the model (fable, opus, sonnet, haiku, each shown with the tier it heads), then the effort among those the phases on that model use (shipped table: fable medium, high, xhigh or max; sonnet low, medium, high or xhigh; opus and haiku have one level each, so no question), then Everywhere or This project only. Rows stay phase names, so the pair maps to an existing phase: the row's current phase wins a tie and the same phase counts as Keep; a pair no phase offers needs a new phase added by hand in routing.json. Everywhere moves the row and records the shipped phase under `changed`; This project only stores an exception under `projects`, keyed by the origin remote reduced to `host/path` (no credentials, no local paths; a remote that cannot be read that way offers no project choice). Keep lands under `confirmed`; Later, a dismissed dialog or a free-text answer asks again next session. After a skill row changes, a toast names the model and effort that skill now runs on, plus the `/route switch on` hint when the main loop holds back a downgrade. One dialog at a time, never in a headless (`-p`) run, never inside a sub-agent. - Only a dialog answer, `/route ask on|off` or `/route forget` writes `routing.json`, never the model. Writes are serialized, capped at 64 KiB, and refused when the file is missing (it is never created). Each write leaves the config repo dirty; a toast reminds you to commit it from there. - The spinner suffix and the status line under the prompt show the route in force. -Config layers: `mods/model-router/routing.json` (tracked: phases with their tier, effort and `about` line, rows, decisions, `ask`), then the optional per-machine `~/.claude/model-router.json`, which wins. A `.claude/model-router.json` inside a project is never read. Machine keys: `models` (alias → full id), `windows` (context window per full id), `tiers` (ordered alias lists per tier: `best` fable>opus>sonnet, `big` opus>fable>sonnet, `work` sonnet>opus, `cheap` haiku>sonnet; the first alias not down is used), `fallback` (the rank order of the aliases, best first, used by the breaker), `cooldownMinutes` (how long a model stays marked down after an availability error, 15 by default, doubling per episode up to 300), `mainUpgrade` (default `true`: the main loop may move up to a phase's tier), `upgradeMaxTokens` (default 200000: no main-loop upgrade above this context size, since an upgrade re-reads the whole context cold), `phases`, `agents`, `skills`, `prompt` (rules), `mainModelSwitch` (default `false`), `verbose` (default `false`; `true` logs transitions only, a non-boolean value falls back to `false`), `spinner` (default `true`), `enabled` (default `true`; `false` turns the mod off on that machine), `ask` (overrides routing.json's `ask` on that machine). A `null` value in `agents` or `skills` drops a row. Edit phases and rows by hand in routing.json (a phase's `about`, 120 characters at most, is read from there only and shown in the dialog; a model and effort pair no phase offers needs a new phase there); without it the mod runs the code's default phases, with no rows and no dialog. `/route reload` re-reads both files. +Config layers: `mods/model-router/routing.json` (tracked: phases with their tier, effort and `about` line, rows, decisions, `ask`), then the optional per-machine `~/.claude/model-router.json`, which wins. A `.claude/model-router.json` inside a project is never read. Machine keys: `models` (alias → full id), `windows` (context window per full id), `tiers` (ordered alias lists per tier: `best` fable>opus>sonnet, `big` opus>fable>sonnet, `work` sonnet>opus, `cheap` haiku>sonnet; the first alias not down is used), `fallback` (the rank order of the aliases, best first, used by the breaker), `cooldownMinutes` (how long a model stays marked down after an availability error, 15 by default, doubling per episode up to 300), `mainUpgrade` (default `true`: the main loop may move up to a phase's tier), `upgradeMaxTokens` (default 200000: no main-loop upgrade above this context size, since an upgrade re-reads the whole context cold), `phases`, `agents`, `skills`, `prompt` (rules), `mainModelSwitch` (default `false`), `classify` (default `true`; `false` stops the prompt classifier on that machine), `verbose` (default `false`; `true` logs transitions only, a non-boolean value falls back to `false`), `spinner` (default `true`), `enabled` (default `true`; `false` turns the mod off on that machine), `ask` (overrides routing.json's `ask` on that machine). A `null` value in `agents` or `skills` drops a row. Edit phases and rows by hand in routing.json (a phase's `about`, 120 characters at most, is read from there only and shown in the dialog; a model and effort pair no phase offers needs a new phase there); without it the mod runs the code's default phases, with no rows and no dialog. `/route reload` re-reads both files. Limits: the main loop changes model only with `mainModelSwitch` on, and each switch into another model costs one cold-cache step. The hooks send full model ids, so the `models` table has to follow new model versions. @@ -234,7 +235,7 @@ a different package, ships its own conflicting `graphify` bin) — see | `/profile` | Activate a skill profile (web / seo / web-full / full / max / backend / design / dev / qa / audit / minimal) (default: full) | | `/tour` | Grouped all-axes sweep — cleanup + security + reconcile + doc, fix and loop until clean | | `/site-motion` | Site-level motion: scroll engine choice, page transitions, pin/scrub sequencing across a page or Astro route (design stack) | -| `/route` | model-router mod: show or set the main-loop route (show, clear, off, on, reload, pending, ask on\|off, , model=… effort=…, switch on\|off, verbose on\|off: transitions only) | +| `/route` | model-router mod: show or set the main-loop route (show, clear, off, on, reload, pending, ask on\|off, , model=… effort=…, switch on\|off, classify on\|off, verbose on\|off: transitions only) | > This table lists personal skills. Gstack skills (investigate, review, retro, > office-hours, cso…) and marketplace plugins add many more — run diff --git a/USAGE.md b/USAGE.md index 21c05cc..8a92314 100644 --- a/USAGE.md +++ b/USAGE.md @@ -163,7 +163,7 @@ Tu veux... | `/pdf-translate` | Traduire un PDF vers une autre langue | Sortie HTML fidèle (images, layout, style préservés) | | `/impeccable` | Audit/polish design + détecteur anti-slop déterministe | 23 verbes ; `npx impeccable detect` (exit 0/2) | | `/tour` | Sweep groupé sur un ou plusieurs projets | Sécu + nettoyage + reconcile + doc, boucle jusqu'à un pass propre | -| `/route` | Voir ou fixer la route du mod model-router | show / clear / off / on / reload / pending / ask on\|off / / model=… effort=… / switch on\|off / verbose on\|off (transitions seules) | +| `/route` | Voir ou fixer la route du mod model-router | show / clear / off / on / reload / pending / ask on\|off / / model=… effort=… / switch on\|off / classify on\|off (classifieur de prompt) / verbose on\|off (transitions seules) | | `/profile` | Changer le profil de skills | web / seo / web-full / full / max / backend / design / dev / qa / audit / minimal | > Cette table couvre les skills personnels principaux. Les plugins (gstack, @@ -187,7 +187,7 @@ l'outil `mcp__model-router__route` (`lib/effort-shift.md`). Les skills externes vendorés (pile design, superpowers, agent-skills, skills scroll MengTo, 21st) ont aussi leur ligne. -Taper un skill qui a une ligne route la boucle principale dessus. Une ligne du tier best (plan, reflect, orchestrate, escalate) tient d'un tour à l'autre pendant tout le run, jusqu'à `/route clear`, `/route off`, un `/model` tapé ou un skill d'un autre tier tapé. Pour relancer un tour bloqué : `ultrathink` dans le prompt (plancher du tour) ou `/route effort=max` (tient jusqu'à `/route clear`). Le `/effort` intégré n'a pas d'effet dans un run : les lignes et les routes passent devant. Les sous-agents reçoivent le modèle de leur ligne au lancement (dans leur tier, jamais en dessous de leur frontmatter) et son niveau à chaque étape ; un `model` ou `effort` explicite sur l'appel gagne. Explore tourne en sonnet/medium, Plan en opus/xhigh. `/route show` affiche la route en cours, slot de run compris (`main: run `). Première utilisation : la première fois qu'un skill avec ligne est tapé, qu'un agent avec ligne est lancé ou qu'une phase est déclarée sur la boucle principale via l'outil `route`, le mod demande une fois si la route convient. La question donne le contexte : la description du skill (première phrase du frontmatter de son `SKILL.md`) ou celle de l'agent, la phase avec sa ligne `about` (champ de la phase dans `routing.json`), puis le modèle et le niveau réels de l'étape suivante ; une phase déclarée dit aussi qui l'utilise. Pour une ligne : Later, Keep ou Change ; pour une phase déclarée : Later ou Keep. Change demande le modèle (fable, opus, sonnet, haiku, chacun avec son tier), puis le niveau parmi ceux des phases de ce modèle (fable : medium, high, xhigh ou max ; sonnet : low, medium, high ou xhigh ; opus et haiku n'en ont qu'un, la question est sautée), puis Everywhere ou This project only (exception rangée sous le remote origin du dépôt réduit à `host/path`, jamais d'identifiants). Une ligne reste un nom de phase : le couple modèle et niveau désigne une phase existante, et la même phase qu'avant vaut Keep. Un couple qu'aucune phase n'offre s'ajoute à la main comme nouvelle phase dans `routing.json`. Après un changement sur un skill, un toast donne le modèle et le niveau réels, et rappelle `/route switch on` quand la boucle principale refuse de descendre. La réponse est écrite dans `routing.json` ; Later, un texte libre (Other) ou un dialogue fermé redemande à une session suivante. Un seul dialogue à la fois, jamais en headless (`-p`) ni dans un sous-agent, et le modèle n'écrit jamais ce fichier. Chaque réponse laisse le repo de config modifié : commite-le depuis ce repo. `/route pending` liste ce qui reste à confirmer, `/route ask off|on` coupe ou rallume le dialogue, `/route forget ` efface une décision (une ligne changée revient à la phase livrée ; `all` et `projects` demandent d'abord). +Taper un skill qui a une ligne route la boucle principale dessus. Une ligne du tier best (plan, reflect, orchestrate, escalate) tient d'un tour à l'autre pendant tout le run, jusqu'à `/route clear`, `/route off`, un `/model` tapé ou un skill d'un autre tier tapé. Pour relancer un tour bloqué : `ultrathink` dans le prompt (plancher du tour) ou `/route effort=max` (tient jusqu'à `/route clear`). Le `/effort` intégré n'a pas d'effet dans un run : les lignes et les routes passent devant. Un prompt tapé sans règle, skill ni route en cours passe par un classifieur (un appel au petit modèle, 3 secondes au plus) : son niveau d'effort devient celui du tour, sans jamais changer de modèle ; `/route classify off` ou `"classify": false` dans la config machine le coupe, et ton propre `/effort` passe devant. Les sous-agents reçoivent le modèle de leur ligne au lancement (dans leur tier, jamais en dessous de leur frontmatter) et son niveau à chaque étape ; un `model` ou `effort` explicite sur l'appel gagne. Explore tourne en sonnet/medium, Plan en opus/xhigh. `/route show` affiche la route en cours, slot de run compris (`main: run `). Première utilisation : la première fois qu'un skill avec ligne est tapé, qu'un agent avec ligne est lancé ou qu'une phase est déclarée sur la boucle principale via l'outil `route`, le mod demande une fois si la route convient. La question donne le contexte : la description du skill (première phrase du frontmatter de son `SKILL.md`) ou celle de l'agent, la phase avec sa ligne `about` (champ de la phase dans `routing.json`), puis le modèle et le niveau réels de l'étape suivante ; une phase déclarée dit aussi qui l'utilise. Pour une ligne : Later, Keep ou Change ; pour une phase déclarée : Later ou Keep. Change demande le modèle (fable, opus, sonnet, haiku, chacun avec son tier), puis le niveau parmi ceux des phases de ce modèle (fable : medium, high, xhigh ou max ; sonnet : low, medium, high ou xhigh ; opus et haiku n'en ont qu'un, la question est sautée), puis Everywhere ou This project only (exception rangée sous le remote origin du dépôt réduit à `host/path`, jamais d'identifiants). Une ligne reste un nom de phase : le couple modèle et niveau désigne une phase existante, et la même phase qu'avant vaut Keep. Un couple qu'aucune phase n'offre s'ajoute à la main comme nouvelle phase dans `routing.json`. Après un changement sur un skill, un toast donne le modèle et le niveau réels, et rappelle `/route switch on` quand la boucle principale refuse de descendre. La réponse est écrite dans `routing.json` ; Later, un texte libre (Other) ou un dialogue fermé redemande à une session suivante. Un seul dialogue à la fois, jamais en headless (`-p`) ni dans un sous-agent, et le modèle n'écrit jamais ce fichier. Chaque réponse laisse le repo de config modifié : commite-le depuis ce repo. `/route pending` liste ce qui reste à confirmer, `/route ask off|on` coupe ou rallume le dialogue, `/route forget ` efface une décision (une ligne changée revient à la phase livrée ; `all` et `projects` demandent d'abord). La config par machine, optionnelle, vit dans `~/.claude/model-router.json` et passe devant `routing.json` ; `"enabled": false` y coupe le mod sur cette machine, `"ask": false` y coupe le dialogue, une ligne à `null` y retire une ligne. Un `.claude/model-router.json` dans un projet n'est jamais lu. Les phases et les lignes se modifient à la main dans `routing.json` ; `/route reload` relit les deux fichiers. diff --git a/lib/effort-shift.md b/lib/effort-shift.md index 84c099c..8d22876 100644 --- a/lib/effort-shift.md +++ b/lib/effort-shift.md @@ -47,6 +47,8 @@ Builtin `/effort` is NOT a lever inside a run: rows and routes outrank it. - A skill typed while a background agent is live routes only through the typed marker (unverified live 2026-10-10). - Headless (`-p`, SDK) runs the hooks, so routing works there too. +- A plain typed prompt with no route in force gets its turn effort from a + small-model classifier (effort only; `/route classify off`; never `-p`). - Mod off: typed agents fall back to their `model:`/`effort:` frontmatter. - First use of a row asks once, with context (Later, Keep or Change: model, then effort, then Everywhere or this project only); the answer is kept in diff --git a/mods/model-router/hooks/register.test.ts b/mods/model-router/hooks/register.test.ts index 96d61cd..734f838 100644 --- a/mods/model-router/hooks/register.test.ts +++ b/mods/model-router/hooks/register.test.ts @@ -4377,3 +4377,622 @@ test('logs: an ended agent forgets its answer marks (reused id)', async ( await runStep($, { ...highStep('a1'), model: SONNET }) expect(answers(logs)).toHaveLength(2) }) + +// ---- wave 4-B: prompt classifier ----------------------------------------- + +type Ask = { text: string; labels: readonly string[]; options: unknown } +type Reply = (n: number) => Promise | string | undefined +type Cls = { + asks: Ask[] + seen: Seen[] + clock: MockClock + logs: string[] + toasts: string[] + status: string[] + world: World +} +type ClsOpts = { + reply?: Reply + model?: string + world?: World + verbose?: boolean +} + +const PLAIN = 'refactor the billing module please' +const PAUSE_MS = 15 * 60000 // cooldownMinutes default, in ms + +/** A ui.status recorder, same shape as recordLogs. */ +function recordStatus(on: On): string[] { + const lines: string[] = [] + on('ui.status', async ($, e, next) => { + lines.push(e.text ?? '') + try { + await next(e) + } catch { + // nothing implements ui.status below the recorder in the kit + } + }) + return lines +} + +/** A bottom `model.classify` answering `reply`; records every ask. */ +async function bootCls($: Engine, on: On, o: ClsOpts = {}): Promise { + const logs = recordLogs(on) + const toasts = recordToasts(on) + const status = recordStatus(on) + const seen: Seen[] = [] + const asks: Ask[] = [] + recordSteps(on, seen) + on('prompt.submit', ($, e) => ({ text: e.text })) + on('turn.complete', () => ({ text: '' })) + on('session.end', ($, e) => ({ sessionId: e.sessionId })) + on('tool.call', { tool: 'Skill' }, () => ({ + result: { success: true, commandName: 'bottom' }, + })) + on('agent.spawn', ($, e) => ({ + model: e.model ?? e.parentModel, + agentId: 'a1', + })) + on('model.classify', async (_$, e) => { + asks.push({ text: e.text, labels: e.labels, options: e.options }) + return { value: await (o.reply ?? (() => 'coding'))(asks.length) } + }) + const world = o.world ?? newWorld() + const clock = await boot($, on, o.model ?? FABLE, 1000, world) + if (o.verbose) await route($, 'verbose on') + logs.splice(0) + return { asks, seen, clock, logs, toasts, status, world } +} + +const clearSession = ($: Engine) => $.session.end({ + reason: 'clear', + sessionId: 's1', + resume: { id: 's1' }, +}) + +/** The effort a main step reaches the bottom with (engine asks high). */ +async function clsEffort($: Engine, c: Cls): Promise { + await runStep($, highStep()) + return c.seen[c.seen.length - 1]?.effort +} + +const never = (): Promise => new Promise(() => undefined) +const boom = (): never => { + throw new Error('overloaded') +} + +/** `n` typed prompts that each fail, a turn end between them. */ +async function failTimes($: Engine, n: number): Promise { + for (let i = 0; i < n; i++) { + await typed($, PLAIN) + await endTurn($) + } +} + +const LABELS_EFFORT: [string, string][] = [ + ['design', 'xhigh'], + ['analysis', 'high'], + ['coding', 'medium'], + ['bookkeeping', 'low'], +] + +test('classify: one call with the five labels and no model option', async ( + $, on) => { + const c = await bootCls($, on) + await typed($, PLAIN) + expect(c.asks).toEqual([{ + text: PLAIN, + labels: ['design', 'analysis', 'coding', 'bookkeeping', 'chat'], + options: undefined, + }]) +}) + +for (const [label, effort] of LABELS_EFFORT) { + test(`classify: ${label} becomes the turn effort ${effort}`, async ( + $, on) => { + const c = await bootCls($, on, { reply: () => label }) + await runStep($, highStep()) + await typed($, PLAIN) + expect(await clsEffort($, c)).toBe(effort) + }) +} + +test('classify: coding lowers the engine effort to medium', async ($, on) => { + const c = await bootCls($, on) + await typed($, PLAIN) + expect(await clsEffort($, c)).toBe('medium') +}) + +test('classify: a chat answer routes nothing', async ($, on) => { + const c = await bootCls($, on, { reply: () => 'chat' }) + await typed($, PLAIN) + expect(await clsEffort($, c)).toBe('high') +}) + +test('classify: no label at all routes nothing', async ($, on) => { + const c = await bootCls($, on, { reply: () => undefined }) + await typed($, PLAIN) + expect(await clsEffort($, c)).toBe('high') +}) + +test('classify: a label naming an object key routes nothing', async ( + $, on) => { + const c = await bootCls($, on, { reply: () => 'constructor' }) + await typed($, PLAIN) + expect(await clsEffort($, c)).toBe('high') +}) + +test('classify: an unknown label neither warns nor counts', async ($, on) => { + const c = await bootCls($, on, { reply: () => 'nonsense' }) + await failTimes($, 4) + expect(c.asks).toHaveLength(4) + expect(c.toasts).toEqual([]) + expect(c.logs).toEqual([]) +}) + +test('classify: the main model never moves for a design guess', async ( + $, on) => { + const c = await bootCls($, on, { model: SONNET, reply: () => 'design' }) + await typed($, PLAIN) + await runStep($, { ...highStep(), model: SONNET }) + expect(c.seen[c.seen.length - 1]).toEqual({ model: SONNET, effort: 'xhigh' }) +}) + +test('classify: the route carries no tier', async ($, on) => { + const c = await bootCls($, on, { reply: () => 'design' }) + await typed($, PLAIN) + const line = mainLine(await route($, 'show')) + expect(line).toContain('classifier plan') + expect(line).not.toContain('[tier') + expect(c.asks).toHaveLength(1) +}) + +test('classify: the route is gone at turn.complete', async ($, on) => { + const c = await bootCls($, on, { reply: () => 'design' }) + await typed($, PLAIN) + await endTurn($) + expect(await clsEffort($, c)).toBe('high') + expect(mainLine(await route($, 'show'))).toContain('session defaults') +}) + +test('classify: a route tool call in the turn replaces it', async ($, on) => { + const c = await bootCls($, on, { reply: () => 'bookkeeping' }) + await typed($, PLAIN) + await $.tool.call({ tool: ROUTE_TOOL, phase: 'plan' }) + expect(await clsEffort($, c)).toBe('xhigh') +}) + +test('classify: a typed skill in the turn replaces it', async ($, on) => { + const c = await bootCls($, on, { reply: () => 'design' }) + await typed($, PLAIN) + await typed($, '/feat add a thing') + await skillPrompt($, 'feat') + expect(await clsEffort($, c)).toBe('high') +}) + +test('classify: a user /effort change beats the guessed effort', async ( + $, on) => { + const c = await bootCls($, on, { reply: () => 'design' }) + await runStep($, highStep()) + await typed($, PLAIN) + await runStep($, { ...highStep(), effort: 'low' as const }) + expect(c.seen[c.seen.length - 1]?.effort).toBe('low') +}) + +test('classify: the baseline survives /clear', async ($, on) => { + const c = await bootCls($, on, { reply: () => 'design' }) + await runStep($, highStep()) + await clearSession($) + await typed($, PLAIN) + await runStep($, { ...highStep(), effort: 'low' as const }) + expect(c.seen[c.seen.length - 1]?.effort).toBe('low') +}) + +test('classify: a baseline-effort step takes the guessed effort', async ( + $, on) => { + const c = await bootCls($, on, { reply: () => 'design' }) + await runStep($, highStep()) + await typed($, PLAIN) + await runStep($, highStep()) + expect(c.seen[c.seen.length - 1]?.effort).toBe('xhigh') +}) + +// ---- the guards: one clause each --- + +/** The prompt is typed, then the asks are counted. */ +async function asksFor($: Engine, c: Cls, text: string): Promise { + await typed($, text) + return c.asks.length +} + +test('classify guard: text under 12 characters is not sent', async ( + $, on) => { + const c = await bootCls($, on) + expect(await asksFor($, c, 'hello world')).toBe(0) +}) + +test('classify guard: 12 characters are sent', async ($, on) => { + const c = await bootCls($, on) + expect(await asksFor($, c, 'hello worlds')).toBe(1) +}) + +test('classify guard: padded short text is not sent', async ($, on) => { + const c = await bootCls($, on) + expect(await asksFor($, c, ' short ')).toBe(0) +}) + +test('classify guard: a padded slash prompt is not sent', async ($, on) => { + const c = await bootCls($, on) + expect(await asksFor($, c, ' /feat add a long thing')).toBe(0) +}) + +test('classify guard: the sample is sent trimmed', async ($, on) => { + const c = await bootCls($, on) + await typed($, ` ${PLAIN} `) + expect(c.asks.map(a => a.text)).toEqual([PLAIN]) +}) + +test('classify guard: a slash prompt is not sent', async ($, on) => { + const c = await bootCls($, on) + expect(await asksFor($, c, '/unknown with a long argument')).toBe(0) +}) + +test('classify guard: a non-composer origin is not sent', async ($, on) => { + const c = await bootCls($, on) + await $.prompt.submit({ text: PLAIN, wait: false, origin: { kind: 'sdk' } }) + expect(c.asks).toHaveLength(0) +}) + +test('classify guard: a mid-turn prompt is not sent', async ($, on) => { + const c = await bootCls($, on) + await queued($, 'composer', PLAIN) + expect(c.asks).toHaveLength(0) +}) + +test('classify guard: a keyword rule match is not sent', async ($, on) => { + const c = await bootCls($, on) + expect(await asksFor($, c, 'please review the billing module')).toBe(0) +}) + +test('classify guard: ultrathink is not sent', async ($, on) => { + const c = await bootCls($, on) + expect(await asksFor($, c, 'ultrathink about the billing module')).toBe(0) +}) + +test('classify guard: a sticky /route is not overridden', async ($, on) => { + const c = await bootCls($, on) + await route($, 'mechanical') + expect(await asksFor($, c, PLAIN)).toBe(0) +}) + +test('classify guard: a held run slot is not sent', async ($, on) => { + const c = await bootCls($, on) + await typed($, '/feat add a thing') + await skillPrompt($, 'feat') + await endTurn($) + expect(await asksFor($, c, PLAIN)).toBe(0) +}) + +test('classify guard: a carried floor is not sent', async ($, on) => { + const c = await bootCls($, on) + await queued($, 'composer', 'ultrathink about the billing module') + await endTurn($) + expect(await asksFor($, c, PLAIN)).toBe(0) +}) + +test('classify guard: a sub-agent never reaches the classifier', async ( + $, on) => { + const c = await bootCls($, on) + await $.agent.spawn(spawnInput()) + await runStep($, highStep('a1')) + expect(c.asks).toHaveLength(0) +}) + +test('classify guard: a config classify=false is not sent', async ($, on) => { + const c = await bootCls($, on, { + world: newWorld({ home: { classify: false } }), + }) + expect(await asksFor($, c, PLAIN)).toBe(0) +}) + +test('classify guard: /route classify off is not sent', async ($, on) => { + const c = await bootCls($, on) + await route($, 'classify off') + expect(await asksFor($, c, PLAIN)).toBe(0) +}) + +test('classify guard: the mod off is not sent', async ($, on) => { + const c = await bootCls($, on) + await route($, 'off') + expect(await asksFor($, c, PLAIN)).toBe(0) +}) + +test('classify guard: a haiku id marked down is not sent', async ($, on) => { + const c = await bootCls($, on) + await switchTo($, HAIKU, SONNET, 'auto') + expect(await asksFor($, c, PLAIN)).toBe(0) +}) + +test('classify guard: a main loop on haiku is not sent', async ($, on) => { + const c = await bootCls($, on, { model: HAIKU }) + expect(await asksFor($, c, PLAIN)).toBe(0) +}) + +test('classify guard: a phase without effort routes nothing', async ( + $, on) => { + const world = newWorld({ edit: f => { + (f.phases as Rec).plan = { tier: 'best' } + } }) + const c = await bootCls($, on, { reply: () => 'design', world }) + await typed($, PLAIN) + expect(await clsEffort($, c)).toBe('high') +}) + +test('classify guard: a non-best skill turn leaves the next open', async ( + $, on) => { + const c = await bootCls($, on) + await typed($, '/status now') + await skillPrompt($, 'status') + await endTurn($) + expect(await asksFor($, c, PLAIN)).toBe(1) +}) + +test('classify guard: /route clear reopens it after a best-tier run', async ( + $, on) => { + const c = await bootCls($, on) + await typed($, '/feat add a thing') + await skillPrompt($, 'feat') + await endTurn($) + await route($, 'clear') + expect(await asksFor($, c, PLAIN)).toBe(1) +}) + +test('classify guard: a route landing during the call wins', async ( + $, on) => { + let release: () => void = () => undefined + const gate = new Promise(resolve => { release = resolve }) + const c = await bootCls($, on, { + reply: async () => { await gate; return 'design' }, + }) + const pending = typed($, PLAIN) + await c.clock.settle() + await $.tool.call({ tool: ROUTE_TOOL, phase: 'mechanical' }) + release() + await pending + expect(mainLine(await route($, 'show'))).toContain('model mechanical') + expect(await clsEffort($, c)).toBe('low') +}) + +// ---- bounded and fail-safe --- + +test('classify bound: a call never answering gives no route at 3 s', async ( + $, on) => { + const c = await bootCls($, on, { reply: never }) + const pending = typed($, PLAIN) + await c.clock.settle() + await c.clock.advance(3000) + await pending + expect(await clsEffort($, c)).toBe('high') +}) + +test('classify bound: the prompt proceeds after a timeout', async ($, on) => { + const c = await bootCls($, on, { reply: never }) + const pending = typed($, PLAIN) + await c.clock.settle() + await c.clock.advance(3000) + expect((await pending).text).toBe(PLAIN) +}) + +test('classify bound: three timeouts pause the classifier', async ($, on) => { + const c = await bootCls($, on, { reply: never }) + for (let i = 0; i < 3; i++) { + const pending = typed($, PLAIN) + await c.clock.settle() + await c.clock.advance(3000) + await pending + } + expect(c.toasts).toEqual(['classifier paused 15 min (3 failures)']) +}) + +test('classify bound: a rejection is not an error for the prompt', async ( + $, on) => { + const c = await bootCls($, on, { reply: boom }) + await typed($, PLAIN) + expect(c.logs.filter(l => l.includes('failed'))).toEqual([]) +}) + +test('classify fail: three rejections pause with one toast', async ($, on) => { + const c = await bootCls($, on, { reply: boom }) + await failTimes($, 3) + expect(c.toasts).toEqual(['classifier paused 15 min (3 failures)']) +}) + +test('classify fail: two rejections do not pause', async ($, on) => { + const c = await bootCls($, on, { reply: boom }) + await failTimes($, 2) + expect(c.toasts).toEqual([]) +}) + +test('classify fail: the status line shows the pause marker', async ( + $, on) => { + const c = await bootCls($, on, { reply: boom }) + await failTimes($, 3) + expect(c.status[c.status.length - 1]).toContain('classifier: paused') +}) + +test('classify fail: no call during the cooldown', async ($, on) => { + const c = await bootCls($, on, { reply: boom }) + await failTimes($, 3) + expect(await asksFor($, c, PLAIN)).toBe(3) +}) + +test('classify fail: a settled answer resets the counter', async ($, on) => { + const c = await bootCls($, on, { + reply: n => (n === 3 ? 'chat' : boom()), + }) + await failTimes($, 5) + expect(c.toasts).toEqual([]) + expect(c.asks).toHaveLength(5) +}) + +test('classify pause: still paused one ms before the cooldown ends', async ( + $, on) => { + const c = await bootCls($, on, { reply: boom }) + await failTimes($, 3) + await c.clock.advance(PAUSE_MS - 1) + expect(await asksFor($, c, PLAIN)).toBe(3) +}) + +test('classify pause: resumes when the cooldown ends', async ($, on) => { + const c = await bootCls($, on, { reply: boom }) + await failTimes($, 3) + await c.clock.advance(PAUSE_MS) + expect(await asksFor($, c, PLAIN)).toBe(4) +}) + +test('classify pause: the marker clears when it ends', async ($, on) => { + const c = await bootCls($, on, { reply: boom }) + await failTimes($, 3) + await c.clock.advance(PAUSE_MS) + await typed($, '/status') + expect(c.status[c.status.length - 1]).not.toContain('paused') +}) + +test('classify pause: /clear keeps it', async ($, on) => { + const c = await bootCls($, on, { reply: boom }) + await failTimes($, 3) + await clearSession($) + expect(await asksFor($, c, PLAIN)).toBe(3) +}) + +test('classify pause: /route reload clears it', async ($, on) => { + const c = await bootCls($, on, { reply: boom }) + await failTimes($, 3) + await route($, 'reload') + expect(await asksFor($, c, PLAIN)).toBe(4) +}) + +// ---- sample --- + +test('classify sample: 2000 code points are sent whole', async ($, on) => { + const c = await bootCls($, on) + const text = 'é'.repeat(2000) + await typed($, text) + expect(c.asks[0]?.text).toBe(text) +}) + +test('classify sample: a longer text is cut to head and tail', async ( + $, on) => { + const c = await bootCls($, on) + await typed($, 'H'.repeat(600) + 'M'.repeat(1000) + 'T'.repeat(1500)) + expect(c.asks[0]?.text).toBe('H'.repeat(500) + 'T'.repeat(1500)) +}) + +test('classify sample: a cut never leaves a lone surrogate', async ( + $, on) => { + const c = await bootCls($, on) + await typed($, '😀'.repeat(2400)) + const sent = c.asks[0]?.text ?? '' + expect(sent.isWellFormed()).toBe(true) + expect(Array.from(sent)).toHaveLength(2000) +}) + +// ---- no dialog, config, visibility --- + +test('classify: no dialog and no file write on a classified prompt', async ( + $, on) => { + const c = await bootCls($, on, { world: newWorld({ ask: true }) }) + await typed($, PLAIN) + expect(c.asks).toHaveLength(1) + expect(c.world.asked).toEqual([]) + expect(c.world.written).toEqual([]) +}) + +test('classify config: an invalid value keeps the default on', async ( + $, on) => { + const c = await bootCls($, on, { + world: newWorld({ home: { classify: 'yes' } }), + }) + expect(await asksFor($, c, PLAIN)).toBe(1) +}) + +test('classify config: an invalid value is logged once', async ($, on) => { + const c = await bootCls($, on, { + world: newWorld({ home: { classify: 'yes' } }), + }) + await route($, 'reload') + const said = c.logs.filter(l => l.includes('"classify" is not a boolean')) + expect(said).toHaveLength(1) +}) + +test('classify config: /route show prints classify: on', async ($, on) => { + await bootCls($, on) + expect(await route($, 'show')).toContain('classify: on') +}) + +test('classify config: /route classify off prints classify: off', async ( + $, on) => { + await bootCls($, on) + expect(await route($, 'classify off')).toContain('classify: off') +}) + +test('classify config: a bad toggle argument prints the usage', async ( + $, on) => { + await bootCls($, on) + expect(await route($, 'classify maybe')).toBe('usage: /route classify on|off') +}) + +test('classify config: the toggle survives a rebuild', async ($, on) => { + const c = await bootCls($, on) + await route($, 'classify off') + expect(await route($, 'reload')).toContain('classify: off') + expect(await asksFor($, c, PLAIN)).toBe(0) +}) + +test('classify config: /route is refused from a plugin origin', async ( + $, on) => { + await bootCls($, on) + const out = await $.command.run({ + command: 'route', + args: 'classify off', + origin: { kind: 'plugin', name: 'x' }, + presentation: { isFullscreen: false, columns: 80 }, + }) + expect(out.text).toContain('user-only') +}) + +test('classify visibility: the status line reads classifier ', async ( + $, on) => { + const c = await bootCls($, on, { reply: () => 'analysis' }) + await typed($, PLAIN) + expect(c.status[c.status.length - 1]).toContain('route: classifier reflect') +}) + +test('classify visibility: verbose says (reflect: classifier)', async ( + $, on) => { + const c = await bootCls($, on, { reply: () => 'analysis', verbose: true }) + await runStep($, { ...highStep(), effort: 'low' as const }) + await typed($, PLAIN) + await runStep($, { ...highStep(), effort: 'low' as const }) + expect(c.logs).toContain( + mainWord('fable/low', 'fable/high', 'reflect: classifier')) +}) + +test('classify: a repeated undefined label never pauses', async ($, on) => { + const c = await bootCls($, on, { reply: () => undefined }) + await failTimes($, 3) + expect(c.asks).toHaveLength(3) + expect(c.toasts).toEqual([]) + expect(c.status.some(l => l.includes('paused'))).toBe(false) +}) + +test('classify: /route advertises classify on|off in its hint', async ( + $, on) => { + const hints: unknown[] = [] + on('command.register', (_$, e) => { + hints.push(e.argumentHint) + return { value: { command: e.name } } + }) + await bootCls($, on) + expect(hints.join('\n')).toContain('classify on|off') +}) diff --git a/mods/model-router/hooks/register.ts b/mods/model-router/hooks/register.ts index 11ba8f9..a0c9c8c 100644 --- a/mods/model-router/hooks/register.ts +++ b/mods/model-router/hooks/register.ts @@ -32,11 +32,13 @@ type Config = { upgradeMaxTokens: number // above this context an upgrade is skipped mainModelSwitch: boolean // gates DOWNGRADES only verbose: boolean + classify: boolean // a small-model guess of the typed prompt's phase spinner: boolean enabled: boolean // false: every hook passes through (per machine) } type Rule = { re: RegExp; phase: string; mode: PromptMode } -type Source = 'user' | 'model' | 'skill' | 'prompt' | 'run' | 'derived' +type Source = + | 'user' | 'model' | 'skill' | 'prompt' | 'run' | 'derived' | 'classifier' type Routed = { phase: string route: Route @@ -91,13 +93,21 @@ type State = { override: Rec | undefined // the machine override's data, if it loaded loaded: boolean // the layers were read once (else: lazily, on first use) rebuildDue: boolean // /clear: re-read the layers at the next prompt - toggles: { switch?: boolean; verbose?: boolean } // session toggles + toggles: Partial> // session toggles + engineBase: NonNullable | null // engine effort of main's first step + classifyFails: number // classifier failures in a row + classifyUntil: number // classifier paused until (ms); 0 = running asking: string | null // key of the one first-use dialog in flight asked: Set // keys answered (or failed) this session writes: Promise // the serialized routing.json write chain } type Log = (text: string) => void type StepIn = Readonly +type PromptIn = { + text: string + turnId?: string + origin: { kind: string } +} type Plan = { model: string; effort: StepIn['effort'] } type RouteInput = { phase?: unknown @@ -107,6 +117,7 @@ type RouteInput = { } type Picked = { phase: string; route: Route } type Effort = StepIn['effort'] +type Toggle = 'switch' | 'verbose' | 'classify' type EffortBy = 'floor' | 'sticky' | 'turn' | 'engine' type Decision = { effort: Effort; by: EffortBy } type Rec = Record @@ -176,6 +187,23 @@ const UNAVAILABLE: ReadonlySet = new Set([ 'rate_limit', 'overloaded', 'billing_error', 'model_not_found', ]) const PHASE_KEY = /^[a-z][a-z0-9_-]{0,31}$/ +// Prompt classifier: label -> the phase whose effort the turn takes. `chat` +// (and any label not here) routes nothing. +const CLASSIFY_LABELS: readonly string[] = + ['design', 'analysis', 'coding', 'bookkeeping', 'chat'] +const LABEL_PHASE: Readonly> = { + design: 'plan', + analysis: 'reflect', + coding: 'implement', + bookkeeping: 'apply', +} +const CLASSIFY_MIN = 12 // code points of a typed prompt worth a call +const CLASSIFY_MS = 3000 // the call is abandoned after this +const CLASSIFY_MAX = 2000 // code points sent whole; longer: head + tail +const CLASSIFY_HEAD = 500 +const CLASSIFY_TAIL = 1500 +const CLASSIFY_FAILS = 3 // failures in a row that pause the classifier +const TIMEOUT: unique symbol = Symbol('classify timeout') const DEFAULT_CONFIG: Config = { models: { @@ -232,6 +260,7 @@ const DEFAULT_CONFIG: Config = { ], mainModelSwitch: false, verbose: false, + classify: true, spinner: true, enabled: true, } @@ -512,6 +541,9 @@ function mergeConfig( mainModelSwitch: pickBool(user.mainModelSwitch, base.mainModelSwitch), verbose: pickSwitch( 'config "verbose" is not a boolean; ignored', user.verbose, base.verbose, log), + classify: pickSwitch( + 'model-router: config "classify" is not a boolean; ignored', + user.classify, base.classify, log), spinner: pickBool(user.spinner, base.spinner), enabled: pickSwitch( 'model-router: config "enabled" is not a boolean; ignored', @@ -845,6 +877,9 @@ function newState(cfg: Config, source: string): State { loaded: false, rebuildDue: false, toggles: {}, + engineBase: null, + classifyFails: 0, + classifyUntil: 0, asking: null, asked: new Set(), writes: Promise.resolve(), @@ -853,7 +888,8 @@ function newState(cfg: Config, source: string): State { /** * /clear rebuilds the state but keeps what is account- or process-wide: - * the breaker (a model's quota outlives the conversation), the model, the + * the breaker, the classifier's pause and effort baseline (a model's quota + * and the user's own /effort outlive the conversation), the model, the * agent definitions' sources (the listing is not offered again), the config * with its session toggles and the dialog in flight. The layers are read * again at the next prompt; the config is swapped only if that read works. @@ -870,6 +906,9 @@ function resetSession(st: State): void { override: st.override, loaded: st.loaded, toggles: st.toggles, + engineBase: st.engineBase, + classifyFails: st.classifyFails, + classifyUntil: st.classifyUntil, asking: st.asking, writes: st.writes, } @@ -1204,11 +1243,12 @@ function higherFloor(cur: Routed | null, next: Routed): Routed { * The one decision of the main loop's effort. The user's floor is the turn's * default (no sticky or turn route names an effort) and its minimum. * `by` names who set the value: the floor when it raised or supplied it. + * `skipTurn`: the turn route yields (a classified effort under a user /effort). */ -function mainEffort(st: State, engine: Effort): Decision { +function mainEffort(st: State, engine: Effort, skipTurn = false): Decision { const sticky = st.userMain?.route.effort - const named = sticky ?? st.turnMain?.route.effort ?? - st.runMain?.route.effort + const turn = skipTurn ? undefined : st.turnMain?.route.effort + const named = sticky ?? turn ?? st.runMain?.route.effort const floor = st.turnFloor?.route.effort const base = named ?? floor ?? engine const effort = floored(base, floor) @@ -1369,7 +1409,8 @@ async function show($: Api, st: State): Promise { mainText(st, s), `router: ${routerWord(st)} · switch: ${flag(c.mainModelSwitch)} ` + `(downgrade) · upgrade: ${flag(c.mainUpgrade)} · verbose: ${ - flag(c.verbose)} · spinner: ${flag(c.spinner)}`, + flag(c.verbose)} · classify: ${flag(c.classify)} · spinner: ${ + flag(c.spinner)}`, downText(st, s.now), `live loops: ${st.loops.size}`, phasesText(st), @@ -1383,8 +1424,10 @@ function statusLine(st: State): string { ? routerWord(st) : r ? `${r.source} ${r.phase}` : 'session defaults' const floor = liveFloor(st) + const paused = st.cfg.classify && st.classifyUntil !== 0 return `route: ${now}${floor ? ` · floor ${floor.level}` : ''}${ - st.cfg.mainModelSwitch ? ' · switch on' : ''}` + st.cfg.mainModelSwitch ? ' · switch on' : ''}${ + paused ? ' · classifier: paused' : ''}` } const refresh = ($: Api, st: State): void => $.ui.status(statusLine(st)) @@ -1431,12 +1474,11 @@ function parseRoute(cfg: Config, args: string): Routed | string { async function toggle( $: Api, st: State, - what: string, + what: Toggle, arg: string | undefined, ): Promise { if (arg !== 'on' && arg !== 'off') return `usage: /route ${what} on|off` - if (what === 'switch') st.toggles.switch = arg === 'on' - else st.toggles.verbose = arg === 'on' + st.toggles[what] = arg === 'on' applyToggles(st) return show($, st) } @@ -1451,9 +1493,10 @@ async function setUserRoute($: Api, st: State, args: string) { /** Session toggles outlive any rebuild of the config. */ function applyToggles(st: State): void { - const { switch: downgrade, verbose } = st.toggles + const { switch: downgrade, verbose, classify } = st.toggles if (downgrade !== undefined) st.cfg.mainModelSwitch = downgrade if (verbose !== undefined) st.cfg.verbose = verbose + if (classify !== undefined) st.cfg.classify = classify } function applyLoaded(st: State, loaded: Loaded, log: Log): void { @@ -1497,6 +1540,8 @@ async function ensureConfig($: Api, st: State, atPrompt = false) { /** The breaker is cleared first, whatever the config read then does. */ async function reloadCommand($: Api, st: State): Promise { clearBreaker(st) + st.classifyFails = 0 + st.classifyUntil = 0 await reloadConfig($, st) refresh($, st) return 'config reloaded\n' + (await show($, st)) @@ -1533,6 +1578,7 @@ async function handleCommand($: Api, st: State, args: string): Promise { return forgetCommand($, st, rest) case 'switch': case 'verbose': + case 'classify': return toggle($, st, head, rest[0]) default: return setUserRoute($, st, args) @@ -2414,7 +2460,8 @@ async function registerCommand($: Api): Promise { argumentHint: '[show|clear|off|on|reload|pending|ask on|off|' + 'forget ||' + - 'model= effort=|switch on|off|verbose on|off]', + 'model= effort=|switch on|off|verbose on|off|' + + 'classify on|off]', immediate: true, }) } catch (err) { @@ -2770,6 +2817,7 @@ function slotWord(slot: Routed | null): string { case 'model': return 'route tool' case 'prompt': return 'prompt rule' case 'derived': return 'dispatch' + case 'classifier': return 'classifier' case 'skill': case 'run': return 'skill' + label default: return 'session' @@ -2921,13 +2969,27 @@ function agentPlan(st: State, e: StepIn): Plan { type Why = { by: EffortBy; call: Call; cur: string } type Step = { plan: Plan; why?: Why } +/** The engine effort of main's first step: what the user left as default. */ +function noteEngineBase(st: State, e: StepIn): void { + if (st.engineBase === null && e.effort !== undefined) { + st.engineBase = e.effort + } +} + +/** A user /effort beats a guessed phase: the engine effort left its base. */ +function userMovedEffort(st: State, e: StepIn): boolean { + return st.turnMain?.source === 'classifier' && e.effort !== undefined && + st.engineBase !== null && e.effort !== st.engineBase +} + /** * The main loop's plan: effort from the floor/sticky/turn decision, model * from `decideMain`. `cur` is the model the router moved this turn, else * the engine's own (verbatim: an engine fallback is respected). */ async function mainPlan($: Api, st: State, e: StepIn): Promise { - const { effort, by } = mainEffort(st, e.effort) + noteEngineBase(st, e) + const { effort, by } = mainEffort(st, e.effort, userMovedEffort(st, e)) await prune($, st) const cur = st.turnModel ?? e.model const { call } = await decideFor($, st, cur) @@ -3309,6 +3371,126 @@ function noteTyped(st: State, kind: string, text: string, midTurn: boolean) { } } +// ---- prompt classifier ----------------------------------------------- +// A typed prompt no rule, skill, route or floor covers is guessed by one +// small-model call: the phase's EFFORT becomes the turn's default (never a +// model move, never a dialog, never a file write). The call is bounded and +// every failure leaves the prompt as it was. + +/** The whole text up to CLASSIFY_MAX code points, else its head and tail. */ +function sampleOf(text: string): string { + const points = Array.from(text.trim()) + if (points.length <= CLASSIFY_MAX) return points.join('') + return points.slice(0, CLASSIFY_HEAD).join('') + + points.slice(-CLASSIFY_TAIL).join('') +} + +/** No route in force on main and the classifier is on and not paused. */ +const classifierIdle = (st: State): boolean => + !st.off && st.cfg.classify && st.classifyUntil === 0 && + mainRoute(st) === null && st.turnFloor === null + +/** A lapsed pause ends here, before any other guard (no stale marker). */ +async function expirePause($: Api, st: State): Promise { + if (st.classifyUntil === 0 || (await $.clock.now()) < st.classifyUntil) { + return + } + st.classifyUntil = 0 + refresh($, st) +} + +/** The small model is the haiku id: marked down, or main already runs it. */ +async function smallModelBusy($: Api, st: State): Promise { + await prune($, st) + const small = st.cfg.models.haiku + const main = st.turnModel ?? st.sessionModel + return (small !== undefined && st.down.has(small)) || isHaiku(st.cfg, main) +} + +/** The guards of D8, in order; the text itself is checked last. */ +async function classifierWanted($: Api, st: State, e: PromptIn) { + if (!st.cfg.classify || e.origin.kind !== 'composer') return false + if (e.turnId !== undefined) return false + await expirePause($, st) + const text = e.text.trim() + if (text.startsWith('/') || Array.from(text).length < CLASSIFY_MIN) { + return false + } + return classifierIdle(st) && !(await smallModelBusy($, st)) +} + +/** One failure; the third in a row pauses the classifier, said once. */ +async function countFailure($: Api, st: State): Promise { + st.classifyFails += 1 + if (st.classifyFails < CLASSIFY_FAILS) return + st.classifyFails = 0 + st.classifyUntil = (await $.clock.now()) + st.cfg.cooldownMinutes * 60000 + $.ui.toast(`classifier paused ${st.cfg.cooldownMinutes} min (${ + CLASSIFY_FAILS} failures)`) + refresh($, st) +} + +/** classify's own abort cause: a user interrupt, not a failure. */ +const isAbort = (err: unknown): boolean => + err instanceof Error && (err.name === 'AbortError' || + /\baborted\b/i.test(err.message)) + +/** The label, or TIMEOUT; rejects like classify or when the dispatch aborts. */ +function raceClassify($: Api, text: string, signal: AbortSignal) { + return Promise.race([ + $.model.classify(sampleOf(text), CLASSIFY_LABELS), + $.clock.sleep(CLASSIFY_MS, { signal }) + .then((): typeof TIMEOUT => TIMEOUT), + ]) +} + +/** The label, or undefined: no answer, an interrupt or a counted failure. */ +async function askClassifier( + $: Api, + st: State, + text: string, + signal: AbortSignal, +): Promise { + let label: string | undefined | typeof TIMEOUT + try { + label = await raceClassify($, text, signal) + } catch (err) { + if (!signal.aborted && !isAbort(err)) await countFailure($, st) + return undefined + } + if (label === TIMEOUT) { + await countFailure($, st) + return undefined + } + st.classifyFails = 0 + return label +} + +/** The turn route of a label: its phase's effort only; false = nothing. */ +function routeFromLabel(st: State, label: string): boolean { + const phase = hasKey(LABEL_PHASE, label) ? LABEL_PHASE[label] : undefined + const effort = phase === undefined + ? undefined + : phaseRoute(st.cfg, phase)?.effort + if (phase === undefined || effort === undefined) return false + st.turnMain = { phase, route: { effort }, source: 'classifier' } + return true +} + +/** Fills an empty route with the guessed phase's effort; never throws on it. */ +async function classifyPrompt( + $: Api, + st: State, + e: PromptIn, + signal: AbortSignal, +): Promise { + if (!(await classifierWanted($, st, e))) return + const label = await askClassifier($, st, e.text, signal) + // a route that landed meanwhile wins; a late answer never crosses turns + if (label === undefined || signal.aborted || !classifierIdle(st)) return + if (routeFromLabel(st, label)) refresh($, st) +} + function registerPrompt(on: On, st: State): void { on('prompt.submit', async ($, e, next) => { await ensureConfig($, st, true) @@ -3317,6 +3499,7 @@ function registerPrompt(on: On, st: State): void { noteTyped(st, e.origin.kind, e.text, midTurn) if (e.origin.kind === 'composer') routeFromPrompt(st, e.text, midTurn) refresh($, st) + await classifyPrompt($, st, e, next.signal) return next(e) }).catch(($, e, next) => { warnOnce(st, $, 'prompt.submit', next.error.kind)