chore(memory): BDR-116 amendment (classifier) + EVAL-045 + journal/TODO/contract w4b — feat model-router wave 4-B

This commit is contained in:
bchanot
2026-10-11 21:07:04 +02:00
parent e54a2b5d61
commit 5389ee53e7
6 changed files with 220 additions and 3 deletions
+5 -1
View File
@@ -1414,4 +1414,8 @@ Branch feature/user-writing-web-rules, UNMERGED (human gate).
- **Amendment (2026-10-11, wave 3-B, commit 604a6c4)**: clause (2) superseded: the dialog carries CONTEXT (skill description = first sentence of its SKILL.md frontmatter, five YAML forms, hardened read: name allowlist, stat kind file, size cap, `clean()` Cc/Cf + code points; agent description from `agent.offer`, bounded cache; phase `about` line, new 120-char field on the 11 routing.json phases, kept in memory not on Route) and the REAL next model id + effort. Options Later / Keep / Change (T3 Later / Keep). Change = model (4 fixed alias labels with tier role) → effort among those the phases headed by that alias offer (ascending, ≤ 4, one → skipped, zero → Later) → scope; the pair maps to an existing PHASE (rows stay phase names; row's current phase wins a tie; same phase = Keep); a pair no phase offers = a new phase by hand. Other anywhere = Later + toast (no echo). A main-row change toasts the real decision + `/route switch on` hint on a held downgrade. Three lenses rejected inline-route rows and dialog phase edits (2 BLOCKERs). First real use: user moved security-auditor to `judge` (opus/xhigh) through it; floor aligned (`model: opus`, 2026-10-11). Links [[EVAL-044]].
- **Amendment (2026-10-11, wave 3-C, commit cd8d72f)**: clause (6) extended: `/route forget <name|all|projects>` (composer only) clears decisions in the tracked file through the same writer: a name in every table (`confirmed`, `changed` → row restored to its recorded `from` when it is a string, the row still equals `to` and `from` is a known phase; `projects[*]` pruned), asked again at once; `all`/`projects` confirm through the engine dialog (Cancel first, anything else = cancelled) and hold the single-dialog slot; nothing written when nothing to forget; the answer names a restored row's file + values to realign (a floor aligned to the decision makes the census FAIL after the restore). No per-user decisions file (user: "seulement le forget"): another user inherits the shared memory and may forget. Residuals (security LOW/MEDIUM): raw error text in the answer, file fragments unsanitized in the answer, `floorFile` name not allowlisted, `__proto__` plan/apply mismatch, `changed.phases` restorable. Links [[EVAL-044]].
- **Amendment (2026-10-11, wave 4-A, commit 68d7ea6)**: logging rule: `verbose` (boolean) logs TRANSITIONS only, built from the decision itself inside `safely` before `next`: main "fable/high → fable/xhigh (<phase>: <cause>[, <model cause>])" once per turn per transition (causes: skill /<name>, route tool, prompt rule, prompt floor, dispatch, sticky /route, session; model: upgrade, switch, fallback: <alias> down); one hand-off line per agent dispatch deduped per turn per (type, target) "main … → agent <type> <alias>/<effort> (row <phase>|explicit|fallback|frontmatter, row|explicit|engine)"; later agent effort moves; an answered-by anomaly (agents only, canonical ids, once per agent+model). The host prefixes the plugin name: new lines carry none. The four stream lines are gone. Residuals W3-A/B/C closed: failure texts to the log, file fragments cleaned at render, floorFile allowlisted, `__proto__` refused, changed scan = rows, dialog Change offers shipped phases only (`mem.filePhases` = validated seed layer), post-answer write failure keeps the key asked. Rejected: a `trace` level (unrequested surface); detecting agent moves at turn.step (decided at spawn). Links [[EVAL-044]].
- **Amendment (2026-10-11, wave 4-B, commit e54a2b5)**: prompt classifier closes the main-loop gap (plain prompt, no skill, no agent, no route in force). Composer `prompt.submit`, trimmed text ≥ 12 chars not starting with `/`, no prompt rule matched, no sticky/turn/run/floor route → one `$.model.classify(sample, [design, analysis, coding, bookkeeping, chat])`, sample = whole text ≤ 2000 code points else head 500 + tail 1500, no `model` option (engine small model). Label → phase (design→plan, analysis→reflect, coding→implement, bookkeeping→apply, chat/unknown → nothing) written as the TURN route, EFFORT ONLY (no tier: main model never moves on a guess), may lower, replaced by any later route tool call or typed skill, ignored at a step whose engine effort ≠ session baseline (`engineBase`: a user `/effort` beats the guess). Bounded: race with `$.clock.sleep(3000)`; timeout/rejection count, user abort neither counts nor writes, every guard re-checked after the await (a route landing meanwhile wins); 3 failures in a row → pause `cooldownMinutes` (×60000, kept across `/clear`, cleared by `/route reload`, one toast + status marker ` · classifier: paused`); skipped while the haiku id is marked down or main runs on haiku. No dialog, no routing.json write on a classified route. Config key `classify` (default true, `pickSwitch`), `/route classify on|off` session toggle surviving a rebuild, `/route show` `classify:` field, status `route: classifier <phase>`, verbose line `(<phase>: classifier)`.
- **Why**: user 2026-10-11: routing must also cover the main conversation when no skill or agent is invoked. Effort-only because a model flip on a 3-s guess would fight the breaker and the user's `/model`; turn-scoped because a guess must never outlive its prompt.
- **Alternatives rejected**: first-use dialog on classified phases (the TODO line said so; a dialog per plain prompt is noise and the phase is already confirmed through T3); writing the guess into routing.json; classifying mid-turn or sub-agent prompts; unbounded await (robustness BLOCKER r1: the classify call sat on the critical path with no clock race); `classify` as a tracked-file writer (session toggle + config key instead).
- **Gates**: plan r1 → r3 (simplicity CONCERNS(4), robustness FATAL(6) BLOCKER unbounded await, correctness CONCERNS(4); confirmation robustness CONCERNS(4)); feater DONE (423) + 1 round (trim, argumentHint, repeated undefined: 428, each mutation-proven); GATE 0 MET; verifier ECARTS(2) coverage → CONFORME on #2; security PASS (0 findings; LOW accepted: pasted text can steer one turn's effort up, README says so). Abort paths judged by reading (`[gated]` clarification). Residual: `smallModelBusy` checks `cfg.models.haiku` while classify uses the engine's default small model (mismatch only when the two differ). Live AC9 open.
- **Refs**: contract `.claude/tasks/contracts/2026-10-11-model-router-w4b-classifier-1858.md`, plan r3 same slug. Links [[EVAL-045]], [[LRN-210]].
+7
View File
@@ -65,6 +65,7 @@ rules:
| EVAL-042 | 2026-10-10 | model-router W2 plan r1 → r4: 3 lenses (1 BLOCKER), 2 confirmations (1 BLOCKER then 0); W2-A verifier 3× ECARTS on coverage clauses only, W2-B ECARTS(7) → CONFORME; gate A→B read from engine records | second confirmation paid again (BLOCKER on my own r2 slot); compound coverage criterion = endless ECARTS; engine jsonl replaces the live log |
| EVAL-043 | 2026-10-11 | model-router W3-A: 3 lenses + 1 confirmation (census BLOCKER, project-tree layer dropped), feater DONE + 4 rounds, verifier 3× then re-verify after a REAL security BLOCK (credential fragment in the tracked key) | challenge + security gates both earned their cost; coverage-shaped criteria still cost 3 verifier rounds; live mod side effects misread as a test leak |
| EVAL-044 | 2026-10-11 | model-router W3-B (dialog with context): 3 lenses converged on 2 BLOCKERs of my r1 (inline rows, phase edits) + 1 confirmation; feater DONE + 2 coverage rounds; verifier 3× to CONFORME; security PASS; user's first real decision through the new dialog | the lenses pay most when the plan adds a data shape; coverage criteria still cost rounds; live dialogs during the gates must be announced and their data reconciled before commit |
| EVAL-045 | 2026-10-11 | model-router W4-B (prompt classifier): 3 lenses (BLOCKER: unbounded classify await) + 1 confirmation; feater DONE + 1 round; verifier ECARTS(2) → CONFORME; security PASS 0 findings; LRN-210 applied (one coverage clause per criterion) | first wave since W2 with no verifier cap: per-clause criteria + abort paths declared unverifiable up front cut the rounds to one |
---
@@ -411,3 +412,9 @@ Dogfood: 3 blind lenses attacked the v1 plan for the plan-challenge feature itse
- **Anomaly**: my r1 proposed a second row shape to satisfy "model then effort" literally; the derived-effort list satisfies it with zero new data shapes. During the gates the user answered the NEW dialog live: security-auditor → opus/xhigh (the request's example), plan-challenger → verify (reset on user go); my reset of `changed` wiped a legitimate entry and left the committed file census-red for one commit.
- **Action**: when a request says "choose X then Y", look for the existing shape that already encodes (X, Y) before adding one; reconcile routing.json decisions one by one (never `changed = {}`), and run the census before every data commit. Links [[EVAL-043]], [[BDR-116]], [[LRN-212]].
## EVAL-045 — model-router W4-B: per-clause criteria and a declared-unverifiable path ended the verifier loop in one round
- **Date**: 2026-10-11
- **Output checked**: plan `.claude/tasks/plans/2026-10-11-model-router-w4b-classifier-1858.md` r1 → r3; diff e54a2b5 (register.ts +211, 428 kit tests, 4 docs).
- **Method**: simplicity CONCERNS(4), robustness FATAL(6) (BLOCKER: the classify await sat unbounded on the prompt's critical path → `$.clock.sleep` race, abort not counted, post-await re-check), correctness CONCERNS(4) → r2; confirmation robustness CONCERNS(4) → r3. Feater DONE (423, 5 mutation proofs); GATE 0 MET; verifier #1 ECARTS(2) (no test on `trim`, none on the argumentHint) → feater (+5 tests, each failing under its mutation; first trim tests passed without the trim because the length guard caught them, lengthened) → verifier #2 CONFORME 8/8, criterion 9 gated. Security PASS, 0 findings, 1 LOW accepted.
- **Anomaly**: none on the loop. Verifier observations worth keeping: the sub-agent test proves only spawn+step never calls classify (`prompt.submit` has no agent form in the API types); the 3-s tests pin the upper bound only; the post-await re-check is tested through the route-landing case alone. Security note: `smallModelBusy` guards `cfg.models.haiku`, classify uses the engine small model.
- **Action**: keep the W4-B contract shape as the template (one clause per criterion, kit-untestable paths listed in CLARIFICATIONS before the first verifier); the live AC9 check is the real oracle for label quality (drop `bookkeeping` if it misfires). Links [[EVAL-044]], [[BDR-116]], [[LRN-210]], [[LRN-212]].
+1
View File
@@ -600,3 +600,4 @@ rules:
- model-router W3-B (dialog with context) landed 604a6c4 + data 32b71d2 + floor chore. User asked after W3-A: explain what is decided, OK/change/later, model then effort then scope. Plan r1→r3: all three lenses killed my inline-row format and the T3 phase edit (2 BLOCKERs avoided); efforts derived from the phases of the chosen model. Feater DONE first pass (229 tests), verifier ECARTS(2)→(1)→CONFORME, security PASS. GATE 0 heredoc CHECK not runnable again → script. Live during the gates: user answered the NEW dialog for security-auditor → opus/xhigh (the request's example; kept, floor aligned to opus, model-routing lock updated), plan-challenger → verify (reset on user go), orchestrate Keep (T3 check done). My `changed = {}` reset wiped a legit entry → census red for one commit → restored. Docs P1-P6 approved (patch in flight), BDR-116 amendment + EVAL-044 written. Next: doc commit, memory commit, user merge of feature/model-router-confirm (W3-A + W3-B) + publish by hand; T1 live check still open.
- model-router W3-C `/route forget` landed cd8d72f (+ data commit: escalate Keep). User asked whether decisions can be cleared and what another user can do; chose the command only, no per-user file. Plan r1→r3: three lenses CONCERNS (false `st.mem` premise, kind order, docs scope, census red after restoring an aligned floor, no confirmation on `all`), confirmation CONCERNS(3). Feater DONE; verifier ECARTS(2) with a REAL typo (`ForgetPlan`), then coverage only ×2 → cap → user accepted; security PASS (1 MEDIUM + 4 LOW parked). My own `route(escalate)` call opened the T3 dialog the user Kept. Branch feature/model-router-confirm now carries W3-A/B/C (14 commits), unmerged, unpushed; merge on signal; full make test running.
- model-router W4-A landed 68d7ea6 on feature/model-router-w4 (+ data commit). User: fewer logs (one line per real change with a cause), residuals, then the classifier. Plan r1→r3: three lenses (BLOCKER: an agent's model move is decided at spawn; `trace` level rejected as unrequested surface; reuse lastPlan/Loop; cause from the decision), confirmation CONCERNS(5) (hand-off line per dispatch deduped per type+target). Feater DONE (341 tests) + 4 rounds; verifier ECARTS(3) real (dedupe on text, double host prefix) then coverage ×2 → cap → user accepted; security PASS + 2 MEDIUM (raw skill names) fixed. Live: the terminal logs changed as soon as the feater saved; user Kept plan-challenger (judge) and the judge phase. Next: W4-B classifier; push-guard hotfix queued.
- model-router W4-B prompt classifier landed e54a2b5 on feature/model-router-w4. Plain composer prompt with no route in force → `$.model.classify` (5 labels, head 500 + tail 1500) → turn effort only, 3-s clock race, 3 failures pause, `/route classify on|off`. Plan r1→r3 (robustness BLOCKER: unbounded await), feater DONE + 1 round (trim, argumentHint), verifier ECARTS(2) → CONFORME, security PASS 0 findings. No live routing.json change this wave. Branch carries W4-A + W4-B (5 commits), unmerged, unpushed: merge on signal, user publishes. Live AC9 (plain prompts labelled) open.