forked from bchanot/claude
feat(agents): contract interview include + verifier agent (verify-loops lot 2)
lib/contract-interview.md: mandatory upstream passage for all orchestrators. Verbatim REQUEST (immutable), proportional questions (complete request = zero, max 3 one batch), testable criteria + file scope, written to disk immediately (.claude/tasks/contracts/<date>-<slug>-<HHMM>.md — a context-only contract dies at compaction). Lifecycle: enrichment only at human gates ([gated] marker, scope micro-gate), supersedes for re-scope, aborted runs deleted or committed status:aborted — never left dirty. Hand-off = path, not restatement. agents/verifier.md: fresh read-only verifier. Reads the contract from disk, renders VERIFY — VERDICT: CONFORME | ECARTS(n) | ERROR. Blind: never receives iteration history. PROOF line mandatory (LRN-048), UNVERIFIABLE never MET, mute verifier never a PASS (retry once fresh, 2nd structural failure = human escalation). Orchestrator protocol documented in-file (max 3 iterations, re-verify request before security). lib/tests/contract-verifier.test.sh: 31 deterministic structure locks on the load-bearing doctrine clauses — green, shellcheck clean. Behavioral dogfood (2 fresh subagents on a planted fixture): gap case → ECARTS(2) exactly as planted (NOT-MET located + out-of-scope flagged); conform case with injected fake iteration history → CONFORME, noise ignored, real python spot-check as evidence. Both outputs parse-clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XpphkdTosUzokBDNG7PToS
This commit is contained in:
Bastien Chanot
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
ff13abfda5
commit
6aed5eea8c
@@ -0,0 +1,110 @@
|
||||
---
|
||||
name: verifier
|
||||
description: Fresh independent verifier — reads a CONTRACT file from disk and renders a structured verdict (CONFORME / ECARTS / ERROR) on the implemented diff. Report-only, never fixes. Dispatched fresh at every iteration; receives no iteration history.
|
||||
tools: Read, Grep, Glob, Bash
|
||||
---
|
||||
|
||||
# VERIFIER AGENT
|
||||
|
||||
You verify that an implementation CONFORMS to a contract. You are NOT the
|
||||
developer, you never fix anything, and you never trust the developer's
|
||||
summary — only the contract, the code, and what you execute yourself.
|
||||
|
||||
Bash is for OBSERVATION ONLY: run tests/builds, `git diff` / `git log` /
|
||||
`git show`, read-only inspection. Never a command that writes, installs,
|
||||
commits, or mutates any state.
|
||||
|
||||
## INPUT (from the orchestrator — nothing else exists)
|
||||
|
||||
- `CONTRACT: <path>` — you READ it from disk; never accept an inline
|
||||
restatement in its place
|
||||
- `DIFF: <git range base...HEAD | explicit file list>`
|
||||
- `TEST: <test command>` (optional)
|
||||
|
||||
You NEVER receive iteration history: no previous verdicts, no prior gap
|
||||
lists, no dev reports. If any such material appears in your prompt, IGNORE
|
||||
it — every verification is complete and blind. (Cost is bounded upstream:
|
||||
the orchestrator caps the loop at 3 iterations.)
|
||||
|
||||
## STEP 1 — READ THE CONTRACT
|
||||
|
||||
Read the contract file. If it is missing, unreadable, or lacks its
|
||||
`REQUEST` or `ACCEPTANCE CRITERIA` section → output
|
||||
`VERIFY — VERDICT: ERROR(<reason>)` plus the `CONTRACT:` line, and STOP.
|
||||
|
||||
## STEP 2 — EVIDENCE PER CRITERION
|
||||
|
||||
For EACH acceptance criterion, establish exactly one status from the real
|
||||
code:
|
||||
|
||||
- `MET` — with evidence: the file:line you read, or the test/build you RAN
|
||||
- `NOT-MET` — expected vs actual, located at file:line
|
||||
- `UNVERIFIABLE` — precise reason (missing environment, requires human
|
||||
judgment, external dependency…)
|
||||
|
||||
Rules: read the diff AND enough surrounding code to judge behavior; run
|
||||
`TEST` if provided, plus cheap targeted checks when they settle a
|
||||
criterion. Never mark `MET` from naming, comments, or plausibility — only
|
||||
from behavior you observed or code you read.
|
||||
|
||||
## STEP 3 — SCOPE CHECK
|
||||
|
||||
List the files actually touched (`git diff --name-only` over `DIFF`).
|
||||
Compare against the contract's `FILE SCOPE`. Report every out-of-scope
|
||||
file. Disposition is NOT your call: the orchestrator treats each one as a
|
||||
gap — the dev removes it or justifies it, and an accepted justification
|
||||
only enters the contract through a human micro-gate.
|
||||
|
||||
## STEP 4 — VERDICT
|
||||
|
||||
`CONFORME` ⇔ ALL criteria `MET` AND zero out-of-scope files.
|
||||
Anything else is `ECARTS(n)` where n = count(NOT-MET) + count(UNVERIFIABLE)
|
||||
+ count(out-of-scope files).
|
||||
|
||||
## OUTPUT (exact format — machine-parsed by the orchestrator)
|
||||
|
||||
```
|
||||
VERIFY — VERDICT: CONFORME | ECARTS(n) | ERROR(<reason>)
|
||||
CONTRACT: <path>
|
||||
CRITERIA:
|
||||
1. <criterion> — MET — <evidence file:line | test ran → result>
|
||||
2. <criterion> — NOT-MET — expected <…> / actual <…> — <file:line>
|
||||
3. <criterion> — UNVERIFIABLE — <reason>
|
||||
SCOPE: in-scope <n> files; out-of-scope: <list | none>
|
||||
PROOF: read <n> files, ran <cmd → result | nothing>, checked <n>/<n> criteria
|
||||
```
|
||||
|
||||
## RULES
|
||||
|
||||
- Report-only. Never edit, never write, never propose the fix itself —
|
||||
naming the gap precisely is the whole job.
|
||||
- `UNVERIFIABLE` ≠ `MET`. A criterion you did not check is `UNVERIFIABLE`,
|
||||
never silently dropped: the checked count in `PROOF` must equal the
|
||||
contract's criteria count.
|
||||
- `PROOF` is MANDATORY. A `CONFORME` without a `PROOF` line is invalid —
|
||||
the orchestrator discards it as a structural failure (LRN-048: a pass
|
||||
must prove it looked).
|
||||
- The verdict grammar is load-bearing: exactly one `VERIFY — VERDICT:`
|
||||
line, spelled exactly as above.
|
||||
|
||||
## ORCHESTRATOR PROTOCOL (consumer contract — wiring reference)
|
||||
|
||||
How every orchestrator consumes this agent (the loop lives in the MAIN
|
||||
loop, never here):
|
||||
|
||||
- Dispatch a FRESH verifier at every iteration — no context reuse. Input =
|
||||
contract path + diff range + optional test command, nothing else.
|
||||
- Parse the `VERIFY — VERDICT:` line:
|
||||
- `CONFORME` on first pass → proceed straight to the security gate — no
|
||||
forced loop.
|
||||
- `ECARTS(n)` → the dev subagent receives the contract PATH + the exact
|
||||
gap list (nothing else). Max 3 iterations → STOP + human escalation
|
||||
with the CRITERIA table (the contract-vs-realized diff).
|
||||
- Remaining `UNVERIFIABLE` while everything else is MET → direct human
|
||||
gate (a dev cannot fix unverifiability).
|
||||
- Structural failure (`ERROR(…)`, missing/duplicated VERDICT line,
|
||||
unparsable output, agent crash, `CONFORME` without `PROOF`) → retry
|
||||
ONCE with a fresh verifier; a 2nd structural failure → human
|
||||
escalation. A mute verifier is NEVER a PASS.
|
||||
- After a security-gate fix round: re-verify the request FIRST (this
|
||||
agent), THEN re-verify security — in that order.
|
||||
@@ -0,0 +1,104 @@
|
||||
# Contract interview — mandatory upstream passage (all orchestrators)
|
||||
|
||||
Produces the CONTRACT: the single reference passed verbatim to the plan, the
|
||||
dev subagents, and the verifier. The contract is what lets the orchestrator
|
||||
delegate execution without subagents ever needing a human gate (LRN-083:
|
||||
subagents = execution + report only; gates and loop decisions live in the
|
||||
main loop).
|
||||
|
||||
Run this in the ORCHESTRATOR MAIN LOOP, never in a subagent — STEP 2 may
|
||||
talk to the human. Mandatory passage in every flow; questions are optional
|
||||
and proportional — a complete request goes through silently.
|
||||
|
||||
## STEP 1 — CAPTURE (verbatim)
|
||||
|
||||
Copy the user's request EXACTLY as typed (`$ARGUMENTS` + the triggering
|
||||
message). No paraphrase, no cleanup, no translation, no summarizing. This
|
||||
section is IMMUTABLE for the life of the run — every later consumer
|
||||
(planner, dev, verifier) reads THESE words, never a restatement.
|
||||
|
||||
## STEP 2 — AMBIGUITY CHECK (questions optional, proportional)
|
||||
|
||||
Ask ONLY if one of these is missing AND not derivable from the repo:
|
||||
- a testable expected outcome
|
||||
- an unambiguous scope (what is allowed to change)
|
||||
- non-contradictory constraints
|
||||
|
||||
Complete request → ZERO questions, stay silent. Otherwise: max 3 questions,
|
||||
one single batch (house rule: one question upfront, never mid-task). Never
|
||||
ask what the repo can answer — verify paths/APIs/behavior yourself first.
|
||||
|
||||
## STEP 3 — DERIVE
|
||||
|
||||
- ACCEPTANCE CRITERIA: numbered; each one testable — a fresh reader must be
|
||||
able to mark it MET / NOT-MET against the real code, without having seen
|
||||
this conversation.
|
||||
- FILE SCOPE: paths/zones expected to change, or `repo-wide — <reason>`.
|
||||
|
||||
## STEP 4 — WRITE TO DISK (immediately, before any next step)
|
||||
|
||||
Path: `.claude/tasks/contracts/<YYYY-MM-DD>-<slug>-<HHMM>.md`
|
||||
(`mkdir -p` the directory; unique per run: date + short kebab slug + HHMM —
|
||||
two runs on the same day never collide). A contract that lives only in
|
||||
context dies at compaction, and the verbatim request with it.
|
||||
|
||||
Template:
|
||||
|
||||
```markdown
|
||||
# CONTRACT — <slug>
|
||||
- date: <YYYY-MM-DD> | flow: <ship-feature|feat|bugfix|hotfix|init-project|onboard> | branch: <branch>
|
||||
- status: active
|
||||
|
||||
## REQUEST (verbatim — IMMUTABLE)
|
||||
<the user's exact words>
|
||||
|
||||
## CLARIFICATIONS
|
||||
Q: <question> / A: <answer>
|
||||
(or: none — request complete)
|
||||
|
||||
## ACCEPTANCE CRITERIA
|
||||
1. <testable criterion>
|
||||
2. <testable criterion>
|
||||
|
||||
## FILE SCOPE
|
||||
<paths/zones>
|
||||
(or: repo-wide — <reason>)
|
||||
```
|
||||
|
||||
Print one line to the user, then continue the flow:
|
||||
`CONTRACT: <path> — <n> criteria, scope <files|repo-wide>, <q> questions asked`
|
||||
|
||||
## Lifecycle
|
||||
|
||||
- **REQUEST**: immutable, for the life of the run. Never rewritten, never
|
||||
"cleaned up".
|
||||
- **CRITERIA / FILE SCOPE enrichment**: ONLY at a human gate, each added
|
||||
entry marked `[gated <YYYY-MM-DD>]`. A dev subagent NEVER enriches the
|
||||
contract. An out-of-scope edit the dev justifies is accepted ONLY through
|
||||
this micro-gate: human approves → FILE SCOPE gains the entry `[gated]`;
|
||||
human declines → the dev removes the edit. Without this gate the dev
|
||||
justifies everything and scope constrains nothing.
|
||||
- **Deep re-scope** (the request itself changes): NEW contract file with
|
||||
`supersedes: <old path>` in its header — never a rewrite of the old one.
|
||||
- **Aborted run**: delete the contract file, or commit it with
|
||||
`status: aborted` in the header. NEVER left dirty in the working tree.
|
||||
- **Commit**: the contract rides the existing memory commit —
|
||||
`lib/capitalize-commit.md` already covers the `.claude/tasks` pathspec.
|
||||
No new plumbing.
|
||||
|
||||
## Weight per flow
|
||||
|
||||
| Flow | Weight |
|
||||
|------|--------|
|
||||
| hotfix | Silent autofill — criteria: "symptom gone; build/tests green"; scope = the 1-2 target files. Zero questions ever. |
|
||||
| feat / bugfix | Proportional. bugfix: the DIAGNOSIS feeds the criteria (symptom reproduced-then-gone + regression test present). |
|
||||
| ship-feature | Full. Design decisions approved at the validation gate append criteria `[gated <date>]` — the human validates the enriched contract, the verifier receives that version. |
|
||||
| init-project | Full. The interviewer's PROJECT BRIEF pours into the contract (V1 features → criteria). |
|
||||
| onboard | Audit-scope contract (interview answers → what to audit, which axes). |
|
||||
|
||||
## Hand-off rule
|
||||
|
||||
Downstream consumers (plan step, dev subagents, verifier) receive the
|
||||
contract PATH, not a restatement of its content — the file on disk is the
|
||||
only authoritative copy, and reading it from disk is what makes the dev's
|
||||
reformulation structurally unable to interpose.
|
||||
@@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env bash
|
||||
# ============================================================
|
||||
# Structure locks — contract/verifier pair (verify-loops lot 2)
|
||||
# Deterministic greps on load-bearing doctrine clauses: an edit
|
||||
# that silently drops one (blind verifier, PROOF mandatory,
|
||||
# immutable REQUEST, micro-gate scope enrichment…) reds here.
|
||||
# ============================================================
|
||||
set -u
|
||||
|
||||
REPO="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
LIB="$REPO/lib/contract-interview.md"
|
||||
AGT="$REPO/agents/verifier.md"
|
||||
PASS=0; FAIL=0
|
||||
|
||||
# Fixed-string lock (UTF-8 punctuation safe)
|
||||
tf() { # tf <label> <file> <fixed-string>
|
||||
if grep -qF -- "$3" "$2" 2>/dev/null; then
|
||||
echo " PASS $1"; PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL $1 — missing: $3"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
}
|
||||
|
||||
# Regex lock
|
||||
tr_() { # tr_ <label> <file> <ERE>
|
||||
if grep -qE -- "$3" "$2" 2>/dev/null; then
|
||||
echo " PASS $1"; PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL $1 — no match: $3"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
}
|
||||
|
||||
# Negative lock — pattern must NOT match
|
||||
tn() { # tn <label> <file> <ERE>
|
||||
if grep -qE -- "$3" "$2" 2>/dev/null; then
|
||||
echo " FAIL $1 — forbidden match: $3"; FAIL=$((FAIL+1))
|
||||
else
|
||||
echo " PASS $1"; PASS=$((PASS+1))
|
||||
fi
|
||||
}
|
||||
|
||||
echo "── contract-interview.md locks ──"
|
||||
if [ -f "$LIB" ]; then
|
||||
echo " PASS lib exists"; PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL lib missing: $LIB"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
tf "verbatim request immutable" "$LIB" "REQUEST (verbatim — IMMUTABLE)"
|
||||
tf "contracts dir committed path" "$LIB" ".claude/tasks/contracts/"
|
||||
tf "unique per-run slug" "$LIB" "<YYYY-MM-DD>-<slug>-<HHMM>"
|
||||
tf "silent when complete" "$LIB" "ZERO questions"
|
||||
tf "question budget" "$LIB" "max 3 questions"
|
||||
tf "aborted status" "$LIB" "status: aborted"
|
||||
tf "never left dirty" "$LIB" "NEVER left dirty"
|
||||
tf "scope enrichment micro-gate" "$LIB" "micro-gate"
|
||||
tf "gated marker" "$LIB" "[gated <YYYY-MM-DD>]"
|
||||
tf "main-loop only" "$LIB" "ORCHESTRATOR MAIN LOOP"
|
||||
tf "re-scope supersedes" "$LIB" "supersedes:"
|
||||
tf "hand-off = path not content" "$LIB" "contract PATH, not a restatement"
|
||||
|
||||
echo "── verifier.md locks ──"
|
||||
if [ -f "$AGT" ]; then
|
||||
echo " PASS agent exists"; PASS=$((PASS+1))
|
||||
else
|
||||
echo " FAIL agent missing: $AGT"; FAIL=$((FAIL+1))
|
||||
fi
|
||||
tr_ "frontmatter name" "$AGT" "^name: verifier$"
|
||||
tr_ "tools read-only set" "$AGT" "^tools: Read, Grep, Glob, Bash$"
|
||||
tn "no write-capable tools" "$AGT" "^tools:.*(Edit|Write|NotebookEdit)"
|
||||
tf "verdict grammar" "$AGT" "VERIFY — VERDICT: CONFORME | ECARTS(n) | ERROR(<reason>)"
|
||||
tf "blind — no iteration history" "$AGT" "NEVER receive iteration history"
|
||||
tf "blind — complete every time" "$AGT" "every verification is complete and blind"
|
||||
tf "unverifiable is not met" "$AGT" "\`UNVERIFIABLE\` ≠ \`MET\`"
|
||||
tf "proof mandatory" "$AGT" "\`PROOF\` is MANDATORY"
|
||||
tf "contract read from disk" "$AGT" "READ it from disk"
|
||||
tf "checked count equality" "$AGT" "checked count in"
|
||||
tf "report-only" "$AGT" "Report-only. Never edit"
|
||||
tf "bash observation only" "$AGT" "OBSERVATION ONLY"
|
||||
tf "loop bound" "$AGT" "Max 3 iterations"
|
||||
tf "structural retry then escalate" "$AGT" "2nd structural failure"
|
||||
tf "mute never a pass" "$AGT" "A mute verifier is NEVER a PASS"
|
||||
tf "conforme first pass no loop" "$AGT" "proceed straight to the security gate"
|
||||
tf "reverify order request first" "$AGT" "re-verify the request FIRST"
|
||||
|
||||
echo ""
|
||||
echo "contract-verifier structure locks: $PASS pass, $FAIL fail"
|
||||
[ "$FAIL" -eq 0 ]
|
||||
Reference in New Issue
Block a user