diff --git a/agents/verifier.md b/agents/verifier.md index 4e902f1..a2bd752 100644 --- a/agents/verifier.md +++ b/agents/verifier.md @@ -71,7 +71,28 @@ You may re-run a `CHECK:` yourself to settle a doubt (Bash is read-only, and these commands are observation). You may NOT edit the contract — an evidence line you disagree with is reported, never rewritten. -## STEP 3 — SCOPE CHECK +## STEP 3 — FLOOR GUARD (mandatory, deterministic) + +Run the floor guard over the diff before rendering any verdict — a red or +skipped run here is a structural gap, never a judgment call: + +```bash +bash ~/.claude/lib/floor-guard.sh -- ... +``` + +`` = the branch's gitflow base (develop; main for a hotfix/release). +Parse the single `FLOOR GUARD:` line: + +- `clean` (rc 0) → no finding, continue to STEP 4. +- ` finding(s), waived` (rc 2) → each `FLOOR : + ` line is a gap for STEP 5's `ECARTS` count, UNLESS the + contract's `CLARIFICATIONS` explicitly authorizes that exact weakening — + quote the authorizing sentence in the verdict instead of counting it as a + gap. `WAIVED` lines are informational only, never a gap. +- rc 3 (usage error) → a structural failure like a missing contract: retry + once (base ref or pathspec likely wrong), a second failure escalates. + +## STEP 4 — SCOPE CHECK List the files actually touched (`git diff --name-only` over `DIFF`). Compare against the contract's `FILE SCOPE`. Report every out-of-scope @@ -79,7 +100,7 @@ file. Disposition is NOT your call: the orchestrator treats each one as a gap — the dev removes it or justifies it, and an accepted justification only enters the contract through a human micro-gate. -## STEP 4 — VERDICT +## STEP 5 — VERDICT Read the contract's `ABANDON:` lines. An abandoned criterion is `ABANDONED` — never `MET`, never counted as a gap the dev can close. @@ -89,11 +110,12 @@ is not: 1. `ERROR()` — the contract is missing or unreadable. 2. `ECARTS(n)` — n = count(NOT-MET) + count(UNVERIFIABLE) + count(out-of-scope - files). Surface any abandonment in the same report. + files) + count(unauthorized FLOOR findings from STEP 3). Surface any + abandonment in the same report. 3. `ABANDONED(n)` — zero gaps remain, but n abandonments stand. This is NOT a pass and NOT a dev loop: it routes straight to the human gate. 4. `CONFORME` — ALL criteria `MET`, zero out-of-scope files, zero - abandonments. + unauthorized FLOOR findings, zero abandonments. ## OUTPUT (exact format — machine-parsed by the orchestrator) @@ -106,6 +128,8 @@ CRITERIA: 3. — UNVERIFIABLE — 4. — ABANDONED — SCOPE: in-scope files; out-of-scope: +FLOOR: clean | finding(s) ( waived) — PROOF: read files, ran , checked / criteria ``` @@ -123,6 +147,9 @@ PROOF: read files, ran , checked / criteria - `PROOF` is MANDATORY. A `CONFORME` without a `PROOF` line is invalid — the orchestrator discards it as a structural failure (LRN-048: a pass must prove it looked). +- STEP 3's floor-guard run is MANDATORY, every dispatch. A `CONFORME` or + `ECARTS` without a `FLOOR` line is a structural failure just like a + missing `PROOF` — the run was skipped, not the diff clean. - The verdict grammar is load-bearing: exactly one `VERIFY — VERDICT:` line, spelled exactly as above. @@ -145,8 +172,8 @@ loop, never here): lifts the abandonment (the criterion was fixable after all) or accepts the partial delivery; the run is never reported as fully complete. - Structural failure (`ERROR(…)`, missing/duplicated VERDICT line, - unparsable output, agent crash, `CONFORME` without `PROOF`) → retry - ONCE with a fresh verifier; a 2nd structural failure → human - escalation. A mute verifier is NEVER a PASS. + unparsable output, agent crash, `CONFORME` without `PROOF` or without + `FLOOR`) → retry ONCE with a fresh verifier; a 2nd structural failure → + human escalation. A mute verifier is NEVER a PASS. - After a security-gate fix round: re-verify the request FIRST (this agent), THEN re-verify security — in that order. diff --git a/lib/floor-guard.sh b/lib/floor-guard.sh new file mode 100755 index 0000000..a573fe0 --- /dev/null +++ b/lib/floor-guard.sh @@ -0,0 +1,297 @@ +#!/usr/bin/env bash +# lib/floor-guard.sh — diff-scoped detector of a quietly weakened quality bar. +# +# bash ~/.claude/lib/floor-guard.sh [-- ...] +# +# rc 0 = clean no floor finding in the diff +# 2 = finding(s), waived +# 3 = usage error (missing , or it does not resolve to a commit) +# +# WHY (BDR-100 class): "no weakened check in this diff" is exactly the kind +# of judgment an LLM verifier can miss, or be talked past one line at a time +# — a single added TS-ignore comment, a skipped test, a dropped assertion, a +# coverage threshold shaved by one point. This makes that judgment +# deterministic: grep the diff for the known ways a change quietly lowers +# the bar, same floor doctrine as gates.sh (contract oracles) and +# doctrine-citers.test.sh (citation census) — a mechanism, not a lesson. +# +# Adapted from addyosmani/agent-skills constraint-driven-development's +# "floor guard" to this repo's own gate model: `git diff` instead of a +# staged-diff assumption, wired into agents/verifier.md STEP 3 rather than a +# pre-commit hook. +# +# Scope: `git diff ` — working tree included (uncommitted changes +# count) — restricted to when given. Every ADDED line is +# classified into one of six kinds (full pattern tables below): +# SUPPRESS a checker-silencing comment added, any file +# SKIP a test disabled or isolated, test files only +# DELETED_TEST a whole test file removed +# ASSERT_DROP a test file's assertion-line count went down +# STUB a not-implemented marker added, any file +# THRESHOLD_DOWN a numeric value lowered on the same key, config files only +# +# Waiver: an added line also carrying `floor-guard: allow ` prints as +# WAIVED and does not count toward the finding total or the rc. +set -uo pipefail + +_usage() { + echo "usage: floor-guard.sh [-- ...]" >&2 + exit 3 +} + +[ $# -ge 1 ] || _usage +BASE_REF="$1"; shift +PATHSPEC=() +if [ $# -gt 0 ]; then + [ "$1" = "--" ] || _usage + shift + PATHSPEC=("$@") +fi +git rev-parse --verify -q "${BASE_REF}^{commit}" >/dev/null 2>&1 || _usage + +TMPDIFF="$(mktemp)" || { echo "floor-guard: mktemp failed" >&2; exit 3; } +trap 'rm -f "$TMPDIFF"' EXIT + +git diff --unified=0 "$BASE_REF" -- "${PATHSPEC[@]}" > "$TMPDIFF" 2>/dev/null +FLOOR_DELETED_FILES="$(git diff --diff-filter=D --name-only \ + "$BASE_REF" -- "${PATHSPEC[@]}" 2>/dev/null)" +export FLOOR_DELETED_FILES + +# "working tree included" means brand-new, still-untracked files too: plain +# `git diff ` never shows them (git only diffs what it already tracks), +# so a file added on this branch and never `git add`-ed would be invisible +# to every kind below. --no-index against /dev/null emits the same unified +# format as the tracked diff above (diff --git / +++ b/path / @@ hunks), +# so the parser needs no separate code path for it. +while IFS= read -r f; do + [ -n "$f" ] || continue + git diff --no-index --unified=0 -- /dev/null "$f" >> "$TMPDIFF" 2>/dev/null +done < <(git ls-files --others --exclude-standard -- "${PATHSPEC[@]}" 2>/dev/null) + +python3 - "$TMPDIFF" <<'PY' +import fnmatch +import os +import re +import sys + +# file classes (CLARIFICATIONS): "path contains test/spec/__tests__" is a +# superset of the explicit globs (*.test.*, *.spec.*, *_test.go, *_test.py, +# test_*.py all contain one of these substrings themselves), so one check +# covers all five. +TEST_SUBSTRINGS = ('test', 'spec', '__tests__') +CONFIG_GLOBS = ('jest.config*', 'vitest.config*', '.nycrc*', 'codecov*', + 'sonar-project.properties', 'lighthouserc*', 'CONSTRAINTS.md') + +# ── pattern tables — the trigger strings themselves, waived on this file's +# own diff so the guard stays clean on itself (also exercises the waiver +# path for real) ──────────────────────────────────────────────────────────── +SUPPRESS_SUBSTRINGS = ( + '@ts-ignore', # floor-guard: allow pattern table + 'eslint-disable', # floor-guard: allow pattern table + '# noqa', # floor-guard: allow pattern table + '# type: ignore', # floor-guard: allow pattern table + 'nosemgrep', # floor-guard: allow pattern table + 'nosec', # floor-guard: allow pattern table + 'shellcheck disable', # floor-guard: allow pattern table +) +TS_EXPECT_ERROR = '@ts-expect-error' # floor-guard: allow pattern table + +SKIP_SUBSTRINGS = ( + '.skip(', '.only(', 'xit(', 'xdescribe(', 'fit(', 'fdescribe(', + 'it.todo(', '@pytest.mark.skip', '@unittest.skip', 't.Skip(', +) + +STUB_SUBSTRINGS = ( + 'not implemented', # floor-guard: allow pattern table + 'NotImplementedError', # floor-guard: allow pattern table +) +EMPTY_CATCH_RE = re.compile(r'catch\s*\([^)]*\)\s*\{\s*\}') +BARE_EXCEPT_RE = re.compile(r'except\b[^:\n]*:\s*pass\b') + +ASSERT_SUBSTRINGS = ('expect(', 'assert', 'should', '.toBe') + +KEYVAL_RE = re.compile(r'["\']?([A-Za-z0-9_.\-]+)["\']?\s*[:=]\s*(-?\d+(?:\.\d+)?)') +WAIVER_RE = re.compile(r'floor-guard:\s*allow\s+(\S.*)$') +HUNK_RE = re.compile(r'^@@ -(\d+)(?:,\d+)? \+(\d+)(?:,\d+)? @@') + + +def is_test_file(path): + return any(s in path for s in TEST_SUBSTRINGS) + + +def is_config_file(path): + base = os.path.basename(path) + return any(fnmatch.fnmatch(base, g) for g in CONFIG_GLOBS) + + +def is_waived(text): + return bool(WAIVER_RE.search(text)) + + +def strip_prefix(raw): + if raw == '/dev/null': + return raw + return raw[2:] if raw[:2] in ('a/', 'b/') else raw + + +def _new_file_entry(files): + entry = {'old_path': None, 'new_path': None, 'adds': [], 'dels': [], + 'first_new': None} + files.append(entry) + return entry + + +def parse_diff(lines): # → list of per-file entries (adds/dels + paths) + files, cur = [], None + old_no = new_no = 0 + for raw in lines: + if raw.startswith('diff --git '): + cur = _new_file_entry(files) + elif raw.startswith('--- '): + cur['old_path'] = strip_prefix(raw[4:]) + elif raw.startswith('+++ '): + cur['new_path'] = strip_prefix(raw[4:]) + elif raw.startswith('@@ '): + m = HUNK_RE.match(raw) + if m: + old_no, new_no = int(m.group(1)), int(m.group(2)) + if cur['first_new'] is None: + cur['first_new'] = new_no + elif raw.startswith('+') and not raw.startswith('+++'): + cur['adds'].append((new_no, raw[1:])); new_no += 1 + elif raw.startswith('-') and not raw.startswith('---'): + cur['dels'].append((old_no, raw[1:])); old_no += 1 + return files + + +def effective_path(entry): + if entry['new_path'] not in (None, '/dev/null'): + return entry['new_path'] + return entry['old_path'] + + +def suppress_kind(text): + if TS_EXPECT_ERROR in text: + after = text.split(TS_EXPECT_ERROR, 1)[1].strip() + return None if after else 'SUPPRESS' + return 'SUPPRESS' if any(p in text for p in SUPPRESS_SUBSTRINGS) else None + + +def stub_kind(text): + if any(p in text for p in STUB_SUBSTRINGS): + return 'STUB' + if EMPTY_CATCH_RE.search(text) or BARE_EXCEPT_RE.search(text): + return 'STUB' + return None + + +def skip_kind(text): + return 'SKIP' if any(p in text for p in SKIP_SUBSTRINGS) else None + + +def line_findings(path, lineno, text, test_file): + out, waived = [], is_waived(text) + for kindfn in (suppress_kind, stub_kind): + kind = kindfn(text) + if kind: + out.append((kind, path, lineno, text, waived)) + if test_file: + kind = skip_kind(text) + if kind: + out.append((kind, path, lineno, text, waived)) + return out + + +def _is_assertion(text): + return any(p in text for p in ASSERT_SUBSTRINGS) + + +def assert_drop_finding(path, entry): + added = sum(1 for _, t in entry['adds'] if _is_assertion(t)) + removed = sum(1 for _, t in entry['dels'] if _is_assertion(t)) + if removed <= added: + return None + lineno = entry['adds'][0][0] if entry['adds'] else (entry['first_new'] or 1) + waived = any(is_waived(t) for _, t in entry['adds']) + snippet = 'assertion lines %d -> %d' % (removed, added) + return ('ASSERT_DROP', path, lineno, snippet, waived) + + +def extract_kv(lines): # → {key: (lineno, value, raw text)} last-wins + kv = {} + for lineno, text in lines: + m = KEYVAL_RE.search(text) + if m: + kv[m.group(1)] = (lineno, float(m.group(2)), text) + return kv + + +def threshold_down_findings(path, entry): + removed_kv = extract_kv(entry['dels']) + added_kv = extract_kv(entry['adds']) + out = [] + for key, (lineno, new_val, text) in added_kv.items(): + old = removed_kv.get(key) + if old and new_val < old[1]: + out.append(('THRESHOLD_DOWN', path, lineno, text, is_waived(text))) + return out + + +def classify_file(entry, deleted_paths): + path = effective_path(entry) + if path is None: + return [] # pure rename/mode-change: no --- / +++ header, no content diff + test_file = is_test_file(path) + if path in deleted_paths and test_file: + return [('DELETED_TEST', path, 1, path, False)] + findings = [] + for lineno, text in entry['adds']: + findings += line_findings(path, lineno, text, test_file) + if test_file: + dropped = assert_drop_finding(path, entry) + if dropped: + findings.append(dropped) + if is_config_file(path): + findings += threshold_down_findings(path, entry) + return findings + + +def load_deleted_paths(): + raw = os.environ.get('FLOOR_DELETED_FILES', '') + return {p for p in raw.splitlines() if p} + + +def snippet_of(text): + return text.strip()[:100] + + +def emit(findings): + ordered = sorted(findings, key=lambda f: (f[1], f[2], f[0])) + n_found = n_waived = 0 + for kind, path, lineno, text, waived in ordered: + tag = 'WAIVED' if waived else 'FLOOR' + print('%s %s %s:%d %s' % (tag, kind, path, lineno, snippet_of(text))) + n_waived += 1 if waived else 0 + n_found += 0 if waived else 1 + if n_found: + print('FLOOR GUARD: %d finding(s), %d waived' % (n_found, n_waived)) + return 2 + print('FLOOR GUARD: clean') + return 0 + + +def main(): + with open(sys.argv[1], 'r', errors='replace') as fh: + lines = fh.read().split('\n') + deleted = load_deleted_paths() + findings = [] + for entry in parse_diff(lines): + findings += classify_file(entry, deleted) + return emit(findings) + + +if __name__ == '__main__': + sys.exit(main()) +PY +rc=$? +exit "$rc" diff --git a/lib/tests/floor-guard.test.sh b/lib/tests/floor-guard.test.sh new file mode 100644 index 0000000..423dde4 --- /dev/null +++ b/lib/tests/floor-guard.test.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash +# lib/tests/floor-guard.test.sh — flip-tests for lib/floor-guard.sh: one RED +# fixture per KIND, one WAIVED fixture, one CLEAN fixture. Each fixture is a +# fresh throwaway repo under $WORK (`make test` exports +# GIT_CONFIG_GLOBAL=/dev/null; core.hooksPath is also pinned per-repo so a +# machine-wide hook never fires here). This file itself carries the trigger +# strings for every kind — the self-run criterion excludes it by pathspec. +set -u +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +LIB="$ROOT/lib/floor-guard.sh" +WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT +pass=0; fail=0 + +# check_kind +check_kind() { + local kind="$1" rc="$2" want_rc="$3" out="$4" want_sub="$5" + if [ "$rc" = "$want_rc" ] && printf '%s\n' "$out" | grep -qF -- "$want_sub"; then + pass=$((pass+1)); echo "PASS $kind" + else + fail=$((fail+1)) + printf 'FAIL %s: rc=%s (want %s), out:\n%s\n' \ + "$kind" "$rc" "$want_rc" "$(printf '%s\n' "$out" | tail -5)" + fi +} + +# mk_repo → path to a fresh throwaway repo: one test file (3 +# assertions), one vitest.config.ts (coverage.lines: 80), one source file. +mk_repo() { + local d="$WORK/$1" + mkdir -p "$d/src" + git init -q "$d" + git -C "$d" config user.email t@t + git -C "$d" config user.name t + git -C "$d" config core.hooksPath /dev/null + printf 'expect(1).toBe(1);\nexpect(2).toBe(2);\nexpect(3).toBe(3);\n' \ + > "$d/sample.test.js" + printf 'export default {\n coverage: {\n lines: 80,\n },\n};\n' \ + > "$d/vitest.config.ts" + printf 'function add(a, b) {\n return a + b;\n}\n' > "$d/src/index.js" + git -C "$d" add -A + git -C "$d" commit -q -m base + echo "$d" +} + +# ── SUPPRESS ────────────────────────────────────────────────────────────── +d=$(mk_repo suppress); base=$(git -C "$d" rev-parse HEAD) +echo '// eslint-disable-next-line no-console' >> "$d/src/index.js" +out=$(cd "$d" && bash "$LIB" "$base" 2>&1); rc=$? +check_kind SUPPRESS "$rc" 2 "$out" 'FLOOR SUPPRESS' + +# ── SKIP ────────────────────────────────────────────────────────────────── +d=$(mk_repo skip); base=$(git -C "$d" rev-parse HEAD) +echo "it.skip('later', () => {});" >> "$d/sample.test.js" +out=$(cd "$d" && bash "$LIB" "$base" 2>&1); rc=$? +check_kind SKIP "$rc" 2 "$out" 'FLOOR SKIP' + +# ── DELETED_TEST ────────────────────────────────────────────────────────── +d=$(mk_repo deleted); base=$(git -C "$d" rev-parse HEAD) +rm "$d/sample.test.js" +out=$(cd "$d" && bash "$LIB" "$base" 2>&1); rc=$? +check_kind DELETED_TEST "$rc" 2 "$out" 'FLOOR DELETED_TEST' + +# ── ASSERT_DROP ─────────────────────────────────────────────────────────── +d=$(mk_repo assertdrop); base=$(git -C "$d" rev-parse HEAD) +printf 'expect(1).toBe(1);\n' > "$d/sample.test.js" +out=$(cd "$d" && bash "$LIB" "$base" 2>&1); rc=$? +check_kind ASSERT_DROP "$rc" 2 "$out" 'FLOOR ASSERT_DROP' + +# ── STUB ────────────────────────────────────────────────────────────────── +d=$(mk_repo stub); base=$(git -C "$d" rev-parse HEAD) +echo "function todo() { throw new Error('not implemented'); }" >> "$d/src/index.js" +out=$(cd "$d" && bash "$LIB" "$base" 2>&1); rc=$? +check_kind STUB "$rc" 2 "$out" 'FLOOR STUB' + +# ── THRESHOLD_DOWN ──────────────────────────────────────────────────────── +d=$(mk_repo threshold); base=$(git -C "$d" rev-parse HEAD) +sed -i 's/lines: 80/lines: 60/' "$d/vitest.config.ts" +out=$(cd "$d" && bash "$LIB" "$base" 2>&1); rc=$? +check_kind THRESHOLD_DOWN "$rc" 2 "$out" 'FLOOR THRESHOLD_DOWN' + +# ── WAIVED ──────────────────────────────────────────────────────────────── +d=$(mk_repo waived); base=$(git -C "$d" rev-parse HEAD) +echo '// eslint-disable-next-line no-console -- floor-guard: allow legacy shim' \ + >> "$d/src/index.js" +out=$(cd "$d" && bash "$LIB" "$base" 2>&1); rc=$? +check_kind WAIVED "$rc" 0 "$out" 'WAIVED' + +# ── CLEAN ───────────────────────────────────────────────────────────────── +d=$(mk_repo clean); base=$(git -C "$d" rev-parse HEAD) +echo '// helper' >> "$d/src/index.js" +out=$(cd "$d" && bash "$LIB" "$base" 2>&1); rc=$? +check_kind CLEAN "$rc" 0 "$out" 'FLOOR GUARD: clean' + +echo "PASS=$pass FAIL=$fail" +[ "$fail" -eq 0 ] diff --git a/lib/verify-secure-loop.md b/lib/verify-secure-loop.md index cd37cbe..0e983c5 100644 --- a/lib/verify-secure-loop.md +++ b/lib/verify-secure-loop.md @@ -55,6 +55,16 @@ Dispatch a FRESH verifier subagent (`subagent_type: verifier`, or load `TEST` command. Never pass the dev's summary, never pass a prior iteration's gaps — the verifier reads the contract from disk and judges blind. +The verifier's STEP 3 (`agents/verifier.md`) runs `lib/floor-guard.sh` +against the diff before it renders any verdict — a deterministic, +diff-scoped check for a quietly weakened quality bar (a suppressed +lint/type check, a skipped or deleted test, a dropped assertion, a lowered +coverage threshold) that an LLM verdict alone can miss or be talked past +one line at a time. Its findings fold straight into that same verifier's +`ECARTS` count unless the contract's `CLARIFICATIONS` explicitly authorizes +the exact weakening; there is no separate gate and no extra dispatch, it +rides this GATE 1 call. + Parse its single `VERIFY — VERDICT:` line: - `CONFORME` → go to GATE 2. (First-pass conforme = no loop.) @@ -74,9 +84,9 @@ Parse its single `VERIFY — VERDICT:` line: micro-gate that appends `[gated ]` to the contract's FILE SCOPE; otherwise the dev removes the file. - Structural failure (`ERROR(…)`, missing/duplicated VERDICT line, - unparsable, crash, `CONFORME` without `PROOF`) → retry ONCE with a fresh - verifier; a 2nd structural failure → human escalation. A mute verifier is - NEVER a PASS. + unparsable, crash, `CONFORME` without `PROOF` or without `FLOOR`) → retry + ONCE with a fresh verifier; a 2nd structural failure → human escalation. + A mute verifier is NEVER a PASS. ## GATE 2 — SECURITY (fresh security-auditor)