forked from bchanot/claude
Merge bugfix/mods-validate-advisory into develop
This commit is contained in:
commit
6168d90fac
8 files changed
+187
-8
No files matched your search
@@ -53,6 +53,7 @@ rules:
|
||||
| BLK-031 | 2026-10-16 | model-router: session starting on Fable w/o credits lost 1st turn + each hold lapse; probe skipped kept cur | resolved |
|
||||
| BLK-032 | 2026-10-19 | Main loop pinned on Fable in long sessions: session model Fable 1M + downgrade cap 120k + prompt rule `plan` → inline code on Fable xhigh; feat A+B shipped (BDR-123), closes after feat C | open |
|
||||
| BLK-033 | 2026-10-21 | Guardrails residuals: probe trusted plan `supabase.url`, redirects free of budget, unread bodies read CLOSED, `str(URLError)` hid budget signal; rules 5 FP / 7 misses | resolved |
|
||||
| BLK-034 | 2026-10-22 | mods.test.sh red on CLI 2.1.306: whole-output `warning` grep caught advisory plugin-directory section (ICON_MISSING, blocking: false); verdict = `--strict` exit code now | resolved |
|
||||
|
||||
---
|
||||
|
||||
@@ -341,3 +342,9 @@ rules:
|
||||
- **Cause**: `supabase.url` read from the plan w/o `_http_url` → planted SUPABASE_URL would receive GUARDRAILS_ANON_KEY / JWT2, `file://` read CLOSED; `_SameHost` redirector followed hops w/o charging `session.budget`; `_rows_state` collapsed non-200 / empty / >1 MB / non-JSON into CLOSED (unobserved = CLOSED, breaks LRN-220); plan r2 mapped budget exhaustion on `str(URLError)`, which wraps the reason → never matched. Rules: SENSITIVE_KEY on metadata names (`token_type`, `has_password`), 403 on first map candidate stopped the walk, hydrated scan read whole route file, NOSQL `$where` fired on GraphQL `_bool_exp`; misses: Go backtick Sprintf, `rf`/triple-quoted f-strings, `sql|db|trx.raw`, empty `.select()`, multi-line hidden input, Rust `{}` shell arg, `"sources"` map head.
|
||||
- **Solution**: 5a98c2c (`_BudgetExhausted(URLError)` + `BUDGET_EXHAUSTED` constant, `_SameHost(session)` charges per hop, `_rows_state` rows|empty|unreadable → SKIPPED, `_map_outcome` full candidate walk, `data:` OPEN w/o decoding, `printable` on remote strings, `_safe_text` in scanner, rule fixes + mutant per language variant) + 633faa2 (checks.md: `IN (…)` placeholder list flagged, judged by hand). Plan r3 after 2 challenge rounds, contract 22 oracles MET, verifier CONFORME, auditor PASS. Suites probe 84, scan 241.
|
||||
- **Residual** (LOW, parked in TODO): table names not `IDENT.fullmatch`-checked per table (`/` → path walk on the approved host); `printable` lets Unicode bidi/format controls through. Links [[LRN-219]], [[LRN-220]], [[LRN-222]], [[LRN-223]].
|
||||
|
||||
## BLK-034 — mods.test.sh red on CLI 2.1.306: whole-output `warning` grep caught the advisory plugin-directory section [resolved]
|
||||
- **Friction**: `make test suite=lib/tests/mods.test.sh` → `FAIL model-router: validate printed a warning`, pre-existing on develop (git archive baseline), surfaced as feat B criterion 8 (NEED-DECISION, BDR-123 run).
|
||||
- **Cause**: CLI 2.1.306 `claude plugin validate` prints "Plugin directory checks" section (submission catalogue): `⚠ Found 1 plugin directory warning:` ICON_MISSING, `--json` `directory.findings[].blocking: false`. mods.test.sh:57 grepped whole output for `warning` (also matched hooks notes quoting register.ts identifiers, checkout paths). Prose scraping of a CLI with an exit-code contract: `--strict` → per-file warnings rc 1, directory findings rc only under `--profile directory`.
|
||||
- **Solution**: 801992a (bugfix/mods-validate-advisory). Helper `validate_strict` = rc of `claude plugin validate --strict`; check_cli + positive control share it; control = scratch manifests `{"name":"warnfix"}` (rc 1) / cleanfix with version+description+author (rc 0) under script-level `mktemp -d`. Plan r3 after 3 lenses + 1 re-challenge (sed range on closing marker killed: CLI prints it only when findings > 0, fail-open to EOF). GATE 0 7/7, verifier ECARTS(2) = out-of-scope routing.json dialog write only, auditor PASS.
|
||||
- **Residual**: 3 historical contracts carry old scrape as CHECK oracle (`.claude/tasks/contracts/2026-10-08-model-router-w1a-1533.md:22`, `2026-10-08-model-router-floor-1835.md:22`, `2026-10-09-model-router-tiers-1237.md:23`), left unedited; re-run fails on ICON_MISSING, not a model-router regression. Unverified: whether `--strict` escalates `advice[]`. Links [[LRN-225]], [[BDR-123]].
|
||||
@@ -635,3 +635,6 @@ rules:
|
||||
- /bugfix guardrails residuals (bugfix/guardrails-residuals): plan r3 after 2 challenge rounds (4 BLOCKERs r1: WARN severity, budget must raise, three-way SKIPPED, vacuous timing; r2: `str(URLError)` wrap), contract 22 oracles, 3 part executors, GATE 0/1/2 green, 5a98c2c + 633faa2 (checks.md IN() wording caught by doc-syncer, not the gates). BLK-033, LRN-222/223. Two LOW parked. Both branches await merge signal. Next: /feat routing policy (BLK-032).
|
||||
- /feat routing policy A (feature/routing-audit-phase, BLK-032): `audit` phase (best/xhigh) for verifier + security-auditor, opus frontmatter = outage floor (LRN-224), census tier_floor, citers. 2 challenge rounds (r1 2 BLOCKERs: dead-Fable floor, test fixture), GATE 0 15/15, verifier CONFORME, auditor PASS. d3635b6 dialog writes + session model opus, a7d747c, dae142b docs. Feat B next: ceiling rule, BDR, README doctrine.
|
||||
- /feat routing policy B (feature/routing-ceiling-rule): ceiling bullet compacted into CLAUDE.global.md (320 budget), README doctrine + challenge citers on `opus` session model, deploy `effort: xhigh`, `changed` cleared. 3 challengers r1→r3, GATE 0 12/12, verifier CONFORME, auditor PASS. ef13713 + docs 887cf89, BDR-123 ([deferred] inline-writing skills, keyword-lift gap → feat C). Merge on user go; then bugfix mods.test.sh, feat C.
|
||||
|
||||
## 2026-10-22
|
||||
- feat B merged (df38fd4). /bugfix mods.test.sh (bugfix/mods-validate-advisory): CLI 2.1.306 advisory ICON_MISSING section read as warning → verdict = `claude plugin validate --strict` rc through one helper + warnfix/cleanfix control; plan r3 (3 lenses + 1 re-challenge, sed-range fail-open killed), GATE 0 7/7, verifier ECARTS = out-of-scope routing.json dialog write (`confirmed.phases.audit`, uncommitted, user call), auditor PASS. 801992a. BLK-034, LRN-225. Next: feat C keyword rules capped at session model, close BLK-032.
|
||||
@@ -244,6 +244,7 @@ rules:
|
||||
| LRN-222 | 2026-10-21 | Map an error to a verdict on a typed exception or carried constant, never on `str(err)`: URLError wraps its reason | probe/fetch error mapping, any err→verdict code |
|
||||
| LRN-223 | 2026-10-21 | Regex timing lock is vacuous when the input matches early: feed a non-matching worst-path input, assert `is None`, keep ≥ 2x margin under the lock | every bounded regex test |
|
||||
| LRN-224 | 2026-10-21 | Best-tier agent row floors on tier 2nd alias: inTier writes head-or-above only, fable frontmatter = no fallback in credit outage | pinning agents on best tier, census floors |
|
||||
| LRN-225 | 2026-10-22 | Gate on tool's exit-code contract, never its prose; positive control through gate's own helper on clean/dirty fixture pair; mktemp EXIT trap script-level | any test that wraps a CLI verdict |
|
||||
|
||||
---
|
||||
|
||||
@@ -1871,3 +1872,9 @@ Rule: when editing a doctrine file under structure locks, grep the test's lock s
|
||||
## LRN-224 — Best-tier agent row floors on tier's 2nd alias: inTier writes head-or-above only
|
||||
- **Context**: feat A (routing policy BLK-032) moved verifier + security-auditor to new `audit` phase (best = fable>opus>sonnet). Plan r1 set frontmatter `model: fable` (census rule floor = tier head) and claimed "Fable down → falls to opus". False: `inTier` (register.ts) writes a model only if it is the head or ranks above it, else logs "tier … down, frontmatter model kept" → fable frontmatter = gates spawn on dead Fable every credit outage (BLK-030/031). Lower tiers never hit it (head sonnet/opus always has a higher alias). Caught by correctness + robustness lenses r1, not by plan author.
|
||||
- **Apply**: an agent on a best-tier row keeps opus frontmatter; router lifts to Fable when up. Census `tier_floor` (effort-routing.test.sh) locks 2nd alias for best. Residual: spawn path unprobed → first gate of an outage window still dies once, retry lands on floor (register.test.ts audit-row case b). Links [[BLK-030]], [[BLK-032]], [[LRN-204]].
|
||||
|
||||
## LRN-225 — Gate on the tool's own exit-code contract, never on its prose; control through the gate's own helper on a clean/dirty pair
|
||||
|
||||
- **Context**: BLK-034. `grep -qi warning` on `claude plugin validate` output broke when CLI 2.1.306 added an advisory section; first plan (sed range delete between two markers) was fail-open: closing marker printed only when findings > 0, else range ran to EOF and hid `✔ Validation passed with warnings`.
|
||||
- **Pattern**: (1) a CLI with `--strict` / `--json` / rc semantics → gate on rc, prose only in the ko message. (2) Positive control runs the SAME helper the gate uses (a copy of the invocation stays green when the gate drifts). (3) Pair of fixtures differing only in the guarded cause (warnfix vs cleanfix): parse error (`error: unknown option`, rc 1 on both) or scrape added back turns the pair red; a lone "rc must be 1" accepts any failure. (4) Inline sample strings test the author's literal, not the code: a check that cannot fail is not a check. (5) `WORK=$(mktemp -d); trap 'rm -rf "$WORK"' EXIT` stays script-level: `local` WORK + EXIT trap → `unbound variable` under `set -u`, dir leaks.
|
||||
- **Apply**: any suite wrapping a third-party CLI verdict (plugin validate, linters, validators). Links [[BLK-034]], [[LRN-223]].
|
||||
@@ -8,7 +8,7 @@
|
||||
- [x] feat A audit phase: d3635b6 (dialog writes, session model opus), a7d747c (audit row, opus floor, census, citers), dae142b (docs); gates 0/1/2 green, LRN-224
|
||||
- [x] feat B (feature/routing-ceiling-rule): ceiling rule CLAUDE.global.md, README doctrine, challenge citers, deploy `effort: xhigh`, routing.json `changed` cleared; ef13713 + 887cf89, gates 0/1/2 green, BDR-123; merge = user go
|
||||
- [ ] feat C: cap the default keyword prompt rules (plan|design|architecture, why|review|audit|analyze, ultrathink) at the session model, effort only like the classifier; close BLK-032 after
|
||||
- [ ] bugfix lib/tests/mods.test.sh:57: CLI 2.1.306 `claude plugin validate` advisory "plugin directory warning" (ICON_MISSING) read as failure; exclude that section, keep other warnings blocking
|
||||
- [x] bugfix lib/tests/mods.test.sh (bugfix/mods-validate-advisory): verdict = `--strict` rc, positive control warnfix/cleanfix; 801992a, gates green, BLK-034 LRN-225; merge = user go. Residual: 3 historical contracts keep the old scrape oracle (BLK-034)
|
||||
- [ ] deferred (BDR-123): remap inline-writing best-tier skills (design Build stack, writing-skills, deprecation-and-migration, /tour single-project, /audit-delta 3d, /harden STEP 3); stale citers skills/feat/SKILL.md:6, skills/code-clean/SKILL.md:36
|
||||
- [ ] residuals (accepted): spawn path unprobed, 1st gate of an outage window dies once (tour/onboard/audit-delta no retry); onAutoSwitch with main on opus marks nothing if engine auto-falls a gate off Fable
|
||||
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# CONTRACT — mods-validate-advisory (bugfix of lib/tests/mods.test.sh)
|
||||
|
||||
## REQUEST (verbatim)
|
||||
lib/tests/mods.test.sh:57 fails on develop: Claude Code CLI 2.1.306 `claude plugin validate` prints an advisory "plugin directory warning" section (ICON_MISSING) and the suite treats any warning as a failure. Exclude that advisory section only; every other validate warning stays blocking. Suite: make test suite=lib/tests/mods.test.sh. Pre-existing on develop (proven via git archive baseline in the previous session).
|
||||
|
||||
## CLARIFICATIONS
|
||||
- Internal (after the challenge, 3 lenses + 1 re-challenge): the "exclusion" is the CLI's own contract, `claude plugin validate --strict`: per-file warnings → rc 1, the plugin-directory advisory never changes the rc (only under `--profile directory`, never requested). No prose scraping is left in the suite.
|
||||
- Internal: the regression proof is a positive control run through the gate's own helper on two scratch manifests (warnfix rc 1, cleanfix rc 0), written by the suite under a script-level `mktemp -d`; no tracked fixture.
|
||||
- Internal: three historical contracts carry the old scrape as oracle (2026-10-08-model-router-w1a-1533.md:22, 2026-10-08-model-router-floor-1835.md:22, 2026-10-09-model-router-tiers-1237.md:23); left unedited, named at CAPITALIZE.
|
||||
- Plan: `.claude/tasks/plans/2026-10-22-mods-validate-advisory-2004.md` (r3).
|
||||
- GATE 1 ECARTS(2) → disposition (main loop): both gaps = criterion 6 on `mods/model-router/routing.json` (`confirmed.phases.audit` written by the mod on the main loop's first `route(phase="audit")`, register.ts:2054, after GATE 0 ran). Not an executor edit, not staged with this fix; commit of that dialog write = user call. Criterion 6 holds on the committed diff.
|
||||
|
||||
## ACCEPTANCE CRITERIA
|
||||
1. The suite is green on CLI 2.1.306 with the ICON_MISSING advisory present in validate's output.
|
||||
CHECK: make test suite=lib/tests/mods.test.sh 2>&1 | grep -E '^mods: ' | sed 's/^/n=/'
|
||||
EXPECT: n=mods: 6 pass, 0 fail
|
||||
EVIDENCE: MET exit=0 marker-found :: n=mods: 6 pass, 0 fail
|
||||
2. The validate verdict is the exit code of `--strict`, through one helper used by check_cli and the control; no `warning` scrape and no `Validation passed` scrape remain.
|
||||
CHECK: f=lib/tests/mods.test.sh; a=$(grep -c 'plugin validate --strict "' $f); b=$(grep -c 'validate_strict "' $f); c=$(grep -ciE "grep -qi? *'?warning|Validation passed" $f); echo "$a/$b/$c"
|
||||
EXPECT: 1/3/0
|
||||
EVIDENCE: MET exit=0 marker-found :: 1/3/0
|
||||
3. Positive control: the clean scratch manifest passes `--strict` and the warning one fails it, both through the helper, both reported as suite checks.
|
||||
CHECK: make test suite=lib/tests/mods.test.sh 2>&1 | grep -cE '^PASS strict control: (clean fixture passes --strict|per-file warnings fail --strict)$'
|
||||
EXPECT: 2
|
||||
EVIDENCE: MET exit=0 marker-found :: 2
|
||||
4. The control is not vacuous: with `--strict` removed from the helper the control goes red (the warning fixture passes).
|
||||
CHECK: S=/tmp/claude-1000/-home-bchanot-Documents-claude/09872630-52ba-43d2-b0a9-f9fe56869e15/scratchpad/chk; mkdir -p "$S"; sed 's/validate --strict "\$1"/validate "$1"/' lib/tests/mods.test.sh > "$S/m.sh"; MODS_ROOT=$PWD bash "$S/m.sh" 2>&1 | grep -c '^FAIL strict control: per-file warnings fail --strict'
|
||||
EXPECT: 1
|
||||
EVIDENCE: MET exit=0 marker-found :: 1
|
||||
5. The scratch dir is script-level with an EXIT trap (never inside a function), and the ko tail names the rerun command.
|
||||
CHECK: f=lib/tests/mods.test.sh; a=$(grep -c "^WORK=\"\$(mktemp -d)\"; trap 'rm -rf \"\$WORK\"' EXIT$" $f); b=$(grep -c 'rerun: claude plugin validate --strict' $f); echo "$a/$b"
|
||||
EXPECT: 1/1
|
||||
EVIDENCE: MET exit=0 marker-found :: 1/1
|
||||
6. Only lib/tests/mods.test.sh changes; shellcheck clean on it.
|
||||
CHECK: a=$(git status --short | grep -v '^?? .claude/' | grep -vc '^ M lib/tests/mods.test.sh$'); b=$(shellcheck lib/tests/mods.test.sh >/dev/null 2>&1 && echo ok || echo ko); echo "$a/$b"
|
||||
EXPECT: 0/ok
|
||||
EVIDENCE: MET exit=0 marker-found :: 0/ok
|
||||
7. The suite still fails loudly when no mod exists (never vacuous): MODS_ROOT on an empty dir.
|
||||
CHECK: d=$(mktemp -d); MODS_ROOT=$d bash lib/tests/mods.test.sh 2>&1 | grep -c '^FAIL no mod found'
|
||||
EXPECT: 1
|
||||
EVIDENCE: MET exit=0 marker-found :: 1
|
||||
|
||||
## FILE SCOPE
|
||||
lib/tests/mods.test.sh
|
||||
@@ -0,0 +1,91 @@
|
||||
# BUGFIX PLAN r3 — mods.test.sh reads the advisory plugin-directory section as a warning
|
||||
|
||||
## DIAGNOSIS
|
||||
BUG: `make test suite=lib/tests/mods.test.sh` fails on develop with
|
||||
`FAIL model-router: validate printed a warning` (pre-existing, proven on
|
||||
develop's copy via git archive in the previous session).
|
||||
ROOT CAUSE: Claude Code CLI 2.1.306 `claude plugin validate` prints a
|
||||
"Plugin directory checks for <dir>" section (the submission catalogue's
|
||||
checks) that says "Found 1 plugin directory warning:" (ICON_MISSING).
|
||||
lib/tests/mods.test.sh:57 greps the WHOLE output case-insensitively for
|
||||
`warning`, so an advisory the CLI itself reports as `blocking: false`
|
||||
(`--json`, `directory.findings[].blocking`) fails the suite. The same grep
|
||||
also matches the hooks notes that quote register.ts identifiers (`warnOnce`
|
||||
today is safe, a future `logWarning` is not) and any checkout path holding
|
||||
"warning": prose scraping, fail-open and fail-closed at once.
|
||||
EVIDENCE: `claude plugin validate mods/model-router` rc 0; `--strict` rc 0
|
||||
(per-file warnings are the only ones `--strict` turns into rc 1:
|
||||
`manifest.warnings` and `contents[].warnings` are empty); `--strict
|
||||
--profile directory` rc 1 (the advisory counts only under that profile).
|
||||
A scratch plugin whose manifest is `{"name":"warnfix"}` gives `⚠ Found N
|
||||
warnings:` (version, description, author) and `--strict` rc 1, plain rc 0.
|
||||
BLAST RADIUS: the fix touches lib/tests/mods.test.sh only. doctor.sh:211
|
||||
names the command in a hint. Three historical contracts carry the same
|
||||
scrape as a CHECK oracle (`grep -q 'Validation passed' && ! … grep -qi
|
||||
'warning'`): .claude/tasks/contracts/2026-10-08-model-router-w1a-1533.md:22,
|
||||
2026-10-08-model-router-floor-1835.md:22, 2026-10-09-model-router-tiers-1237.md:23.
|
||||
Re-run on CLI 2.1.306 they fail on ICON_MISSING: known-stale oracles, left
|
||||
unedited (historical), named at CAPITALIZE so a re-run is not read as a
|
||||
model-router regression. `advice[]` (third top-level `--json` field) is
|
||||
empty on every fixture tried; whether `--strict` escalates it is unverified.
|
||||
PRIOR: none bears. RELATED MEMORY: BLK-030..033 are model-router runtime /
|
||||
guardrails, not the CLI validate output; no BDR constrains the suite shape.
|
||||
|
||||
## CHALLENGE r1 → r2 → r3
|
||||
Three lenses (correctness, robustness, simplicity), 0 BLOCKER, 5 MAJOR, all
|
||||
closed by this revision: the sed range on a closing marker the CLI prints
|
||||
only when findings > 0 (fail-open to EOF, hides `✔ Validation passed with
|
||||
warnings`) is dropped for the CLI's own contract, the `--strict` exit code;
|
||||
the tautological SAMPLE_UNTERMINATED and the invented warning samples are
|
||||
dropped; the blocking branch is now exercised by a real CLI run on a
|
||||
fixture plugin instead of a copy of the grep. r3 (one re-challenge, 1 MAJOR):
|
||||
the control now runs through the gate's own helper, a clean sibling fixture
|
||||
pins the cause (only per-file warnings differ), the ko tail names the rerun
|
||||
command, WORK is script-level.
|
||||
|
||||
## FIX PLAN (one file: lib/tests/mods.test.sh)
|
||||
1. New helper, next to `bounded`:
|
||||
`validate_strict() { STRICT_OUT=$(bounded claude plugin validate --strict "$1"); }`
|
||||
returns the rc of `bounded` (last command), output in the script-level
|
||||
`STRICT_OUT` (same pattern as LAST_OUT in lib/tests/a11y-scan.test.sh:13).
|
||||
2. check_cli, validate half (lines 53-59): `validate_strict "$dir" || rc=$?`.
|
||||
rc 124 → `ko "$name: validate timed out after ${CLI_TIMEOUT}s"` (unchanged);
|
||||
rc ≠ 0 → `ko "$name: validate --strict rc=$rc: $(printf '%s' "$STRICT_OUT" | grep -E '^(✘|error:|⚠ Found)' | head -2 | tr '\n' ' ') (rerun: claude plugin validate --strict $dir)"`;
|
||||
rc 0 → `ok "$name: validate --strict passed"`.
|
||||
The `Validation passed` substring check and the `grep -qi warning` are
|
||||
removed: the exit code is the verdict. The plugin-test half (lines 60-65)
|
||||
is unchanged.
|
||||
3. Positive control. Script-level, right after `pass=0; fail=0`:
|
||||
`WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT` (top-level, as in
|
||||
lib/tests/a11y-scan.test.sh:12, never inside a function). New function
|
||||
`strict_discriminates`, called once after the `use_cli` block, only when
|
||||
`use_cli` = 1. It writes two scratch plugins:
|
||||
- `$WORK/warnfix/.claude-plugin/plugin.json` = `{"name":"warnfix"}`
|
||||
(per-file warnings: version, description, author);
|
||||
- `$WORK/cleanfix/.claude-plugin/plugin.json` =
|
||||
`{"name":"cleanfix","version":"0.1.0","description":"d","author":{"name":"x"}}`
|
||||
(no per-file warning; the directory findings README_MISSING,
|
||||
LICENSE_MISSING, ICON_MISSING are printed for both).
|
||||
Then, through the helper of step 1:
|
||||
- `validate_strict "$WORK/cleanfix"` → `check "strict control: clean fixture passes --strict" "$rc" 0`;
|
||||
- `validate_strict "$WORK/warnfix"` → `check "strict control: per-file warnings fail --strict" "$rc" 1`;
|
||||
rc 124 on either → `ko "strict control: validate timed out"`.
|
||||
The pair differs only in the per-file warnings, so a parse error (CLI
|
||||
without `--strict`: rc 1 on both) or a scrape added back turns the
|
||||
control red; the control covers the gate's own code path.
|
||||
4. Header comment (lines 4-5): "passes `claude plugin validate --strict`
|
||||
(per-file warnings are errors; the advisory plugin-directory section
|
||||
does not change the exit code) and `claude plugin test`".
|
||||
5. No new tracked file; the two manifests are written by the suite.
|
||||
|
||||
## EDGE CASES
|
||||
- CLI without `--strict` (older than its introduction): commander rejects
|
||||
the unknown option with rc 1 on every call → check_cli red with the
|
||||
`error: unknown option` line in its tail, and the clean-fixture control
|
||||
red too. Accepted: `cli_unavailable` already requires `claude plugin
|
||||
test`, which post-dates `--strict`.
|
||||
- Icon added later to mods/model-router: no effect, the directory profile
|
||||
is never requested.
|
||||
- The fixture's directory findings never affect rc (no `--profile`).
|
||||
|
||||
## RISK: low. Test-only file, no runtime behaviour.
|
||||
@@ -42,6 +42,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/) and this project
|
||||
- `gitflow delete` (and `finish`) land on the base that contains the branch and drop the branch's upstream before `git branch -d`, so a branch whose upstream lags (manual-push mode) is deleted instead of refused by git (T18k).
|
||||
- An invalid `gitflow.autopush` value (not a boolean, or a config read that fails) no longer pushes. The post-commit / post-merge hooks and every push site of `lib/gitflow.sh` (`start`, `finish`, the `origin/` cleanup of `delete`) read it as auto and pushed; they now push nothing and say why. Each hook run prints `gitflow post-commit: gitflow.autopush unreadable (git rc <n>) — NOT pushed, treated as manual push mode; fix the value by hand` (post-merge likewise), and the lib passes through the `push-mode` verb's line, `gitflow.sh push-mode: gitflow.autopush='<value>' is not a boolean (git rc <n>)`. A repo with its own committed `.githooks/` (onboarded projects) keeps running its old hooks, which still push on an invalid value, until a session start runs `reconcile-hooks` and rewrites them; commit that refresh so other clones get it. Tests: `lib/gitflow-test.sh` T18q block.
|
||||
- `/guardrails` residuals. The live probe fails closed in more cases: every followed same-host redirect is charged to the request budget, and a budget that runs out mid-redirect reads ERROR; an anon-read or cross-user answer that is not a 200 or a refusal, or a 200 whose body is empty, over 1 MB or not JSON, reads SKIPPED instead of CLOSED; a rate-limit burst cut short by the budget or the transport reads SKIPPED; the source-map probe tries every candidate map of a script instead of stopping at the first it cannot confirm, and a `data:` map is OPEN whatever its encoding; the fields probe no longer flags metadata keys (`token_type`, `has_password`, `next_page_token`). The plan's Supabase `url` must be an http(s) url when tables or cross-user paths are set, `supabase.tables` must be a list of strings, and the `--authorized` refusal names both hosts. Scanner and probe text output strip control characters from file names and remote strings. Static rules lose false positives (hydrated state read per export chunk, webpack dev configs, `data-*` names with a UI suffix such as `-toggle` or `-strength`, GraphQL `_bool_exp` and capitalised types under `$where`) and gain detections: Go backtick SQL in `fmt.Sprintf`, `rf` / `fr` and triple-quoted Python f-strings, `sql.raw` / `db.raw` / `trx.raw`, an empty `.select()` on a users table, a hidden input spanning lines, a Rust `{}` argument to a DB shell tool, a large public `.map` whose head holds `"sources"`. Tests: `lib/tests/guardrails-scan.test.sh`, `lib/tests/guardrails-probe.test.sh`.
|
||||
- `lib/tests/mods.test.sh` judges `claude plugin validate` by the exit code of `--strict` instead of scraping its output: per-file warnings exit 1 and fail the mod, and the advisory plugin-directory section that CLI 2.1.306 prints (`ICON_MISSING`, not blocking) no longer counts as a failure, as the old whole-output `warning` grep did. A failure prints the `✘` / `error:` lines and the `--strict` command to rerun. A positive control writes two scratch manifests in a temp dir and checks that `--strict` passes the clean one and fails the one with warnings, so the gate cannot pass vacuously.
|
||||
|
||||
## [2.0.0] — 2026-10-06
|
||||
|
||||
|
||||
+33
-7
@@ -2,13 +2,16 @@
|
||||
# lib/tests/mods.test.sh — every mods/<name>/ plugin: the manifest name
|
||||
# equals the folder, skills/<name> is the relative loading symlink
|
||||
# ../mods/<name>, and (when the CLI offers `claude plugin test`) the mod
|
||||
# passes `claude plugin validate` without warning and `claude plugin test`.
|
||||
# passes `claude plugin validate --strict` (per-file warnings are errors; the
|
||||
# advisory plugin-directory section does not change the exit code) and
|
||||
# `claude plugin test`.
|
||||
# MODS_ROOT overrides the repo root (fixture controls). Fails when no mod
|
||||
# is found, so it can never pass vacuously.
|
||||
set -u
|
||||
ROOT="${MODS_ROOT:-$(cd "$(dirname "$0")/../.." && pwd)}"
|
||||
CLI_TIMEOUT=120
|
||||
pass=0; fail=0
|
||||
WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
|
||||
ok() { pass=$((pass+1)); echo "PASS $1"; }
|
||||
ko() { fail=$((fail+1)); echo "FAIL $1"; }
|
||||
check() { if [ "$2" = "$3" ]; then ok "$1"; else ko "$1: got[$2] want[$3]"; fi; }
|
||||
@@ -29,6 +32,11 @@ bounded() {
|
||||
cat "$out"; rm -f "$out"; return "$rc"
|
||||
}
|
||||
|
||||
# validate_strict DIR: output in STRICT_OUT, rc of the CLI (124 = timeout).
|
||||
validate_strict() {
|
||||
STRICT_OUT=$(bounded claude plugin validate --strict "$1")
|
||||
}
|
||||
|
||||
manifest_name() {
|
||||
python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("name",""))' \
|
||||
"$1" 2>/dev/null
|
||||
@@ -50,13 +58,15 @@ cli_unavailable() {
|
||||
|
||||
check_cli() {
|
||||
local name="$1" dir="$2" out rc=0
|
||||
out=$(bounded claude plugin validate "$dir") || rc=$?
|
||||
validate_strict "$dir" || rc=$?
|
||||
if [ "$rc" -eq 124 ]; then ko "$name: validate timed out after ${CLI_TIMEOUT}s"
|
||||
elif ! printf '%s' "$out" | grep -q 'Validation passed'; then
|
||||
ko "$name: validate did not pass: $(printf '%s' "$out" | head -3 | tr '\n' ' ')"
|
||||
elif printf '%s' "$out" | grep -qi 'warning'; then
|
||||
ko "$name: validate printed a warning"
|
||||
else ok "$name: validate passed, no warning"; fi
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
local detail
|
||||
detail=$(printf '%s' "$STRICT_OUT" | grep -E '^(✘|error:|⚠ Found)' \
|
||||
| head -2 | tr '\n' ' ')
|
||||
detail="$detail (rerun: claude plugin validate --strict $dir)"
|
||||
ko "$name: validate --strict rc=$rc: $detail"
|
||||
else ok "$name: validate --strict passed"; fi
|
||||
rc=0
|
||||
out=$(bounded claude plugin test "$dir") || rc=$?
|
||||
if [ "$rc" -eq 124 ]; then ko "$name: plugin test timed out after ${CLI_TIMEOUT}s"
|
||||
@@ -65,6 +75,21 @@ check_cli() {
|
||||
else ok "$name: plugin test passed"; fi
|
||||
}
|
||||
|
||||
# strict_discriminates: --strict must fail per-file warnings, pass a clean one.
|
||||
strict_discriminates() {
|
||||
local rc
|
||||
mkdir -p "$WORK/warnfix/.claude-plugin" "$WORK/cleanfix/.claude-plugin"
|
||||
echo '{"name":"warnfix"}' > "$WORK/warnfix/.claude-plugin/plugin.json"
|
||||
echo '{"name":"cleanfix","version":"0.1.0","description":"d",'\
|
||||
'"author":{"name":"x"}}' > "$WORK/cleanfix/.claude-plugin/plugin.json"
|
||||
rc=0; validate_strict "$WORK/cleanfix" || rc=$?
|
||||
if [ "$rc" -eq 124 ]; then ko "strict control: validate timed out"
|
||||
else check "strict control: clean fixture passes --strict" "$rc" 0; fi
|
||||
rc=0; validate_strict "$WORK/warnfix" || rc=$?
|
||||
if [ "$rc" -eq 124 ]; then ko "strict control: validate timed out"
|
||||
else check "strict control: per-file warnings fail --strict" "$rc" 1; fi
|
||||
}
|
||||
|
||||
manifests=()
|
||||
for m in "$ROOT"/mods/*/.claude-plugin/plugin.json; do
|
||||
[ -f "$m" ] && manifests+=("$m")
|
||||
@@ -76,6 +101,7 @@ if reason=$(cli_unavailable); then
|
||||
use_cli=0
|
||||
echo "SKIP: claude plugin test unavailable ($reason) — validate/test not run"
|
||||
fi
|
||||
[ "$use_cli" -eq 1 ] && strict_discriminates
|
||||
|
||||
for m in ${manifests[@]+"${manifests[@]}"}; do
|
||||
dir="$(dirname "$(dirname "$m")")"; name="$(basename "$dir")"
|
||||
|
||||
Reference in new issue
Block a user