forked from bchanot/claude
feat(guardrails): read-only live probe to confirm a hole from outside
probe.py observes a running target the user owns and names, driven by a plan built from the static findings. Five probes: anon-read, cross-user, no-auth route, CORS reflection, capped rate-limit burst. GET/HEAD/OPTIONS only, --authorized gate, secrets from the environment, same-host redirect only, bounded request budget. SKILL STEP 5, checks.md reference, docs. Suite guardrails-probe.test.sh (16 checks) against a localhost fixture.
This commit is contained in:
1 parent
dbf409f6e6
commit
f0bcbfef46
8 files changed
+572
-16
No files matched your search
+1
-1
@@ -7,7 +7,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/) and this project
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
- **Secure-by-default guardrails**: `rules/secure-by-default.md` (always on) makes five domains baseline on every project, plus systematic API and access hygiene (auth on every route, CORS, secrets, passwords hashed with argon2id or bcrypt, never compared in clear, logged or hardcoded): hosting (rate limit, CDN and firewall, hard spend cap), accessibility (keyboard, visible focus, alt text, field labels), Supabase (RLS on every table, owner-scoped policies, no service role key client-side), SMS and email (recorded consent, unsubscribe, STOP), cloud functions (no self-trigger, instance cap, budget stop). `/guardrails` audits them: a stdlib Python scanner (`skills/guardrails/scripts/scan.py`, FAIL / WARN / MANUAL, exit 1 on FAIL) then judgment, report in `.claude/audits/GUARDRAILS.md`. The security-auditor gate runs the scanner's `api` and `supabase` domains and blocks on FAIL. Items only a provider dashboard can set stay MANUAL, never claimed done. Suite: `lib/tests/guardrails-scan.test.sh` (clean fixture plus one mutant per rule).
|
||||
- **Secure-by-default guardrails**: `rules/secure-by-default.md` (always on) makes five domains baseline on every project, plus systematic API and access hygiene (auth on every route, CORS, secrets, passwords hashed with argon2id or bcrypt, never compared in clear, logged or hardcoded): hosting (rate limit, CDN and firewall, hard spend cap), accessibility (keyboard, visible focus, alt text, field labels), Supabase (RLS on every table, owner-scoped policies, no service role key client-side), SMS and email (recorded consent, unsubscribe, STOP), cloud functions (no self-trigger, instance cap, budget stop). `/guardrails` audits them: a stdlib Python scanner (`skills/guardrails/scripts/scan.py`, FAIL / WARN / MANUAL, exit 1 on FAIL) then judgment, report in `.claude/audits/GUARDRAILS.md`. The security-auditor gate runs the scanner's `api` and `supabase` domains and blocks on FAIL. Items only a provider dashboard can set stay MANUAL, never claimed done. An optional read-only live probe (`skills/guardrails/scripts/probe.py`, `--live`) confirms a finding from outside on a target the user owns and names: anon-read, cross-user, no-auth route, CORS reflection and a capped rate-limit burst, verdicts OPEN / CLOSED / SKIPPED / ERROR. It sends GET, HEAD and OPTIONS only (one opt-in POST burst, capped at 25), requires `--authorized`, takes secrets from the environment never the command line, never follows a redirect to another host, and stops at a request budget. Suites: `lib/tests/guardrails-scan.test.sh` (clean fixture plus one mutant per rule) and `lib/tests/guardrails-probe.test.sh` (probe logic against a localhost fixture server).
|
||||
- **model-router mod**: `mods/model-router/`, a Claude Code mod (function-hooks plugin), routes every repo skill and agent from phase rows (`plan`, `reflect`, `orchestrate`, `escalate`, `judge`, `implement`, `write`, `verify`, `explore`, `apply`, `mechanical`; built-ins Explore on sonnet/medium, Plan on opus/xhigh). A typed skill routes the main loop to its row, and a best-tier row holds across turns in a run slot until `/route clear`, `/route off`, a user `/model` or a typed skill on a non-best row. Agents get their row's model at spawn (within the tier, upward only) and its effort on every step; explicit Agent params win. Orchestrators declare their phases through the `mcp__model-router__route` tool. `ultrathink` in a prompt sets the turn's minimum effort and `/route effort=max` holds until `/route clear`; the built-in `/effort` is not a lever inside a run. `/route show` names the run slot (`main: run <phase>`), and a `null` row in the override drops a default row. The model gets a `route` tool and the user a `/route` command (`show|clear|off|on|reload|pending|ask on|off|forget <name|all|projects>|<phase>|model=<alias|id> effort=<level>|switch on|off|verbose on|off`). Optional per-machine config `~/.claude/model-router.json`, where `"enabled": false` turns it off on that machine. The spinner suffix and the status line show the route in force. It loads in every session through the tracked symlink `skills/model-router` (`model-router@skills-dir`). `make doctor` gains a Mods section; suite `make test suite=lib/tests/mods.test.sh`. Known limits: a main-loop switch runs only while the cold read it costs (the context minus what the target model's prompt cache holds) fits `upgradeMaxTokens` (200000) or `downgradeMaxTokens` (120000), and a downgrade also needs `mainModelSwitch` (on by default, off on any non-boolean value); the hooks send full model ids, so the `models` table has to follow new versions.
|
||||
- **model-router first-use confirmation**: `mods/model-router/routing.json` (tracked) is the single source of the phase table and of every skill and agent row, and keeps the decisions: `confirmed`, `changed` (`from`/`to`), `projects` exceptions keyed by the origin remote reduced to `host/path` (credentials and local paths never stored), and `ask`. The first use of a rowed typed skill, a rowed agent spawn or a main-loop phase declared through the `route` tool opens a dialog with context: the skill's description (first sentence of its `SKILL.md` frontmatter) or the agent's, the phase with its `about` line (a new field on each of the 11 phases) and the model id and effort the next step really runs on. A row offers Later, Keep or Change; a main-loop phase Later or Keep. Change asks the model (fable, opus, sonnet, haiku with their tier), then the effort among those the phases of that model use (skipped when there is only one), then Everywhere or This project only; the pair maps to an existing phase (rows stay phase names, the same phase counts as Keep, a pair no phase offers is added by hand as a new phase). A skill-row change toasts the model and effort it now runs on, with the `/route switch on` hint when the main loop holds back a downgrade; a free-text answer counts as Later. One dialog at a time, never in headless (`-p`) runs or inside a sub-agent. Only a dialog answer, `/route ask on|off` or `/route forget` (which takes decisions back; a changed row returns to its shipped phase, the frontmatter floors stay yours to realign) writes the file (serialized, 64 KiB cap, never created when absent); each write asks you to commit it from the config repo. `/route pending` lists unconfirmed rows and phases. Layers: routing.json < `~/.claude/model-router.json` (its `ask` wins); a project's `.claude/model-router.json` is never read. Tests: `mods/model-router/hooks/register.test.ts`.
|
||||
- **Manual-push mode**: `git config gitflow.autopush false` (human-set) now stops every push the gitflow lib makes, not only the post-commit / post-merge hooks. `gitflow start` and `finish` branch, commit and merge locally and push nothing; `gitflow delete` leaves the `origin/` copy in place and prints `git push origin --delete <br>` for the user to run. `hooks/unpushed-guard.sh` stays silent at turn end in this mode and opens each session with one `ℹ manual push mode:` line counting the commits no remote holds across every local branch; an unparseable or unreadable `gitflow.autopush` value is treated as manual push mode too, and that line names it. `hooks/push-guard.sh` (PreToolUse, `Bash|Monitor`) refuses any `git push` Claude types while `gitflow.autopush` reads false in the session cwd or in a literal `-C`/`cd` directory the command names (global config counts outside a repo); the refusal tells the user to run it with `! git push`. It reads the mode through the same lib verb as every other reader and fails closed: an unparseable or unreadable value reads as manual, and an internal error, a missing `lib/gitflow.sh`, more than 20 distinct directory tokens in one command (capped before any token is classified), a `cd`/`-C` directory token mixing quoted and unquoted parts, or a payload jq cannot parse whose raw text looks like a push refuse the push (these pathological cases fire in auto mode too). Directory tokens are read as whole shell words, adjacent quoted segments and backslash escapes included. Once the text looks like a push, the command is split into simple commands (quotes, escapes, separators, heredoc bodies dropped) and a push must be git's own subcommand in one of them: `grep "git push"`, `git log -S "git push"`, `git config --get push.default`, `git stash push`, a heredoc quoting a push or a commit message naming one are allowed; `bash -c` scripts are re-checked, wrappers (`sudo`, `env`, `eval`, `xargs`, `timeout`, `git subtree`...) and assignments keep the text verdict. The misses listed in its header fall to a new `autoMode.soft_deny` rule that no request in the turn clears. The session banner adds `🔒 push : manual (autopush=false) — ! git push` when the key reads false, and `🔒 push : manual (autopush bad) — ! git push` when the value is invalid. Skills read the mode through a new lib verb, `bash ~/.claude/lib/gitflow.sh push-mode`: it prints `auto`, `manual` or `invalid` (rc 0) and names an invalid value on stderr (printable characters only, 64 at most). It is the one reader a skill may call, since the `git config` read of the key is denied to Claude. Skills push nothing on their own, except the `/release-candidate` tag in auto-push mode on an explicit go. Every "on origin" or "not pushed" line they print comes from `git rev-list --count origin/<br>..<br>` read after the fact, with the complete `! git …` command when something is left for the user to push. An invalid value (anything but unset, true or false, or a read that fails) is manual push mode for every reader and is named where it is read (see Fixed). Tests: `lib/gitflow-test.sh` T11b (push-mode verb), T18m and T18q blocks, `lib/tests/unpushed-guard.test.sh` T10-T16, `lib/tests/push-guard.test.sh` (98 checks).
|
||||
|
||||
@@ -228,7 +228,7 @@ a different package, ships its own conflicting `graphify` bin) — see
|
||||
| `/reconcile` | Confront declared status (TODO, registries) against real git/fs state — surface stale items |
|
||||
| `/pdf-translate` | Translate a PDF to another language, output as HTML (via Vision) |
|
||||
| `/explain-diff` | Explain a diff, commit, branch or PR as one self-contained HTML page (background, intuition, code walkthrough, 5-question quiz), saved outside the repo |
|
||||
| `/guardrails` | Audit the five baseline guardrails (hosting, a11y, Supabase RLS, SMS/email consent, cloud functions) plus API hygiene (auth, CORS, secrets, password storage): deterministic scanner, then judgment, report in `.claude/audits/GUARDRAILS.md` |
|
||||
| `/guardrails` | Audit the five baseline guardrails (hosting, a11y, Supabase RLS, SMS/email consent, cloud functions) plus API hygiene (auth, CORS, secrets, password storage): deterministic scanner, then judgment, report in `.claude/audits/GUARDRAILS.md`. `--live <url you own>` adds a read-only probe that confirms a hole from outside |
|
||||
| `/close` | End-of-session ritual — alias for `/capitalize --ritual` (dedup + TODO reconcile + 3-question reflection) |
|
||||
| `/harden` | Web hardening audit — HTTPS/TLS, HSTS, CSP, security headers |
|
||||
| `/web-validate` | W3C HTML/CSS validity + WCAG 2.1 accessibility audit |
|
||||
|
||||
@@ -121,6 +121,7 @@ Tu veux...
|
||||
| Traduire un PDF | `/pdf-translate` |
|
||||
| Comprendre un changement pas suivi | `/explain-diff` |
|
||||
| Auditer garde-fous (RLS, a11y, consentement, fonctions, mots de passe) | `/guardrails` |
|
||||
| Sonder le projet en direct (lecture seule, cible à toi) | `/guardrails --live <url>` |
|
||||
| Changer profil skills | `/profile` |
|
||||
| Audit/polish design (anti-slop) | `/impeccable` |
|
||||
| Sweep groupé tous axes (nettoyage + sécu + reconcile + doc) | `/tour` |
|
||||
@@ -164,7 +165,7 @@ Tu veux...
|
||||
| `/client-handover` | Livraison client | Audits finaux + livrable brandé |
|
||||
| `/pdf-translate` | Traduire un PDF vers une autre langue | Sortie HTML fidèle (images, layout, style préservés) |
|
||||
| `/explain-diff` | Comprendre un commit, une branche ou une PR qu'on n'a pas suivis | Page HTML autonome hors du repo : contexte, intuition, code, quiz de 5 questions |
|
||||
| `/guardrails` | Vérifier hébergement, a11y, Supabase RLS, consentement SMS/email, fonctions cloud, stockage des mots de passe | Scanner déterministe puis jugement ; rapport `.claude/audits/GUARDRAILS.md` |
|
||||
| `/guardrails` | Vérifier hébergement, a11y, Supabase RLS, consentement SMS/email, fonctions cloud, stockage des mots de passe | Scanner déterministe puis jugement ; rapport `.claude/audits/GUARDRAILS.md` ; `--live <url à toi>` sonde en lecture seule |
|
||||
| `/impeccable` | Audit/polish design + détecteur anti-slop déterministe | 23 verbes ; `npx impeccable detect` (exit 0/2) |
|
||||
| `/tour` | Sweep groupé sur un ou plusieurs projets | Sécu + nettoyage + reconcile + doc, boucle jusqu'à un pass propre |
|
||||
| `/route` | Voir ou fixer la route du mod model-router | show / clear / off / on / reload / pending / ask on\|off / <phase> / model=… effort=… / switch on\|off / classify on\|off (classifieur de prompt) / verbose on\|off (transitions seules) |
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
#!/usr/bin/env bash
|
||||
# lib/tests/guardrails-probe.test.sh — floor of the /guardrails live probe.
|
||||
# A throwaway http.server on 127.0.0.1 serves scripted responses; the probe
|
||||
# runs against it and each verdict is asserted. Nothing leaves the machine.
|
||||
# The probe sends read-only methods; this suite proves the OPEN/CLOSED logic,
|
||||
# the --authorized gate, the same-host redirect guard and the request budget.
|
||||
set -u
|
||||
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
PROBE="$ROOT/skills/guardrails/scripts/probe.py"
|
||||
WORK="$(mktemp -d)"
|
||||
PASS=0; FAIL=0; SRV=""
|
||||
cleanup() { [ -n "$SRV" ] && kill "$SRV" 2>/dev/null; rm -rf "$WORK"; }
|
||||
trap cleanup EXIT
|
||||
|
||||
check() { # check <label> <got> <want>
|
||||
if [ "$2" = "$3" ]; then PASS=$((PASS+1)); echo " PASS $1"
|
||||
else FAIL=$((FAIL+1)); echo " FAIL $1 (got '$2', want '$3')"; fi
|
||||
}
|
||||
|
||||
# Fixture server: a fixed map of path -> (status, body, headers). Read-only,
|
||||
# bound to the loopback, on an OS-chosen port it prints on the first line.
|
||||
cat > "$WORK/server.py" <<'PY'
|
||||
import json
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
|
||||
HITS = {"rl": 0}
|
||||
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
def _send(self, code, body=b"", headers=None):
|
||||
self.send_response(code)
|
||||
for k, v in (headers or {}).items():
|
||||
self.send_header(k, v)
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def do_OPTIONS(self):
|
||||
origin = self.headers.get("Origin", "")
|
||||
self._send(204, b"", {
|
||||
"Access-Control-Allow-Origin": origin,
|
||||
"Access-Control-Allow-Credentials": "true"})
|
||||
|
||||
def do_GET(self):
|
||||
path = self.path.split("?")[0]
|
||||
if path == "/rest/v1/open_table":
|
||||
return self._send(200, b'[{"id": 1}]')
|
||||
if path == "/rest/v1/closed_table":
|
||||
return self._send(200, b'[]')
|
||||
if path == "/api/private":
|
||||
return self._send(200, b"secret")
|
||||
if path == "/api/guarded":
|
||||
return self._send(401, b"no")
|
||||
if path == "/rl":
|
||||
HITS["rl"] += 1
|
||||
return self._send(429 if HITS["rl"] > 5 else 200, b"")
|
||||
if path == "/nolimit":
|
||||
return self._send(200, b"")
|
||||
return self._send(404, b"")
|
||||
|
||||
|
||||
srv = HTTPServer(("127.0.0.1", 0), H)
|
||||
print(srv.server_port, flush=True)
|
||||
srv.serve_forever()
|
||||
PY
|
||||
|
||||
python3 -B "$WORK/server.py" > "$WORK/port" &
|
||||
SRV=$!
|
||||
for _ in $(seq 1 50); do [ -s "$WORK/port" ] && break; sleep 0.1; done
|
||||
PORT="$(head -1 "$WORK/port")"
|
||||
if [ -z "$PORT" ]; then echo " FAIL fixture server did not start"; exit 1; fi
|
||||
BASE="http://127.0.0.1:$PORT"
|
||||
|
||||
probe() { python3 -B "$PROBE" "$@"; }
|
||||
verdict() { printf '%s\n' "$1" | grep -A1 "$2" | grep -oE 'OPEN|CLOSED|SKIPPED|ERROR' | head -1; }
|
||||
|
||||
echo "== authorization gate"
|
||||
cat > "$WORK/plan.json" <<EOF
|
||||
{"base_url": "$BASE",
|
||||
"supabase": {"url": "$BASE", "tables": ["open_table", "closed_table"],
|
||||
"cross_user_paths": ["/rest/v1/open_table"]},
|
||||
"routes": [{"path": "/api/private"}, {"path": "/api/guarded"},
|
||||
{"path": "/api/public", "public": true}],
|
||||
"cors": true}
|
||||
EOF
|
||||
probe "$WORK/plan.json" >/dev/null 2>&1
|
||||
check "refuses without --authorized (exit 2)" "$?" 2
|
||||
|
||||
echo "== read-only probes against the fixture"
|
||||
export GUARDRAILS_ANON_KEY="anon-test-key"
|
||||
export GUARDRAILS_JWT2="second-user-test-token"
|
||||
OUT="$(probe --authorized "$WORK/plan.json")"
|
||||
RC=$?
|
||||
check "anon read of an open table is OPEN" \
|
||||
"$(verdict "$OUT" 'anon-read:open_table')" OPEN
|
||||
check "anon read of a closed table is CLOSED" \
|
||||
"$(verdict "$OUT" 'anon-read:closed_table')" CLOSED
|
||||
check "cross-user read returning rows is OPEN" \
|
||||
"$(verdict "$OUT" 'cross-user:')" OPEN
|
||||
check "unauth route returning 200 is OPEN" \
|
||||
"$(verdict "$OUT" 'no-auth:/api/private')" OPEN
|
||||
check "unauth route returning 401 is CLOSED" \
|
||||
"$(verdict "$OUT" 'no-auth:/api/guarded')" CLOSED
|
||||
check "public route is not probed" \
|
||||
"$(printf '%s\n' "$OUT" | grep -c 'no-auth:/api/public')" 0
|
||||
check "reflected origin with credentials is OPEN" \
|
||||
"$(verdict "$OUT" 'cors-reflect')" OPEN
|
||||
check "a found OPEN verdict exits 1" "$RC" 1
|
||||
|
||||
echo "== skips when a secret is absent"
|
||||
unset GUARDRAILS_ANON_KEY
|
||||
OUT="$(probe --authorized "$WORK/plan.json")"
|
||||
check "anon-read skips without the key" \
|
||||
"$(verdict "$OUT" 'anon-read')" SKIPPED
|
||||
export GUARDRAILS_ANON_KEY="anon-test-key"
|
||||
|
||||
echo "== rate limit"
|
||||
cat > "$WORK/rl.json" <<EOF
|
||||
{"base_url": "$BASE", "rate_limit": {"path": "/rl", "count": 10}}
|
||||
EOF
|
||||
OUT="$(probe --authorized "$WORK/rl.json")"
|
||||
check "a route that starts sending 429 is CLOSED" \
|
||||
"$(verdict "$OUT" 'rate-limit')" CLOSED
|
||||
cat > "$WORK/nolimit.json" <<EOF
|
||||
{"base_url": "$BASE", "rate_limit": {"path": "/nolimit", "count": 10}}
|
||||
EOF
|
||||
OUT="$(probe --authorized "$WORK/nolimit.json")"
|
||||
check "a route with no 429 is OPEN" \
|
||||
"$(verdict "$OUT" 'rate-limit')" OPEN
|
||||
|
||||
echo "== budget caps the burst"
|
||||
cat > "$WORK/burst.json" <<EOF
|
||||
{"base_url": "$BASE", "rate_limit": {"path": "/nolimit", "count": 99}}
|
||||
EOF
|
||||
OUT="$(probe --authorized "$WORK/burst.json")"
|
||||
check "the burst is capped, never 99 requests" \
|
||||
"$(printf '%s\n' "$OUT" | grep -oE 'sent [0-9]+' | grep -oE '[0-9]+')" 25
|
||||
|
||||
echo "== usage errors"
|
||||
probe --authorized "$WORK/missing.json" >/dev/null 2>&1
|
||||
check "a missing plan exits 2" "$?" 2
|
||||
printf '{"base_url": "%s", "rate_limit": {"path": "/x", "method": "DELETE"}}\n' \
|
||||
"$BASE" > "$WORK/bad.json"
|
||||
probe --authorized "$WORK/bad.json" >/dev/null 2>&1
|
||||
check "a write method in the plan exits 2" "$?" 2
|
||||
|
||||
echo "== JSON output"
|
||||
check "json reports the open count" \
|
||||
"$(probe --authorized --json "$WORK/plan.json" \
|
||||
| python3 -B -c 'import json,sys; print(json.load(sys.stdin)["open"])')" 4
|
||||
|
||||
echo "---"
|
||||
echo "passed: $PASS failed: $FAIL"
|
||||
[ "$FAIL" -eq 0 ]
|
||||
+50
-12
@@ -1,18 +1,20 @@
|
||||
---
|
||||
name: guardrails
|
||||
effort: high
|
||||
description: 'Use when auditing a project against the baseline guardrails: hosting (rate limit, CDN and firewall, spend cap), accessibility (keyboard, visible focus, alt text, field labels), Supabase (RLS on every table, owner-scoped policies, no service_role key client-side), SMS and email (recorded consent, unsubscribe, STOP), cloud functions (no self-trigger, instance cap, budget stop), plus API and access hygiene (auth on every route, CORS, secrets, password storage). Runs a deterministic scanner, judges each hit, writes .claude/audits/GUARDRAILS.md. Triggers: "guardrails", "garde-fous", "RLS activée ?", "audit supabase", "consentement sms", "plafond de dépense", "mots de passe en clair ?", "teste la sécurité de base du projet". Headers and TLS → /harden; secrets, CVE, OWASP → /cso; full WCAG → /web-validate.'
|
||||
argument-hint: '[project path] [--domain api,hosting,a11y,supabase,messaging,functions] [--exclude paths]'
|
||||
description: 'Use when auditing a project against the baseline guardrails: hosting (rate limit, CDN and firewall, spend cap), accessibility (keyboard, visible focus, alt text, field labels), Supabase (RLS on every table, owner-scoped policies, no service_role key client-side), SMS and email (recorded consent, unsubscribe, STOP), cloud functions (no self-trigger, instance cap, budget stop), plus API and access hygiene (auth on every route, CORS, secrets, password storage). Runs a deterministic scanner, judges each hit, writes .claude/audits/GUARDRAILS.md. Optional read-only live probe confirms a hole from outside on a target the user owns and names. Triggers: "guardrails", "garde-fous", "RLS activée ?", "audit supabase", "consentement sms", "plafond de dépense", "mots de passe en clair ?", "teste la sécurité de base du projet", "teste-le en conditions réelles", "sonde le projet en direct". Headers and TLS → /harden; secrets, CVE, OWASP → /cso; full WCAG → /web-validate.'
|
||||
argument-hint: '[project path] [--domain api,hosting,a11y,supabase,messaging,functions] [--exclude paths] [--live <url you own>]'
|
||||
---
|
||||
|
||||
# /guardrails
|
||||
|
||||
Checks a project against `rules/secure-by-default.md`. A scanner reads the
|
||||
tree as data and finds what text can prove. This skill then judges the hits,
|
||||
covers what text cannot prove, and lists what only a dashboard can set.
|
||||
covers what text cannot prove, lists what only a dashboard can set, and, on
|
||||
request, probes a running target the user owns to confirm a hole from outside.
|
||||
|
||||
The scanner proves absence of a protection, never presence of security. A
|
||||
clean scan means "no static finding", and the report says exactly that.
|
||||
clean scan means "no static finding", and the report says exactly that. A
|
||||
live probe proves the opposite direction: that a door is actually open.
|
||||
|
||||
## STEP 1: Scope
|
||||
|
||||
@@ -62,21 +64,57 @@ Read the code for the domains that apply and answer, with file:line evidence:
|
||||
- **A11y**: with a dev server running, walk the main flow with Tab only
|
||||
(`/browse`). Otherwise mark the keyboard walk-through as not done.
|
||||
|
||||
## STEP 5: Manual checklist
|
||||
## STEP 5: Live probe (only on `--live`, read-only)
|
||||
|
||||
The static scan shows a protection is missing in the code. A probe shows the
|
||||
door is actually open. Run it only when the user asks, and only against a
|
||||
target the user owns and names: their local dev server first, a staging or
|
||||
production URL only on an explicit, specific go. Never a URL you inferred,
|
||||
never a third party.
|
||||
|
||||
1. **Build the plan** `references/checks.md` describes. From the static
|
||||
findings fill a JSON plan: `base_url`, the Supabase `url` and `tables`,
|
||||
the `routes` (mark the ones meant to be public), `cors`, and a
|
||||
`rate_limit` route if the user wants that one burst. Secrets never go in
|
||||
the plan or the command line: export `GUARDRAILS_ANON_KEY` and, for the
|
||||
cross-user probe, `GUARDRAILS_JWT2`, from test keys, never `service_role`.
|
||||
2. **Run it**, confirming ownership with `--authorized`:
|
||||
```
|
||||
python3 -B "$HOME/.claude/skills/guardrails/scripts/probe.py" \
|
||||
plan.json --authorized
|
||||
```
|
||||
It sends GET, HEAD and OPTIONS only (the rate-limit burst is the one
|
||||
POST, opt-in in the plan, capped at 25 requests). It never writes data,
|
||||
never follows a redirect to another host, and stops at a request budget.
|
||||
Exit 1 means at least one OPEN verdict.
|
||||
3. **Cross the verdicts with the static findings.** A verdict is an
|
||||
observation, not a conclusion: OPEN means the request returned data or a
|
||||
200, CLOSED means it was refused or empty. A static FAIL confirmed OPEN in
|
||||
the probe leads the report. A CLOSED probe on a static FAIL still needs the
|
||||
fix: the dev target may differ from production.
|
||||
4. **Guardrails on the probe itself**: read-only, test keys only, the
|
||||
loopback or a URL the user named. A login burst can lock the user's own
|
||||
account, so run it only with a throwaway test account and the user's go.
|
||||
Never fuzz, never POST to anything but a named login route, never probe a
|
||||
target you were not told to. That is pentest territory, another contract.
|
||||
|
||||
## STEP 6: Manual checklist
|
||||
|
||||
Print each MANUAL item as a step the user performs: spend cap, WAF and DDoS
|
||||
rules, budget alert wired to a stop, live RLS queries. Provider limits change:
|
||||
name the setting, tell the user to confirm it in the current provider docs.
|
||||
Never run a live probe against production, and never claim a MANUAL item done.
|
||||
Never run a live probe against production without the user's explicit go, and
|
||||
never claim a MANUAL item done.
|
||||
|
||||
## STEP 6: Report
|
||||
## STEP 7: Report
|
||||
|
||||
Write `.claude/audits/GUARDRAILS.md`: one table per domain with status PASS,
|
||||
FAIL, WARN, MANUAL or N/A, the evidence, and the fix in one line. PASS means
|
||||
"no static finding". Keep it short; no filler sections.
|
||||
"no static finding". When a live probe ran, add a line for its verdict next
|
||||
to the matching static finding. Keep it short; no filler sections.
|
||||
|
||||
## STEP 7: Hand over
|
||||
## STEP 8: Hand over
|
||||
|
||||
Reply with counts, the FAILs with file:line and a one-line fix each, and the
|
||||
MANUAL items. This skill audits; it does not fix. A fix goes through
|
||||
`/bugfix`, `/feat` or `/hotfix`.
|
||||
Reply with counts, the FAILs with file:line and a one-line fix each, any OPEN
|
||||
probe verdicts, and the MANUAL items. This skill audits; it does not fix. A
|
||||
fix goes through `/bugfix`, `/feat` or `/hotfix`.
|
||||
@@ -153,3 +153,54 @@ user's token, a GET on the first user's rows must return nothing.
|
||||
a retention rule for consent proof.
|
||||
- **Supabase**: the live queries above; spend cap on the plan; redirect URLs
|
||||
and email confirmation settings reviewed.
|
||||
|
||||
## Live probe (`--live`, read-only)
|
||||
|
||||
`probe.py` observes a running target the user owns and names. It is driven by
|
||||
a plan JSON built from the static findings; secrets come from the environment,
|
||||
never the plan or the command line. It sends GET, HEAD and OPTIONS only, with
|
||||
one opt-in POST burst for the rate-limit check, and stops at a request budget.
|
||||
|
||||
```json
|
||||
{
|
||||
"base_url": "http://localhost:5173",
|
||||
"supabase": {
|
||||
"url": "http://localhost:54321",
|
||||
"tables": ["notes", "contacts"],
|
||||
"cross_user_paths": ["/rest/v1/notes?select=*"]
|
||||
},
|
||||
"routes": [
|
||||
{"path": "/api/v1/notes"},
|
||||
{"path": "/api/v1/health", "public": true}
|
||||
],
|
||||
"cors": true,
|
||||
"cors_path": "/api/v1/notes",
|
||||
"rate_limit": {"path": "/api/v1/auth/login", "method": "POST",
|
||||
"count": 20, "body": "{\"email\":\"t@t.io\",\"password\":\"x\"}"}
|
||||
}
|
||||
```
|
||||
|
||||
Environment, from test keys only, never `service_role`:
|
||||
`GUARDRAILS_ANON_KEY` (anon-read), `GUARDRAILS_JWT2` (cross-user).
|
||||
|
||||
| Probe | id | OPEN means | Read it against |
|
||||
|---|---|---|---|
|
||||
| anon-read | `anon-read:<table>` | the anon key read a row | SB-RLS-MISSING, SB-POLICY-* |
|
||||
| cross-user | `cross-user:<path>` | a second user's token read these rows | policy owner column (STEP 4) |
|
||||
| no-auth route | `no-auth:<path>` | the route answered 200 with no token | API-ROUTE-NO-AUTH, object authz |
|
||||
| cors-reflect | `cors-reflect` | a foreign origin was reflected with credentials | API-CORS-WILDCARD |
|
||||
| rate-limit | `rate-limit` | the burst saw no 429 | HOST-NO-RATE-LIMIT |
|
||||
|
||||
A verdict is an observation, not a verdict on the project. OPEN is a returned
|
||||
200 or returned rows; CLOSED is a refusal or an empty body; SKIPPED is a
|
||||
missing key; ERROR is a transport failure. A static FAIL confirmed OPEN leads
|
||||
the report. A CLOSED probe on a static FAIL still needs the fix: the dev
|
||||
target may differ from production.
|
||||
|
||||
Safety, enforced by the script and by the skill: `--authorized` is required;
|
||||
read-only methods only; the loopback or a URL the user named, never one you
|
||||
inferred; a login burst only with a throwaway test account on the user's go,
|
||||
because it can lock the user's own account; a redirect to another host is
|
||||
never followed, so a key never leaves the named host. Fuzzing, writes beyond
|
||||
the named login route, or probing a third party are out of scope: that is
|
||||
pentest, another contract.
|
||||
@@ -0,0 +1,309 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Live guardrails probe: observe a running target from outside, read-only.
|
||||
|
||||
The static scan shows a protection is absent in the code. This observes what
|
||||
a running target the user owns and names actually returns. It sends GET, HEAD
|
||||
and OPTIONS only; a login burst is opt-in in the plan. It never writes data,
|
||||
never uses a service_role key, never follows a redirect to another host, and
|
||||
stops at a fixed request budget.
|
||||
|
||||
Input is a plan JSON (built by the skill from the static findings) plus
|
||||
secrets from the environment, never the command line:
|
||||
GUARDRAILS_ANON_KEY Supabase anon/publishable key for the anon-read probe
|
||||
GUARDRAILS_JWT2 a second test user's token for the cross-user probe
|
||||
|
||||
A verdict is a factual observation, not a conclusion: the skill reads it
|
||||
against the static findings. OPEN means the request returned data or a 200,
|
||||
CLOSED means it was refused or empty. Exit codes: 0 no OPEN, 1 at least one
|
||||
OPEN, 2 usage error.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from dataclasses import asdict, dataclass
|
||||
from urllib.parse import quote, urlsplit
|
||||
|
||||
READ_METHODS = {"GET", "HEAD", "OPTIONS"}
|
||||
FAKE_ORIGIN = "https://guardrails-probe.invalid"
|
||||
DEFAULT_BUDGET = 200
|
||||
BURST_CAP = 25
|
||||
BODY_CAP = 1_000_000
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Response:
|
||||
status: int # None on a transport error
|
||||
headers: dict
|
||||
body: bytes
|
||||
error: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Probe:
|
||||
domain: str
|
||||
id: str
|
||||
verdict: str # OPEN | CLOSED | SKIPPED | ERROR
|
||||
target: str
|
||||
message: str
|
||||
|
||||
def as_dict(self):
|
||||
return asdict(self)
|
||||
|
||||
|
||||
class _SameHost(urllib.request.HTTPRedirectHandler):
|
||||
"""Follow a same-host redirect, drop one that leaves the host."""
|
||||
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
if urlsplit(newurl).netloc != urlsplit(req.full_url).netloc:
|
||||
return None
|
||||
return super().redirect_request(req, fp, code, msg, headers, newurl)
|
||||
|
||||
|
||||
class Session:
|
||||
"""Holds the redirect-guarded opener and the shared request budget."""
|
||||
|
||||
def __init__(self, timeout=8, budget=DEFAULT_BUDGET):
|
||||
self.opener = urllib.request.build_opener(_SameHost())
|
||||
self.timeout = timeout
|
||||
self.budget = budget
|
||||
|
||||
|
||||
def _response_from(resp):
|
||||
return Response(resp.status, dict(resp.headers), resp.read(BODY_CAP))
|
||||
|
||||
|
||||
def fetch(session, method, url, headers=None, body=None):
|
||||
if session.budget <= 0:
|
||||
return Response(None, {}, b"", "request budget exhausted")
|
||||
session.budget -= 1
|
||||
req = urllib.request.Request(
|
||||
url, data=body, method=method, headers=headers or {})
|
||||
try:
|
||||
with session.opener.open(req, timeout=session.timeout) as resp:
|
||||
return _response_from(resp)
|
||||
except urllib.error.HTTPError as exc:
|
||||
return Response(exc.code, dict(exc.headers), b"")
|
||||
except (urllib.error.URLError, OSError, ValueError) as exc:
|
||||
return Response(None, {}, b"", str(exc))
|
||||
|
||||
|
||||
def _verdict(ctx, response, is_open, msgs):
|
||||
"""ctx is (domain, id, target); msgs is (open_note, closed_note)."""
|
||||
domain, pid, target = ctx
|
||||
if response.error:
|
||||
return Probe(domain, pid, "ERROR", target, response.error)
|
||||
verdict = "OPEN" if is_open else "CLOSED"
|
||||
note = f"HTTP {response.status}: {msgs[0] if is_open else msgs[1]}"
|
||||
return Probe(domain, pid, verdict, target, note)
|
||||
|
||||
|
||||
def _nonempty_rows(body):
|
||||
try:
|
||||
data = json.loads(body or b"[]")
|
||||
except ValueError:
|
||||
return False
|
||||
return isinstance(data, list) and len(data) > 0
|
||||
|
||||
|
||||
def _table_probe(session, base, table, headers):
|
||||
target = f"{base}/rest/v1/{quote(table)}?select=*&limit=1"
|
||||
resp = fetch(session, "GET", target, headers)
|
||||
return _verdict(
|
||||
("supabase", f"anon-read:{table}", target), resp,
|
||||
resp.status == 200 and _nonempty_rows(resp.body),
|
||||
("anon request returned rows", "anon request returned no row"))
|
||||
|
||||
|
||||
def probe_supabase(session, plan):
|
||||
cfg = plan.get("supabase")
|
||||
anon = os.environ.get("GUARDRAILS_ANON_KEY")
|
||||
if not cfg or not cfg.get("tables"):
|
||||
return []
|
||||
if not anon:
|
||||
return [Probe("supabase", "anon-read", "SKIPPED", cfg.get("url", ""),
|
||||
"set GUARDRAILS_ANON_KEY to run the anon-read probe")]
|
||||
base = cfg["url"].rstrip("/")
|
||||
headers = {"apikey": anon, "Authorization": f"Bearer {anon}"}
|
||||
return [_table_probe(session, base, t, headers) for t in cfg["tables"]]
|
||||
|
||||
|
||||
def _cross_user_probe(session, base, path, headers):
|
||||
target = base + path
|
||||
resp = fetch(session, "GET", target, headers)
|
||||
return _verdict(
|
||||
("supabase", f"cross-user:{path}", target), resp,
|
||||
resp.status == 200 and _nonempty_rows(resp.body),
|
||||
("second user's token returned these rows",
|
||||
"second user's token returned nothing"))
|
||||
|
||||
|
||||
def probe_cross_user(session, plan):
|
||||
cfg = plan.get("supabase") or {}
|
||||
paths = cfg.get("cross_user_paths") or []
|
||||
jwt2 = os.environ.get("GUARDRAILS_JWT2")
|
||||
if not paths:
|
||||
return []
|
||||
if not jwt2:
|
||||
return [Probe("supabase", "cross-user", "SKIPPED", cfg.get("url", ""),
|
||||
"set GUARDRAILS_JWT2 to run the cross-user probe")]
|
||||
headers = {"apikey": os.environ.get("GUARDRAILS_ANON_KEY", ""),
|
||||
"Authorization": f"Bearer {jwt2}"}
|
||||
base = cfg["url"].rstrip("/")
|
||||
return [_cross_user_probe(session, base, p, headers) for p in paths]
|
||||
|
||||
|
||||
def _route_probe(session, base, route):
|
||||
target = base + route["path"]
|
||||
resp = fetch(session, "GET", target)
|
||||
return _verdict(
|
||||
("api", f"no-auth:{route['path']}", target), resp,
|
||||
resp.status == 200,
|
||||
("unauthenticated request returned 200",
|
||||
"unauthenticated request was refused"))
|
||||
|
||||
|
||||
def probe_routes(session, plan):
|
||||
base = plan["base_url"].rstrip("/")
|
||||
routes = [r for r in plan.get("routes", []) if not r.get("public")]
|
||||
return [_route_probe(session, base, r) for r in routes]
|
||||
|
||||
|
||||
def _cors_open(headers):
|
||||
allow = headers.get("Access-Control-Allow-Origin", "")
|
||||
creds = headers.get("Access-Control-Allow-Credentials", "").lower()
|
||||
return allow in (FAKE_ORIGIN, "*") and creds == "true"
|
||||
|
||||
|
||||
def probe_cors(session, plan):
|
||||
if not plan.get("cors"):
|
||||
return []
|
||||
base = plan["base_url"].rstrip("/")
|
||||
headers = {"Origin": FAKE_ORIGIN, "Access-Control-Request-Method": "GET"}
|
||||
resp = fetch(session, "OPTIONS", base + plan.get("cors_path", "/"), headers)
|
||||
return [_verdict(
|
||||
("api", "cors-reflect", base), resp, _cors_open(resp.headers),
|
||||
("foreign origin reflected with credentials",
|
||||
"foreign origin not reflected with credentials"))]
|
||||
|
||||
|
||||
def _burst_codes(session, url, method, body, count):
|
||||
data = body.encode() if body else None
|
||||
codes = []
|
||||
for _ in range(count):
|
||||
if session.budget <= 0:
|
||||
break
|
||||
codes.append(fetch(session, method, url, None, data).status)
|
||||
return codes
|
||||
|
||||
|
||||
def _rate_verdict(target, codes):
|
||||
sent = len([c for c in codes if c is not None])
|
||||
throttled = 429 in codes
|
||||
note = (f"sent {sent}, {'got a 429' if throttled else 'no 429'}: "
|
||||
+ ("throttled" if throttled else "no 429 seen on this route"))
|
||||
return Probe("hosting", "rate-limit",
|
||||
"CLOSED" if throttled else "OPEN", target, note)
|
||||
|
||||
|
||||
def probe_rate_limit(session, plan):
|
||||
cfg = plan.get("rate_limit")
|
||||
if not cfg:
|
||||
return []
|
||||
target = plan["base_url"].rstrip("/") + cfg["path"]
|
||||
codes = _burst_codes(
|
||||
session, target, cfg.get("method", "GET").upper(),
|
||||
cfg.get("body"), min(cfg.get("count", 20), BURST_CAP))
|
||||
return [_rate_verdict(target, codes)]
|
||||
|
||||
|
||||
PROBES = (probe_supabase, probe_cross_user, probe_routes,
|
||||
probe_cors, probe_rate_limit)
|
||||
|
||||
|
||||
def run_probes(session, plan):
|
||||
results = []
|
||||
for probe in PROBES:
|
||||
results += probe(session, plan)
|
||||
return results
|
||||
|
||||
|
||||
def _tally(results):
|
||||
return {v: sum(p.verdict == v for p in results)
|
||||
for v in ("OPEN", "CLOSED", "SKIPPED", "ERROR")}
|
||||
|
||||
|
||||
def _probe_row(probe):
|
||||
return (f" {probe.verdict:<8} {probe.domain:<9} {probe.id} "
|
||||
f"{probe.target}\n {probe.message}")
|
||||
|
||||
|
||||
def _summary(results):
|
||||
counts = _tally(results)
|
||||
return f"\nopen: {counts['OPEN']} " + ", ".join(
|
||||
f"{v} {n}" for v, n in counts.items())
|
||||
|
||||
|
||||
def _render_text(plan, results):
|
||||
body = [_probe_row(p) for p in results]
|
||||
head = [f"guardrails live probe: {plan['base_url']}",
|
||||
f"probes: {len(results)} (read-only, bounded budget)"]
|
||||
return "\n".join(head + [""] + body + [_summary(results)])
|
||||
|
||||
|
||||
def _render_json(plan, results):
|
||||
return json.dumps({
|
||||
"base_url": plan["base_url"],
|
||||
"open": _tally(results)["OPEN"],
|
||||
"probes": [p.as_dict() for p in results],
|
||||
}, indent=2)
|
||||
|
||||
|
||||
def _load_plan(path):
|
||||
try:
|
||||
plan = json.loads(open(path, encoding="utf-8").read())
|
||||
except (OSError, ValueError) as exc:
|
||||
return None, f"cannot read plan: {exc}"
|
||||
if not isinstance(plan, dict) or not plan.get("base_url"):
|
||||
return None, "plan needs an object with a base_url"
|
||||
return plan, ""
|
||||
|
||||
|
||||
def _plan_error(plan):
|
||||
method = plan.get("rate_limit", {}).get("method", "GET").upper()
|
||||
if method not in READ_METHODS and method != "POST":
|
||||
return f"rate_limit method {method} is not a probe method"
|
||||
return ""
|
||||
|
||||
|
||||
def parse_args(argv):
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("plan", help="path to the probe plan JSON")
|
||||
parser.add_argument(
|
||||
"--authorized", action="store_true",
|
||||
help="confirm you own the target and may test it (required)")
|
||||
parser.add_argument("--json", action="store_true", help="JSON output")
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def main(argv):
|
||||
args = parse_args(argv)
|
||||
plan, error = _load_plan(args.plan)
|
||||
error = error or _plan_error(plan or {})
|
||||
if error:
|
||||
print(f"usage error: {error}", file=sys.stderr)
|
||||
return 2
|
||||
if not args.authorized:
|
||||
print("refused: pass --authorized to confirm you own "
|
||||
f"{plan['base_url']} and may probe it", file=sys.stderr)
|
||||
return 2
|
||||
results = run_probes(Session(), plan)
|
||||
render = _render_json if args.json else _render_text
|
||||
print(render(plan, results))
|
||||
return 1 if any(p.verdict == "OPEN" for p in results) else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -2,5 +2,6 @@
|
||||
{"id": 1, "prompt": "Vérifie que la RLS est bien activée partout sur mon projet Supabase dans ~/Documents/game et qu'aucune clé service_role n'est côté client.", "expected": "Runs the scanner on the path, reads each supabase FAIL and WARN in context, gives the live RLS queries as a manual step, writes .claude/audits/GUARDRAILS.md, does not fix anything"},
|
||||
{"id": 2, "prompt": "teste les garde-fous sur ce projet, il envoie des emails et des SMS de relance", "expected": "Scans messaging and api, checks consent record, unsubscribe link, List-Unsubscribe and STOP handling, traces the opt-out path from the provider webhook, lists the dashboard items as MANUAL"},
|
||||
{"id": 3, "prompt": "Audit my Firebase functions: can any of them loop on itself or run up the bill?", "expected": "Scans functions domain only, reports max instances status and any trigger that writes its own collection, MANUAL item for a budget alert wired to an automatic stop, no live call to the provider"},
|
||||
{"id": 4, "prompt": "est-ce que les mots de passe sont stockés en clair quelque part dans ~/Documents/zenquality ?", "expected": "Runs the scanner's api domain, reads every API-PASSWORD-* hit in context, traces the sign-up and password-change write points to the password_hash column by hand, names the hashing function found, reports with file:line, fixes nothing"}
|
||||
{"id": 4, "prompt": "est-ce que les mots de passe sont stockés en clair quelque part dans ~/Documents/zenquality ?", "expected": "Runs the scanner's api domain, reads every API-PASSWORD-* hit in context, traces the sign-up and password-change write points to the password_hash column by hand, names the hashing function found, reports with file:line, fixes nothing"},
|
||||
{"id": 5, "prompt": "teste les garde-fous sur ~/Documents/game puis sonde-le en direct, le dev tourne sur http://localhost:5173", "expected": "Runs the static scan first, then builds a probe plan from the findings, exports test keys from the environment, runs probe.py --authorized against the localhost target only, reports OPEN verdicts next to the matching static FAIL, never writes data, never probes a target the user did not name"}
|
||||
]
|
||||
Reference in new issue
Block a user