feat(guardrails): read-only live probe to confirm a hole from outside

probe.py observes a running target the user owns and names, driven by a
plan built from the static findings. Five probes: anon-read, cross-user,
no-auth route, CORS reflection, capped rate-limit burst. GET/HEAD/OPTIONS
only, --authorized gate, secrets from the environment, same-host redirect
only, bounded request budget. SKILL STEP 5, checks.md reference, docs.
Suite guardrails-probe.test.sh (16 checks) against a localhost fixture.
This commit is contained in:
bchanot committed 2026-10-18 18:49:51 +02:00
1 parent dbf409f6e6
commit f0bcbfef46
8 files changed
+572 -16

No files matched your search

+1 -1
View File
@@ -7,7 +7,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/) and this project
## [Unreleased]
### Added
- **Secure-by-default guardrails**: `rules/secure-by-default.md` (always on) makes five domains baseline on every project, plus systematic API and access hygiene (auth on every route, CORS, secrets, passwords hashed with argon2id or bcrypt, never compared in clear, logged or hardcoded): hosting (rate limit, CDN and firewall, hard spend cap), accessibility (keyboard, visible focus, alt text, field labels), Supabase (RLS on every table, owner-scoped policies, no service role key client-side), SMS and email (recorded consent, unsubscribe, STOP), cloud functions (no self-trigger, instance cap, budget stop). `/guardrails` audits them: a stdlib Python scanner (`skills/guardrails/scripts/scan.py`, FAIL / WARN / MANUAL, exit 1 on FAIL) then judgment, report in `.claude/audits/GUARDRAILS.md`. The security-auditor gate runs the scanner's `api` and `supabase` domains and blocks on FAIL. Items only a provider dashboard can set stay MANUAL, never claimed done. Suite: `lib/tests/guardrails-scan.test.sh` (clean fixture plus one mutant per rule).
- **Secure-by-default guardrails**: `rules/secure-by-default.md` (always on) makes five domains baseline on every project, plus systematic API and access hygiene (auth on every route, CORS, secrets, passwords hashed with argon2id or bcrypt, never compared in clear, logged or hardcoded): hosting (rate limit, CDN and firewall, hard spend cap), accessibility (keyboard, visible focus, alt text, field labels), Supabase (RLS on every table, owner-scoped policies, no service role key client-side), SMS and email (recorded consent, unsubscribe, STOP), cloud functions (no self-trigger, instance cap, budget stop). `/guardrails` audits them: a stdlib Python scanner (`skills/guardrails/scripts/scan.py`, FAIL / WARN / MANUAL, exit 1 on FAIL) then judgment, report in `.claude/audits/GUARDRAILS.md`. The security-auditor gate runs the scanner's `api` and `supabase` domains and blocks on FAIL. Items only a provider dashboard can set stay MANUAL, never claimed done. An optional read-only live probe (`skills/guardrails/scripts/probe.py`, `--live`) confirms a finding from outside on a target the user owns and names: anon-read, cross-user, no-auth route, CORS reflection and a capped rate-limit burst, verdicts OPEN / CLOSED / SKIPPED / ERROR. It sends GET, HEAD and OPTIONS only (one opt-in POST burst, capped at 25), requires `--authorized`, takes secrets from the environment never the command line, never follows a redirect to another host, and stops at a request budget. Suites: `lib/tests/guardrails-scan.test.sh` (clean fixture plus one mutant per rule) and `lib/tests/guardrails-probe.test.sh` (probe logic against a localhost fixture server).
- **model-router mod**: `mods/model-router/`, a Claude Code mod (function-hooks plugin), routes every repo skill and agent from phase rows (`plan`, `reflect`, `orchestrate`, `escalate`, `judge`, `implement`, `write`, `verify`, `explore`, `apply`, `mechanical`; built-ins Explore on sonnet/medium, Plan on opus/xhigh). A typed skill routes the main loop to its row, and a best-tier row holds across turns in a run slot until `/route clear`, `/route off`, a user `/model` or a typed skill on a non-best row. Agents get their row's model at spawn (within the tier, upward only) and its effort on every step; explicit Agent params win. Orchestrators declare their phases through the `mcp__model-router__route` tool. `ultrathink` in a prompt sets the turn's minimum effort and `/route effort=max` holds until `/route clear`; the built-in `/effort` is not a lever inside a run. `/route show` names the run slot (`main: run <phase>`), and a `null` row in the override drops a default row. The model gets a `route` tool and the user a `/route` command (`show|clear|off|on|reload|pending|ask on|off|forget <name|all|projects>|<phase>|model=<alias|id> effort=<level>|switch on|off|verbose on|off`). Optional per-machine config `~/.claude/model-router.json`, where `"enabled": false` turns it off on that machine. The spinner suffix and the status line show the route in force. It loads in every session through the tracked symlink `skills/model-router` (`model-router@skills-dir`). `make doctor` gains a Mods section; suite `make test suite=lib/tests/mods.test.sh`. Known limits: a main-loop switch runs only while the cold read it costs (the context minus what the target model's prompt cache holds) fits `upgradeMaxTokens` (200000) or `downgradeMaxTokens` (120000), and a downgrade also needs `mainModelSwitch` (on by default, off on any non-boolean value); the hooks send full model ids, so the `models` table has to follow new versions.
- **model-router first-use confirmation**: `mods/model-router/routing.json` (tracked) is the single source of the phase table and of every skill and agent row, and keeps the decisions: `confirmed`, `changed` (`from`/`to`), `projects` exceptions keyed by the origin remote reduced to `host/path` (credentials and local paths never stored), and `ask`. The first use of a rowed typed skill, a rowed agent spawn or a main-loop phase declared through the `route` tool opens a dialog with context: the skill's description (first sentence of its `SKILL.md` frontmatter) or the agent's, the phase with its `about` line (a new field on each of the 11 phases) and the model id and effort the next step really runs on. A row offers Later, Keep or Change; a main-loop phase Later or Keep. Change asks the model (fable, opus, sonnet, haiku with their tier), then the effort among those the phases of that model use (skipped when there is only one), then Everywhere or This project only; the pair maps to an existing phase (rows stay phase names, the same phase counts as Keep, a pair no phase offers is added by hand as a new phase). A skill-row change toasts the model and effort it now runs on, with the `/route switch on` hint when the main loop holds back a downgrade; a free-text answer counts as Later. One dialog at a time, never in headless (`-p`) runs or inside a sub-agent. Only a dialog answer, `/route ask on|off` or `/route forget` (which takes decisions back; a changed row returns to its shipped phase, the frontmatter floors stay yours to realign) writes the file (serialized, 64 KiB cap, never created when absent); each write asks you to commit it from the config repo. `/route pending` lists unconfirmed rows and phases. Layers: routing.json < `~/.claude/model-router.json` (its `ask` wins); a project's `.claude/model-router.json` is never read. Tests: `mods/model-router/hooks/register.test.ts`.
- **Manual-push mode**: `git config gitflow.autopush false` (human-set) now stops every push the gitflow lib makes, not only the post-commit / post-merge hooks. `gitflow start` and `finish` branch, commit and merge locally and push nothing; `gitflow delete` leaves the `origin/` copy in place and prints `git push origin --delete <br>` for the user to run. `hooks/unpushed-guard.sh` stays silent at turn end in this mode and opens each session with one `ℹ manual push mode:` line counting the commits no remote holds across every local branch; an unparseable or unreadable `gitflow.autopush` value is treated as manual push mode too, and that line names it. `hooks/push-guard.sh` (PreToolUse, `Bash|Monitor`) refuses any `git push` Claude types while `gitflow.autopush` reads false in the session cwd or in a literal `-C`/`cd` directory the command names (global config counts outside a repo); the refusal tells the user to run it with `! git push`. It reads the mode through the same lib verb as every other reader and fails closed: an unparseable or unreadable value reads as manual, and an internal error, a missing `lib/gitflow.sh`, more than 20 distinct directory tokens in one command (capped before any token is classified), a `cd`/`-C` directory token mixing quoted and unquoted parts, or a payload jq cannot parse whose raw text looks like a push refuse the push (these pathological cases fire in auto mode too). Directory tokens are read as whole shell words, adjacent quoted segments and backslash escapes included. Once the text looks like a push, the command is split into simple commands (quotes, escapes, separators, heredoc bodies dropped) and a push must be git's own subcommand in one of them: `grep "git push"`, `git log -S "git push"`, `git config --get push.default`, `git stash push`, a heredoc quoting a push or a commit message naming one are allowed; `bash -c` scripts are re-checked, wrappers (`sudo`, `env`, `eval`, `xargs`, `timeout`, `git subtree`...) and assignments keep the text verdict. The misses listed in its header fall to a new `autoMode.soft_deny` rule that no request in the turn clears. The session banner adds `🔒 push : manual (autopush=false) — ! git push` when the key reads false, and `🔒 push : manual (autopush bad) — ! git push` when the value is invalid. Skills read the mode through a new lib verb, `bash ~/.claude/lib/gitflow.sh push-mode`: it prints `auto`, `manual` or `invalid` (rc 0) and names an invalid value on stderr (printable characters only, 64 at most). It is the one reader a skill may call, since the `git config` read of the key is denied to Claude. Skills push nothing on their own, except the `/release-candidate` tag in auto-push mode on an explicit go. Every "on origin" or "not pushed" line they print comes from `git rev-list --count origin/<br>..<br>` read after the fact, with the complete `! git …` command when something is left for the user to push. An invalid value (anything but unset, true or false, or a read that fails) is manual push mode for every reader and is named where it is read (see Fixed). Tests: `lib/gitflow-test.sh` T11b (push-mode verb), T18m and T18q blocks, `lib/tests/unpushed-guard.test.sh` T10-T16, `lib/tests/push-guard.test.sh` (98 checks).
+1 -1
View File
@@ -228,7 +228,7 @@ a different package, ships its own conflicting `graphify` bin) — see
| `/reconcile` | Confront declared status (TODO, registries) against real git/fs state — surface stale items |
| `/pdf-translate` | Translate a PDF to another language, output as HTML (via Vision) |
| `/explain-diff` | Explain a diff, commit, branch or PR as one self-contained HTML page (background, intuition, code walkthrough, 5-question quiz), saved outside the repo |
| `/guardrails` | Audit the five baseline guardrails (hosting, a11y, Supabase RLS, SMS/email consent, cloud functions) plus API hygiene (auth, CORS, secrets, password storage): deterministic scanner, then judgment, report in `.claude/audits/GUARDRAILS.md` |
| `/guardrails` | Audit the five baseline guardrails (hosting, a11y, Supabase RLS, SMS/email consent, cloud functions) plus API hygiene (auth, CORS, secrets, password storage): deterministic scanner, then judgment, report in `.claude/audits/GUARDRAILS.md`. `--live <url you own>` adds a read-only probe that confirms a hole from outside |
| `/close` | End-of-session ritual — alias for `/capitalize --ritual` (dedup + TODO reconcile + 3-question reflection) |
| `/harden` | Web hardening audit — HTTPS/TLS, HSTS, CSP, security headers |
| `/web-validate` | W3C HTML/CSS validity + WCAG 2.1 accessibility audit |
+2 -1
View File
@@ -121,6 +121,7 @@ Tu veux...
| Traduire un PDF | `/pdf-translate` |
| Comprendre un changement pas suivi | `/explain-diff` |
| Auditer garde-fous (RLS, a11y, consentement, fonctions, mots de passe) | `/guardrails` |
| Sonder le projet en direct (lecture seule, cible à toi) | `/guardrails --live <url>` |
| Changer profil skills | `/profile` |
| Audit/polish design (anti-slop) | `/impeccable` |
| Sweep groupé tous axes (nettoyage + sécu + reconcile + doc) | `/tour` |
@@ -164,7 +165,7 @@ Tu veux...
| `/client-handover` | Livraison client | Audits finaux + livrable brandé |
| `/pdf-translate` | Traduire un PDF vers une autre langue | Sortie HTML fidèle (images, layout, style préservés) |
| `/explain-diff` | Comprendre un commit, une branche ou une PR qu'on n'a pas suivis | Page HTML autonome hors du repo : contexte, intuition, code, quiz de 5 questions |
| `/guardrails` | Vérifier hébergement, a11y, Supabase RLS, consentement SMS/email, fonctions cloud, stockage des mots de passe | Scanner déterministe puis jugement ; rapport `.claude/audits/GUARDRAILS.md` |
| `/guardrails` | Vérifier hébergement, a11y, Supabase RLS, consentement SMS/email, fonctions cloud, stockage des mots de passe | Scanner déterministe puis jugement ; rapport `.claude/audits/GUARDRAILS.md` ; `--live <url à toi>` sonde en lecture seule |
| `/impeccable` | Audit/polish design + détecteur anti-slop déterministe | 23 verbes ; `npx impeccable detect` (exit 0/2) |
| `/tour` | Sweep groupé sur un ou plusieurs projets | Sécu + nettoyage + reconcile + doc, boucle jusqu'à un pass propre |
| `/route` | Voir ou fixer la route du mod model-router | show / clear / off / on / reload / pending / ask on\|off / <phase> / model=… effort=… / switch on\|off / classify on\|off (classifieur de prompt) / verbose on\|off (transitions seules) |
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env bash
# lib/tests/guardrails-probe.test.sh — floor of the /guardrails live probe.
# A throwaway http.server on 127.0.0.1 serves scripted responses; the probe
# runs against it and each verdict is asserted. Nothing leaves the machine.
# The probe sends read-only methods; this suite proves the OPEN/CLOSED logic,
# the --authorized gate, the same-host redirect guard and the request budget.
set -u
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
PROBE="$ROOT/skills/guardrails/scripts/probe.py"
WORK="$(mktemp -d)"
PASS=0; FAIL=0; SRV=""
cleanup() { [ -n "$SRV" ] && kill "$SRV" 2>/dev/null; rm -rf "$WORK"; }
trap cleanup EXIT
check() { # check <label> <got> <want>
if [ "$2" = "$3" ]; then PASS=$((PASS+1)); echo " PASS $1"
else FAIL=$((FAIL+1)); echo " FAIL $1 (got '$2', want '$3')"; fi
}
# Fixture server: a fixed map of path -> (status, body, headers). Read-only,
# bound to the loopback, on an OS-chosen port it prints on the first line.
cat > "$WORK/server.py" <<'PY'
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
HITS = {"rl": 0}
class H(BaseHTTPRequestHandler):
def log_message(self, *a):
pass
def _send(self, code, body=b"", headers=None):
self.send_response(code)
for k, v in (headers or {}).items():
self.send_header(k, v)
self.end_headers()
self.wfile.write(body)
def do_OPTIONS(self):
origin = self.headers.get("Origin", "")
self._send(204, b"", {
"Access-Control-Allow-Origin": origin,
"Access-Control-Allow-Credentials": "true"})
def do_GET(self):
path = self.path.split("?")[0]
if path == "/rest/v1/open_table":
return self._send(200, b'[{"id": 1}]')
if path == "/rest/v1/closed_table":
return self._send(200, b'[]')
if path == "/api/private":
return self._send(200, b"secret")
if path == "/api/guarded":
return self._send(401, b"no")
if path == "/rl":
HITS["rl"] += 1
return self._send(429 if HITS["rl"] > 5 else 200, b"")
if path == "/nolimit":
return self._send(200, b"")
return self._send(404, b"")
srv = HTTPServer(("127.0.0.1", 0), H)
print(srv.server_port, flush=True)
srv.serve_forever()
PY
python3 -B "$WORK/server.py" > "$WORK/port" &
SRV=$!
for _ in $(seq 1 50); do [ -s "$WORK/port" ] && break; sleep 0.1; done
PORT="$(head -1 "$WORK/port")"
if [ -z "$PORT" ]; then echo " FAIL fixture server did not start"; exit 1; fi
BASE="http://127.0.0.1:$PORT"
probe() { python3 -B "$PROBE" "$@"; }
verdict() { printf '%s\n' "$1" | grep -A1 "$2" | grep -oE 'OPEN|CLOSED|SKIPPED|ERROR' | head -1; }
echo "== authorization gate"
cat > "$WORK/plan.json" <<EOF
{"base_url": "$BASE",
"supabase": {"url": "$BASE", "tables": ["open_table", "closed_table"],
"cross_user_paths": ["/rest/v1/open_table"]},
"routes": [{"path": "/api/private"}, {"path": "/api/guarded"},
{"path": "/api/public", "public": true}],
"cors": true}
EOF
probe "$WORK/plan.json" >/dev/null 2>&1
check "refuses without --authorized (exit 2)" "$?" 2
echo "== read-only probes against the fixture"
export GUARDRAILS_ANON_KEY="anon-test-key"
export GUARDRAILS_JWT2="second-user-test-token"
OUT="$(probe --authorized "$WORK/plan.json")"
RC=$?
check "anon read of an open table is OPEN" \
"$(verdict "$OUT" 'anon-read:open_table')" OPEN
check "anon read of a closed table is CLOSED" \
"$(verdict "$OUT" 'anon-read:closed_table')" CLOSED
check "cross-user read returning rows is OPEN" \
"$(verdict "$OUT" 'cross-user:')" OPEN
check "unauth route returning 200 is OPEN" \
"$(verdict "$OUT" 'no-auth:/api/private')" OPEN
check "unauth route returning 401 is CLOSED" \
"$(verdict "$OUT" 'no-auth:/api/guarded')" CLOSED
check "public route is not probed" \
"$(printf '%s\n' "$OUT" | grep -c 'no-auth:/api/public')" 0
check "reflected origin with credentials is OPEN" \
"$(verdict "$OUT" 'cors-reflect')" OPEN
check "a found OPEN verdict exits 1" "$RC" 1
echo "== skips when a secret is absent"
unset GUARDRAILS_ANON_KEY
OUT="$(probe --authorized "$WORK/plan.json")"
check "anon-read skips without the key" \
"$(verdict "$OUT" 'anon-read')" SKIPPED
export GUARDRAILS_ANON_KEY="anon-test-key"
echo "== rate limit"
cat > "$WORK/rl.json" <<EOF
{"base_url": "$BASE", "rate_limit": {"path": "/rl", "count": 10}}
EOF
OUT="$(probe --authorized "$WORK/rl.json")"
check "a route that starts sending 429 is CLOSED" \
"$(verdict "$OUT" 'rate-limit')" CLOSED
cat > "$WORK/nolimit.json" <<EOF
{"base_url": "$BASE", "rate_limit": {"path": "/nolimit", "count": 10}}
EOF
OUT="$(probe --authorized "$WORK/nolimit.json")"
check "a route with no 429 is OPEN" \
"$(verdict "$OUT" 'rate-limit')" OPEN
echo "== budget caps the burst"
cat > "$WORK/burst.json" <<EOF
{"base_url": "$BASE", "rate_limit": {"path": "/nolimit", "count": 99}}
EOF
OUT="$(probe --authorized "$WORK/burst.json")"
check "the burst is capped, never 99 requests" \
"$(printf '%s\n' "$OUT" | grep -oE 'sent [0-9]+' | grep -oE '[0-9]+')" 25
echo "== usage errors"
probe --authorized "$WORK/missing.json" >/dev/null 2>&1
check "a missing plan exits 2" "$?" 2
printf '{"base_url": "%s", "rate_limit": {"path": "/x", "method": "DELETE"}}\n' \
"$BASE" > "$WORK/bad.json"
probe --authorized "$WORK/bad.json" >/dev/null 2>&1
check "a write method in the plan exits 2" "$?" 2
echo "== JSON output"
check "json reports the open count" \
"$(probe --authorized --json "$WORK/plan.json" \
| python3 -B -c 'import json,sys; print(json.load(sys.stdin)["open"])')" 4
echo "---"
echo "passed: $PASS failed: $FAIL"
[ "$FAIL" -eq 0 ]
+50 -12
View File
@@ -1,18 +1,20 @@
---
name: guardrails
effort: high
description: 'Use when auditing a project against the baseline guardrails: hosting (rate limit, CDN and firewall, spend cap), accessibility (keyboard, visible focus, alt text, field labels), Supabase (RLS on every table, owner-scoped policies, no service_role key client-side), SMS and email (recorded consent, unsubscribe, STOP), cloud functions (no self-trigger, instance cap, budget stop), plus API and access hygiene (auth on every route, CORS, secrets, password storage). Runs a deterministic scanner, judges each hit, writes .claude/audits/GUARDRAILS.md. Triggers: "guardrails", "garde-fous", "RLS activée ?", "audit supabase", "consentement sms", "plafond de dépense", "mots de passe en clair ?", "teste la sécurité de base du projet". Headers and TLS → /harden; secrets, CVE, OWASP → /cso; full WCAG → /web-validate.'
argument-hint: '[project path] [--domain api,hosting,a11y,supabase,messaging,functions] [--exclude paths]'
description: 'Use when auditing a project against the baseline guardrails: hosting (rate limit, CDN and firewall, spend cap), accessibility (keyboard, visible focus, alt text, field labels), Supabase (RLS on every table, owner-scoped policies, no service_role key client-side), SMS and email (recorded consent, unsubscribe, STOP), cloud functions (no self-trigger, instance cap, budget stop), plus API and access hygiene (auth on every route, CORS, secrets, password storage). Runs a deterministic scanner, judges each hit, writes .claude/audits/GUARDRAILS.md. Optional read-only live probe confirms a hole from outside on a target the user owns and names. Triggers: "guardrails", "garde-fous", "RLS activée ?", "audit supabase", "consentement sms", "plafond de dépense", "mots de passe en clair ?", "teste la sécurité de base du projet", "teste-le en conditions réelles", "sonde le projet en direct". Headers and TLS → /harden; secrets, CVE, OWASP → /cso; full WCAG → /web-validate.'
argument-hint: '[project path] [--domain api,hosting,a11y,supabase,messaging,functions] [--exclude paths] [--live <url you own>]'
---
# /guardrails
Checks a project against `rules/secure-by-default.md`. A scanner reads the
tree as data and finds what text can prove. This skill then judges the hits,
covers what text cannot prove, and lists what only a dashboard can set.
covers what text cannot prove, lists what only a dashboard can set, and, on
request, probes a running target the user owns to confirm a hole from outside.
The scanner proves absence of a protection, never presence of security. A
clean scan means "no static finding", and the report says exactly that.
clean scan means "no static finding", and the report says exactly that. A
live probe proves the opposite direction: that a door is actually open.
## STEP 1: Scope
@@ -62,21 +64,57 @@ Read the code for the domains that apply and answer, with file:line evidence:
- **A11y**: with a dev server running, walk the main flow with Tab only
(`/browse`). Otherwise mark the keyboard walk-through as not done.
## STEP 5: Manual checklist
## STEP 5: Live probe (only on `--live`, read-only)
The static scan shows a protection is missing in the code. A probe shows the
door is actually open. Run it only when the user asks, and only against a
target the user owns and names: their local dev server first, a staging or
production URL only on an explicit, specific go. Never a URL you inferred,
never a third party.
1. **Build the plan** `references/checks.md` describes. From the static
findings fill a JSON plan: `base_url`, the Supabase `url` and `tables`,
the `routes` (mark the ones meant to be public), `cors`, and a
`rate_limit` route if the user wants that one burst. Secrets never go in
the plan or the command line: export `GUARDRAILS_ANON_KEY` and, for the
cross-user probe, `GUARDRAILS_JWT2`, from test keys, never `service_role`.
2. **Run it**, confirming ownership with `--authorized`:
```
python3 -B "$HOME/.claude/skills/guardrails/scripts/probe.py" \
plan.json --authorized
```
It sends GET, HEAD and OPTIONS only (the rate-limit burst is the one
POST, opt-in in the plan, capped at 25 requests). It never writes data,
never follows a redirect to another host, and stops at a request budget.
Exit 1 means at least one OPEN verdict.
3. **Cross the verdicts with the static findings.** A verdict is an
observation, not a conclusion: OPEN means the request returned data or a
200, CLOSED means it was refused or empty. A static FAIL confirmed OPEN in
the probe leads the report. A CLOSED probe on a static FAIL still needs the
fix: the dev target may differ from production.
4. **Guardrails on the probe itself**: read-only, test keys only, the
loopback or a URL the user named. A login burst can lock the user's own
account, so run it only with a throwaway test account and the user's go.
Never fuzz, never POST to anything but a named login route, never probe a
target you were not told to. That is pentest territory, another contract.
## STEP 6: Manual checklist
Print each MANUAL item as a step the user performs: spend cap, WAF and DDoS
rules, budget alert wired to a stop, live RLS queries. Provider limits change:
name the setting, tell the user to confirm it in the current provider docs.
Never run a live probe against production, and never claim a MANUAL item done.
Never run a live probe against production without the user's explicit go, and
never claim a MANUAL item done.
## STEP 6: Report
## STEP 7: Report
Write `.claude/audits/GUARDRAILS.md`: one table per domain with status PASS,
FAIL, WARN, MANUAL or N/A, the evidence, and the fix in one line. PASS means
"no static finding". Keep it short; no filler sections.
"no static finding". When a live probe ran, add a line for its verdict next
to the matching static finding. Keep it short; no filler sections.
## STEP 7: Hand over
## STEP 8: Hand over
Reply with counts, the FAILs with file:line and a one-line fix each, and the
MANUAL items. This skill audits; it does not fix. A fix goes through
`/bugfix`, `/feat` or `/hotfix`.
Reply with counts, the FAILs with file:line and a one-line fix each, any OPEN
probe verdicts, and the MANUAL items. This skill audits; it does not fix. A
fix goes through `/bugfix`, `/feat` or `/hotfix`.
+51
View File
@@ -153,3 +153,54 @@ user's token, a GET on the first user's rows must return nothing.
a retention rule for consent proof.
- **Supabase**: the live queries above; spend cap on the plan; redirect URLs
and email confirmation settings reviewed.
## Live probe (`--live`, read-only)
`probe.py` observes a running target the user owns and names. It is driven by
a plan JSON built from the static findings; secrets come from the environment,
never the plan or the command line. It sends GET, HEAD and OPTIONS only, with
one opt-in POST burst for the rate-limit check, and stops at a request budget.
```json
{
"base_url": "http://localhost:5173",
"supabase": {
"url": "http://localhost:54321",
"tables": ["notes", "contacts"],
"cross_user_paths": ["/rest/v1/notes?select=*"]
},
"routes": [
{"path": "/api/v1/notes"},
{"path": "/api/v1/health", "public": true}
],
"cors": true,
"cors_path": "/api/v1/notes",
"rate_limit": {"path": "/api/v1/auth/login", "method": "POST",
"count": 20, "body": "{\"email\":\"t@t.io\",\"password\":\"x\"}"}
}
```
Environment, from test keys only, never `service_role`:
`GUARDRAILS_ANON_KEY` (anon-read), `GUARDRAILS_JWT2` (cross-user).
| Probe | id | OPEN means | Read it against |
|---|---|---|---|
| anon-read | `anon-read:<table>` | the anon key read a row | SB-RLS-MISSING, SB-POLICY-* |
| cross-user | `cross-user:<path>` | a second user's token read these rows | policy owner column (STEP 4) |
| no-auth route | `no-auth:<path>` | the route answered 200 with no token | API-ROUTE-NO-AUTH, object authz |
| cors-reflect | `cors-reflect` | a foreign origin was reflected with credentials | API-CORS-WILDCARD |
| rate-limit | `rate-limit` | the burst saw no 429 | HOST-NO-RATE-LIMIT |
A verdict is an observation, not a verdict on the project. OPEN is a returned
200 or returned rows; CLOSED is a refusal or an empty body; SKIPPED is a
missing key; ERROR is a transport failure. A static FAIL confirmed OPEN leads
the report. A CLOSED probe on a static FAIL still needs the fix: the dev
target may differ from production.
Safety, enforced by the script and by the skill: `--authorized` is required;
read-only methods only; the loopback or a URL the user named, never one you
inferred; a login burst only with a throwaway test account on the user's go,
because it can lock the user's own account; a redirect to another host is
never followed, so a key never leaves the named host. Fuzzing, writes beyond
the named login route, or probing a third party are out of scope: that is
pentest, another contract.
+309
View File
@@ -0,0 +1,309 @@
#!/usr/bin/env python3
"""Live guardrails probe: observe a running target from outside, read-only.
The static scan shows a protection is absent in the code. This observes what
a running target the user owns and names actually returns. It sends GET, HEAD
and OPTIONS only; a login burst is opt-in in the plan. It never writes data,
never uses a service_role key, never follows a redirect to another host, and
stops at a fixed request budget.
Input is a plan JSON (built by the skill from the static findings) plus
secrets from the environment, never the command line:
GUARDRAILS_ANON_KEY Supabase anon/publishable key for the anon-read probe
GUARDRAILS_JWT2 a second test user's token for the cross-user probe
A verdict is a factual observation, not a conclusion: the skill reads it
against the static findings. OPEN means the request returned data or a 200,
CLOSED means it was refused or empty. Exit codes: 0 no OPEN, 1 at least one
OPEN, 2 usage error.
"""
import argparse
import json
import os
import sys
import urllib.error
import urllib.request
from dataclasses import asdict, dataclass
from urllib.parse import quote, urlsplit
READ_METHODS = {"GET", "HEAD", "OPTIONS"}
FAKE_ORIGIN = "https://guardrails-probe.invalid"
DEFAULT_BUDGET = 200
BURST_CAP = 25
BODY_CAP = 1_000_000
@dataclass(frozen=True)
class Response:
status: int # None on a transport error
headers: dict
body: bytes
error: str = ""
@dataclass(frozen=True)
class Probe:
domain: str
id: str
verdict: str # OPEN | CLOSED | SKIPPED | ERROR
target: str
message: str
def as_dict(self):
return asdict(self)
class _SameHost(urllib.request.HTTPRedirectHandler):
"""Follow a same-host redirect, drop one that leaves the host."""
def redirect_request(self, req, fp, code, msg, headers, newurl):
if urlsplit(newurl).netloc != urlsplit(req.full_url).netloc:
return None
return super().redirect_request(req, fp, code, msg, headers, newurl)
class Session:
"""Holds the redirect-guarded opener and the shared request budget."""
def __init__(self, timeout=8, budget=DEFAULT_BUDGET):
self.opener = urllib.request.build_opener(_SameHost())
self.timeout = timeout
self.budget = budget
def _response_from(resp):
return Response(resp.status, dict(resp.headers), resp.read(BODY_CAP))
def fetch(session, method, url, headers=None, body=None):
if session.budget <= 0:
return Response(None, {}, b"", "request budget exhausted")
session.budget -= 1
req = urllib.request.Request(
url, data=body, method=method, headers=headers or {})
try:
with session.opener.open(req, timeout=session.timeout) as resp:
return _response_from(resp)
except urllib.error.HTTPError as exc:
return Response(exc.code, dict(exc.headers), b"")
except (urllib.error.URLError, OSError, ValueError) as exc:
return Response(None, {}, b"", str(exc))
def _verdict(ctx, response, is_open, msgs):
"""ctx is (domain, id, target); msgs is (open_note, closed_note)."""
domain, pid, target = ctx
if response.error:
return Probe(domain, pid, "ERROR", target, response.error)
verdict = "OPEN" if is_open else "CLOSED"
note = f"HTTP {response.status}: {msgs[0] if is_open else msgs[1]}"
return Probe(domain, pid, verdict, target, note)
def _nonempty_rows(body):
try:
data = json.loads(body or b"[]")
except ValueError:
return False
return isinstance(data, list) and len(data) > 0
def _table_probe(session, base, table, headers):
target = f"{base}/rest/v1/{quote(table)}?select=*&limit=1"
resp = fetch(session, "GET", target, headers)
return _verdict(
("supabase", f"anon-read:{table}", target), resp,
resp.status == 200 and _nonempty_rows(resp.body),
("anon request returned rows", "anon request returned no row"))
def probe_supabase(session, plan):
cfg = plan.get("supabase")
anon = os.environ.get("GUARDRAILS_ANON_KEY")
if not cfg or not cfg.get("tables"):
return []
if not anon:
return [Probe("supabase", "anon-read", "SKIPPED", cfg.get("url", ""),
"set GUARDRAILS_ANON_KEY to run the anon-read probe")]
base = cfg["url"].rstrip("/")
headers = {"apikey": anon, "Authorization": f"Bearer {anon}"}
return [_table_probe(session, base, t, headers) for t in cfg["tables"]]
def _cross_user_probe(session, base, path, headers):
target = base + path
resp = fetch(session, "GET", target, headers)
return _verdict(
("supabase", f"cross-user:{path}", target), resp,
resp.status == 200 and _nonempty_rows(resp.body),
("second user's token returned these rows",
"second user's token returned nothing"))
def probe_cross_user(session, plan):
cfg = plan.get("supabase") or {}
paths = cfg.get("cross_user_paths") or []
jwt2 = os.environ.get("GUARDRAILS_JWT2")
if not paths:
return []
if not jwt2:
return [Probe("supabase", "cross-user", "SKIPPED", cfg.get("url", ""),
"set GUARDRAILS_JWT2 to run the cross-user probe")]
headers = {"apikey": os.environ.get("GUARDRAILS_ANON_KEY", ""),
"Authorization": f"Bearer {jwt2}"}
base = cfg["url"].rstrip("/")
return [_cross_user_probe(session, base, p, headers) for p in paths]
def _route_probe(session, base, route):
target = base + route["path"]
resp = fetch(session, "GET", target)
return _verdict(
("api", f"no-auth:{route['path']}", target), resp,
resp.status == 200,
("unauthenticated request returned 200",
"unauthenticated request was refused"))
def probe_routes(session, plan):
base = plan["base_url"].rstrip("/")
routes = [r for r in plan.get("routes", []) if not r.get("public")]
return [_route_probe(session, base, r) for r in routes]
def _cors_open(headers):
allow = headers.get("Access-Control-Allow-Origin", "")
creds = headers.get("Access-Control-Allow-Credentials", "").lower()
return allow in (FAKE_ORIGIN, "*") and creds == "true"
def probe_cors(session, plan):
if not plan.get("cors"):
return []
base = plan["base_url"].rstrip("/")
headers = {"Origin": FAKE_ORIGIN, "Access-Control-Request-Method": "GET"}
resp = fetch(session, "OPTIONS", base + plan.get("cors_path", "/"), headers)
return [_verdict(
("api", "cors-reflect", base), resp, _cors_open(resp.headers),
("foreign origin reflected with credentials",
"foreign origin not reflected with credentials"))]
def _burst_codes(session, url, method, body, count):
data = body.encode() if body else None
codes = []
for _ in range(count):
if session.budget <= 0:
break
codes.append(fetch(session, method, url, None, data).status)
return codes
def _rate_verdict(target, codes):
sent = len([c for c in codes if c is not None])
throttled = 429 in codes
note = (f"sent {sent}, {'got a 429' if throttled else 'no 429'}: "
+ ("throttled" if throttled else "no 429 seen on this route"))
return Probe("hosting", "rate-limit",
"CLOSED" if throttled else "OPEN", target, note)
def probe_rate_limit(session, plan):
cfg = plan.get("rate_limit")
if not cfg:
return []
target = plan["base_url"].rstrip("/") + cfg["path"]
codes = _burst_codes(
session, target, cfg.get("method", "GET").upper(),
cfg.get("body"), min(cfg.get("count", 20), BURST_CAP))
return [_rate_verdict(target, codes)]
PROBES = (probe_supabase, probe_cross_user, probe_routes,
probe_cors, probe_rate_limit)
def run_probes(session, plan):
results = []
for probe in PROBES:
results += probe(session, plan)
return results
def _tally(results):
return {v: sum(p.verdict == v for p in results)
for v in ("OPEN", "CLOSED", "SKIPPED", "ERROR")}
def _probe_row(probe):
return (f" {probe.verdict:<8} {probe.domain:<9} {probe.id} "
f"{probe.target}\n {probe.message}")
def _summary(results):
counts = _tally(results)
return f"\nopen: {counts['OPEN']} " + ", ".join(
f"{v} {n}" for v, n in counts.items())
def _render_text(plan, results):
body = [_probe_row(p) for p in results]
head = [f"guardrails live probe: {plan['base_url']}",
f"probes: {len(results)} (read-only, bounded budget)"]
return "\n".join(head + [""] + body + [_summary(results)])
def _render_json(plan, results):
return json.dumps({
"base_url": plan["base_url"],
"open": _tally(results)["OPEN"],
"probes": [p.as_dict() for p in results],
}, indent=2)
def _load_plan(path):
try:
plan = json.loads(open(path, encoding="utf-8").read())
except (OSError, ValueError) as exc:
return None, f"cannot read plan: {exc}"
if not isinstance(plan, dict) or not plan.get("base_url"):
return None, "plan needs an object with a base_url"
return plan, ""
def _plan_error(plan):
method = plan.get("rate_limit", {}).get("method", "GET").upper()
if method not in READ_METHODS and method != "POST":
return f"rate_limit method {method} is not a probe method"
return ""
def parse_args(argv):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("plan", help="path to the probe plan JSON")
parser.add_argument(
"--authorized", action="store_true",
help="confirm you own the target and may test it (required)")
parser.add_argument("--json", action="store_true", help="JSON output")
return parser.parse_args(argv)
def main(argv):
args = parse_args(argv)
plan, error = _load_plan(args.plan)
error = error or _plan_error(plan or {})
if error:
print(f"usage error: {error}", file=sys.stderr)
return 2
if not args.authorized:
print("refused: pass --authorized to confirm you own "
f"{plan['base_url']} and may probe it", file=sys.stderr)
return 2
results = run_probes(Session(), plan)
render = _render_json if args.json else _render_text
print(render(plan, results))
return 1 if any(p.verdict == "OPEN" for p in results) else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+2 -1
View File
@@ -2,5 +2,6 @@
{"id": 1, "prompt": "Vérifie que la RLS est bien activée partout sur mon projet Supabase dans ~/Documents/game et qu'aucune clé service_role n'est côté client.", "expected": "Runs the scanner on the path, reads each supabase FAIL and WARN in context, gives the live RLS queries as a manual step, writes .claude/audits/GUARDRAILS.md, does not fix anything"},
{"id": 2, "prompt": "teste les garde-fous sur ce projet, il envoie des emails et des SMS de relance", "expected": "Scans messaging and api, checks consent record, unsubscribe link, List-Unsubscribe and STOP handling, traces the opt-out path from the provider webhook, lists the dashboard items as MANUAL"},
{"id": 3, "prompt": "Audit my Firebase functions: can any of them loop on itself or run up the bill?", "expected": "Scans functions domain only, reports max instances status and any trigger that writes its own collection, MANUAL item for a budget alert wired to an automatic stop, no live call to the provider"},
{"id": 4, "prompt": "est-ce que les mots de passe sont stockés en clair quelque part dans ~/Documents/zenquality ?", "expected": "Runs the scanner's api domain, reads every API-PASSWORD-* hit in context, traces the sign-up and password-change write points to the password_hash column by hand, names the hashing function found, reports with file:line, fixes nothing"}
{"id": 4, "prompt": "est-ce que les mots de passe sont stockés en clair quelque part dans ~/Documents/zenquality ?", "expected": "Runs the scanner's api domain, reads every API-PASSWORD-* hit in context, traces the sign-up and password-change write points to the password_hash column by hand, names the hashing function found, reports with file:line, fixes nothing"},
{"id": 5, "prompt": "teste les garde-fous sur ~/Documents/game puis sonde-le en direct, le dev tourne sur http://localhost:5173", "expected": "Runs the static scan first, then builds a probe plan from the findings, exports test keys from the environment, runs probe.py --authorized against the localhost target only, reports OPEN verdicts next to the matching static FAIL, never writes data, never probes a target the user did not name"}
]