feat(guardrails): read-only live probe to confirm a hole from outside

probe.py observes a running target the user owns and names, driven by a
plan built from the static findings. Five probes: anon-read, cross-user,
no-auth route, CORS reflection, capped rate-limit burst. GET/HEAD/OPTIONS
only, --authorized gate, secrets from the environment, same-host redirect
only, bounded request budget. SKILL STEP 5, checks.md reference, docs.
Suite guardrails-probe.test.sh (16 checks) against a localhost fixture.
This commit is contained in:
bchanot committed 2026-10-18 18:49:51 +02:00
1 parent dbf409f6e6
commit f0bcbfef46
8 files changed
+572 -16

No files matched your search

+50 -12
View File
@@ -1,18 +1,20 @@
---
name: guardrails
effort: high
description: 'Use when auditing a project against the baseline guardrails: hosting (rate limit, CDN and firewall, spend cap), accessibility (keyboard, visible focus, alt text, field labels), Supabase (RLS on every table, owner-scoped policies, no service_role key client-side), SMS and email (recorded consent, unsubscribe, STOP), cloud functions (no self-trigger, instance cap, budget stop), plus API and access hygiene (auth on every route, CORS, secrets, password storage). Runs a deterministic scanner, judges each hit, writes .claude/audits/GUARDRAILS.md. Triggers: "guardrails", "garde-fous", "RLS activée ?", "audit supabase", "consentement sms", "plafond de dépense", "mots de passe en clair ?", "teste la sécurité de base du projet". Headers and TLS → /harden; secrets, CVE, OWASP → /cso; full WCAG → /web-validate.'
argument-hint: '[project path] [--domain api,hosting,a11y,supabase,messaging,functions] [--exclude paths]'
description: 'Use when auditing a project against the baseline guardrails: hosting (rate limit, CDN and firewall, spend cap), accessibility (keyboard, visible focus, alt text, field labels), Supabase (RLS on every table, owner-scoped policies, no service_role key client-side), SMS and email (recorded consent, unsubscribe, STOP), cloud functions (no self-trigger, instance cap, budget stop), plus API and access hygiene (auth on every route, CORS, secrets, password storage). Runs a deterministic scanner, judges each hit, writes .claude/audits/GUARDRAILS.md. Optional read-only live probe confirms a hole from outside on a target the user owns and names. Triggers: "guardrails", "garde-fous", "RLS activée ?", "audit supabase", "consentement sms", "plafond de dépense", "mots de passe en clair ?", "teste la sécurité de base du projet", "teste-le en conditions réelles", "sonde le projet en direct". Headers and TLS → /harden; secrets, CVE, OWASP → /cso; full WCAG → /web-validate.'
argument-hint: '[project path] [--domain api,hosting,a11y,supabase,messaging,functions] [--exclude paths] [--live <url you own>]'
---
# /guardrails
Checks a project against `rules/secure-by-default.md`. A scanner reads the
tree as data and finds what text can prove. This skill then judges the hits,
covers what text cannot prove, and lists what only a dashboard can set.
covers what text cannot prove, lists what only a dashboard can set, and, on
request, probes a running target the user owns to confirm a hole from outside.
The scanner proves absence of a protection, never presence of security. A
clean scan means "no static finding", and the report says exactly that.
clean scan means "no static finding", and the report says exactly that. A
live probe proves the opposite direction: that a door is actually open.
## STEP 1: Scope
@@ -62,21 +64,57 @@ Read the code for the domains that apply and answer, with file:line evidence:
- **A11y**: with a dev server running, walk the main flow with Tab only
(`/browse`). Otherwise mark the keyboard walk-through as not done.
## STEP 5: Manual checklist
## STEP 5: Live probe (only on `--live`, read-only)
The static scan shows a protection is missing in the code. A probe shows the
door is actually open. Run it only when the user asks, and only against a
target the user owns and names: their local dev server first, a staging or
production URL only on an explicit, specific go. Never a URL you inferred,
never a third party.
1. **Build the plan** `references/checks.md` describes. From the static
findings fill a JSON plan: `base_url`, the Supabase `url` and `tables`,
the `routes` (mark the ones meant to be public), `cors`, and a
`rate_limit` route if the user wants that one burst. Secrets never go in
the plan or the command line: export `GUARDRAILS_ANON_KEY` and, for the
cross-user probe, `GUARDRAILS_JWT2`, from test keys, never `service_role`.
2. **Run it**, confirming ownership with `--authorized`:
```
python3 -B "$HOME/.claude/skills/guardrails/scripts/probe.py" \
plan.json --authorized
```
It sends GET, HEAD and OPTIONS only (the rate-limit burst is the one
POST, opt-in in the plan, capped at 25 requests). It never writes data,
never follows a redirect to another host, and stops at a request budget.
Exit 1 means at least one OPEN verdict.
3. **Cross the verdicts with the static findings.** A verdict is an
observation, not a conclusion: OPEN means the request returned data or a
200, CLOSED means it was refused or empty. A static FAIL confirmed OPEN in
the probe leads the report. A CLOSED probe on a static FAIL still needs the
fix: the dev target may differ from production.
4. **Guardrails on the probe itself**: read-only, test keys only, the
loopback or a URL the user named. A login burst can lock the user's own
account, so run it only with a throwaway test account and the user's go.
Never fuzz, never POST to anything but a named login route, never probe a
target you were not told to. That is pentest territory, another contract.
## STEP 6: Manual checklist
Print each MANUAL item as a step the user performs: spend cap, WAF and DDoS
rules, budget alert wired to a stop, live RLS queries. Provider limits change:
name the setting, tell the user to confirm it in the current provider docs.
Never run a live probe against production, and never claim a MANUAL item done.
Never run a live probe against production without the user's explicit go, and
never claim a MANUAL item done.
## STEP 6: Report
## STEP 7: Report
Write `.claude/audits/GUARDRAILS.md`: one table per domain with status PASS,
FAIL, WARN, MANUAL or N/A, the evidence, and the fix in one line. PASS means
"no static finding". Keep it short; no filler sections.
"no static finding". When a live probe ran, add a line for its verdict next
to the matching static finding. Keep it short; no filler sections.
## STEP 7: Hand over
## STEP 8: Hand over
Reply with counts, the FAILs with file:line and a one-line fix each, and the
MANUAL items. This skill audits; it does not fix. A fix goes through
`/bugfix`, `/feat` or `/hotfix`.
Reply with counts, the FAILs with file:line and a one-line fix each, any OPEN
probe verdicts, and the MANUAL items. This skill audits; it does not fix. A
fix goes through `/bugfix`, `/feat` or `/hotfix`.
+51
View File
@@ -153,3 +153,54 @@ user's token, a GET on the first user's rows must return nothing.
a retention rule for consent proof.
- **Supabase**: the live queries above; spend cap on the plan; redirect URLs
and email confirmation settings reviewed.
## Live probe (`--live`, read-only)
`probe.py` observes a running target the user owns and names. It is driven by
a plan JSON built from the static findings; secrets come from the environment,
never the plan or the command line. It sends GET, HEAD and OPTIONS only, with
one opt-in POST burst for the rate-limit check, and stops at a request budget.
```json
{
"base_url": "http://localhost:5173",
"supabase": {
"url": "http://localhost:54321",
"tables": ["notes", "contacts"],
"cross_user_paths": ["/rest/v1/notes?select=*"]
},
"routes": [
{"path": "/api/v1/notes"},
{"path": "/api/v1/health", "public": true}
],
"cors": true,
"cors_path": "/api/v1/notes",
"rate_limit": {"path": "/api/v1/auth/login", "method": "POST",
"count": 20, "body": "{\"email\":\"t@t.io\",\"password\":\"x\"}"}
}
```
Environment, from test keys only, never `service_role`:
`GUARDRAILS_ANON_KEY` (anon-read), `GUARDRAILS_JWT2` (cross-user).
| Probe | id | OPEN means | Read it against |
|---|---|---|---|
| anon-read | `anon-read:<table>` | the anon key read a row | SB-RLS-MISSING, SB-POLICY-* |
| cross-user | `cross-user:<path>` | a second user's token read these rows | policy owner column (STEP 4) |
| no-auth route | `no-auth:<path>` | the route answered 200 with no token | API-ROUTE-NO-AUTH, object authz |
| cors-reflect | `cors-reflect` | a foreign origin was reflected with credentials | API-CORS-WILDCARD |
| rate-limit | `rate-limit` | the burst saw no 429 | HOST-NO-RATE-LIMIT |
A verdict is an observation, not a verdict on the project. OPEN is a returned
200 or returned rows; CLOSED is a refusal or an empty body; SKIPPED is a
missing key; ERROR is a transport failure. A static FAIL confirmed OPEN leads
the report. A CLOSED probe on a static FAIL still needs the fix: the dev
target may differ from production.
Safety, enforced by the script and by the skill: `--authorized` is required;
read-only methods only; the loopback or a URL the user named, never one you
inferred; a login burst only with a throwaway test account on the user's go,
because it can lock the user's own account; a redirect to another host is
never followed, so a key never leaves the named host. Fuzzing, writes beyond
the named login route, or probing a third party are out of scope: that is
pentest, another contract.
+309
View File
@@ -0,0 +1,309 @@
#!/usr/bin/env python3
"""Live guardrails probe: observe a running target from outside, read-only.
The static scan shows a protection is absent in the code. This observes what
a running target the user owns and names actually returns. It sends GET, HEAD
and OPTIONS only; a login burst is opt-in in the plan. It never writes data,
never uses a service_role key, never follows a redirect to another host, and
stops at a fixed request budget.
Input is a plan JSON (built by the skill from the static findings) plus
secrets from the environment, never the command line:
GUARDRAILS_ANON_KEY Supabase anon/publishable key for the anon-read probe
GUARDRAILS_JWT2 a second test user's token for the cross-user probe
A verdict is a factual observation, not a conclusion: the skill reads it
against the static findings. OPEN means the request returned data or a 200,
CLOSED means it was refused or empty. Exit codes: 0 no OPEN, 1 at least one
OPEN, 2 usage error.
"""
import argparse
import json
import os
import sys
import urllib.error
import urllib.request
from dataclasses import asdict, dataclass
from urllib.parse import quote, urlsplit
READ_METHODS = {"GET", "HEAD", "OPTIONS"}
FAKE_ORIGIN = "https://guardrails-probe.invalid"
DEFAULT_BUDGET = 200
BURST_CAP = 25
BODY_CAP = 1_000_000
@dataclass(frozen=True)
class Response:
status: int # None on a transport error
headers: dict
body: bytes
error: str = ""
@dataclass(frozen=True)
class Probe:
domain: str
id: str
verdict: str # OPEN | CLOSED | SKIPPED | ERROR
target: str
message: str
def as_dict(self):
return asdict(self)
class _SameHost(urllib.request.HTTPRedirectHandler):
"""Follow a same-host redirect, drop one that leaves the host."""
def redirect_request(self, req, fp, code, msg, headers, newurl):
if urlsplit(newurl).netloc != urlsplit(req.full_url).netloc:
return None
return super().redirect_request(req, fp, code, msg, headers, newurl)
class Session:
"""Holds the redirect-guarded opener and the shared request budget."""
def __init__(self, timeout=8, budget=DEFAULT_BUDGET):
self.opener = urllib.request.build_opener(_SameHost())
self.timeout = timeout
self.budget = budget
def _response_from(resp):
return Response(resp.status, dict(resp.headers), resp.read(BODY_CAP))
def fetch(session, method, url, headers=None, body=None):
if session.budget <= 0:
return Response(None, {}, b"", "request budget exhausted")
session.budget -= 1
req = urllib.request.Request(
url, data=body, method=method, headers=headers or {})
try:
with session.opener.open(req, timeout=session.timeout) as resp:
return _response_from(resp)
except urllib.error.HTTPError as exc:
return Response(exc.code, dict(exc.headers), b"")
except (urllib.error.URLError, OSError, ValueError) as exc:
return Response(None, {}, b"", str(exc))
def _verdict(ctx, response, is_open, msgs):
"""ctx is (domain, id, target); msgs is (open_note, closed_note)."""
domain, pid, target = ctx
if response.error:
return Probe(domain, pid, "ERROR", target, response.error)
verdict = "OPEN" if is_open else "CLOSED"
note = f"HTTP {response.status}: {msgs[0] if is_open else msgs[1]}"
return Probe(domain, pid, verdict, target, note)
def _nonempty_rows(body):
try:
data = json.loads(body or b"[]")
except ValueError:
return False
return isinstance(data, list) and len(data) > 0
def _table_probe(session, base, table, headers):
target = f"{base}/rest/v1/{quote(table)}?select=*&limit=1"
resp = fetch(session, "GET", target, headers)
return _verdict(
("supabase", f"anon-read:{table}", target), resp,
resp.status == 200 and _nonempty_rows(resp.body),
("anon request returned rows", "anon request returned no row"))
def probe_supabase(session, plan):
cfg = plan.get("supabase")
anon = os.environ.get("GUARDRAILS_ANON_KEY")
if not cfg or not cfg.get("tables"):
return []
if not anon:
return [Probe("supabase", "anon-read", "SKIPPED", cfg.get("url", ""),
"set GUARDRAILS_ANON_KEY to run the anon-read probe")]
base = cfg["url"].rstrip("/")
headers = {"apikey": anon, "Authorization": f"Bearer {anon}"}
return [_table_probe(session, base, t, headers) for t in cfg["tables"]]
def _cross_user_probe(session, base, path, headers):
target = base + path
resp = fetch(session, "GET", target, headers)
return _verdict(
("supabase", f"cross-user:{path}", target), resp,
resp.status == 200 and _nonempty_rows(resp.body),
("second user's token returned these rows",
"second user's token returned nothing"))
def probe_cross_user(session, plan):
cfg = plan.get("supabase") or {}
paths = cfg.get("cross_user_paths") or []
jwt2 = os.environ.get("GUARDRAILS_JWT2")
if not paths:
return []
if not jwt2:
return [Probe("supabase", "cross-user", "SKIPPED", cfg.get("url", ""),
"set GUARDRAILS_JWT2 to run the cross-user probe")]
headers = {"apikey": os.environ.get("GUARDRAILS_ANON_KEY", ""),
"Authorization": f"Bearer {jwt2}"}
base = cfg["url"].rstrip("/")
return [_cross_user_probe(session, base, p, headers) for p in paths]
def _route_probe(session, base, route):
target = base + route["path"]
resp = fetch(session, "GET", target)
return _verdict(
("api", f"no-auth:{route['path']}", target), resp,
resp.status == 200,
("unauthenticated request returned 200",
"unauthenticated request was refused"))
def probe_routes(session, plan):
base = plan["base_url"].rstrip("/")
routes = [r for r in plan.get("routes", []) if not r.get("public")]
return [_route_probe(session, base, r) for r in routes]
def _cors_open(headers):
allow = headers.get("Access-Control-Allow-Origin", "")
creds = headers.get("Access-Control-Allow-Credentials", "").lower()
return allow in (FAKE_ORIGIN, "*") and creds == "true"
def probe_cors(session, plan):
if not plan.get("cors"):
return []
base = plan["base_url"].rstrip("/")
headers = {"Origin": FAKE_ORIGIN, "Access-Control-Request-Method": "GET"}
resp = fetch(session, "OPTIONS", base + plan.get("cors_path", "/"), headers)
return [_verdict(
("api", "cors-reflect", base), resp, _cors_open(resp.headers),
("foreign origin reflected with credentials",
"foreign origin not reflected with credentials"))]
def _burst_codes(session, url, method, body, count):
data = body.encode() if body else None
codes = []
for _ in range(count):
if session.budget <= 0:
break
codes.append(fetch(session, method, url, None, data).status)
return codes
def _rate_verdict(target, codes):
sent = len([c for c in codes if c is not None])
throttled = 429 in codes
note = (f"sent {sent}, {'got a 429' if throttled else 'no 429'}: "
+ ("throttled" if throttled else "no 429 seen on this route"))
return Probe("hosting", "rate-limit",
"CLOSED" if throttled else "OPEN", target, note)
def probe_rate_limit(session, plan):
cfg = plan.get("rate_limit")
if not cfg:
return []
target = plan["base_url"].rstrip("/") + cfg["path"]
codes = _burst_codes(
session, target, cfg.get("method", "GET").upper(),
cfg.get("body"), min(cfg.get("count", 20), BURST_CAP))
return [_rate_verdict(target, codes)]
PROBES = (probe_supabase, probe_cross_user, probe_routes,
probe_cors, probe_rate_limit)
def run_probes(session, plan):
results = []
for probe in PROBES:
results += probe(session, plan)
return results
def _tally(results):
return {v: sum(p.verdict == v for p in results)
for v in ("OPEN", "CLOSED", "SKIPPED", "ERROR")}
def _probe_row(probe):
return (f" {probe.verdict:<8} {probe.domain:<9} {probe.id} "
f"{probe.target}\n {probe.message}")
def _summary(results):
counts = _tally(results)
return f"\nopen: {counts['OPEN']} " + ", ".join(
f"{v} {n}" for v, n in counts.items())
def _render_text(plan, results):
body = [_probe_row(p) for p in results]
head = [f"guardrails live probe: {plan['base_url']}",
f"probes: {len(results)} (read-only, bounded budget)"]
return "\n".join(head + [""] + body + [_summary(results)])
def _render_json(plan, results):
return json.dumps({
"base_url": plan["base_url"],
"open": _tally(results)["OPEN"],
"probes": [p.as_dict() for p in results],
}, indent=2)
def _load_plan(path):
try:
plan = json.loads(open(path, encoding="utf-8").read())
except (OSError, ValueError) as exc:
return None, f"cannot read plan: {exc}"
if not isinstance(plan, dict) or not plan.get("base_url"):
return None, "plan needs an object with a base_url"
return plan, ""
def _plan_error(plan):
method = plan.get("rate_limit", {}).get("method", "GET").upper()
if method not in READ_METHODS and method != "POST":
return f"rate_limit method {method} is not a probe method"
return ""
def parse_args(argv):
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("plan", help="path to the probe plan JSON")
parser.add_argument(
"--authorized", action="store_true",
help="confirm you own the target and may test it (required)")
parser.add_argument("--json", action="store_true", help="JSON output")
return parser.parse_args(argv)
def main(argv):
args = parse_args(argv)
plan, error = _load_plan(args.plan)
error = error or _plan_error(plan or {})
if error:
print(f"usage error: {error}", file=sys.stderr)
return 2
if not args.authorized:
print("refused: pass --authorized to confirm you own "
f"{plan['base_url']} and may probe it", file=sys.stderr)
return 2
results = run_probes(Session(), plan)
render = _render_json if args.json else _render_text
print(render(plan, results))
return 1 if any(p.verdict == "OPEN" for p in results) else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+2 -1
View File
@@ -2,5 +2,6 @@
{"id": 1, "prompt": "Vérifie que la RLS est bien activée partout sur mon projet Supabase dans ~/Documents/game et qu'aucune clé service_role n'est côté client.", "expected": "Runs the scanner on the path, reads each supabase FAIL and WARN in context, gives the live RLS queries as a manual step, writes .claude/audits/GUARDRAILS.md, does not fix anything"},
{"id": 2, "prompt": "teste les garde-fous sur ce projet, il envoie des emails et des SMS de relance", "expected": "Scans messaging and api, checks consent record, unsubscribe link, List-Unsubscribe and STOP handling, traces the opt-out path from the provider webhook, lists the dashboard items as MANUAL"},
{"id": 3, "prompt": "Audit my Firebase functions: can any of them loop on itself or run up the bill?", "expected": "Scans functions domain only, reports max instances status and any trigger that writes its own collection, MANUAL item for a budget alert wired to an automatic stop, no live call to the provider"},
{"id": 4, "prompt": "est-ce que les mots de passe sont stockés en clair quelque part dans ~/Documents/zenquality ?", "expected": "Runs the scanner's api domain, reads every API-PASSWORD-* hit in context, traces the sign-up and password-change write points to the password_hash column by hand, names the hashing function found, reports with file:line, fixes nothing"}
{"id": 4, "prompt": "est-ce que les mots de passe sont stockés en clair quelque part dans ~/Documents/zenquality ?", "expected": "Runs the scanner's api domain, reads every API-PASSWORD-* hit in context, traces the sign-up and password-change write points to the password_hash column by hand, names the hashing function found, reports with file:line, fixes nothing"},
{"id": 5, "prompt": "teste les garde-fous sur ~/Documents/game puis sonde-le en direct, le dev tourne sur http://localhost:5173", "expected": "Runs the static scan first, then builds a probe plan from the findings, exports test keys from the environment, runs probe.py --authorized against the localhost target only, reports OPEN verdicts next to the matching static FAIL, never writes data, never probes a target the user did not name"}
]