fix(guardrails): probe fails closed on budget, redirects and unread bodies; rules lose five false positives and gain seven misses
The probe trusted the plan's Supabase url, followed redirects for free, crashed on deep JSON and stopped at the first unconfirmed map candidate. Rules now scope hydrated state to the export chunk, silence dev configs, UI-suffixed data attributes, GraphQL types and metadata keys, and catch Go raw strings, triple-quoted f-strings, sql.raw, empty select() and multi-line hidden inputs; text output is sanitised; per-variant locks.
This commit is contained in:
1 parent
f586304ba1
commit
5a98c2c095
9 files changed
+530
-96
No files matched your search
@@ -56,6 +56,10 @@ it ever reached git history is a separate check (`gitleaks git <repo>`).
|
||||
Heuristic blind spots are in `references/checks.md`: check them before
|
||||
calling a hit false.
|
||||
|
||||
- INJ-SQL-CONCAT on `IN (${placeholders})` where the list is generated
|
||||
`?`/`$n` markers and the values travel in a separate array: parameterised,
|
||||
confirm by hand and drop it from the proposals.
|
||||
|
||||
## STEP 4: Cover what the scanner cannot
|
||||
|
||||
Read the code for the domains that apply and answer, with file:line evidence:
|
||||
@@ -93,8 +97,10 @@ never a third party.
|
||||
the `routes` (mark the ones meant to be public), `cors`, and a
|
||||
`rate_limit` route if the user wants that one burst. Add an `exposure` block (`{}` for the defaults, `error_routes` for a route you own that answers an error to a malformed id, never a path you guess) so the probe reads source maps, error bodies and response keys; every path in the plan starts with `/`. Secrets never go in
|
||||
the plan or the command line: export `GUARDRAILS_ANON_KEY` and, for the
|
||||
cross-user probe, `GUARDRAILS_JWT2`, from test keys, never `service_role`.
|
||||
2. **Run it**, confirming ownership with `--authorized`:
|
||||
cross-user probe, `GUARDRAILS_JWT2`, from test keys, never `service_role`. The fields probe sends
|
||||
`GUARDRAILS_JWT2` as a bearer when set, so an OPEN there is what a
|
||||
logged-in user sees.
|
||||
2. **Run it**, confirming ownership with `--authorized`. Before adding `--authorized`, show the user both hosts the plan names (`base_url` and the Supabase `url`, the latter read from the project's config and so untrusted) and get their go on both:
|
||||
```
|
||||
python3 -B "$HOME/.claude/skills/guardrails/scripts/probe.py" \
|
||||
plan.json --authorized
|
||||
@@ -201,7 +207,8 @@ file before anything else. Accepted set empty → stop.
|
||||
proof the scanner cannot run (probe, cross-user, manual SQL) is a
|
||||
criterion without CHECK: STEP 11 settles it. FILE SCOPE = the files the
|
||||
accepted proposals name. Paths go in single quotes; a path holding a
|
||||
single quote is named to the user, never written into a CHECK.
|
||||
single quote is named to the user, never written into a CHECK. The
|
||||
RULE:path pairs come from the `--json` output, never from the text report.
|
||||
4. **Positive control.** Run each oracle once on the unfixed tree: it must
|
||||
fail. An oracle that passes before the fix proves nothing; drop it, find
|
||||
why, and say so.
|
||||
|
||||
@@ -38,25 +38,25 @@ match (an interpolation inside a raw escape stays FAIL).
|
||||
| Rule | Sev | Catches | Fix |
|
||||
|---|---|---|---|
|
||||
| INJ-SQL-CONCAT | FAIL | a SQL statement (select … from … where, insert into, update … set, delete … where, create/drop/alter) built with `${}`, `+`, an f-string, `%`, `.format`, PHP `$var`, Ruby `#{}` (ActiveRecord `where("… #{}")` included), `format!`, `Sprintf` | placeholders and a parameter list |
|
||||
| INJ-SQL-RAW-UNSAFE | WARN | `$queryRawUnsafe`, `$executeRawUnsafe`, `knex.raw`, `sequelize.query`, `Sequelize.literal`, Django `.objects.raw()` / `.extra()`, SQLAlchemy `text()` / `sa.text(` / `sqlalchemy.text(` fed a variable (a variable may hold a constant)| show it is a constant with bound values, else bind them; or the tagged `$queryRaw` |
|
||||
| INJ-SQL-RAW-UNSAFE | WARN | `$queryRawUnsafe`, `$executeRawUnsafe`, `knex.raw`, `sql.raw`, `db.raw`, `trx.raw`, `sequelize.query`, `Sequelize.literal`, Django `.objects.raw()` / `.extra()`, SQLAlchemy `text()` / `sa.text(` / `sqlalchemy.text(` fed a variable (a variable may hold a constant)| show it is a constant with bound values, else bind them; or the tagged `$queryRaw` |
|
||||
| INJ-SQL-DYNAMIC-EXEC | WARN | plpgsql `EXECUTE '…' \|\| x` | `format('… %I %L', …)` and `USING` |
|
||||
| INJ-POSTGREST-FILTER | WARN | supabase-js `.or()`, `.filter()`, `.not()`, `.match()`, `.textSearch()` with an interpolated string, in a file that uses supabase | `.eq()` and friends, validated values |
|
||||
| INJ-NOSQL-OPERATOR | WARN | Mongo `$where` with a string; a filter taken from `req.body`, `req.query` or `req.params` | allowlist the fields, cast to string, `sanitizeFilter` |
|
||||
| INJ-SHELL-DB-TOOL | FAIL | a shell call naming psql, mysql, mysqldump, pg_dump, mongosh, redis-cli, sqlite3… with an interpolated argument (an argv array and a bare `$ENV_VAR` stay silent; in PHP every `$name` counts) | an argv array, no shell, validated input |
|
||||
| INJ-NOSQL-OPERATOR | WARN | Mongo `$where` with a string, an identifier or a function (a GraphQL `_bool_exp` type and a Capitalised type are excluded); a filter taken from `req.body`, `req.query` or `req.params` | allowlist the fields, cast to string, `sanitizeFilter` |
|
||||
| INJ-SHELL-DB-TOOL | FAIL | a shell call naming psql, mysql, mysqldump, pg_dump, mongosh, redis-cli, sqlite3… with an interpolated argument (`${x}`, `#{x}`, `{x}`, a positional `{}` closing a Rust format string, `%s`; an argv array and a bare `$ENV_VAR` stay silent; in PHP every `$name` counts) | an argv array, no shell, validated input |
|
||||
|
||||
## exposure
|
||||
|
||||
| Rule | Sev | Catches | Fix |
|
||||
|---|---|---|---|
|
||||
| EXP-CONSOLE-DATA | WARN | `console.log/info/debug/dir/table` in client-reachable code with a data argument (user, session, response, token, body…); string literals are ignored | strip console calls in the production build, log server-side |
|
||||
| EXP-SOURCEMAP-PUBLIC | FAIL / WARN | `productionBrowserSourceMaps: true`, vite `sourcemap: true` or `'inline'`, `GENERATE_SOURCEMAP=true` in a committed non-dev env file, webpack `devtool` readable maps (FAIL in `webpack.prod*`, WARN elsewhere), a real `.map` under `public/` or `static/` (any size; vendored and `.min.` maps excluded); angular `sourceMap: true` (WARN) | off, or `hidden` and upload to the error tracker |
|
||||
| EXP-SOURCEMAP-PUBLIC | FAIL / WARN | `productionBrowserSourceMaps: true`, vite `sourcemap: true` or `'inline'`, `GENERATE_SOURCEMAP=true` in a committed non-dev env file, webpack `devtool` readable maps (FAIL in `webpack.prod*`, WARN elsewhere, dev configs silent), a real `.map` under `public/` or `static/` (any size; vendored and `.min.` maps excluded); angular `sourceMap: true` (WARN) | off, or `hidden` and upload to the error tracker |
|
||||
| EXP-HYDRATED-SENSITIVE | WARN | a token, secret, key, password or ssn-like key returned by `getServerSideProps`, `getStaticProps`, a Remix `loader`, a SvelteKit `load` or set on `window.__STATE__`; a TS type member is skipped | return only the fields the page renders |
|
||||
| EXP-MARKUP-COMMENT | WARN | `<!-- -->` shipped to the browser (html, astro, php, ejs, hbs, twig, njk, liquid) with todo, fixme, debug, password, token, internal, staging, admin… | delete it, or move it to a code comment the build strips |
|
||||
| EXP-DATA-ATTR-SENSITIVE | WARN | `data-*` attribute or hidden input named secret, api-key, password, jwt, private, credential (csrf, xsrf, authenticity, nonce excluded) | keep it server-side, send an opaque id |
|
||||
| EXP-DATA-ATTR-SENSITIVE | WARN | `data-*` attribute or hidden input named secret, api-key, password, jwt, private, credential (csrf, xsrf, authenticity, nonce excluded; a UI suffix such as `-toggle`, `-visible` or `-strength` is silent) | keep it server-side, send an opaque id |
|
||||
| EXP-ERROR-STACK | FAIL | in any server file: `err.stack`, the error object, `traceback.format_exc()` or `getTraceAsString()` in a response | generic message out, full error in the server log |
|
||||
| EXP-ERROR-MESSAGE | WARN | in any server file: `err.message`, `str(e)`, `$e->getMessage()` sent as is | map to a safe message; validation errors are fine, DB errors are not |
|
||||
| EXP-DEBUG-ON | FAIL | `DEBUG = True` in a settings module (not dev/local/test), `APP_DEBUG=true` / `FLASK_DEBUG=1` in a committed non-dev env file, `app.debug = True`, `consider_all_requests_local = true`, `errorhandler()` mounted without an environment guard | env-driven, false in production |
|
||||
| EXP-RESPONSE-WIDE | WARN | a route returning `select *` / `select('*')` on a users-like table, or a single-line Prisma `findUnique/findFirst/findMany` without `select` or `omit` | an explicit field list |
|
||||
| EXP-RESPONSE-WIDE | WARN | a route returning `select *` / `select('*')` / `.select()` on a users-like table, or a single-line Prisma `findUnique/findFirst/findMany` without `select` or `omit` | an explicit field list |
|
||||
|
||||
## supabase
|
||||
|
||||
@@ -126,6 +126,14 @@ The scanner is regex over text. It cannot see:
|
||||
elsewhere: the column rule fires only when no library is found anywhere,
|
||||
so trace the write point by hand (STEP 4)
|
||||
- a log call spanning several lines, or `req.body` dumped to a log
|
||||
- hydrated state is read per export chunk: a non-exported helper between two
|
||||
exports is still scanned (false positive), a sensitive object built in a
|
||||
helper above the loader is missed, and moving a key above the loader clears
|
||||
the oracle, so the STEP 3 judge re-reads the whole loader
|
||||
- `$where: PascalCase` and `*_exp` identifiers are read as GraphQL types
|
||||
- a raw SQL literal whose statement keyword does not open it (Go backtick,
|
||||
Python triple-quoted f-string with a comment first) is not seen
|
||||
- INJ-SQL-CONCAT costs about 3 s on a 1 MB file: accepted
|
||||
|
||||
Sender classification is by file path only: names with contact, reset, verify,
|
||||
otp, receipt and the like count as transactional, names with newsletter,
|
||||
@@ -138,7 +146,7 @@ The scanner skips log files and cache directories, and never follows symlinks.
|
||||
By design the scanner ignores: policies granted only to `service_role`,
|
||||
UI route components, keys
|
||||
inside git-ignored env files, and password literals in test, fixture and
|
||||
i18n files, tagged SQL templates, placeholders with a separate parameter list, an argv array handed to a DB tool, console calls in server code, logging calls that carry a stack, `sourcemap: 'hidden'`, a debug flag read from the environment, dev, local and test env files and settings modules, minified files and built directories, and a hidden input or data attribute named csrf, xsrf, authenticity or nonce.
|
||||
i18n files, tagged SQL templates, placeholders with a separate parameter list, an argv array handed to a DB tool, console calls in server code, logging calls that carry a stack, `sourcemap: 'hidden'`, a debug flag read from the environment, dev, local and test env files and settings modules, minified files and built directories, a hidden input or data attribute named csrf, xsrf, authenticity or nonce, a generated `?` list in `IN (…)`, webpack dev configs, data-* names ending in a UI suffix, a loader scoped to its export chunk, and metadata keys in the fields probe.
|
||||
|
||||
## Live Supabase checks (the user runs them in the SQL editor)
|
||||
|
||||
@@ -221,13 +229,14 @@ The fields probe sends `GUARDRAILS_JWT2` as a bearer when it is set. Every path
|
||||
| no-auth route | `no-auth:<path>` | the route answered 200 with no token | API-ROUTE-NO-AUTH, object authz |
|
||||
| cors-reflect | `cors-reflect` | a foreign origin was reflected with credentials | API-CORS-WILDCARD |
|
||||
| rate-limit | `rate-limit` | the burst saw no 429 | HOST-NO-RATE-LIMIT |
|
||||
| sourcemap | `sourcemap:<script>` | a source map with sources answered 200 for a page script, or the script carries an inline map | EXP-SOURCEMAP-PUBLIC |
|
||||
| sourcemap | `sourcemap:<script>` | a source map with sources answered 200 for a page script, or the script carries an inline map (a line-start `data:` directive is OPEN whatever its encoding) | EXP-SOURCEMAP-PUBLIC |
|
||||
| error-body | `error-body:<path>` | a 404 or a named error route returned a stack trace or an internal path | EXP-ERROR-STACK, EXP-DEBUG-ON |
|
||||
| fields | `fields:<path>` | a JSON 200 carried a password, salt, token, hash, key or secret-like key (`private` only as private_key; identifier-shaped names printed, never values) | EXP-RESPONSE-WIDE, EXP-HYDRATED-SENSITIVE |
|
||||
| fields | `fields:<path>` | a JSON 200 carried a password, salt, token, hash, key or secret-like key (`private` only as private_key; metadata names such as `token_type`, `has_password` or `next_page_token` are excluded; identifier-shaped names printed, never values) | EXP-RESPONSE-WIDE, EXP-HYDRATED-SENSITIVE |
|
||||
|
||||
A verdict is an observation, not a verdict on the project. OPEN is a returned
|
||||
200 or returned rows; CLOSED is a refusal or an empty body; SKIPPED is a
|
||||
missing key; ERROR is a transport failure. A static FAIL confirmed OPEN leads
|
||||
missing key or a body the probe could not read; ERROR is a transport
|
||||
failure or an exhausted budget. A static FAIL confirmed OPEN leads
|
||||
the report. A CLOSED probe on a static FAIL still needs the fix: the dev
|
||||
target may differ from production. An exposure target the probe could not observe (unreachable page, no same-host script, a refused or non-JSON route, a map fetch that failed) is SKIPPED or ERROR, never CLOSED. Run the exposure block against a production build or a staging URL: a dev server serves maps and tracebacks by design, and its OPEN verdicts are dev artefacts, not findings.
|
||||
|
||||
|
||||
@@ -26,6 +26,11 @@ SOURCE_EXTS = {
|
||||
}
|
||||
|
||||
|
||||
def printable(text):
|
||||
"""Control characters of a remote string never reach the tty."""
|
||||
return re.sub(r"[\x00-\x1f\x7f-\x9f\udc80-\udcff]", "?", str(text))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Rule:
|
||||
domain: str
|
||||
|
||||
@@ -22,6 +22,7 @@ SENSITIVE = (
|
||||
r"(?:token|accessToken|refreshToken|secret|apiKey|api_key|password"
|
||||
r"|passwordHash|password_hash|serviceRole|service_role|privateKey"
|
||||
r"|private_key|ssn|iban|stripeSecret|clientSecret)")
|
||||
EXPORT = re.compile(r"^export\b", re.M)
|
||||
LOADER = re.compile(
|
||||
r"getServerSideProps|getStaticProps"
|
||||
r"|export\s+(?:async\s+)?function\s+(?:loader|load)\b"
|
||||
@@ -35,10 +36,15 @@ COMMENT_WORDS = re.compile(
|
||||
r"\b(?:todo|fixme|hack|xxx|debug|password|passwd|secret|token"
|
||||
r"|api[-_ ]?key|internal|staging|localhost|admin|remove before"
|
||||
r"|do not commit|don'?t commit|temporary)\b", re.I)
|
||||
SENSITIVE_WORD = (r"(?:secret|api-?_?key|password|passwd|jwt|private"
|
||||
r"|credential|service-?role)")
|
||||
DATA_ATTR = re.compile(
|
||||
r"\b(data-[\w-]{0,60}(?:secret|api-?_?key|password|passwd|jwt|private"
|
||||
r"|credential|service-?role)[\w-]{0,60})\s*=", re.I)
|
||||
HIDDEN_INPUT = re.compile(r"<input\b[^>\n]{0,300}>", re.I)
|
||||
rf"\b(data-[\w-]{{0,60}}{SENSITIVE_WORD}[\w-]{{0,60}})\s*=", re.I)
|
||||
UI_ATTR = re.compile(
|
||||
rf"{SENSITIVE_WORD}-(?:toggle|visible|visibility|show|hide|reveal|mask"
|
||||
r"|strength|meter|hint|label|placeholder|confirm|error|required"
|
||||
r"|pattern)$", re.I)
|
||||
HIDDEN_INPUT = re.compile(r"<input\b[^>]{0,300}>", re.I)
|
||||
SENSITIVE_NAME = re.compile(
|
||||
r"secret|api[-_]?key|password|passwd|jwt|private|credential", re.I)
|
||||
SAFE_NAME = re.compile(r"csrf|xsrf|authenticity|nonce", re.I)
|
||||
@@ -76,7 +82,7 @@ VENDORED = re.compile(r"\.min\.|vendor|bootstrap|jquery", re.I)
|
||||
USERS = (r"(?:users|profiles|accounts|customers|members|employees"
|
||||
r"|patients|clients)")
|
||||
WIDE = re.compile(
|
||||
rf"\.from\(\s*['\"]{USERS}['\"]\s*\)\s*\.select\(\s*['\"]\*['\"]\s*\)"
|
||||
rf"\.from\(\s*['\"]{USERS}['\"]\s*\)\s*\.select\(\s*(?:['\"]\*['\"]\s*)?\)"
|
||||
rf"|select\s+\*\s+from\s+{USERS}\b"
|
||||
r"|\b(?:user|profile|account|customer|member)\.(?:findUnique|findFirst"
|
||||
r"|findMany)\(\s*\{(?![^\n]{0,300}\b(?:select|omit)\s*:)[^\n]{0,300}\}"
|
||||
@@ -124,7 +130,8 @@ def _map_config(project):
|
||||
out += hits(project, _named(project, "vite.config"), VITE_MAPS, fail,
|
||||
"build source maps enabled: use 'hidden' or false")
|
||||
prod = _named(project, "webpack.prod")
|
||||
other = [p for p in _named(project, "webpack") if p not in prod]
|
||||
other = [p for p in _named(project, "webpack")
|
||||
if p not in prod and not DEV_TOKEN.search(p.name)]
|
||||
out += hits(project, prod, WEBPACK_DEVTOOL, fail,
|
||||
"devtool emits readable source maps in production")
|
||||
out += hits(project, other, WEBPACK_DEVTOOL, warn,
|
||||
@@ -148,12 +155,13 @@ def _public_map(project, path):
|
||||
|
||||
|
||||
def _large_map(project, path):
|
||||
"""A large public .map whose first 4 KB already hold a mappings key."""
|
||||
"""A large public .map whose first 4 KB hold a mappings or sources key."""
|
||||
if path.suffix.lower() != ".map" or not _public_map(project, path):
|
||||
return False
|
||||
try:
|
||||
with path.open("rb") as handle:
|
||||
return b'"mappings"' in handle.read(4096)
|
||||
head = handle.read(4096)
|
||||
return b'"mappings"' in head or b'"sources"' in head
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
@@ -169,11 +177,27 @@ def _map_files(project):
|
||||
for p in small + large]
|
||||
|
||||
|
||||
def _hydrated_spans(text):
|
||||
"""(start, end) of each export chunk holding a loader, from its line."""
|
||||
bounds = sorted({0, *(m.start() for m in EXPORT.finditer(text))})
|
||||
spans = []
|
||||
for start, end in zip(bounds, bounds[1:] + [len(text)]):
|
||||
found = LOADER.search(text, start, end)
|
||||
if found:
|
||||
spans.append((text.rfind("\n", 0, found.start()) + 1, end))
|
||||
return spans
|
||||
|
||||
|
||||
def _hydrated(project, files):
|
||||
loaders = [p for p in files if LOADER.search(project.text(p))]
|
||||
return hits(project, loaders, HYDRATED,
|
||||
Rule(DOMAIN, "EXP-HYDRATED-SENSITIVE", "WARN"),
|
||||
"sensitive key in server-to-client state")
|
||||
rule = Rule(DOMAIN, "EXP-HYDRATED-SENSITIVE", "WARN")
|
||||
out = []
|
||||
for path in files:
|
||||
text = project.text(path)
|
||||
out += [project.found(rule, path, m.start(),
|
||||
"sensitive key in server-to-client state")
|
||||
for begin, end in _hydrated_spans(text)
|
||||
for m in HYDRATED.finditer(text, begin, end)]
|
||||
return dedupe_lines(out)
|
||||
|
||||
|
||||
def _markup_comments(project):
|
||||
@@ -202,7 +226,8 @@ def _data_attrs(project):
|
||||
out += [project.found(rule, path, m.start(),
|
||||
"sensitive value in a data-* attribute")
|
||||
for m in DATA_ATTR.finditer(text)
|
||||
if not SAFE_NAME.search(m.group(1))]
|
||||
if not SAFE_NAME.search(m.group(1))
|
||||
and not UI_ATTR.search(m.group(1))]
|
||||
out += [project.found(rule, path, m.start(),
|
||||
"sensitive value in a hidden input")
|
||||
for m in HIDDEN_INPUT.finditer(text)
|
||||
|
||||
@@ -32,7 +32,8 @@ CONCAT = {
|
||||
rf"(?<![\w$)\]>])`[^`]{{0,400}}?{STMT}[^`]{{0,400}}?\$\{{|"
|
||||
+ _quoted(r"\s*\+\s*[\w$(]"), re.I),
|
||||
(".py",): re.compile(
|
||||
rf"\bf['\"][^'\"\n]{{0,200}}?{STMT}|"
|
||||
rf"\b(?:rf|fr|f)['\"][^'\"\n]{{0,200}}?{STMT}"
|
||||
rf"|\b(?:rf|fr|f)(?:\"\"\"|''')\s{{0,40}}{STMT}|"
|
||||
+ _quoted(r"\s*(?:%\s*[\w(]|\.format\(|\+\s*[\w(])"), re.I),
|
||||
(".php",): re.compile(
|
||||
rf"\"{DQ}?{STMT}{DQ}\$[A-Za-z_]\w*|"
|
||||
@@ -40,11 +41,12 @@ CONCAT = {
|
||||
(".rb",): re.compile(
|
||||
rf"\"{DQ}?{STMT}{DQ}#\{{|\.where\(\s*\"{DQ}#\{{", re.I),
|
||||
(".rs",): re.compile(rf"format!\(\s*\"{DQ}?{STMT}", re.I),
|
||||
(".go",): re.compile(rf"fmt\.Sprintf\(\s*\"{DQ}?{STMT}", re.I),
|
||||
(".go",): re.compile(
|
||||
rf"fmt\.Sprintf\(\s*(?:\"{DQ}?{STMT}|`\s{{0,40}}{STMT})", re.I),
|
||||
}
|
||||
RAW = {
|
||||
JS_EXTS: re.compile(
|
||||
r"(?:\$queryRawUnsafe|\$executeRawUnsafe|\bknex\.raw"
|
||||
r"(?:\$queryRawUnsafe|\$executeRawUnsafe|\b(?:sql|db|trx|knex)\.raw"
|
||||
r"|\bsequelize\.query|\b[Ss]equelize\.literal)\s*\(\s*(?![\s'\"`)])"),
|
||||
(".py",): re.compile(
|
||||
r"\.objects\.raw\(\s*(?![\s'\"`)])|\.extra\(\s*(?![\s'\"`)])"),
|
||||
@@ -56,7 +58,8 @@ POSTGREST = re.compile(
|
||||
r"\.(?:or|filter|not|match|textSearch)\(\s*(?:`[^`\n]{0,200}\$\{"
|
||||
r"|['\"][^'\"\n]{0,200}['\"]\s*\+\s*[\w$])")
|
||||
NOSQL = re.compile(
|
||||
r"\$where\s*:\s*(?:['\"`]|\w)"
|
||||
r"\$where\s*:\s*(?!\w{0,64}_(?:bool_)?exp\b|[A-Z]\w{0,64}!?\s*[),])"
|
||||
r"(?:['\"`]|\w)"
|
||||
r"|\.(?:find|findOne|findOneAndUpdate|findOneAndDelete|updateOne"
|
||||
r"|updateMany|deleteOne|deleteMany|countDocuments|replaceOne)\(\s*"
|
||||
r"(?:req\.(?:body|query|params)\b"
|
||||
@@ -67,7 +70,8 @@ SHELL_DB_HEAD = (
|
||||
r"[^\n]{0,300}?\b(?:psql|mysql|mysqldump|pg_dump|pg_restore|mongosh"
|
||||
r"|mongo|mongodump|redis-cli|sqlite3|clickhouse-client)"
|
||||
r"\b(?![\"']\s*[,\]])[^\n]{0,300}?")
|
||||
SHELL_INTERP = r"(?:\$\{|\+\s*[\w$]|#\{|\{\w+\}|%s|"
|
||||
SHELL_INTERP = (r"(?:\$\{|\+\s*[\w$]|#\{|\{\w+\}|%s"
|
||||
r"|\{\}(?=[^\"{}\n]{0,12}\"\s*,)|")
|
||||
# An uppercase $NAME is a shell env reference, except in PHP where every
|
||||
# $name is a variable.
|
||||
SHELL_DB = re.compile(SHELL_DB_HEAD + SHELL_INTERP + r"\$[a-z_])")
|
||||
|
||||
@@ -31,12 +31,15 @@ import urllib.request
|
||||
from dataclasses import asdict, dataclass
|
||||
from urllib.parse import quote, urljoin, urlsplit
|
||||
|
||||
from checks.common import printable
|
||||
|
||||
READ_METHODS = {"GET", "HEAD", "OPTIONS"}
|
||||
FAKE_ORIGIN = "https://guardrails-probe.invalid"
|
||||
DEFAULT_BUDGET = 200
|
||||
BURST_CAP = 25
|
||||
BODY_CAP = 1_000_000
|
||||
SCRIPT_CAP = 10
|
||||
BUDGET_EXHAUSTED = "request budget exhausted"
|
||||
SCRIPT_SRC = re.compile(r"<script[^>]{1,300}src=[\"']?([^\"'\s>]+)", re.I)
|
||||
MAP_COMMENT = re.compile(
|
||||
r"^[ \t]*//[#@][ \t]*sourceMappingURL=(\S+)", re.M)
|
||||
@@ -50,6 +53,10 @@ SENSITIVE_KEY = re.compile(
|
||||
r"|service_?role"
|
||||
r"|private_?key|(?:access|refresh|id|auth|api|session)_token|jwt|ssn"
|
||||
r"|iban|cvv|card_?number)(?:$|_)")
|
||||
METADATA_KEY = re.compile(
|
||||
r"^(?:has|is|needs|requires)_"
|
||||
r"|_(?:type|count|length|url|expires?(?:_in)?|expiry|ttl|at)$"
|
||||
r"|^(?:next_page_token|page_token|cursor|csrf_?token|xsrf_?token)$")
|
||||
IDENT = re.compile(r"[A-Za-z_][\w-]{0,31}") # used with fullmatch
|
||||
|
||||
|
||||
@@ -84,12 +91,25 @@ def _stays_on_host(old_url, new_url):
|
||||
== ("http", "https"))
|
||||
|
||||
|
||||
class _BudgetExhausted(urllib.error.URLError):
|
||||
"""The shared request budget ran out in the middle of a redirect."""
|
||||
|
||||
|
||||
class _SameHost(urllib.request.HTTPRedirectHandler):
|
||||
"""Follow a same-host redirect, drop one that leaves the host."""
|
||||
"""Follow a same-host redirect, drop one that leaves the host.
|
||||
|
||||
Each followed hop is a request: it is charged to the session budget.
|
||||
"""
|
||||
|
||||
def __init__(self, session):
|
||||
self.session = session
|
||||
|
||||
def redirect_request(self, req, fp, code, msg, headers, newurl):
|
||||
if not _stays_on_host(req.full_url, newurl):
|
||||
return None
|
||||
if self.session.budget <= 0:
|
||||
raise _BudgetExhausted(BUDGET_EXHAUSTED)
|
||||
self.session.budget -= 1
|
||||
return super().redirect_request(req, fp, code, msg, headers, newurl)
|
||||
|
||||
|
||||
@@ -97,7 +117,7 @@ class Session:
|
||||
"""Holds the redirect-guarded opener and the shared request budget."""
|
||||
|
||||
def __init__(self, timeout=8, budget=DEFAULT_BUDGET):
|
||||
self.opener = urllib.request.build_opener(_SameHost())
|
||||
self.opener = urllib.request.build_opener(_SameHost(self))
|
||||
self.timeout = timeout
|
||||
self.budget = budget
|
||||
|
||||
@@ -115,7 +135,7 @@ def _response_from(resp):
|
||||
|
||||
def fetch(session, method, url, headers=None, body=None):
|
||||
if session.budget <= 0:
|
||||
return Response(None, {}, b"", "request budget exhausted")
|
||||
return Response(None, {}, b"", BUDGET_EXHAUSTED)
|
||||
session.budget -= 1
|
||||
req = urllib.request.Request(
|
||||
url, data=body, method=method, headers=headers or {})
|
||||
@@ -124,6 +144,8 @@ def fetch(session, method, url, headers=None, body=None):
|
||||
return _response_from(resp)
|
||||
except urllib.error.HTTPError as exc:
|
||||
return Response(exc.code, exc.headers, _error_body(exc))
|
||||
except _BudgetExhausted:
|
||||
return Response(None, {}, b"", BUDGET_EXHAUSTED)
|
||||
except (urllib.error.URLError, OSError, ValueError,
|
||||
http.client.HTTPException) as exc:
|
||||
reason = str(exc)
|
||||
@@ -140,20 +162,37 @@ def _verdict(ctx, response, is_open, msgs):
|
||||
return Probe(domain, pid, verdict, target, note)
|
||||
|
||||
|
||||
def _nonempty_rows(body):
|
||||
try:
|
||||
data = json.loads(body or b"[]")
|
||||
except ValueError:
|
||||
return False
|
||||
return isinstance(data, list) and len(data) > 0
|
||||
def _rows_state(resp):
|
||||
"""rows | empty | unreadable: what a 200 body says about table rows."""
|
||||
if not resp.body or len(resp.body) >= BODY_CAP:
|
||||
return "unreadable"
|
||||
data, is_json = _json_body(resp)
|
||||
if not is_json:
|
||||
return "unreadable"
|
||||
return "rows" if isinstance(data, list) and data else "empty"
|
||||
|
||||
|
||||
def _rows_verdict(ctx, resp, msgs):
|
||||
"""ctx is (domain, id, target); msgs is (rows_note, empty_note)."""
|
||||
if resp.error or resp.status in (401, 403, 404):
|
||||
return _verdict(ctx, resp, False, msgs)
|
||||
domain, pid, target = ctx
|
||||
if resp.status != 200:
|
||||
note = f"HTTP {resp.status}: not confirmed"
|
||||
return Probe(domain, pid, "SKIPPED", target, note)
|
||||
state = _rows_state(resp)
|
||||
if state == "unreadable":
|
||||
note = (f"HTTP {resp.status}: body not readable as JSON rows, "
|
||||
"not confirmed")
|
||||
return Probe(domain, pid, "SKIPPED", target, note)
|
||||
return _verdict(ctx, resp, state == "rows", msgs)
|
||||
|
||||
|
||||
def _table_probe(session, base, table, headers):
|
||||
target = f"{base}/rest/v1/{quote(table)}?select=*&limit=1"
|
||||
resp = fetch(session, "GET", target, headers)
|
||||
return _verdict(
|
||||
return _rows_verdict(
|
||||
("supabase", f"anon-read:{table}", target), resp,
|
||||
resp.status == 200 and _nonempty_rows(resp.body),
|
||||
("anon request returned rows", "anon request returned no row"))
|
||||
|
||||
|
||||
@@ -173,9 +212,8 @@ def probe_supabase(session, plan):
|
||||
def _cross_user_probe(session, base, path, headers):
|
||||
target = base + path
|
||||
resp = fetch(session, "GET", target, headers)
|
||||
return _verdict(
|
||||
return _rows_verdict(
|
||||
("supabase", f"cross-user:{path}", target), resp,
|
||||
resp.status == 200 and _nonempty_rows(resp.body),
|
||||
("second user's token returned these rows",
|
||||
"second user's token returned nothing"))
|
||||
|
||||
@@ -239,9 +277,13 @@ def _burst_codes(session, url, method, body, count):
|
||||
return codes
|
||||
|
||||
|
||||
def _rate_verdict(target, codes):
|
||||
def _rate_verdict(target, codes, count):
|
||||
sent = len([c for c in codes if c is not None])
|
||||
throttled = 429 in codes
|
||||
if not throttled and sent < count:
|
||||
return Probe("hosting", "rate-limit", "SKIPPED", target,
|
||||
"burst cut short (budget or transport), "
|
||||
"rate limit not confirmed")
|
||||
note = (f"sent {sent}, {'got a 429' if throttled else 'no 429'}: "
|
||||
+ ("throttled" if throttled else "no 429 seen on this route"))
|
||||
return Probe("hosting", "rate-limit",
|
||||
@@ -253,10 +295,11 @@ def probe_rate_limit(session, plan):
|
||||
if not cfg:
|
||||
return []
|
||||
target = plan["base_url"].rstrip("/") + cfg["path"]
|
||||
count = min(cfg.get("count", 20), BURST_CAP)
|
||||
codes = _burst_codes(
|
||||
session, target, cfg.get("method", "GET").upper(),
|
||||
cfg.get("body"), min(cfg.get("count", 20), BURST_CAP))
|
||||
return [_rate_verdict(target, codes)]
|
||||
cfg.get("body"), count)
|
||||
return [_rate_verdict(target, codes, count)]
|
||||
|
||||
|
||||
def _same_host(context, url):
|
||||
@@ -292,30 +335,47 @@ def _map_candidates(script_url, resp):
|
||||
return list(dict.fromkeys(urls))
|
||||
|
||||
|
||||
def _status_outcome(resp, target):
|
||||
"""Outcome of a fetched map candidate that answered with a status."""
|
||||
if resp.status == 200 and len(resp.body) >= BODY_CAP:
|
||||
return ("note", target, "map answered 200 but its body is over "
|
||||
"1 MB: sources not confirmed, fetch it by hand")
|
||||
if resp.status == 200 and b'"sources"' in resp.body:
|
||||
return "open", target, "source map served with its sources"
|
||||
if resp.status in (200, 404, 410):
|
||||
return "negative", target, ""
|
||||
return ("note", target,
|
||||
f"HTTP {resp.status}: map not readable, not confirmed")
|
||||
|
||||
|
||||
def _map_outcome(session, script_url, url):
|
||||
"""(kind, target, note), kind in open | negative | note | error."""
|
||||
if url.startswith("data:"):
|
||||
return "open", script_url, "inline source map in the script"
|
||||
target = _same_host(script_url, url)
|
||||
if not target:
|
||||
return "note", url, "map url on another host, not fetched"
|
||||
resp = fetch(session, "GET", target)
|
||||
if resp.error or resp.status is None:
|
||||
return "error", target, resp.error or "no response"
|
||||
return _status_outcome(resp, target)
|
||||
|
||||
|
||||
def _map_verdict(session, script_url, candidates):
|
||||
"""(verdict, target, note) over the candidate map urls, fail closed."""
|
||||
"""(verdict, target, note) over every candidate map url, fail closed."""
|
||||
seen, notes = set(), []
|
||||
for url in candidates:
|
||||
if url.startswith("data:"):
|
||||
if "base64," not in url:
|
||||
continue
|
||||
return "OPEN", script_url, "inline source map in the script"
|
||||
target = _same_host(script_url, url)
|
||||
if not target:
|
||||
return "SKIPPED", url, "map url on another host, not fetched"
|
||||
resp = fetch(session, "GET", target)
|
||||
if resp.error or resp.status is None:
|
||||
return "ERROR", target, resp.error or "no response"
|
||||
if 300 <= resp.status < 400:
|
||||
return ("SKIPPED", target, f"HTTP {resp.status}: map redirect "
|
||||
"dropped (off host or downgraded), not followed")
|
||||
if resp.status == 403 or (resp.status or 0) >= 500:
|
||||
return ("SKIPPED", target,
|
||||
f"HTTP {resp.status}: map not readable, not confirmed")
|
||||
if resp.status == 200 and len(resp.body) >= BODY_CAP:
|
||||
return ("SKIPPED", target, "map answered 200 but its body is "
|
||||
"over 1 MB: sources not confirmed, fetch it by hand")
|
||||
if resp.status == 200 and b'"sources"' in resp.body:
|
||||
return "OPEN", target, "source map served with its sources"
|
||||
key = _same_host(script_url, url) or url
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
kind, target, note = _map_outcome(session, script_url, url)
|
||||
if kind in ("open", "error"):
|
||||
return kind.upper(), target, note
|
||||
if kind == "note":
|
||||
notes.append(("SKIPPED", target, note))
|
||||
if notes:
|
||||
return notes[0]
|
||||
return "CLOSED", script_url, "no source map served for this script"
|
||||
|
||||
|
||||
@@ -379,8 +439,13 @@ def _snake(key):
|
||||
return re.sub(r"(?<=[a-z0-9])(?=[A-Z])", "_", key).replace("-", "_").lower()
|
||||
|
||||
|
||||
def _is_sensitive(key):
|
||||
name = _snake(key)
|
||||
return bool(SENSITIVE_KEY.search(name)) and not METADATA_KEY.search(name)
|
||||
|
||||
|
||||
def _sensitive_keys(data):
|
||||
return sorted({k for k in _keys(data) if SENSITIVE_KEY.search(_snake(k))})
|
||||
return sorted({k for k in _keys(data) if _is_sensitive(k)})
|
||||
|
||||
|
||||
def _key_note(keys):
|
||||
@@ -447,15 +512,10 @@ def _tally(results):
|
||||
for v in ("OPEN", "CLOSED", "SKIPPED", "ERROR")}
|
||||
|
||||
|
||||
def _printable(text):
|
||||
"""Control characters of a remote or plan string never reach the tty."""
|
||||
return re.sub(r"[\x00-\x1f\x7f-\x9f]", "?", str(text))
|
||||
|
||||
|
||||
def _probe_row(probe):
|
||||
return (f" {probe.verdict:<8} {probe.domain:<9} "
|
||||
f"{_printable(probe.id)} {_printable(probe.target)}\n"
|
||||
f" {_printable(probe.message)}")
|
||||
f"{printable(probe.id)} {printable(probe.target)}\n"
|
||||
f" {printable(probe.message)}")
|
||||
|
||||
|
||||
def _summary(results):
|
||||
@@ -466,7 +526,7 @@ def _summary(results):
|
||||
|
||||
def _render_text(plan, results):
|
||||
body = [_probe_row(p) for p in results]
|
||||
head = [_printable(f"guardrails live probe: {plan['base_url']}"),
|
||||
head = [printable(f"guardrails live probe: {plan['base_url']}"),
|
||||
f"probes: {len(results)} (read-only, bounded budget)"]
|
||||
return "\n".join(head + [""] + body + [_summary(results)])
|
||||
|
||||
@@ -527,6 +587,23 @@ def _exposure_error(cfg):
|
||||
return ""
|
||||
|
||||
|
||||
def _supabase_error(plan):
|
||||
cfg = plan.get("supabase")
|
||||
if cfg is None:
|
||||
return ""
|
||||
if not isinstance(cfg, dict):
|
||||
return "supabase must be an object"
|
||||
paths, tables = cfg.get("cross_user_paths"), cfg.get("tables")
|
||||
if paths is not None and not isinstance(paths, list):
|
||||
return "supabase.cross_user_paths must be a list"
|
||||
if tables is not None and not (isinstance(tables, list) and all(
|
||||
isinstance(t, str) for t in tables)):
|
||||
return "supabase.tables must be a list of strings"
|
||||
if (tables or paths) and not _http_url(cfg.get("url")):
|
||||
return "supabase.url must be an http(s) url"
|
||||
return ""
|
||||
|
||||
|
||||
def _shape_error(plan):
|
||||
rate = plan.get("rate_limit", {})
|
||||
if not isinstance(rate, dict):
|
||||
@@ -536,13 +613,7 @@ def _shape_error(plan):
|
||||
if plan.get("routes") is not None and \
|
||||
not isinstance(plan["routes"], list):
|
||||
return "routes must be a list"
|
||||
if plan.get("supabase") is not None and \
|
||||
not isinstance(plan["supabase"], dict):
|
||||
return "supabase must be an object"
|
||||
paths = _obj(plan, "supabase").get("cross_user_paths")
|
||||
if paths is not None and not isinstance(paths, list):
|
||||
return "supabase.cross_user_paths must be a list"
|
||||
return ""
|
||||
return _supabase_error(plan)
|
||||
|
||||
|
||||
def _plan_error(plan):
|
||||
@@ -567,17 +638,25 @@ def parse_args(argv):
|
||||
return parser.parse_args(argv)
|
||||
|
||||
|
||||
def _refusal(plan):
|
||||
hosts = [plan["base_url"]]
|
||||
supabase_url = _obj(plan, "supabase").get("url")
|
||||
if supabase_url:
|
||||
hosts.append(supabase_url)
|
||||
owned = " and ".join(printable(h) for h in hosts)
|
||||
return ("refused: pass --authorized to confirm you own "
|
||||
f"{owned} and may probe {'them' if len(hosts) > 1 else 'it'}")
|
||||
|
||||
|
||||
def main(argv):
|
||||
args = parse_args(argv)
|
||||
plan, error = _load_plan(args.plan)
|
||||
error = error or _plan_error(plan or {})
|
||||
if error:
|
||||
print(f"usage error: {error}", file=sys.stderr)
|
||||
print(f"usage error: {printable(error)}", file=sys.stderr)
|
||||
return 2
|
||||
if not args.authorized:
|
||||
print("refused: pass --authorized to confirm you own "
|
||||
f"{_printable(plan['base_url'])} and may probe it",
|
||||
file=sys.stderr)
|
||||
print(_refusal(plan), file=sys.stderr)
|
||||
return 2
|
||||
results = run_probes(Session(), plan)
|
||||
render = _render_json if args.json else _render_text
|
||||
|
||||
@@ -19,7 +19,7 @@ from collections import Counter
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from checks.common import Project
|
||||
from checks.common import Project, printable
|
||||
|
||||
SEVERITY_RANK = {"FAIL": 0, "WARN": 1, "MANUAL": 2}
|
||||
GATED = ("FAIL", "WARN")
|
||||
@@ -101,6 +101,11 @@ def _summary_line(findings):
|
||||
f"{n} {sev}" for sev, n in totals.items())
|
||||
|
||||
|
||||
def _safe_text(text):
|
||||
"""Control characters from file names and messages never reach the tty."""
|
||||
return "\n".join(printable(line) for line in text.split("\n"))
|
||||
|
||||
|
||||
def render_text(project, cat, result):
|
||||
lines = [
|
||||
f"{cat.name} scan: {project.root} ({len(project.files)} files)",
|
||||
@@ -110,7 +115,7 @@ def render_text(project, cat, result):
|
||||
lines.append("not applicable: " + ", ".join(result.skipped))
|
||||
for domain in result.applicable:
|
||||
lines += _domain_lines(domain, result.findings)
|
||||
return "\n".join(lines + [_summary_line(result.findings)])
|
||||
return _safe_text("\n".join(lines + [_summary_line(result.findings)]))
|
||||
|
||||
|
||||
def render_json(project, result):
|
||||
@@ -236,7 +241,7 @@ def _scan(cat, args):
|
||||
print(text)
|
||||
return 1 if any(f.severity == "FAIL" for f in result.findings) else 0
|
||||
for _, lines in checks:
|
||||
print("\n".join(lines))
|
||||
print(_safe_text("\n".join(lines)))
|
||||
return 0 if all(ok for ok, _ in checks) else 1
|
||||
|
||||
|
||||
@@ -244,6 +249,6 @@ def main(argv, cat):
|
||||
args = parse_args(argv, cat)
|
||||
error = _usage_error(cat, args)
|
||||
if error:
|
||||
print(f"usage error: {error}", file=sys.stderr)
|
||||
print(_safe_text(f"usage error: {error}"), file=sys.stderr)
|
||||
return 2
|
||||
return _scan(cat, args)
|
||||
Reference in new issue
Block a user