feat(seo-data): safe_fetch — resolve-then-pin, close DNS-rebinding + SSRF

By-principle hardening. H1's url-guard validates the NAME; urlopen then resolved
AND connected — two DNS lookups with a window a hostile authority uses to answer
PUBLIC to validation and PRIVATE (169.254.169.254 metadata, 127.0.0.1, the LAN)
to the connect. A name-level guard cannot see that rebind.

safe_fetch collapses the two lookups into one: resolve ONCE, validate every IP
(ipaddress, dual-stack v4+v6), refuse if ANY is non-public (the multi-A vector),
connect to the exact validated IP with Host+SNI+cert for the real host — no
second resolution to poison. Redirects re-validate each hop (urlopen followed
them blind). One seam: sitemap._fetch, which linkgraph/render_check/drift all
call, so every network verb inherits it.

The load-bearing property (confirmed by the security review): classification is
on the OS-resolved address (sockaddr[0]), never the URL text — so octal/hex/
decimal literals, IPv4-mapped IPv6, NAT64, 6to4 are all defeated structurally,
not by enumeration.

Better than the source idea (claude-seo url_safety.py, MIT): dual-stack (theirs
IPv4-only), no global monkeypatch so thread-safe by construction (theirs locks a
patched getaddrinfo), stdlib-only (no requests). Proven end-to-end before
writing: pinned connect keeps SNI+cert for the real host.

NOT covered, stated not silent: shell `curl` in the agent specs (separate
process, unpinnable here). Smaller surface; `curl --resolve` is a separate change.

REVIEW-SURFACED (fresh security-auditor, adversarial, VERDICT PASS) — two real
holes it found while attacking the diff, both fixed here:
- billion-laughs REOPENED in C1b: _refuse_dtd scanned only raw[:4096], so a
  >4KB leading comment pushed <!DOCTYPE past the window while ET parsed AND
  EXPANDED the entities. Proven (&lol2; → "lollollollollol"), now a full-doc
  case-insensitive scan. This is a genuine fix to already-merged C1b, not this
  feature — fixed here rather than filed, per root-cause discipline.
- 192.88.99.0/24 (6to4-relay anycast) passed is_global as public — added to an
  extra special-use deny list.

Verified: rebind-to-metadata refused BEFORE any connect (injected resolver),
multi-A public+private refused, classifier fuzzed dual-stack incl. CGNAT/6to4,
non-http scheme refused, both review fixes proven with no false positive; real
fetch still works (zenquality 86 loc, lavageangels 24) through the pinned path;
all 4 verbs work end-to-end via fetch.sh; seo-data 210 → 221 pass, 0 fail; full
suite green; shellcheck + py_compile clean.
This commit is contained in:
Bastien Chanot
2026-07-17 19:58:59 +02:00
parent 4ccfb606a8
commit b00e8ef442
5 changed files with 252 additions and 11 deletions
+17
View File
@@ -120,6 +120,23 @@ fetch.sh cannibal --account client-a --property … [--days 90] [--rows 1000]
Rows gained a `keys` list; `key` stays as keys[0], so the single-dim
consumer is untouched.
safe_fetch.py — NOT a verb; the SSRF/DNS-rebinding-safe fetcher behind
sitemap._fetch, so every network verb (sitemap, linkgraph, rendercheck,
drift) inherits it. urlopen resolved then connected — two DNS lookups, a
window a hostile authority uses to answer PUBLIC to validation and PRIVATE
(169.254.169.254 metadata, 127.0.0.1, the LAN) to the connect. This resolves
ONCE, validates every IP (ipaddress, dual-stack v4+v6), refuses if ANY is
non-public (the multi-A vector), and connects to the exact validated IP with
Host+SNI+cert for the real host — no second resolution to poison. Redirects
are followed with each hop RE-VALIDATED (urlopen followed them blind).
• Better than the source idea (claude-seo url_safety.py, MIT): dual-stack
(theirs IPv4-only), no global monkeypatch so thread-safe by construction
(theirs locks a patched socket.getaddrinfo), stdlib-only (no requests).
• Refusal raises UnsafeTarget; callers already degrade → fail-open kept.
• NOT covered, and said so: the shell `curl` in the agent specs runs in
another process, unpinnable from here. Smaller surface (fixed set vs an
operator-confirmed $DOMAIN); `curl --resolve` would close it, separate change.
fetch.sh sitemap --url https://ex.com/sitemap.xml
→ {"status":"ok","source":"sitemap","index":false,"count":86,"dropped":0,
"urls":["https://ex.com/", …]}