feat(seo-data): C1b — sitemap verb, the denominator COVERAGE never had
I5 made a COVERAGE line mandatory in STEP 9 and told the agent to "count the
URLs in sitemap.xml" without giving it a command. STEP 4 only ever did
`curl … | head -50` — a preview, not a count. This closes that.
fetch.sh sitemap --url … → {count, urls[], index, dropped}. Stdlib only
(urllib + xml.etree + gzip): no auth, no Google, no venv, so it runs wherever
the mock/degrade paths run. Follows <sitemapindex> one level, dedupes, strips
whitespace, handles .xml.gz. Every cap REPORTS what it cut (children_skipped,
truncated) rather than truncating silently — same rule as COVERAGE itself.
PLAN CORRECTION: the proposal said the verb would "validate each URL via the
H1 guard". Wrong. urllib fetches these, so nothing here reaches a shell and
there is no injection surface to guard. The guard belongs at the point of
use, where seo-analyzer interpolates a URL into curl — which is the contract
the sameAs check already established. A second copy of url-guard here would
only drift from the first. The module carries a garbage filter, named as such.
SECURITY: the security-guidance hook asked for defusedxml. Taken seriously,
not obeyed — it would drag a venv into a module whose whole point is being
stdlib-only. Split the threat instead: xml.etree does NOT expand external
entities (XXE is not the vector), but it IS billion-laughs-vulnerable, and
the 20 MB read ceiling bounds the input, not the expansion. A sitemap NEVER
has a DTD — sitemaps.org is <?xml?> then <urlset xmlns=> — so any
doctype/entity is refused BEFORE parsing, with its own reason
(unsafe_xml_dtd, distinct from parse_failed: it is a finding, not a glitch).
Refusing the construct beats depending on parser internals. Fixture is a real
billion-laughs payload.
Verified against the live target, not just fixtures: zenquality's sitemap
returns count=86, dropped=0, matching `grep -c '<loc>'` on the raw XML
exactly. Dead URL → {"status":"degraded","reason":"fetch_failed"}, exit 0.
seo-data 95 -> 110 pass, 0 fail; full suite green; shellcheck + py_compile
clean.
Note: no config-edit sentinel was needed after all — config-protection guards
lib/tests, not lib/seo-data. I posted one, found it uncommitted-and-unconsumed
afterwards, and removed it rather than leave an open one-shot gate lying
around. Worth knowing: seo-data.test.sh is 110 assertions and is NOT covered
by that hook, while lib/tests/*.test.sh is.
This commit is contained in:
@@ -81,6 +81,36 @@ has "gsc degrades w/o creds" "$DEG" '"status": "degraded"'
|
||||
has "gsc degrade reason" "$DEG" 'no_credentials'
|
||||
rm -rf "$TMP2"
|
||||
|
||||
echo "── sitemap ──"
|
||||
SM="$(SEO_DATA_MOCK_DIR="$MOCK" python3 "$SD/sitemap.py" --url https://ex.com/sitemap.xml)"
|
||||
has "sitemap ok" "$SM" '"status": "ok"'
|
||||
has "sitemap not an index" "$SM" '"index": false'
|
||||
# fixture holds 8 <loc>: 1 empty, blog twice, ftp:// and a quoted one to drop
|
||||
has "sitemap dedupes" "$SM" '"count": 4'
|
||||
has "sitemap counts drops" "$SM" '"dropped": 2'
|
||||
has "sitemap strips whitespace" "$SM" '"https://ex.com/spaced"'
|
||||
hasnt "sitemap drops non-http" "$SM" 'ftp://'
|
||||
hasnt "sitemap drops shell-meta" "$SM" 'bad"quote'
|
||||
# namespace-agnostic: real sitemaps carry sitemaps.org xmlns (+ xhtml here)
|
||||
has "sitemap reads namespaced" "$SM" '"https://ex.com/services"'
|
||||
|
||||
IDX="$(SEO_DATA_MOCK_DIR="$SD/fixtures-sitemap-index" python3 "$SD/sitemap.py" \
|
||||
--url https://ex.com/sitemap.xml)"
|
||||
has "sitemapindex detected" "$IDX" '"index": true'
|
||||
has "sitemapindex fans out" "$IDX" '"children_read": 2'
|
||||
has "sitemapindex no child fail" "$IDX" '"children_failed": 0'
|
||||
has "sitemapindex yields urls" "$IDX" '"https://ex.com/child-a"'
|
||||
|
||||
# A sitemap NEVER has a DTD. Refused at the door: xml.etree does not expand
|
||||
# external entities but IS billion-laughs-vulnerable, and the 20MB read ceiling
|
||||
# bounds the input, not the expansion. Refusing beats depending on the parser,
|
||||
# and keeps this module stdlib-only (no defusedxml, no venv).
|
||||
DTD="$(SEO_DATA_MOCK_DIR="$SD/fixtures-sitemap-dtd" python3 "$SD/sitemap.py" \
|
||||
--url https://ex.com/sitemap.xml)"
|
||||
has "billion-laughs refused" "$DTD" '"status": "degraded"'
|
||||
has "dtd reason is distinct" "$DTD" 'unsafe_xml_dtd'
|
||||
hasnt "dtd never parsed" "$DTD" '"count"'
|
||||
|
||||
echo "── fetch.sh ──"
|
||||
FETCH="$SD/fetch.sh"
|
||||
# SEO_DATA_ENV_FILE=/dev/null: tests must NEVER source the real ~/.claude/.env —
|
||||
|
||||
Reference in New Issue
Block a user