feat(seo-data): C1b — sitemap verb, the denominator COVERAGE never had
I5 made a COVERAGE line mandatory in STEP 9 and told the agent to "count the
URLs in sitemap.xml" without giving it a command. STEP 4 only ever did
`curl … | head -50` — a preview, not a count. This closes that.
fetch.sh sitemap --url … → {count, urls[], index, dropped}. Stdlib only
(urllib + xml.etree + gzip): no auth, no Google, no venv, so it runs wherever
the mock/degrade paths run. Follows <sitemapindex> one level, dedupes, strips
whitespace, handles .xml.gz. Every cap REPORTS what it cut (children_skipped,
truncated) rather than truncating silently — same rule as COVERAGE itself.
PLAN CORRECTION: the proposal said the verb would "validate each URL via the
H1 guard". Wrong. urllib fetches these, so nothing here reaches a shell and
there is no injection surface to guard. The guard belongs at the point of
use, where seo-analyzer interpolates a URL into curl — which is the contract
the sameAs check already established. A second copy of url-guard here would
only drift from the first. The module carries a garbage filter, named as such.
SECURITY: the security-guidance hook asked for defusedxml. Taken seriously,
not obeyed — it would drag a venv into a module whose whole point is being
stdlib-only. Split the threat instead: xml.etree does NOT expand external
entities (XXE is not the vector), but it IS billion-laughs-vulnerable, and
the 20 MB read ceiling bounds the input, not the expansion. A sitemap NEVER
has a DTD — sitemaps.org is <?xml?> then <urlset xmlns=> — so any
doctype/entity is refused BEFORE parsing, with its own reason
(unsafe_xml_dtd, distinct from parse_failed: it is a finding, not a glitch).
Refusing the construct beats depending on parser internals. Fixture is a real
billion-laughs payload.
Verified against the live target, not just fixtures: zenquality's sitemap
returns count=86, dropped=0, matching `grep -c '<loc>'` on the raw XML
exactly. Dead URL → {"status":"degraded","reason":"fetch_failed"}, exit 0.
seo-data 95 -> 110 pass, 0 fail; full suite green; shellcheck + py_compile
clean.
Note: no config-edit sentinel was needed after all — config-protection guards
lib/tests, not lib/seo-data. I posted one, found it uncommitted-and-unconsumed
afterwards, and removed it rather than leave an open one-shot gate lying
around. Worth knowing: seo-data.test.sh is 110 assertions and is NOT covered
by that hook, while lib/tests/*.test.sh is.
This commit is contained in:
@@ -98,6 +98,32 @@ fetch.sh inspect --account client-a --property … --url https://ex.com/page
|
||||
• errors/warnings count issue INSTANCES; issues[] is deduped — the same
|
||||
issueMessage repeats across every affected item.
|
||||
|
||||
fetch.sh sitemap --url https://ex.com/sitemap.xml
|
||||
→ {"status":"ok","source":"sitemap","index":false,"count":86,"dropped":0,
|
||||
"urls":["https://ex.com/", …]}
|
||||
→ {"status":"ok","index":true,"children_total":4,"children_read":4,
|
||||
"children_failed":0,"count":312,…} # <sitemapindex>, one level deep
|
||||
→ {"status":"degraded","reason":"fetch_failed"|"parse_failed"|"no_urls"
|
||||
|"unsafe_xml_dtd"}
|
||||
|
||||
No auth, no Google, no venv: stdlib only (urllib + xml.etree + gzip).
|
||||
Gives STEP 9's COVERAGE line the denominator it was told to print and never
|
||||
had, and STEP 5 a real sampling frame. Dedupes, strips whitespace, handles
|
||||
.xml.gz. Caps: 50 children of an index, 50k URLs, 20 MB read — each cut is
|
||||
REPORTED (children_skipped / truncated), never silent.
|
||||
|
||||
• NOT a security boundary. urllib fetches these, so nothing here reaches a
|
||||
shell. The CONSUMER interpolates them into curl, so seo-analyzer runs
|
||||
lib/url-guard.sh at the point of use — same contract as the sameAs check.
|
||||
A second copy of the guard here would only drift.
|
||||
• `unsafe_xml_dtd`: a sitemap NEVER has a DTD (sitemaps.org is <?xml?> then
|
||||
<urlset xmlns=>). Any doctype/entity is refused BEFORE parsing. xml.etree
|
||||
does not expand external entities, but it IS billion-laughs-vulnerable —
|
||||
1 KB expands to gigabytes, and the 20 MB read ceiling bounds the input,
|
||||
not the expansion. Refusing the construct beats depending on parser
|
||||
internals AND keeps this stdlib-only; defusedxml would drag in a venv for
|
||||
a document type that has no legitimate DTD.
|
||||
|
||||
fetch.sh forget --label client-a
|
||||
→ {"status":"ok","removed":true|false} # false = label wasn't in the store
|
||||
|
||||
|
||||
Reference in New Issue
Block a user