feat(seo-data): C1b — sitemap verb, the denominator COVERAGE never had
I5 made a COVERAGE line mandatory in STEP 9 and told the agent to "count the
URLs in sitemap.xml" without giving it a command. STEP 4 only ever did
`curl … | head -50` — a preview, not a count. This closes that.
fetch.sh sitemap --url … → {count, urls[], index, dropped}. Stdlib only
(urllib + xml.etree + gzip): no auth, no Google, no venv, so it runs wherever
the mock/degrade paths run. Follows <sitemapindex> one level, dedupes, strips
whitespace, handles .xml.gz. Every cap REPORTS what it cut (children_skipped,
truncated) rather than truncating silently — same rule as COVERAGE itself.
PLAN CORRECTION: the proposal said the verb would "validate each URL via the
H1 guard". Wrong. urllib fetches these, so nothing here reaches a shell and
there is no injection surface to guard. The guard belongs at the point of
use, where seo-analyzer interpolates a URL into curl — which is the contract
the sameAs check already established. A second copy of url-guard here would
only drift from the first. The module carries a garbage filter, named as such.
SECURITY: the security-guidance hook asked for defusedxml. Taken seriously,
not obeyed — it would drag a venv into a module whose whole point is being
stdlib-only. Split the threat instead: xml.etree does NOT expand external
entities (XXE is not the vector), but it IS billion-laughs-vulnerable, and
the 20 MB read ceiling bounds the input, not the expansion. A sitemap NEVER
has a DTD — sitemaps.org is <?xml?> then <urlset xmlns=> — so any
doctype/entity is refused BEFORE parsing, with its own reason
(unsafe_xml_dtd, distinct from parse_failed: it is a finding, not a glitch).
Refusing the construct beats depending on parser internals. Fixture is a real
billion-laughs payload.
Verified against the live target, not just fixtures: zenquality's sitemap
returns count=86, dropped=0, matching `grep -c '<loc>'` on the raw XML
exactly. Dead URL → {"status":"degraded","reason":"fetch_failed"}, exit 0.
seo-data 95 -> 110 pass, 0 fail; full suite green; shellcheck + py_compile
clean.
Note: no config-edit sentinel was needed after all — config-protection guards
lib/tests, not lib/seo-data. I posted one, found it uncommitted-and-unconsumed
afterwards, and removed it rather than leave an open one-shot gate lying
around. Worth knowing: seo-data.test.sh is 110 assertions and is NOT covered
by that hook, while lib/tests/*.test.sh is.
This commit is contained in:
+32
-6
@@ -444,12 +444,38 @@ Fetch rendered HTML. Extract and analyze:
|
||||
## STEP 5 — ON-PAGE AUDIT `[both]`
|
||||
|
||||
**Record the denominator BEFORE sampling.** This step samples; the report
|
||||
says "audit". Count the URLs in `sitemap.xml` (fetch it in full — the
|
||||
`head -50` in STEP 4 is a preview, not a count). That count is the coverage
|
||||
denominator, and it feeds the mandatory COVERAGE line in STEP 9. No sitemap
|
||||
→ denominator unknown: say so, never let silence imply full coverage. On a
|
||||
500-page site a 12-page sample is 2.4% — the On-page score is an
|
||||
extrapolation from it, and the reader cannot know that unless you print it.
|
||||
says "audit". On a 500-page site a 12-page sample is 2.4% — the On-page score
|
||||
is an extrapolation from it, and the reader cannot know unless you print it.
|
||||
|
||||
```bash
|
||||
bash ~/.claude/lib/seo-data/fetch.sh sitemap --url "https://$DOMAIN/sitemap.xml"
|
||||
```
|
||||
|
||||
Returns `{count, urls[], index, dropped, ...}` — the coverage denominator and
|
||||
your sampling frame. It follows a `<sitemapindex>` one level, dedupes, strips
|
||||
whitespace, and handles `.xml.gz`. No auth, no venv, no Google.
|
||||
|
||||
Read it honestly:
|
||||
- `count` → the denominator for the STEP 9 COVERAGE line.
|
||||
- `dropped > 0` → entries that were not usable URLs. Worth a §14 line: a
|
||||
sitemap emitting junk is a tooling finding.
|
||||
- `children_failed > 0` or `children_skipped` → the frame is incomplete. Say
|
||||
so; do NOT present a partial denominator as the total.
|
||||
- `status: degraded` → denominator UNKNOWN. Print that, never let silence
|
||||
imply full coverage. `reason: unsafe_xml_dtd` is not a glitch — a sitemap
|
||||
carrying a DTD is broken tooling or a billion-laughs aimed at the auditor.
|
||||
Report it as a finding.
|
||||
|
||||
**Guard every URL before it reaches curl.** These come from the target's own
|
||||
server, not from the operator — the one place in this audit where a remote
|
||||
file's bytes flow into a shell:
|
||||
|
||||
```bash
|
||||
U="$(bash ~/.claude/lib/url-guard.sh url "$RAW_FROM_SITEMAP")" || continue
|
||||
```
|
||||
|
||||
The verb applies a garbage filter, not that guard; the guard belongs at the
|
||||
point of use (same contract as the sameAs check in geo-analyzer).
|
||||
|
||||
### Meta tags per page (sample 5-15 key pages)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user