forked from bchanot/claude
feat(seo-data): C3 — internal link graph; orphans + click depth, measured
seo-analyzer.md:613 asks "Every important page reachable within 3 clicks?"
and :616 asks "Orphan pages (no inbound internal links)?". Neither ever had a
command — same shape as the sameAs check before W3. This is that command.
My earlier reservation ("costs a lot of network") was wrong and the
measurement killed it: 24 pages in 2.7s, 86 in 3.8s. Cheap enough to always
run on FULL.
EXHAUSTIVE OR NOTHING is the design constraint, not a nicety. Orphans cannot
be sampled: proving a page has no inbound link means having read every other
page. So when the crawl is capped or any page fails, orphans are WITHHELD —
`orphans_withheld: true` and no list. A false orphan ("page X has no inbound
links" when it does) sends a client fixing what is not broken; that is the
worst finding this tool could emit. The cap does not degrade the result, it
invalidates it.
SPA refusal: on a client-rendered site the links are not in the HTML and
every page reads as orphaned. That is catastrophic, so an empty graph returns
degraded/no_links_in_html instead of a full false-positive list. No JS
rendering by design — that is the R1/R2 arbitration, not something to smuggle
in here.
Verified against BOTH live sites and against a planted failure, because two
clean results are not evidence a detector detects:
- native PHP: 24 pages, 335 links, depth 2, 0 orphans
- Astro: 86 pages, 2015 links, depth 2, 0 orphans
- fixture with a planted orphan + a 4-click chain: both found. Filters proven
on real shapes seen live — /css/main.css?v=1778157313, #anchors, mailto:,
tel:, external hosts, .png. /b/ in markup vs /b in sitemap unify to one node
rather than a phantom orphan pair.
Fixed a flaw in my own mock while writing that test: a single page.html
fixture cannot express a GRAPH (every node gets identical links), so the mock
is now pages.json = {url: html}.
Verified: seo-data 122 -> 136 pass, 0 fail; full suite green; shellcheck +
py_compile clean.
This commit is contained in:
@@ -146,6 +146,30 @@ fetch.sh sitemap --url https://ex.com/sitemap.xml
|
||||
internals AND keeps this stdlib-only; defusedxml would drag in a venv for
|
||||
a document type that has no legitimate DTD.
|
||||
|
||||
fetch.sh linkgraph --url https://ex.com/sitemap.xml [--max 500]
|
||||
→ {"status":"ok","source":"linkgraph","pages_crawled":86,"pages_failed":0,
|
||||
"total_internal_links":2015,"capped":false,"max_depth":2,
|
||||
"orphans":[…],"beyond_3_clicks":[…],"unreachable":[…]}
|
||||
→ {"status":"ok",…,"orphans_withheld":true,"reason_withheld":"crawl incomplete…"}
|
||||
→ {"status":"degraded","reason":"no_links_in_html"|"no_pages_fetched"|…}
|
||||
|
||||
Answers seo-analyzer.md:613 ("reachable within 3 clicks?") and :616 ("orphan
|
||||
pages?") — asked since forever, never computed. Stdlib only (urllib +
|
||||
html.parser + urljoin), no auth. Measured: 24 pages in 2.7s, 86 in 3.8s.
|
||||
• EXHAUSTIVE OR NOTHING. Orphans cannot be sampled: proving no inbound
|
||||
link means having read every other page. If the crawl is capped or any
|
||||
page failed, orphans are WITHHELD, never truncated — a false orphan
|
||||
sends a client fixing what is not broken.
|
||||
• no_links_in_html = a JS-rendered site, not a link-less one. Every page
|
||||
would read as orphaned, so it REFUSES rather than report that. Does not
|
||||
render JS by design (see the R1/R2 arbitration).
|
||||
• Filters what a link graph must never hold: assets (seen live:
|
||||
/css/main.css?v=1778157313), #anchors, mailto:/tel:/javascript:, other
|
||||
hosts. Normalises the trailing slash so /blog and /blog/ are one node
|
||||
rather than a phantom orphan pair.
|
||||
• Mock is pages.json ({url: html}), not a single page.html: one fixture
|
||||
cannot express a graph — every node would carry identical links.
|
||||
|
||||
fetch.sh forget --label client-a
|
||||
→ {"status":"ok","removed":true|false} # false = label wasn't in the store
|
||||
|
||||
|
||||
Reference in New Issue
Block a user