Bastien Chanot
fe41986be9
seo-analyzer.md:613 asks "Every important page reachable within 3 clicks?"
and :616 asks "Orphan pages (no inbound internal links)?". Neither ever had a
command — same shape as the sameAs check before W3. This is that command.
My earlier reservation ("costs a lot of network") was wrong and the
measurement killed it: 24 pages in 2.7s, 86 in 3.8s. Cheap enough to always
run on FULL.
EXHAUSTIVE OR NOTHING is the design constraint, not a nicety. Orphans cannot
be sampled: proving a page has no inbound link means having read every other
page. So when the crawl is capped or any page fails, orphans are WITHHELD —
`orphans_withheld: true` and no list. A false orphan ("page X has no inbound
links" when it does) sends a client fixing what is not broken; that is the
worst finding this tool could emit. The cap does not degrade the result, it
invalidates it.
SPA refusal: on a client-rendered site the links are not in the HTML and
every page reads as orphaned. That is catastrophic, so an empty graph returns
degraded/no_links_in_html instead of a full false-positive list. No JS
rendering by design — that is the R1/R2 arbitration, not something to smuggle
in here.
Verified against BOTH live sites and against a planted failure, because two
clean results are not evidence a detector detects:
- native PHP: 24 pages, 335 links, depth 2, 0 orphans
- Astro: 86 pages, 2015 links, depth 2, 0 orphans
- fixture with a planted orphan + a 4-click chain: both found. Filters proven
on real shapes seen live — /css/main.css?v=1778157313, #anchors, mailto:,
tel:, external hosts, .png. /b/ in markup vs /b in sitemap unify to one node
rather than a phantom orphan pair.
Fixed a flaw in my own mock while writing that test: a single page.html
fixture cannot express a GRAPH (every node gets identical links), so the mock
is now pages.json = {url: html}.
Verified: seo-data 122 -> 136 pass, 0 fail; full suite green; shellcheck +
py_compile clean.
52 lines
2.4 KiB
Bash
52 lines
2.4 KiB
Bash
#!/usr/bin/env bash
|
|
# Stable entrypoint for the seo-data engine. JSON on stdout; exit 0 on ok/degrade,
|
|
# exit 2 on bad usage. Never prints secrets.
|
|
set -uo pipefail
|
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
ENV_FILE="${SEO_DATA_ENV_FILE:-${HOME}/.claude/.env}" # canonical; tests override to /dev/null
|
|
STORE="${SEO_DATA_STORE:-${HOME}/.claude/seo-data/tokens.json}"
|
|
VENV_PY="${HOME}/.claude/.venv-seo-data/bin/python3"
|
|
|
|
# Library stderr must never leak a secret into agent context — suppress it
|
|
# globally unless explicitly debugging (SEO_DATA_DEBUG=1 restores it).
|
|
[ -n "${SEO_DATA_DEBUG:-}" ] || exec 2>/dev/null
|
|
|
|
# Load secrets quietly (sourced, never echoed).
|
|
if [ -f "$ENV_FILE" ]; then
|
|
set -a; # shellcheck source=/dev/null
|
|
. "$ENV_FILE"; set +a
|
|
fi
|
|
# Prefer the isolated venv (has google-auth); fall back to system python3 for
|
|
# stdlib-only paths (accounts / mock / degrade).
|
|
PY="python3"; [ -x "$VENV_PY" ] && PY="$VENV_PY"
|
|
|
|
# Whole-string label guard (shell-safe ASCII). POSIX `case` in a C-locale
|
|
# subshell — newline-proof and locale-independent, unlike a per-line grep.
|
|
_label_safe() ( LC_ALL=C; case "$1" in ''|[!A-Za-z0-9]*|*[!A-Za-z0-9._-]*) exit 1;; esac )
|
|
|
|
cmd="${1:-}"; shift || true
|
|
case "$cmd" in
|
|
accounts) exec "$PY" "$HERE/tokenstore.py" list --file "$STORE" ;;
|
|
crux|queries|inspect|cannibal)
|
|
exec "$PY" "$HERE/google_seo.py" "$cmd" --store "$STORE" "$@" ;;
|
|
# No auth, no Google: stdlib-only, runs even without the venv.
|
|
sitemap)
|
|
exec "$PY" "$HERE/sitemap.py" --store "$STORE" "$@" ;;
|
|
linkgraph)
|
|
exec "$PY" "$HERE/linkgraph.py" --store "$STORE" "$@" ;;
|
|
forget)
|
|
# forget --label <label> → drop one account; forget --all → empty the store.
|
|
# Local removal only — does NOT revoke the grant at Google's end.
|
|
# Label charset guard: store keys stay shell-safe wherever an agent
|
|
# interpolates them into a command line (defense-in-depth vs injection).
|
|
if [ "${1:-}" = "--all" ]; then
|
|
exec "$PY" "$HERE/tokenstore.py" clear --file "$STORE"
|
|
elif [ "${1:-}" = "--label" ] && _label_safe "${2:-}"; then
|
|
exec "$PY" "$HERE/tokenstore.py" remove --file "$STORE" --label "$2"
|
|
fi
|
|
echo '{"status":"error","reason":"usage: fetch.sh forget {--label <label>|--all} (label charset: A-Za-z0-9._-)"}'
|
|
exit 2 ;;
|
|
*) echo '{"status":"error","reason":"usage: fetch.sh {accounts|crux|queries|inspect|cannibal|sitemap|linkgraph|forget} [flags]"}'
|
|
exit 2 ;;
|
|
esac
|