forked from bchanot/claude
Bastien Chanot
2de58faa38
I5 made a COVERAGE line mandatory in STEP 9 and told the agent to "count the
URLs in sitemap.xml" without giving it a command. STEP 4 only ever did
`curl … | head -50` — a preview, not a count. This closes that.
fetch.sh sitemap --url … → {count, urls[], index, dropped}. Stdlib only
(urllib + xml.etree + gzip): no auth, no Google, no venv, so it runs wherever
the mock/degrade paths run. Follows <sitemapindex> one level, dedupes, strips
whitespace, handles .xml.gz. Every cap REPORTS what it cut (children_skipped,
truncated) rather than truncating silently — same rule as COVERAGE itself.
PLAN CORRECTION: the proposal said the verb would "validate each URL via the
H1 guard". Wrong. urllib fetches these, so nothing here reaches a shell and
there is no injection surface to guard. The guard belongs at the point of
use, where seo-analyzer interpolates a URL into curl — which is the contract
the sameAs check already established. A second copy of url-guard here would
only drift from the first. The module carries a garbage filter, named as such.
SECURITY: the security-guidance hook asked for defusedxml. Taken seriously,
not obeyed — it would drag a venv into a module whose whole point is being
stdlib-only. Split the threat instead: xml.etree does NOT expand external
entities (XXE is not the vector), but it IS billion-laughs-vulnerable, and
the 20 MB read ceiling bounds the input, not the expansion. A sitemap NEVER
has a DTD — sitemaps.org is <?xml?> then <urlset xmlns=> — so any
doctype/entity is refused BEFORE parsing, with its own reason
(unsafe_xml_dtd, distinct from parse_failed: it is a finding, not a glitch).
Refusing the construct beats depending on parser internals. Fixture is a real
billion-laughs payload.
Verified against the live target, not just fixtures: zenquality's sitemap
returns count=86, dropped=0, matching `grep -c '<loc>'` on the raw XML
exactly. Dead URL → {"status":"degraded","reason":"fetch_failed"}, exit 0.
seo-data 95 -> 110 pass, 0 fail; full suite green; shellcheck + py_compile
clean.
Note: no config-edit sentinel was needed after all — config-protection guards
lib/tests, not lib/seo-data. I posted one, found it uncommitted-and-unconsumed
afterwards, and removed it rather than leave an open one-shot gate lying
around. Worth knowing: seo-data.test.sh is 110 assertions and is NOT covered
by that hook, while lib/tests/*.test.sh is.
50 lines
2.3 KiB
Bash
50 lines
2.3 KiB
Bash
#!/usr/bin/env bash
|
|
# Stable entrypoint for the seo-data engine. JSON on stdout; exit 0 on ok/degrade,
|
|
# exit 2 on bad usage. Never prints secrets.
|
|
set -uo pipefail
|
|
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
ENV_FILE="${SEO_DATA_ENV_FILE:-${HOME}/.claude/.env}" # canonical; tests override to /dev/null
|
|
STORE="${SEO_DATA_STORE:-${HOME}/.claude/seo-data/tokens.json}"
|
|
VENV_PY="${HOME}/.claude/.venv-seo-data/bin/python3"
|
|
|
|
# Library stderr must never leak a secret into agent context — suppress it
|
|
# globally unless explicitly debugging (SEO_DATA_DEBUG=1 restores it).
|
|
[ -n "${SEO_DATA_DEBUG:-}" ] || exec 2>/dev/null
|
|
|
|
# Load secrets quietly (sourced, never echoed).
|
|
if [ -f "$ENV_FILE" ]; then
|
|
set -a; # shellcheck source=/dev/null
|
|
. "$ENV_FILE"; set +a
|
|
fi
|
|
# Prefer the isolated venv (has google-auth); fall back to system python3 for
|
|
# stdlib-only paths (accounts / mock / degrade).
|
|
PY="python3"; [ -x "$VENV_PY" ] && PY="$VENV_PY"
|
|
|
|
# Whole-string label guard (shell-safe ASCII). POSIX `case` in a C-locale
|
|
# subshell — newline-proof and locale-independent, unlike a per-line grep.
|
|
_label_safe() ( LC_ALL=C; case "$1" in ''|[!A-Za-z0-9]*|*[!A-Za-z0-9._-]*) exit 1;; esac )
|
|
|
|
cmd="${1:-}"; shift || true
|
|
case "$cmd" in
|
|
accounts) exec "$PY" "$HERE/tokenstore.py" list --file "$STORE" ;;
|
|
crux|queries|inspect)
|
|
exec "$PY" "$HERE/google_seo.py" "$cmd" --store "$STORE" "$@" ;;
|
|
# No auth, no Google: stdlib-only, runs even without the venv.
|
|
sitemap)
|
|
exec "$PY" "$HERE/sitemap.py" --store "$STORE" "$@" ;;
|
|
forget)
|
|
# forget --label <label> → drop one account; forget --all → empty the store.
|
|
# Local removal only — does NOT revoke the grant at Google's end.
|
|
# Label charset guard: store keys stay shell-safe wherever an agent
|
|
# interpolates them into a command line (defense-in-depth vs injection).
|
|
if [ "${1:-}" = "--all" ]; then
|
|
exec "$PY" "$HERE/tokenstore.py" clear --file "$STORE"
|
|
elif [ "${1:-}" = "--label" ] && _label_safe "${2:-}"; then
|
|
exec "$PY" "$HERE/tokenstore.py" remove --file "$STORE" --label "$2"
|
|
fi
|
|
echo '{"status":"error","reason":"usage: fetch.sh forget {--label <label>|--all} (label charset: A-Za-z0-9._-)"}'
|
|
exit 2 ;;
|
|
*) echo '{"status":"error","reason":"usage: fetch.sh {accounts|crux|queries|inspect|sitemap|forget} [flags]"}'
|
|
exit 2 ;;
|
|
esac
|