feat(seo-data): H2 — drift baseline; regressions vs changes, not prose
seo-analyzer.md:1365 keeps history as "date + score + key changes" — prose the
LLM writes about its own previous prose. Lossy, unreproducible, and
machine-uncomparable, so "the redesign silently dropped 40 canonicals" is
invisible unless someone happens to notice.
drift snapshots title/description/canonical/robots/h1_count/jsonld_types per
URL and diffs them. Stdlib only, no auth.
The classification IS the feature: LOSING a signal is a regression, CHANGING
one is a change that may well be intended. The engine says which kind; the
agent judges. A reworded title is not an alert; an evaporated canonical is.
Runs over the WHOLE sitemap, never a sample — caught while designing: a drift
computed over a sample that changes between runs compares nothing.
NOT rank tracking. That is the common misread of this same feature elsewhere;
positions come from GSC `queries`. This is on-page regression detection.
Also caught in my own draft before testing: _capture reused
sm._mock("page.html"), the exact single-fixture flaw I had already fixed in
linkgraph — one fixture cannot express a multi-page snapshot, every URL would
read identical. Now pages.json, same convention.
Proved on a planted failure rather than a happy path — two clean sites would
look identical to a detector that always returns []:
v1 -> v2: canonical lost on /a, h1 + jsonld lost on /, title reworded,
/gone removed, /neuve added
→ 3 regressions, 1 change, gone/new both detected, title correctly NOT a
regression.
Store is ~/.claude/seo-data/drift/<host>.json, 0700, written via os.replace so
a crash never leaves a half-written baseline; a corrupt store degrades to
"first run" instead of killing the audit.
Verified: seo-data 144 -> 155 pass, 0 fail; full suite green.
This commit is contained in:
@@ -178,6 +178,32 @@ has "cap is reported" "$CAP" '"capped": true'
|
||||
has "capped withholds orphans" "$CAP" '"orphans_withheld": true'
|
||||
hasnt "capped emits no orphans" "$CAP" '"orphans":'
|
||||
|
||||
echo "── drift (H2) ──"
|
||||
DH="$(mktemp -d)"
|
||||
D1="$(HOME="$DH" SEO_DATA_MOCK_DIR="$SD/fixtures-drift-v1" python3 "$SD/drift.py" \
|
||||
--url https://ex.com/sitemap.xml)"
|
||||
has "first run is a baseline" "$D1" '"baseline": true'
|
||||
has "baseline captures pages" "$D1" '"pages": 3'
|
||||
hasnt "baseline diffs nothing" "$D1" '"regressions"'
|
||||
# v2: canonical lost on /a, h1+jsonld lost on /, title reworded, /gone removed,
|
||||
# /neuve added. Losses are regressions; a reworded title is not.
|
||||
D2="$(HOME="$DH" SEO_DATA_MOCK_DIR="$SD/fixtures-drift-v2" python3 "$SD/drift.py" \
|
||||
--url https://ex.com/sitemap.xml)"
|
||||
has "second run diffs" "$D2" '"baseline": false'
|
||||
has "detects removed url" "$D2" '"https://ex.com/gone"'
|
||||
has "detects added url" "$D2" '"https://ex.com/neuve"'
|
||||
has "lost canonical = regression" "$D2" '"canonical"'
|
||||
has "lost h1 = regression" "$D2" '"h1_count"'
|
||||
has "lost jsonld = regression" "$D2" '"jsonld_types"'
|
||||
# the classification IS the feature: losing a signal != changing one
|
||||
NREG="$(printf '%s' "$D2" | python3 -c 'import sys,json; print(len(json.load(sys.stdin)["regressions"]))')"
|
||||
NCHG="$(printf '%s' "$D2" | python3 -c 'import sys,json; print(len(json.load(sys.stdin)["changes"]))')"
|
||||
[ "$NREG" = "3" ] && ok "3 losses classed as regressions" \
|
||||
|| no "3 losses classed as regressions" "got $NREG"
|
||||
[ "$NCHG" = "1" ] && ok "reworded title is a change, not a regression" \
|
||||
|| no "reworded title is a change, not a regression" "got $NCHG"
|
||||
rm -rf "$DH"
|
||||
|
||||
echo "── fetch.sh ──"
|
||||
FETCH="$SD/fetch.sh"
|
||||
# SEO_DATA_ENV_FILE=/dev/null: tests must NEVER source the real ~/.claude/.env —
|
||||
|
||||
Reference in New Issue
Block a user