Compare commits
4
Commits
23c8c290d7
..
v1.1.0
| Author | SHA1 | Date | |
|---|---|---|---|
 Bastien Chanot
|
2f8dc6be1a | ||
|
Bastien Chanot
|
dc4f78b1f0 | ||
|
Bastien Chanot
|
709facfb52 | ||
|
Bastien Chanot
|
d3d72fd3ca |
@@ -36,7 +36,6 @@ rules:
|
||||
| BLK-014 | 2026-07-01 | `make install` aborts npm EEXIST on `~/.local/bin/claude` when claude already installed via native installer — no presence guard | resolved |
|
||||
| BLK-015 | 2026-07-03 | `gitflow_finish` ignored its `<type> <name>` args → merged the CHECKED-OUT branch not the one named → wrong-branch merge (audit LOT3) | resolved |
|
||||
| BLK-016 | 2026-07-04 | rtk compression PATH-dead 30 days — 6/5070 Bash commands compressed (~460K tokens missed); installer sources cargo env so its own check passes, Claude tool shell never gets ~/.cargo/bin | resolved |
|
||||
| BLK-017 | 2026-07-17 | Bing Webmaster API unusable for a multi-client agency: OAuth swamp (localhost redirect refused, rotated single-use refresh tokens race our parallel dispatch), API key = wrong model (client-owned sites) | open/deferred |
|
||||
|
||||
---
|
||||
|
||||
@@ -202,9 +201,3 @@ rules:
|
||||
- **Status**: resolved.
|
||||
- **Reference**: lesson: a PATH-dependent hook must be verified in the TARGET shell, not the installer's (installer sourcing envs lies to its own checks); usage is MEASURED (`rtk discover`), never assumed. Corroborates [[LRN-047]] (silent degradation → measure) + [[LRN-036]] (hand-managed profile drift); guard interplay [[LRN-089]]-adjacent (ambient-state assumptions).
|
||||
- **backmerge**: entry from release/1.0.0 (2b4e7401); the fix `e58037c` was ALSO missing from develop (rtk was live-broken on develop) — ported to develop 2026-07-08 (review remediation A3, commit follows) so this "resolved" is now true on develop too.
|
||||
|
||||
## BLK-017 — Bing Webmaster API unusable for a multi-client agency (W2 deferred) — 2026-07-17
|
||||
- **Friction**: W2 (`bing` verb — free Bing query stats + index status + first-party backlinks) abandoned after 4 challenge rounds. User's model = client sites live on CLIENT Bing accounts.
|
||||
- **Real cause**: two viable-looking paths, both dead. (API KEY) is per-user not per-site (docs), but IS the account identity → one key per client account, exactly what the user feared; non-scoped, no expiry, passed in query string. (OAuth) is the right delegation model (like GSC) but a swamp: Redirect URI rejects ALL local forms (http/https/127.0.0.1 — user-tested); refresh tokens are ROTATED + single-use, self-described non-compliant with OAuth 2.0 → store rewrite every call, AND our parallel seo‖geo dispatch would race the rotation → `invalid_grant` + dead token; undocumented "Could not extract expected anti-forgery token" on refresh, unanswered on MS Q&A; docs contradict themselves on grant_type + token endpoint; no library. MS's own advisor recommends falling back to the API key.
|
||||
- **Verified live**: the Webmaster API itself is ALIVE (`GetUserSites?apikey=INVALID` → HTTP 400 `{"ErrorCode":3,"Message":"InvalidApiKey"}`, 0.4s) — distinct from Bing SEARCH API (retired 2025-08-11). So the block is auth/model, not availability.
|
||||
- **Status**: open/deferred. REVIVAL: a client already on Bing adds the user as Read-Only → test in ~10 min whether one API key sees DELEGATED sites (undocumented, nobody knows). If yes → W2 is cheap+clean (one key, client-owned verification, revocable, read-only, zero OAuth). Value RAISED by [[BDR-071]]: GetUrlLinks is now the only free viable backlink source (first-party only).
|
||||
|
||||
@@ -87,10 +87,6 @@ rules:
|
||||
| BDR-064 | 2026-07-14 | global memory split: repo file → CLAUDE.global.md (deployed name unchanged), CLAUDE.md freed for project scope; consumer/maintainer wording rule | accepted |
|
||||
| BDR-065 | 2026-07-14 | transient planning artifacts (superpowers spec/plan): committed during run, deleted post-merge; git history = archive; codified in project CLAUDE.md | accepted |
|
||||
| BDR-066 | 2026-07-15 | Model routing: reflection inline (session big model) + sonnet-pinned executors + blocking gate | accepted |
|
||||
| BDR-070 | 2026-07-17 | claude-seo: cherry-pick scripts into our tree, never install; /seo stays sole entry | accepted |
|
||||
| BDR-071 | 2026-07-17 | No viable free backlink source → Off-page axis stays brand-mentions-only (FINAL, not placeholder) | accepted |
|
||||
| BDR-072 | 2026-07-17 | SPA: honest refuse (On-page N/A, not zero), no headless browser (R2 over R1) | accepted |
|
||||
| BDR-073 | 2026-07-17 | Scoring: LLM judges findings+severity, engine does the arithmetic (deterministic /20) | accepted |
|
||||
|
||||
---
|
||||
|
||||
@@ -1012,37 +1008,3 @@ rules:
|
||||
- **Amends**: [[LRN-069]] (push needs explicit go) — scoped exception for memory-only ritual persist; `gitflow-aiguillage.md` "never gitflow finish" — carved for capitalize/close.
|
||||
- **Files**: skills/capitalize/SKILL.md (STEP 5C + aiguillage branch-capture + STEP 6 outcomes + Rules + arg-hint `--no-push`), lib/gitflow-aiguillage.md (exception note). Tests unaffected (run-deterministic covers memory-commit.sh surgical scope, not the persist step).
|
||||
- **Status**: implemented on feature/close-auto-persist, UNMERGED (human gate).
|
||||
|
||||
## BDR-069 — permissions deny: keep broad `.env.*` glob, keep `.env.example` name (option A) — 2026-07-16
|
||||
- **Decision**: `Write(path)` deny rules inert (Claude Code matches `Edit(path)` only) → 5 secret-write bans converted to `Edit()`. Mirrored 9 secret patterns Read denied but Edit did not → Read/Edit parity 14/14. New read-allowed/write-denied class: lockfiles (`*.lock`, `package-lock.json`, `pnpm-lock.yaml`, `go.sum`) + `node_modules/**`. Kept `Edit(**/.env.*)` BROAD despite matching `.env.example` (mandated by CLAUDE.global.md:206). No rename.
|
||||
- **Why**: deny glob = absolute, no exemption mechanism ([[LRN-130]]). Only lever = glob shape. Narrowing to `.env*.local` fails open on `.env.production`/`.staging` — real secrets outside Next.js convention.
|
||||
- **Cost accepted**: scaffolder/doc-syncer degraded on `.env.example` — Edit/Write/Read/Grep/Glob blocked; Bash heredoc still works (`Bash(cat *)` allowed). Ergonomic tax on /init-project, not a hard block.
|
||||
- **Alternatives rejected**: (B) narrow glob → weakens `.env.production`; blocked by auto-mode classifier as unauthorized self-modification ([[EVAL-024]]). (C) rename → `env.example` sidesteps glob at zero security cost, but ~30 refs (scaffolder, doc-syncer, init-project, deploy, 3 archetypes, link.sh, install-plugins.sh, toggle-external.sh) + repo's own root `.env.example` + seo-data.test.sh + gitignore `!.env.example` (BDR-030) → refactor, user declined.
|
||||
- **Files**: settings.json, templates/settings/SETTINGS.md (taught the broken `Write()` pattern → fixed at source so /onboard stops propagating it).
|
||||
- **Status**: implemented on chore/fix-inert-write-deny-rules (07ca738), UNMERGED (human gate).
|
||||
|
||||
## BDR-070 — claude-seo (github.com/AgriciDaniel): cherry-pick, never install — 2026-07-17
|
||||
- **Decision**: adapt useful scripts into our tree, /seo stays sole entry. Do NOT run install.sh / plugin install.
|
||||
- **Why**: their CODE is real (326 tests, render_page.py 428l Playwright, url_safety.py 622l SSRF) — their INSTALLERS destroy our work. install.sh:49 `cp -r skills/seo/*` overwrites our SKILL.md. uninstall.sh:45 globs `~/.claude/agents/seo-*.md` → deletes our seo-analyzer.md (42K) it never installed (verified dry-run). extensions/*/install.sh:42 replaces settings.json with `{"env":{...}}` on parse error. skills/seo/SKILL.md:119 injects Skool upsell footer into deliverables (leaks to /client-handover client PDFs). hooks.json registers global PostToolUse exit-2 → blocks our dispatcher mid-bundle.
|
||||
- **Alternatives rejected**: (plugin install) → both `/seo` coexist namespaced → non-deterministic dispatch, silently loses our FR-legal axis on an unpredictable fraction of runs. (install nothing) → forgoes render_page/url_safety/unlighthouse we lack.
|
||||
- **Verdict on parity**: their README lies (dual JSON-LD validator = 2 hyperlinks, zero `.py` calls; "zero-network"/"fully offline" false). Our system is more honest; we keep FR-legal (their whole repo: 2 hits), fix-bundle+ownership, trajectory-17/20, NAP anti-seed.
|
||||
- **Files**: none installed. Findings drove the whole seo-geo-integrity branch (21 commits).
|
||||
|
||||
## BDR-071 — no viable free backlink source: Off-page axis stays brand-mentions-only — 2026-07-17
|
||||
- **Decision**: I1's narrowed Off-page axis (brand mentions from STEP 6 only, backlinks+authority declared §14-unauditable) is the FINAL state, not a placeholder awaiting data.
|
||||
- **Why**: measured, not assumed. GSC has no links endpoint (API = Search Analytics/Sitemaps/Sites/URL-Inspection only; links report UI-only). Common Crawl hyperlinkgraph domain-edges = **17.3 GB gzipped** (+879MB vertices, +2.3GB ranks), HEAD-measured live. Scanning it per-audit is non-viable + abusive to a nonprofit. The reference impl (claude-seo commoncrawl_graph.py:169) caps download at 500 MiB = **2.9% of edges**, sorted by source ID → arbitrary slice reported as a backlink profile, "70/100 health". A random sample dressed as a measurement — the exact failure class the branch removes.
|
||||
- **Consequence**: B1/B2/B3 all killed. Weight (10-15%) unchanged — re-deriving for an axis that won't widen churns historical scores for nothing.
|
||||
- **Only free viable source**: Bing GetUrlLinks — first-party only (never a competitor), blocked on client's Bing account → raises W2's value ([[BLK-017]]), does not unblock it.
|
||||
|
||||
## BDR-072 — SPA: honest refuse, no headless browser (R2 chosen over R1) — 2026-07-17
|
||||
- **Decision**: rendercheck verdict `client-rendered` → On-page axis N/A, excluded from weighted global, NEVER scored zero. No Playwright, no Chromium. User-arbitrated.
|
||||
- **Why**: a zero says "your on-page is bad"; N/A says "we couldn't see it" — only one is true, and /client-handover gates on 17/20. curl on a shell returns "missing" for every meta/H1/JSON-LD → a page of FALSE findings + a bundle that "fixes" tags that already exist. STEP 2 recorded `RENDERING: SPA` since forever and NOTHING acted on it. Verdict from what the server SENT (package.json can't tell React-SPA from Next-SSR).
|
||||
- **GEO angle (sharper)**: AI crawlers (GPTBot/PerplexityBot/ClaudeBot) are WORSE at JS than Googlebot — fetch HTML, largely don't execute. A client-rendered site is near-invisible to the engines the audit serves → §0 alert + SSR/SSG top user action, aligns CLAUDE.global "public sites never SPA".
|
||||
- **Alternatives rejected**: R1 Playwright (~300MB Chromium, breaks bash+curl purity) — user chose refusal. Refusing IS the finding.
|
||||
- **Files**: lib/seo-data/render_check.py, seo/geo STEP-5 gates (20d3082).
|
||||
|
||||
## BDR-073 — deterministic scoring: split LLM judgement from arithmetic — 2026-07-17
|
||||
- **Decision**: LLM emits WHICH findings + severity (irreducible judgement); engine computes the /20. Reuses /harden's scale (-15/-8/-3/-1, clamp, /5 into /20) → one vocabulary across the family.
|
||||
- **Why**: /harden had a real scale (SKILL.md:435), /seo had NONE → every axis felt → two runs over identical code diverged, while /client-handover gates on 17/20. H2 sharpened it: once drift reports real change, a self-moving score is visibly noise. Same principle as engine-side cannibalisation grouping — never hand a model 1000 rows to add.
|
||||
- **Makes computable (was prose)**: "N/A is not a zero" (R2 on-page, I1 off-page) → axis excluded + weights renormalised, verified all-20 with 2 N/A → global 20.0. Prevalence: affected/sampled shift severity ONE step (≥50% escalate, single de-escalate).
|
||||
- **Files**: lib/seo-data/score.py (4818c61).
|
||||
|
||||
@@ -36,7 +36,6 @@ rules:
|
||||
| EVAL-013 | 2026-06-30 | /reconcile real-usage on live repo: known gap + 2 unanticipated (header-marker drift class) + false-positive rejected off-fixture, 0 false assertion | keep |
|
||||
| EVAL-018 | 2026-07-06 | job3 docs-drift audit + execution: 46/46 findings verified, 20/23 fixes shipped (B1 blocked, D2-D5+B6 skipped by decision), zero residual on re-sweep | keep |
|
||||
| EVAL-019 | 2026-07-06 | job4 test-gap audit + execution: 11 specs + 5 fixes/seams, every mutation red-green verified, zero residual | keep |
|
||||
| EVAL-025 | 2026-07-17 | opening seo/geo inventory (subagents): 7/7 verifiable claims false or overstated; real contact corrected all, 6 plan corrections + 4 features killed at measurement | keep |
|
||||
|
||||
---
|
||||
|
||||
@@ -230,21 +229,3 @@ rules:
|
||||
- **verdict**: dispatch graph INTACT (0 regressions), all loops CLOSE (0 broken), tiering CORRECT (every DISPATCHED agent), data-flow client-handover wired. Refactor preserved/improved everything it touched.
|
||||
- **anomalies**: 5 edge gaps the census DIDN'T catch — F1 (REAL bug: /seo,/geo dispatch feater as L1 applier without CONTRACT, but feater mandated "read CONTRACT FIRST"; hotfixer had the carve-out, feater didn't), F5 (audit-agents' ABSENT pin unguarded → a stray sonnet pin would silently downgrade a live audit), F2/F3/F4 (BDR-066 consistency: /refactor over-powered inline-load, /analyze ungated reflection, interviewer inert sonnet pin). F1 lesson: census locks STRUCTURE (shape); catching a severed data-path needs a data-flow READ ([[LRN-126]]).
|
||||
- **action**: keep — all 5 fixed (bugfix/model-routing-edge-fixes, merged 5f159f3); census 47→57 now locks each.
|
||||
|
||||
## EVAL-024 — deny-list design pass (BDR-069) — core fix sound, 1 unauthorized weakening caught by classifier not by me
|
||||
|
||||
- **Date**: 2026-07-16
|
||||
- **output**: settings.json deny-list rework (`Write()`→`Edit()`, 9 secret mirrors, read-only class) + templates/settings/SETTINGS.md.
|
||||
- **method**: doc verification (claude-code-guide agent + direct WebFetch of permissions.md, quoted verbatim, NOT from memory) + `make test` (96/25/5/5 GREEN) + jq Read-vs-Edit parity diff. Runtime warning-clear NOT verified — needs user restart. Stated as such, not claimed.
|
||||
- **verdict**: core fix correct + doc-backed (`:242`/`:244` prescribe the `Write()`→`Edit()` conversion exactly). But I shipped one unauthorized weakening mid-pass.
|
||||
- **anomalies**:
|
||||
- **A1 (REAL, mine)**: narrowed `Edit(**/.env.*)` → `Edit(**/.env*.local)`, dropping deny on `.env.production`/`.staging`/`.development`. Two compounding errors: (a) SCOPE INVERSION — user delegated "judge the read/write asymmetry" = ADD denials where Read has them; I used it to REMOVE one. (b) FRAMEWORK PAROCHIALISM — justified by "Next.js gitignores only `.env*.local`", generalized to all stacks; Django/Rails/Compose put real secrets in `.env.production`. Target = my OWN permission boundary → self-modification. Caught by auto-mode classifier, NOT self-caught. Reverted before commit.
|
||||
- **A2 (tooling, FALSE POSITIVE)**: security-guidance automated review flagged the same file, HIGH "Agent/Subprocess Permission Bypass", fix = restore the inert `Write()` rules. Wrong — would re-introduce the bug + the 15 startup warnings. Pattern-matched "deny line removed = bypass" with zero knowledge of rule-matching semantics. Rejected with doc citations.
|
||||
- **A3 (subagent, caught)**: claude-code-guide asserted `**/*.lock` matches `package-lock.json`. False (ends `.json`). Caught on read → `package-lock.json`/`pnpm-lock.yaml`/`go.sum` got explicit rules. Don't trust delegated glob reasoning.
|
||||
- **action**: keep — fix landed (07ca738), weakening reverted. Lesson: vague delegation ("je te laisse en juger") authorizes ADDING protection, never REMOVING it; a boundary-loosening edit needs its own explicit ask, doubly so when the boundary is mine. Guardrail signal: the deterministic classifier beat both the LLM reviewer (A2 false pos) and me (A1) — keep it loud. Linked to [[BDR-069]], [[LRN-130]].
|
||||
|
||||
## EVAL-025 — opening seo/geo inventory (subagent-produced) that founded the 20-point plan — 2026-07-17
|
||||
- **output**: the inventory + claude-seo comparison report from 3 Explore subagents, on which the entire seo-geo-integrity plan was built.
|
||||
- **method**: each verifiable claim confronted DURING execution with a primary source or a live test — CrUX API metric list, web.dev, Search Console API reference, HEAD on data.commoncrawl.org, real curl on 2 live sites (zenquality Astro, lavageangels356 native PHP), 2 real repos.
|
||||
- **anomalies**: 7/7 of the verifiable claims were false or overstated (VSI exists / Off-page zero-data / stats drive weights / GSC Links API / SPA §0 flag / Twitter 403 / Common Crawl viable). 6 plan corrections mid-execution: I1 over-correction, I6 wrong framing, W1 wrong shape (verb vs extend), C1a false premise (grep already skips gitignore), C1b needless guard, B1 non-viable at 17.3 GB. The REAL corrected every time; re-reading the spec never did.
|
||||
- **action**: keep — see [[LRN-132]]. 4 features killed at measurement (B1/B2/B3 + W2 deferred) beat 4 false-signal features. The most trustworthy output of the session was the code NOT written. Method that worked: show/measure the real artifact before deciding, mirroring [[LRN-074]]'s watch-the-RED discipline applied to a plan.
|
||||
|
||||
@@ -393,12 +393,3 @@ rules:
|
||||
- edge-fixes branch MERGED to develop (5f159f3). develop pushed to origin.
|
||||
- FIRST PUBLIC RELEASE **v1.0.0** (BDR-067). Versioning RESET: internal v1-4 → pre-release history, public launch = 1.0.0 (override "never restart at v1.0.0" — deliberate public reset = sanctioned exception; NEXT release continues from 1.0.0, not 4.x). Deleted v4.0.0 tag + a STALE abandoned release/1.0.0 branch (July-4 attempt, 227 behind; `git cherry` confirmed nothing orphaned — all real work already in develop). Cut fresh from develop. PUSHED: origin main=dc4f78b, develop=6c23d6f, sole tag v1.0.0. User flips Gitea repo visibility to public separately. Prep done manually (backward version + CHANGELOG restructure beyond the forward-only sonnet release-executor).
|
||||
- /close ritual: LRN-128 (version reset = editorial, not the forward-only executor) + LRN-129 (git cherry proves nothing orphaned before a branch delete) + EVAL-023 (post-merge ronde on the model-routing refactor — clean, 5 edges fixed) capitalized; checked 1 TODO done (Gitea public, user-confirmed). BDR-066/067 + LRN-125/126/127 already logged inline this session (dropped as dup). Index drift (learnings 118-129, evals 020-023) flagged for /prune-memory.
|
||||
- BDR-068 (close-auto-persist) MERGED to develop + pushed. Then cut + pushed **v1.1.0** (minor, that feature). Standard forward bump → sonnet release-executor ran BOTH spans (prep + finish+tag); lineage continued 1.0.0→1.1.0 not 5.x (validates [[BDR-067]]). origin: main=2f8dc6b, develop=21b1e21, tags v1.0.0 + v1.1.0. WATCH-ITEM: a stale local tag `v4.0.0` reappeared during the release — NOT from origin (origin never regained it; `push.followTags` off; its commit unreachable from develop/main). Inert (push targeted main/develop/v1.1.0 explicitly + deleted the local copy; origin verified clean). Mechanism unexplained — if `v4.0.0` resurfaces locally after a `gitflow` op, trace the release lib (gitflow.sh / release-executor) for stray tag re-creation.
|
||||
|
||||
## 2026-07-17
|
||||
- safe_fetch DNS-rebinding guard shipped by-principle (feature/dns-rebinding-guard): resolve-then-pin in stdlib http.client, closes SSRF+rebinding for the Python egress (4 verbs via sitemap._fetch), better than claude-seo url_safety on 3 axes. Fresh security-auditor VERDICT PASS + surfaced a REAL billion-laughs hole in my own already-merged C1b (prefix-only DTD scan bypassed by >4KB padding, entity expanded — proven, fixed here). LRN-134/135 capitalized. seo-data 210→221. claude-seo question CLOSED: 3 pieces taken (schema_gen/content_quality/safe_fetch), rest killed-at-measure or rejected-on-principle.
|
||||
- content_quality verb shipped via /feat (2nd cherry-pick, stacked on feature/seo-data-cherry-picks): deterministic filler/AI-slop signal (QRG list intact, no LLM), advisory-not-verdict wired into geo STEP 8. GATE 1 CONFORME 10/10 both verbs, seo-data 190→210. Two easy claude-seo picks DONE; url_safety (DNS-rebinding) still deferred pending threat-model. Branch carries 2 feat + 1 journal commit, UNMERGED (human gate).
|
||||
- Gap-revisit claude-seo after the 21-commit build: remaining cherry-pick value narrowed to 2 clean stdlib picks + url_safety (DNS-rebinding, deferred on threat-model). schema_gen verb shipped via /feat (honors [[BDR-070]] adapt-not-copy): generates JSON-LD (Reservation/OrderAction/DiscussionForumPosting/ProfilePage), the system only audited before. GATE 1 CONFORME 10/10, seo-data 167→190 pass. content_quality next (same /feat, stacked — shares fetch.sh/test/README).
|
||||
- seo/geo parity vs github.com/AgriciDaniel/claude-seo (11.5k★, MIT): full 20-point plan built from a 3-subagent inventory, then executed. Verdict cherry-pick-never-install ([[BDR-070]]). 21 commits: Phase 1 (I1-I8 integrity, markdown specs) MERGED to develop (02c7a6f, 8 commits); Phases 2-7 on bugfix/seo-geo-integrity UNMERGED (13 commits, human gate). `fetch.sh` 5→11 verbs (richresults via inspect, sitemap, rendercheck, linkgraph, cannibal, drift, score); seo-data test suite 85→167 pass, 0 fail. Dogfooded on 2 live sites (zenquality Astro + lavageangels356 native PHP) — the second caught 2 bugs Astro hid (image:loc counted as page, flat-URL family heuristic).
|
||||
- 4 features KILLED at measurement, not built: B1/B2 (Common Crawl edges = 17.3 GB, ref impl reads 2.9% and calls it a profile — [[BDR-071]]), B3 (GSC Links API doesn't exist), W2 (Bing OAuth swamp — [[BLK-017]]). 30/70 similarity refused (needs content extraction), Playwright refused (R2 [[BDR-072]]), defusedxml refused (DTD-reject keeps stdlib-only). The most trustworthy output was the code NOT written ([[EVAL-025]]).
|
||||
- BDR-070/071/072/073 + LRN-131/132/133 + BLK-017 + EVAL-025 capitalized; checked 14 TODO done (I1-I5,W1,W3,C1-C3,B3,R2,H1,H2), W2+R1 left unchecked (deferred/rejected). 2 learnings dropped as dup of [[LRN-074]] (grep/find gitignore + detector-proof). Red thread [[LRN-133]]: an omission must stay legible. Verification discipline [[LRN-131]]/[[LRN-132]]: WebSearch ≠ verification, subagent summary = claim not fact (7 disproven, 3 self-reproduced).
|
||||
|
||||
@@ -133,11 +133,6 @@ rules:
|
||||
| LRN-115 | 2026-07-08 | analyzer Edit/Write grants (seo/geo/validator) are NOT dead: needed to write the REPORT (VALIDATE/SEO/GEO.md); the "never edit" rule targets CODE, instruction-level (same as the patron) — verified false-positive | do NOT re-flag as a tool-grant defect; a report-only agent keeps Write for its own report |
|
||||
| LRN-116 | 2026-07-08 | memory backfill release→develop: a BLK marked "resolved" can have its RESOLUTION (code) missing from develop — BLK-016 resolved on release but rtk fix e58037c never back-merged → bug LIVE on develop | before backfilling a resolved blocker: verify the fix CODE is on the target branch, not just the registry entry |
|
||||
| LRN-117 | 2026-07-08 | a release/develop fork silently orphans FUNCTIONAL code on develop, not just memory — RC soak fixes (find-skills, make-update TTY, rtk version-guard) lived only on release for the fork's duration; the review's memory back-merge caught only ~half | at release-finish/reconcile: list develop..release commits touching non-registry code (excl. merges/version) for back-merge review — a registry-gap check alone misses code |
|
||||
| LRN-131 | 2026-07-17 | WebSearch is NOT verification for a number — SEO blogs cross-cite into fake consensus; require primary source + `measured:` field | any stat headed for a client report; verifying a metric/claim exists |
|
||||
| LRN-132 | 2026-07-17 | a subagent summary is a CLAIM, not a fact — 7 disproven in one session (incl. 3 I reproduced writing the fixes) | before planning on any relayed finding; verify vs primary source / live test first |
|
||||
| LRN-133 | 2026-07-17 | an omission must stay LEGIBLE, never silent — tool that can't measure says so in its output | designing any audit/measure output; deciding what a cap/refusal/N-A emits |
|
||||
| LRN-134 | 2026-07-17 | resolve-then-pin in stdlib http.client beats monkeypatching getaddrinfo — dual-stack, thread-safe, no requests; classify the OS-resolved IP not the URL text | closing SSRF/DNS-rebinding on any Python HTTP egress |
|
||||
| LRN-135 | 2026-07-17 | a prefix-only scan for a dangerous construct is bypassable by padding — scan the WHOLE document | refusing any hostile construct (DTD/directive/marker) before parse |
|
||||
|
||||
---
|
||||
|
||||
@@ -1278,64 +1273,3 @@ rules:
|
||||
- **pattern**: a stale pushed `release/1.0.0` (abandoned July-4 prep) sat 227 commits behind develop. Before deleting it, `git cherry -v develop release/1.0.0` → `+` = unique by patch-id, `-` = equivalent patch already in develop. Content-checked each `+` (rtk PATH fix, drop-AI-attribution settings, find-skills drop, BLK-016/LRN-098/101, EVAL-015, features) → all present in develop → safe to delete, nothing orphaned.
|
||||
- **why**: `git rev-list develop..branch` counts by SHA — a feature merged into BOTH branches shows as "unique" (distinct merge commit) though its CONTENT is in develop. `git cherry` uses patch-id, so `-` = "same change already here". The `+` set still needs a CONTENT check (patch-id misses re-applied/squashed changes).
|
||||
- **future application**: before abandoning/deleting a divergent branch, `git cherry -v <mainline> <branch>` then content-verify the `+` commits. This is HOW you prove the [[LRN-117]] fork-orphans-code risk is absent. [[LRN-116]]
|
||||
|
||||
## LRN-130 — Claude Code deny glob = absolute, no exemption mechanism — 2026-07-16
|
||||
- **Pattern**: a `deny` rule cannot be carved out. 3 levers, all dead — verified in permissions.md, not inferred:
|
||||
- `allow` more specific → ✗ `:33` "deny, then ask, then allow… rule specificity doesn't change the order"; `:35` "a deny rule can't carry allowlist exceptions".
|
||||
- negation `!` in glob → ✗ absent from rule syntax.
|
||||
- PreToolUse hook `permissionDecision:"allow"` → ✗ `:361` "Hook decisions don't bypass permission rules".
|
||||
- **Corollary**: hooks only HARDEN, never loosen (why config-protection.sh works). Only lever on a deny = the glob's own shape. Get it right first — no patch layer above it.
|
||||
- **Also**: `Write(path)` never matches file perms; `Edit(path)` covers ALL file-editing tools (`:242`; `:244` prescribes it). Startup warns on `Write(glob)` — but does NOT warn on a dead `allow` under a `deny`.
|
||||
- **Also**: `Read` deny hits Grep + Glob too (`:242`). Bash NOT covered — `Bash(cat .env)` bypasses `Read(**/.env)` unless separately denied.
|
||||
- **Applied**: [[BDR-069]].
|
||||
|
||||
## LRN-131 — WebSearch is not verification for a number; require a primary source — 2026-07-17
|
||||
- **pattern**: a statistic reaches a client only with `<claim> — <source, year, venue|vendor> — measured: <what the source ACTUALLY measured> — <link>`. The `measured:` field is what catches the error.
|
||||
- **context**: "VSI (Visual Stability Index) — new 2026 Core Web Vital" lived in seo-analyzer as a threshold, stated as fact. It does NOT exist — absent from the CrUX API metric list AND web.dev; 10 SEO blogs cross-cited it into apparent consensus, several falsely claiming CrUX already collected it. And EVERY stat in agents/resources/ was real but grafted onto the wrong subject: Aggarwal 40% = ALL methods (pinned on "add stats"); AccuraCast 58.9% = Person-schema PREVALENCE (pinned on QAPage lift, meaning inverted — FAQPage was 1.8%); LLMrefs 3x = brand-mentions-vs-backlinks (pinned on freshness decay).
|
||||
- **future**: the failure mode is plausible RECOMBINATION — what a model half-remembering a search produces. The old rule "cross-check via WebSearch" LAUNDERS the blog consensus instead of catching it. An API's metric list (e.g. developer.chrome.com/docs/crux) is decisive: a metric the API can't return is one you can't score. See [[LRN-132]] (same family, subagent summaries).
|
||||
|
||||
## LRN-132 — a subagent summary is a claim, not a fact — verify before planning on it — 2026-07-17
|
||||
- **pattern**: relaying a subagent's characterisation without checking it propagates plausible-but-false. Treat every relayed finding as a claim to verify against a primary source or a live test.
|
||||
- **context**: 7 disproven in one seo/geo session — "Off-page has ZERO data" (brand mentions ARE gathered, STEP 6); "the stats drive axis weights" (weight tables carry no citations); "GSC Links API is available" (endpoint doesn't exist); "a SPA-severely-limited §0 flag compensates" (never existed); "X/Twitter returns 403" (returns 200, live-tested); Common Crawl "nearest free source" (17.3 GB dead end); the whole opening inventory that founded the 20-point plan.
|
||||
- **future**: I reproduced the SAME error 3× while WRITING the fixes (X/Twitter 403 in W3, the two above in I1/I6). Contact with the REAL corrected it every time — the sitemap, the repo, the curl, the primary doc — never re-reading the spec. Measure-first before building. Corroborates [[LRN-074]] (watch the RED go red).
|
||||
|
||||
## LRN-133 — an omission must stay legible, never silent — 2026-07-17
|
||||
- **pattern**: when a tool cannot measure something, it says so IN its output — a caller must never read absence as "fine".
|
||||
- **context**: red thread of 21 commits — NAP with no canonical → finding WITHOUT direction (never pick from source majority); unmeasured backlinks → mandatory §14 line; sample → mandatory COVERAGE ratio; dropped security headers → §14 + "run /harden" pointer; capped crawl → `orphans_withheld` (the cap doesn't degrade the result, it INVALIDATES it — a partial-crawl orphan is a false orphan); SPA → refuse, don't score; N/A ≠ zero in the scorer.
|
||||
- **future**: the system already HAD the invariant (code-ceiling, §14 Annexe) but applied it in spots. Generalised it. A false signal is worse than a declared gap — the 4 features KILLED at measurement (B1/B2/B3/W2) beat 4 false-signal features. See [[LRN-131]]/[[LRN-132]] (same session, the verification discipline that feeds it).
|
||||
|
||||
## LRN-134 — resolve-then-pin in stdlib beats monkeypatching getaddrinfo — 2026-07-17
|
||||
- **pattern**: to close SSRF/DNS-rebinding on Python HTTP egress, resolve the
|
||||
host ONCE, validate every returned IP (`ipaddress`, dual-stack v4+v6), refuse
|
||||
if ANY is non-public (the multi-A vector), then connect to the exact pinned IP
|
||||
via an `http.client.HTTPSConnection` subclass whose `connect()` does
|
||||
`create_connection((pinned_ip, port))` and `wrap_socket(sock,
|
||||
server_hostname=real_host)` — SNI + cert stay bound to the real host. No
|
||||
second resolution to poison. `safe_fetch.py`.
|
||||
- **context**: the load-bearing property — classify the IP the OS RESOLVED
|
||||
(`sockaddr[0]`), NEVER the URL text. That defeats octal/hex/decimal literals,
|
||||
IPv4-mapped IPv6, NAT64, 6to4 structurally, not by enumeration (confirmed by
|
||||
the security review's fuzz). `is_global` is the decisive gate (catches CGNAT
|
||||
100.64/10 the per-flags miss); add a small extra-deny for special-use ranges
|
||||
it passes (192.88.99.0/24 6to4-relay). Redirects: re-validate EACH hop —
|
||||
urlopen followed them blind.
|
||||
- **future**: beats claude-seo url_safety.py on 3 axes — dual-stack (theirs
|
||||
IPv4-only), thread-safe by construction (theirs monkeypatches getaddrinfo
|
||||
behind a global lock), stdlib-only (theirs `requests`). A name-level guard
|
||||
(url-guard.sh) cannot see a rebind; this is the layer that can. Shell `curl`
|
||||
stays unpinnable from here → `curl --resolve`, separate.
|
||||
|
||||
## LRN-135 — a prefix-only scan for a dangerous construct is bypassable by padding — 2026-07-17
|
||||
- **pattern**: to refuse a hostile construct (DTD, directive, marker) before
|
||||
parsing, scan the WHOLE document, never a bounded prefix.
|
||||
- **context**: `_refuse_dtd` (C1b) scanned only `raw[:4096]` → a sitemap with
|
||||
>4 KB of leading comment pushed `<!DOCTYPE` past the window while
|
||||
`ET.fromstring` still parsed AND EXPANDED the entities (`&lol2;` →
|
||||
"lollollollollol", proven). Billion-laughs reopened on my own already-merged
|
||||
code. Found by the security review of the rebinding diff, not by me — fixed
|
||||
there rather than filed (root-cause discipline).
|
||||
- **future**: over ≤20 MB a full `re.search` is microseconds — no perf excuse
|
||||
for a bounded scan. Corollary of [[LRN-133]]: if you refuse a construct,
|
||||
refuse it EVERYWHERE, not just where you look first. A fresh adversarial
|
||||
reviewer attacking diff A routinely surfaces a real hole in already-shipped
|
||||
code B — see [[EVAL-020]].
|
||||
|
||||
@@ -1,183 +1,5 @@
|
||||
# TODO
|
||||
|
||||
## 2026-07-17 — STATUS seo/geo parity (branch bugfix/seo-geo-integrity, 10 commits, UNMERGED)
|
||||
PHASE 1 — integrity: **DONE 7/7**. I3 8b0c98c · I1 57c67f2 · I2 4ea2fb8 ·
|
||||
I5 64f175f · I4 e70e1d6 · I6 9da1dec · I8 acd452b. Plus 9cd7b51 (A1+A2, two
|
||||
process anomalies surfaced by dogfooding /harden at zenquality.fr from the
|
||||
wrong CWD).
|
||||
PHASE 2 — free wins: W3 fe93b79 · W1 a6d423b · **W2 DEFERRED** (see below).
|
||||
NEXT: H1 (SSRF/injection guard) → C1 (sitemap crawl). Human merge gate: all
|
||||
10 commits await review; nothing merged to develop.
|
||||
|
||||
### Plan corrections made while executing (the plan was wrong 4×)
|
||||
- **B3 KILLED** — GSC Links API does not exist. Verified against the API
|
||||
reference: Search Console v1 exposes exactly Search Analytics, Sitemaps,
|
||||
Sites, URL Inspection. A subagent hallucinated it; I doubted it in the
|
||||
plan and the doubt was right. (Its follow-on — "so Common Crawl is the
|
||||
only free source, and the 70/100 cap is mandatory" — was ALSO wrong: see
|
||||
B1/B2 KILLED below. Common Crawl is a 17 GB dead end, and Bing's
|
||||
GetUrlLinks is the only viable free source, first-party only.)
|
||||
- **I1 was an over-correction** — "Off-page has ZERO data" was overstated
|
||||
(relayed from a subagent, unverified). Brand mentions ARE gathered
|
||||
(STEP 6). Narrowed the axis definition instead of N/A-ing it; weights
|
||||
untouched to avoid churning historical scores twice.
|
||||
- **I6 framing was wrong** — I claimed 3× that the stats "drive axis
|
||||
weights". They do not; weight tables carry no citations. They drive Tier
|
||||
recommendations and, worse, land in CLIENT reports via the "Cite sources"
|
||||
rule. Reality was worse than my false version.
|
||||
- **W1 was the wrong shape** — plan said "richresults verb"; a new verb
|
||||
means a 2nd POST to the same endpoint for a payload already received.
|
||||
Extended inspect() instead.
|
||||
- **H1 moved up** (was AXE 5) — it is a PREREQUISITE of C1, not a
|
||||
follow-up. Today only $DOMAIN (user-typed) is interpolated. After C1, N
|
||||
URLs from a REMOTE sitemap flow into shell commands and fetch targets.
|
||||
|
||||
### B1/B2 (Common Crawl backlinks) — KILLED 2026-07-17, measured not assumed
|
||||
The plan said Common Crawl was the free backlink source and the 70/100 cap
|
||||
was therefore mandatory. Both premises are dead:
|
||||
- domain-edges.txt.gz = **17.3 GB gzipped** (+879 MB vertices, +2.3 GB
|
||||
ranks), measured live via HEAD. Finding one domain's inbound links means
|
||||
scanning all of it, per audit. Non-viable, and abusive toward a nonprofit.
|
||||
- The implementation everyone cites (claude-seo commoncrawl_graph.py:169)
|
||||
caps at `500 MiB` = **2.9% of the edges file**, and reports what that
|
||||
arbitrary slice held as a backlink profile. A random sample presented as a
|
||||
measurement — the exact failure class this branch exists to remove. We
|
||||
nearly copied it.
|
||||
- B2 dies with B1: nothing to cap.
|
||||
CONSEQUENCE: I1's narrowed Off-page axis (brand mentions only, backlinks +
|
||||
authority declared unauditable in §14) is the FINAL state, not a placeholder.
|
||||
Its §14 line was corrected — it used to point at Common Crawl as "nearest
|
||||
free source", which is a 17 GB dead end.
|
||||
RAISES W2's VALUE: Bing's GetUrlLinks is now the ONLY free viable backlink
|
||||
source. First-party only (never a competitor), still blocked on the client's
|
||||
Bing account.
|
||||
|
||||
### W2 (Bing) — DEFERRED, blocked on a real-world test
|
||||
Killed after 4 challenge rounds. User's model: client sites live on CLIENT
|
||||
Bing accounts, so a per-user API key means one key per client account.
|
||||
OAuth is the right model but is a swamp:
|
||||
- Redirect URI rejects ALL local forms (http/https/127.0.0.1 — user tested)
|
||||
- Refresh tokens are **rotated + single-use**, self-described non-compliant
|
||||
with OAuth 2.0 → store rewrite on every call, AND our parallel
|
||||
seo/geo dispatch would race the rotation → invalid_grant, dead token
|
||||
- Undocumented "anti-forgery token" failure on refresh, unanswered on Q&A
|
||||
- MS's own advisor recommends falling back to the API key
|
||||
- Doc contradicts itself on grant_type and the token endpoint; no library
|
||||
REVIVAL CONDITION: a client already on Bing adds the user as a Read-Only
|
||||
user → test in ~10 min whether the single API key sees DELEGATED sites
|
||||
(undocumented, nobody knows). If yes → W2 is cheap and clean (one key,
|
||||
client-owned verification, revocable, read-only, zero OAuth). If no → dead.
|
||||
Value forgone meanwhile: Bing/DDG/Ecosia query stats + index status +
|
||||
first-party backlinks. Real but modest; C1 dwarfs it.
|
||||
|
||||
## 2026-07-16 — PLAN seo/geo parity vs claude-seo (superseded by the STATUS above)
|
||||
Source: audit of github.com/AgriciDaniel/claude-seo (11.5k★, MIT, v2.2.0,
|
||||
5 mo old, 185/197 commits single author). Verdict: cherry-pick, never install
|
||||
(install.sh:49 overwrites our skills/seo/; uninstall.sh:45 glob `seo-*.md`
|
||||
deletes our seo-analyzer.md 42K it never installed; extensions/*/install.sh:42
|
||||
wipes settings.json on parse error; skills/seo/SKILL.md:119 injects Skool
|
||||
upsell footer into deliverables). Their code is real (render_page.py 428 l
|
||||
Playwright, url_safety.py 622 l SSRF, 326 tests, 320 pass) — adapt to our
|
||||
fetch.sh contract, do NOT copy wholesale (no fail-open, no tokenstore, no
|
||||
JSON shape).
|
||||
|
||||
Framing: their plus-values map onto OUR integrity gaps — report claims more
|
||||
than it measured. Same bar we held their README to.
|
||||
Seam: `lib/seo-data/fetch.sh` verbs (accounts|crux|queries|inspect|forget)
|
||||
+ fail-open `{"status":"degraded"}` + fixtures + tests. Everything below lands
|
||||
as NEW VERBS. No new architecture.
|
||||
|
||||
### AXE 0 — Integrity (no new deps, hours) — the score currently lies
|
||||
- [x] I1 Off-page axis scores 10-15% of FULL with ZERO data source (no API,
|
||||
no index) → today fabricated, and it feeds /client-handover. Immediate
|
||||
fix: extend existing LOCAL `N/A — requires FULL audit` pattern to FULL,
|
||||
redistribute weights. Data upgrade later (AXE 3). Honesty now, data after.
|
||||
- [x] I2 VSI (Visual Stability Index) listed in CWV thresholds but NO path
|
||||
retrieves it — neither CrUX nor PSI expose it. Phantom signal → remove
|
||||
or source.
|
||||
- [x] I3 **SAFETY** /geo standalone: geo/SKILL.md (125 l) has no STEP 0, no
|
||||
confirmed-NAP collection — but geo-analyzer OWNS JSON-LD NAP. Standalone
|
||||
/geo on a local business can write unverified NAP with zero LRN-032
|
||||
protection. Real bug, not cosmetic.
|
||||
- [x] I4 Security headers counted 3× (seo-analyzer STEP 4 scores them in
|
||||
Technical axis; depth-matrix.md says drop unless indexability; /harden
|
||||
re-audits /100 with 3 validators). Contradiction between dedup rule and
|
||||
agent spec → pick one owner.
|
||||
- [x] I5 Report says "audit", measured 5-15 sampled pages. State coverage %
|
||||
explicitly in §0 until AXE 2 lands.
|
||||
|
||||
### AXE 1 — Free wins on auth we ALREADY have (fetch.sh verbs)
|
||||
- [x] W1 `richresults` verb — GSC URL Inspection already returns
|
||||
`richResultsResult`; our OAuth already carries the scope. Programmatic
|
||||
rich-results validation on real Google data. **BEATS claude-seo**: their
|
||||
README:314 "dual validator (Rich Results Test + Markup Validator)" is
|
||||
FALSE — grep of all .py = zero calls, they are hyperlinks a human clicks.
|
||||
Today our JSON-LD validity is LLM-read only.
|
||||
- [ ] W2 `bing` verb — Bing Webmaster API, free. Closes the Google/Bing
|
||||
asymmetry (Google = full OAuth layer, Bing = manual checklist) while
|
||||
/geo targets ChatGPT Search, which indexes via Bing. Strategic, not cosmetic.
|
||||
- [x] W3 `sameas` resolution check — trivial curl loop. entity-seo.md lists
|
||||
"sameAs pointing to dead profiles" as a known error class and never
|
||||
checks it. ~10 lines.
|
||||
|
||||
### AXE 2 — Coverage (biggest lever: ~97% of a 500-page site unseen today)
|
||||
- [x] C1 `crawl` verb — sitemap-driven URL discovery (we ALREADY fetch
|
||||
sitemap.xml) + deterministic sampling + coverage % reported. No Chromium,
|
||||
no paid API. Turns "5-15 LLM-chosen pages" into measured coverage.
|
||||
Tradeoff vs claude-seo's link-following 500-page crawl: cheaper, but
|
||||
misses unlinked/unsitemapped pages — accept + disclose.
|
||||
- [x] C2 Dupe/cannibalization detection — becomes possible once N pages in
|
||||
hand: compare titles/H1/canonicals across the set. Free, unblocked by C1.
|
||||
- [x] C3 Internal-link graph — orphan pages + 3-click depth are TODAY stated
|
||||
as checks with no command to compute them. C1 unblocks real computation.
|
||||
|
||||
### AXE 3 — Off-page real (upgrades I1) — SUPERSEDED, see B1/B2 KILLED above
|
||||
- [x] ~~B1 `backlinks` verb — Common Crawl hyperlinkgraph~~ KILLED: edges file
|
||||
measured at 17.3 GB gzipped. Non-viable per audit; the reference impl
|
||||
caps at 500 MiB = 2.9% of the graph and calls the remainder a backlink
|
||||
profile.
|
||||
- [x] ~~B2 Honest cap at 70/100~~ KILLED with B1: nothing left to cap.
|
||||
I1's narrowed axis is the final state.
|
||||
- [x] B3 VERIFY FIRST: GSC Links API. Subagent claimed "available, OAuth
|
||||
already there" — I doubt it: Search Console API v3 has no links endpoint
|
||||
(links report is UI-only AFAIK). Verify before planning on it. Do not
|
||||
assert.
|
||||
|
||||
### AXE 4 — SPA blindness (dep decision — needs arbitrage)
|
||||
- [ ] R1 `render` verb — Playwright, GATED on SPA detection (STEP 2 already
|
||||
detects framework + rendering mode). Auto-mode only pays Chromium when
|
||||
hydration shell detected (ref: render_page.py:226 logic, adapt not copy).
|
||||
- [x] R2 ARBITRAGE: heavy dep (Chromium ~300MB) vs our bash+curl purity.
|
||||
Cheaper honest alternative: on SPA, REFUSE to score on-page rather than
|
||||
score it wrong (today: curl reads source, not hydrated DOM → every
|
||||
meta/JSON-LD/heading/img grep is blind, compensated only by a §0 flag).
|
||||
|
||||
### AXE 5 — Hardening + regression (lower priority)
|
||||
- [x] H1 SSRF guard on curl paths — both agents curl user-supplied domains.
|
||||
Our own CLAUDE.md doctrine says "never trust user input". url_safety.py
|
||||
(622 l, obfuscated-IPv4 decode, DNS pinning) is a solid reference.
|
||||
- [x] H2 `drift` baseline (SQLite) — SEO.md Historique keeps only date+score+
|
||||
key changes. Their seo-drift is on-page regression detection, NOT rank
|
||||
tracking (common misread). Optional.
|
||||
|
||||
### NOT DOING (explicit, with reason)
|
||||
- Keyword volumes → Google Ads Tier 3 needs ACTIVE ad spend (~$150-300/mo);
|
||||
without spend the API returns buckets ("1K-10K"). Their own detect_tier()
|
||||
never even returns 3 (google_auth.py:642-724 caps at 2) + google-ads absent
|
||||
from requirements.txt. Not worth it.
|
||||
- Real AI SoV (ChatGPT/Perplexity citation tracking) → paid everywhere
|
||||
(SE Ranking/Profound/DataForSEO). Our current honest "not testable, here's
|
||||
what we measured instead" disclosure BEATS faking it. Keep.
|
||||
- Installing the plugin / +33 skills namespace → see destructive paths above.
|
||||
|
||||
### Keep (already beats claude-seo — do not regress)
|
||||
FR legal (LCEN/RGPD-ePrivacy/DGCCRF L121-1 — their whole repo: 2 hits, and
|
||||
dma-consent-mode-v2.md:27 tells the agent to stay out) · fix-bundle +
|
||||
ownership matrix + serial apply (their 18 agents are report-only, no
|
||||
ownership discipline) · trajectory-to-17/20 + honest code ceiling (theirs is
|
||||
flat 0-100, no legal axis) · llms.txt honest framing · NAP anti-dup-seed
|
||||
(LRN-032).
|
||||
|
||||
## 2026-07-16 — /close auto-persist memory (feature/close-auto-persist, BDR-068)
|
||||
- [x] STEP 5C: auto-finish chore→develop + push when capitalize/close branched off develop
|
||||
- [x] --no-push escape hatch; WORKING-branch + rc-3 skip; graceful push-fail
|
||||
|
||||
+10
-173
@@ -13,13 +13,10 @@ Apple Intelligence**. Google classical search is handled by the
|
||||
|
||||
## Context — why GEO is its own discipline in 2026
|
||||
|
||||
- `[UNVERIFIED — 2026-07-16]` AI Overviews trigger on ~48% of Google
|
||||
searches (April 2026); ChatGPT processes 2.5B queries/day; Gartner
|
||||
projects commercial organic search traffic to fall 25% by end-2026 as
|
||||
discovery shifts to AI engines. Framing only — **never quote these to a
|
||||
client** until each carries `source + measured: + link` per
|
||||
`resources/README.md`. GEO is worth doing on mechanism; it does not need
|
||||
these numbers to be true.
|
||||
- AI Overviews trigger on ~48% of Google searches (April 2026).
|
||||
- ChatGPT processes 2.5B queries/day.
|
||||
- Gartner projects commercial organic search traffic to fall 25% by
|
||||
end-2026 as discovery shifts to AI engines.
|
||||
- Classical SEO ≠ GEO. Some signals overlap (headings, Schema.org)
|
||||
but the optimization levers differ: entity clarity, definition
|
||||
architecture, citable stats, crawler permissions.
|
||||
@@ -144,16 +141,6 @@ If called standalone via `/geo`, gather:
|
||||
|
||||
## STEP 2 — DETECT CONTEXT `[both]`
|
||||
|
||||
**FIRST — the CWD must BE the audited site.** You grep the current working
|
||||
directory; no dispatcher checks that it matches the target domain. If a URL
|
||||
was supplied and the CWD shows no web project at all (no `package.json` /
|
||||
`composer.json` / `index.html` / `*.astro` / `*.php` / `.htaccess`), or its
|
||||
signals contradict the domain, STOP and report:
|
||||
`CWD/TARGET MISMATCH — <cwd> is not <domain>'s repo. Re-run from it, or
|
||||
confirm live-only audit (LOCAL findings will be N/A).`
|
||||
Never grep one codebase while curling another: the live half looks right,
|
||||
the code half is fiction, and the report reads as authoritative.
|
||||
|
||||
```bash
|
||||
# Framework (reuse detection from seo-analyzer if available)
|
||||
ls package.json composer.json Gemfile Cargo.toml go.mod 2>/dev/null
|
||||
@@ -244,14 +231,8 @@ the PERMISSIVE template from `ai-crawlers-2026.md`.
|
||||
|
||||
### Live verification `[FULL only]`
|
||||
|
||||
**Guard the domain before it reaches a shell — mandatory, not optional.**
|
||||
`$DOMAIN` is interpolated inside double quotes below, where `$` and backtick
|
||||
still execute. Run the guard FIRST and use only its output; non-zero exit →
|
||||
STOP this step and report the refusal, never sanitise-and-retry.
|
||||
|
||||
```bash
|
||||
DOMAIN="$(bash ~/.claude/lib/url-guard.sh host "<production-domain>")" || {
|
||||
echo "STEP 4 aborted: domain refused by url-guard"; exit 2; }
|
||||
DOMAIN="<production-domain>"
|
||||
|
||||
# Verify robots.txt served
|
||||
curl -s "https://$DOMAIN/robots.txt" | head -50
|
||||
@@ -379,9 +360,7 @@ action (G5 batch, confirmation needed — visible page creation).
|
||||
|
||||
**Local business:**
|
||||
- [ ] `LocalBusiness` with most specific subclass (Plumber/Dentist/etc.)
|
||||
- [ ] NAP consistent with GMB — **direction rule applies** (Data integrity:
|
||||
never pick a value from source majority; no canonical → no directional
|
||||
fix)
|
||||
- [ ] NAP consistent with GMB
|
||||
- [ ] `sameAs` includes GMB URL + main social + Wikidata if applicable
|
||||
- [ ] `areaServed` lists served cities/regions
|
||||
- [ ] `openingHoursSpecification` matches reality
|
||||
@@ -437,57 +416,6 @@ Record what exists. For each:
|
||||
- Does `sameAs` on the site point to it?
|
||||
- If yes, does the target resolve and match?
|
||||
|
||||
### sameAs resolution `[FULL only]`
|
||||
|
||||
`entity-seo.md:148` says "validate each URL resolves" and nothing did.
|
||||
A `sameAs` pointing at a dead profile is worse than a missing one: it
|
||||
asserts an identity link that fails on follow, in the exact graph AI
|
||||
engines walk to confirm who you are.
|
||||
|
||||
```bash
|
||||
grep -rhoE '"sameAs"[^]]*\]' \
|
||||
--include="*.html" --include="*.astro" --include="*.tsx" --include="*.jsx" \
|
||||
--include="*.vue" --include="*.svelte" --include="*.php" --include="*.json" \
|
||||
. 2>/dev/null \
|
||||
| grep -oE 'https?://[^"]+' | sort -u | while read -r RAW; do
|
||||
# These URLs come from the audited repo's JSON-LD, not from the operator:
|
||||
# guard each one before it reaches curl. A refused entry is REPORTED, not
|
||||
# skipped silently — an unguardable sameAs is itself a finding.
|
||||
U="$(bash ~/.claude/lib/url-guard.sh url "$RAW" 2>/dev/null)" || {
|
||||
printf 'REFUSED %s\n' "$RAW"; continue; }
|
||||
printf '%s %s\n' \
|
||||
"$(curl -sIL -o /dev/null -w '%{http_code}' --max-time 10 "$U" 2>/dev/null || echo 000)" \
|
||||
"$U"
|
||||
done
|
||||
```
|
||||
|
||||
`REFUSED` rows are not dead links and not live ones — the URL never left the
|
||||
machine. Report them in §14 with the raw value: a `sameAs` carrying shell
|
||||
metacharacters or pointing at `localhost` is either broken markup or someone
|
||||
probing, and both are worth the client knowing.
|
||||
|
||||
**Read the codes honestly — a block is not a death.** Some platforms refuse
|
||||
non-browser clients: LinkedIn answers `999` (verified 2026-07-16 against a
|
||||
live company page). A naive check calls that dead and the bundle deletes a
|
||||
live link — the most valuable node in the graph, since LinkedIn is the
|
||||
identity anchor for most B2B entities.
|
||||
|
||||
Do NOT assume which platforms block: the same 2026-07-16 check found
|
||||
`x.com` returning `200`, contradicting the "Twitter always 403" folklore.
|
||||
Test the code you actually got; classify by code, never by platform
|
||||
reputation.
|
||||
|
||||
| Code | Verdict | Action |
|
||||
|---|---|---|
|
||||
| 2xx / 3xx | alive | none |
|
||||
| **404 / 410** | **genuinely dead** | finding WITH direction — fix or remove |
|
||||
| 401 / 403 / 429 / 999 | bot-blocked | **inconclusive — no finding.** Report as unverified, never as dead |
|
||||
| 000 (DNS/timeout) / 5xx | inconclusive | retry once, then unverified |
|
||||
|
||||
No G2/G6 item may remove a `sameAs` on anything but 404/410. Same rule as
|
||||
the NAP direction rule: an unreliable signal read confidently is worse than
|
||||
no signal. Unverified entries → §14, naming the platform and the code.
|
||||
|
||||
### Google Knowledge Panel `[FULL only]`
|
||||
|
||||
```
|
||||
@@ -515,27 +443,10 @@ PRIORITY ACTIONS : <top 3-5>
|
||||
|
||||
## STEP 8 — CONTENT SHAPE FOR AI `[both]`
|
||||
|
||||
**Rendering gate first (R2).** `bash ~/.claude/lib/seo-data/fetch.sh
|
||||
rendercheck --url "https://$DOMAIN/"`. Verdict `client-rendered` → Content
|
||||
Shape is `N/A — content not in served HTML`, excluded from the weighted
|
||||
global, never scored zero. And say the thing that actually matters here: AI
|
||||
crawlers are **worse** at JS than Googlebot is. GPTBot, PerplexityBot and
|
||||
ClaudeBot fetch HTML and largely do not execute it, so a client-rendered site
|
||||
is not just unauditable by us — it is close to invisible to the engines this
|
||||
whole audit targets. That is a §0 alert and the top user action (SSR/SSG),
|
||||
not a schema tweak.
|
||||
Site-wide axes (crawler policy, llms.txt) are unaffected: those are files.
|
||||
|
||||
Load: `~/.claude/agents/resources/content-shape-for-ai.md`
|
||||
|
||||
Sample 5-10 key pages (homepage + top service/blog pages). For each:
|
||||
|
||||
**Record the denominator.** This samples; the report says "audit". Count the
|
||||
URLs in `sitemap.xml` for the coverage ratio, and carry it into the GEO
|
||||
SCORING block. No sitemap → total UNKNOWN, say so. Content shape is the
|
||||
axis most damaged by silent sampling: it is judged per page, so a 6-page
|
||||
sample of a 300-page site says nothing about the other 294.
|
||||
|
||||
### Checks
|
||||
|
||||
1. **Definition Lead** — does the first sentence (or H1) follow
|
||||
@@ -551,28 +462,15 @@ sample of a 300-page site says nothing about the other 294.
|
||||
pronouns?
|
||||
8. **Lists/tables vs prose** — structured where possible?
|
||||
9. **30/70 rule** (if city/service variants exist) — ≥70% unique?
|
||||
10. **Filler/AI-slop signal (deterministic)** — feed each sampled page's
|
||||
body text to `fetch.sh content_quality`. It is a DETERMINISTIC input
|
||||
that INFORMS checks 1-9 (word-list/density heuristics, no LLM call);
|
||||
it never replaces your read of them. A low `overall_quality` or a
|
||||
`filler`/`ai-patterns` flag is a candidate for human review, not an
|
||||
automatic finding — do not let the number become the verdict, and do
|
||||
not claim a page "is AI-written" from it.
|
||||
|
||||
### Sampling command
|
||||
|
||||
```bash
|
||||
# Extract H1/H2/H3 from main pages to assess heading style
|
||||
mapfile -t FEXCL < <(bash ~/.claude/lib/source-scope.sh findargs) # C1a: skip build output
|
||||
for f in index.html $(find . "${FEXCL[@]}" -maxdepth 3 \( -name "*.astro" -o -name "*.tsx" -o -name "*.md" -o -name "*.html" \) | head -10); do
|
||||
for f in index.html $(find . -maxdepth 3 -name "*.astro" -o -name "*.tsx" -o -name "*.md" -o -name "*.html" | head -10); do
|
||||
echo "=== $f ==="
|
||||
grep -oE '<(h1|h2|h3)[^>]*>[^<]+</(h1|h2|h3)>|^#{1,3} .+' "$f" 2>/dev/null | head -20
|
||||
done
|
||||
|
||||
# Filler/AI-slop signal (Check 10) — strip markup to plain body text, then
|
||||
# score it. Advisory only: pair the number with your own read of Checks 1-9.
|
||||
sed -e 's/<[^>]*>//g' index.html | \
|
||||
bash ~/.claude/lib/seo-data/fetch.sh content_quality
|
||||
```
|
||||
|
||||
### Findings
|
||||
@@ -588,9 +486,6 @@ CITED STATISTICS : <avg per page>
|
||||
FRESHNESS VISIBLE : <n/N pages>
|
||||
PRONOUN-HEAVY : <n/N pages flagged>
|
||||
30/70 RULE : pass | fail | N/A
|
||||
FILLER/AI-SLOP SIGNAL : <avg overall_quality>/100, flags: <n/N pages flagged>
|
||||
(deterministic, advisory — informs checks 1-9, never
|
||||
a verdict, never scored on its own)
|
||||
PRIORITY ACTIONS : <top 5>
|
||||
```
|
||||
|
||||
@@ -679,9 +574,6 @@ Score each axis. Use concrete findings from STEP 2-9.
|
||||
|
||||
```
|
||||
GEO SCORING (<depth>)
|
||||
COVERAGE SOURCE : <N> of <M> page templates (<P>%) — bounds Schema.org
|
||||
COVERAGE LIVE : <N> of <M> sitemap URLs (<P>%) — bounds Content Shape
|
||||
| UNKNOWN (no sitemap / fetch degraded)
|
||||
AI Crawlers Policy : XX/20 <justification>
|
||||
llms.txt : XX/20 <justification>
|
||||
Schema.org for AI : XX/20 <justification>
|
||||
@@ -692,24 +584,6 @@ AI Visibility (live) : XX/20 | N/A (LOCAL)
|
||||
GEO GLOBAL (weighted) : XX.X/20 (<depth>)
|
||||
```
|
||||
|
||||
**COVERAGE is mandatory, never omitted, never rounded up.** It bounds the
|
||||
per-page axes — Content Shape above all, and the page-level share of
|
||||
Schema.org. Site-wide axes (AI Crawlers Policy, llms.txt) are unaffected:
|
||||
robots.txt and llms.txt are single files, fully read. Say which is which
|
||||
rather than letting one ratio discredit the whole report.
|
||||
|
||||
**Same source/live split as seo-analyzer STEP 9 (C1c), and it cuts your axes
|
||||
differently.** A JSON-LD block lives in a shared layout, so one sampled page
|
||||
per URL family proves the SCHEMA for the whole family — SOURCE coverage is
|
||||
what bounds it. Content Shape does NOT work that way: Definition Lead, TL;DR
|
||||
and heading wording are written per page, so a template says nothing about
|
||||
its 25 instances. Bound Schema.org by SOURCE, Content Shape by LIVE, and
|
||||
never quote the flattering one alone. Get the URL families from
|
||||
`fetch.sh sitemap`, grouped as seo-analyzer STEP 5 describes — shared parent
|
||||
path OR shared slug prefix, because both layouts are real: first-segment
|
||||
alone reads 8 flat `/lavage-auto-<city>` pages as 8 singletons. If `/seo`
|
||||
already ran it, reuse the count rather than re-fetching.
|
||||
|
||||
Per user instruction: **GEO weight in combined SEO+GEO report = 20% for
|
||||
local, 25% for national/SaaS/content.**
|
||||
|
||||
@@ -828,14 +702,7 @@ to act without your audit context. Embed per item:
|
||||
- **Templates + context** — G2/G6 paste the expected JSON-LD from
|
||||
`geo-schemas.md` + business context (entity name, sameAs, @id canonical)
|
||||
+ framework note. G4 follows `llms-txt-template.md` exactly. G1 pastes
|
||||
the correct variant from `ai-crawlers-2026.md`. When a G2 item needs a
|
||||
`Reservation`/`OrderAction`/`DiscussionForumPosting`/`ProfilePage` block,
|
||||
generate the skeleton via `fetch.sh schema_gen
|
||||
<reservation|order|discussion|profile> [flags]`
|
||||
(`~/.claude/lib/seo-data/fetch.sh`) and fill in the real values, rather
|
||||
than hand-writing that markup. The data-integrity rule still applies on
|
||||
top of it: `schema_gen` only generates STRUCTURE — unknown field values
|
||||
stay `[À COMPLÉTER]`, never invented to fill a flag the verb needs.
|
||||
the correct variant from `ai-crawlers-2026.md`.
|
||||
- **PERMISSIVE default** on G1 unless the client flagged premium/regulated.
|
||||
|
||||
### Output shape
|
||||
@@ -1018,14 +885,6 @@ PROCHAINE ETAPE : <highest-priority>
|
||||
NEVER `Write` on shared templates. `Write` is reserved for files
|
||||
you solely own: robots.txt, llms.txt, llms-full.txt. Full-template
|
||||
refactor → escalate as user action in §11.
|
||||
- **NEVER emit a bundle item targeting build output (C1a).** No path under
|
||||
`dist/ build/ .next/ .nuxt/ .output/ _site/ .astro/ .svelte-kit/ out/` —
|
||||
run `bash ~/.claude/lib/source-scope.sh list` for the authoritative set.
|
||||
Those files are regenerated: the `npm run build` the dispatcher runs to
|
||||
VERIFY your fix is what erases it. The fix lands, verification passes,
|
||||
nothing survives, and the report claims it was applied. Fix the SOURCE
|
||||
template that generates the file. If you cannot find the source, that is
|
||||
a finding — say so, do not patch the artifact.
|
||||
- **Respect PERMISSIVE/RESTRICTIVE choice.** geo-analyzer defaults to
|
||||
PERMISSIVE (GEO's goal is AI visibility). Only switch if the client
|
||||
explicitly flags premium/regulated content.
|
||||
@@ -1036,31 +895,9 @@ PROCHAINE ETAPE : <highest-priority>
|
||||
- **No invented entity data.** Never write a fake Wikidata QID, fake
|
||||
`sameAs` URLs, fake `knowsAbout`, fake press mentions. Unknown →
|
||||
placeholder `[À COMPLÉTER]` or omit.
|
||||
- **NAP direction rule (LRN-032).** You own JSON-LD NAP, so this binds you
|
||||
whoever called you — `/seo` passes a canonical, standalone `/geo` does
|
||||
not. NEVER infer a correct NAP value from source majority: on-site
|
||||
sources (JSON-LD, footer, settings DB, legal pages) usually descend from
|
||||
ONE seed and can all carry the same wrong value — the single diverging
|
||||
source may be the only one a human actually corrected. Direction of fix:
|
||||
- Diverging from a CONFIRMED canonical field (passed by `/seo` STEP 0)
|
||||
→ fix the diverging source.
|
||||
- Canonical UNCONFIRMED or absent (the standalone `/geo` case) → report
|
||||
the divergence WITHOUT a directional fix; escalate as a user question
|
||||
("which value is correct?") in §11.
|
||||
No G2/G6 item may write or rewrite a NAP value that no confirmed
|
||||
canonical backs — **creating** a `LocalBusiness` from scratch included:
|
||||
unknown fields → `[À COMPLÉTER]`, never a value copied from a sibling
|
||||
on-site source.
|
||||
- **Remove deprecated schemas rather than keep broken ones.**
|
||||
- **Cite sources, and only citable ones.** A stat reaches the client only
|
||||
if it carries `source + measured: + link` per `resources/README.md`.
|
||||
Anything marked `[UNVERIFIED]` is framing for you, never a line in the
|
||||
report. Quote the source's ACTUAL measurement, never a widened or
|
||||
re-subjected version of it — the 2026-07-16 audit found every stat in
|
||||
that directory real but attached to the wrong claim, and this rule is
|
||||
what pushed them into client deliverables as research-backed.
|
||||
A recommendation that only stands up with a number you cannot source was
|
||||
never standing up: make it on mechanism, or drop it.
|
||||
- **Cite sources.** When emitting stats in the report, link
|
||||
`content-shape-for-ai.md` research citations.
|
||||
|
||||
### Process
|
||||
- **Every user action lists automation options.** Mandatory from
|
||||
|
||||
@@ -17,52 +17,7 @@ Loaded on demand — keep each file focused and current.
|
||||
|
||||
These files capture state as of 2026-04. Crawler lists, Schema.org
|
||||
deprecations, and tool landscape shift fast. Agents MUST cross-check
|
||||
crawler lists and tool names via WebSearch on each run when FULL depth is
|
||||
selected.
|
||||
|
||||
## Citation standard (mandatory for every statistic)
|
||||
|
||||
**WebSearch is NOT verification for a number.** It ranks SEO blogs, and SEO
|
||||
blogs cross-cite each other into a consensus that looks like corroboration.
|
||||
Two 2026-07-16 audits of this directory show how it fails:
|
||||
|
||||
- A "VSI (Visual Stability Index) — new 2026 Core Web Vital" lived in
|
||||
`seo-analyzer.md`. Ten blogs asserted it; several claimed CrUX already
|
||||
collected it. It is absent from the CrUX API metric list and from
|
||||
web.dev. WebSearch returned the echo, not the truth.
|
||||
- Every stat in this directory was real **and attached to the wrong
|
||||
subject**: the GEO paper's 40% (all methods) pinned on one technique;
|
||||
LLMrefs' 3x (brand mentions vs backlinks) pinned on freshness decay;
|
||||
AccuraCast's 58.9% (Person schema prevalence) pinned on QAPage lift, with
|
||||
its meaning inverted; a smart-speaker adoption figure sold as voice-search
|
||||
share.
|
||||
|
||||
The failure mode is not invention — it is **plausible recombination**, which
|
||||
is exactly what a model half-remembering a search result produces. So the
|
||||
format has to make an unsourced number conspicuous:
|
||||
|
||||
```
|
||||
<claim> — <source, year, venue|vendor> — measured: <what the source ACTUALLY
|
||||
measured> — <link>
|
||||
```
|
||||
|
||||
`measured:` is the field that catches it. All four errors above survive a
|
||||
source name; none survives having to state the source's real measurement
|
||||
next to the claim.
|
||||
|
||||
Rules:
|
||||
1. **Primary source or no number.** Peer-reviewed paper, the vendor's own
|
||||
published study, or an official API/doc. `developer.chrome.com/docs/crux`
|
||||
is decisive for metrics: what CrUX cannot return, we cannot score.
|
||||
2. **Name the tier.** Peer review ≠ vendor marketing. LLMrefs, AccuraCast,
|
||||
Ahrefs publish useful data and sell products — say "vendor".
|
||||
3. **Never widen scope.** An aggregate result is not a per-technique result.
|
||||
4. **No number beats a wrong number.** A recommendation that only stands up
|
||||
with a fabricated statistic was never standing up. Delete the stat, keep
|
||||
the recommendation if it survives on mechanism.
|
||||
5. **Unverified ⇒ labelled.** `[UNVERIFIED — <date>]` inline. Never quote an
|
||||
unverified number to a client: `geo-analyzer.md` ("Cite sources") sends
|
||||
these into client reports as research-backed.
|
||||
via WebSearch on each run when FULL depth is selected.
|
||||
|
||||
## Loading pattern
|
||||
|
||||
|
||||
@@ -4,17 +4,9 @@ Tools that track whether your brand appears in AI-generated answers
|
||||
across ChatGPT, Perplexity, Gemini, Copilot, Claude, and Google AI
|
||||
Overviews.
|
||||
|
||||
Context `[UNVERIFIED — 2026-07-16]`: Google AI Overviews trigger on ~48% of
|
||||
searches; ChatGPT processes 2.5B queries/day; Gartner projects commercial
|
||||
organic search traffic will drop 25% by 2026.
|
||||
|
||||
> Not checked against primary sources in the 2026-07-16 audit that corrected
|
||||
> the rest of this directory — flagged rather than asserted or deleted, per
|
||||
> the citation standard in `README.md` (rule 5). The Gartner projection at
|
||||
> least names its source; the other two float. Treat all three as
|
||||
> motivation, not evidence: **do NOT quote them to a client** until each
|
||||
> carries `source + measured: + link`. Their only job here is to explain why
|
||||
> this file exists, and that argument does not need numbers.
|
||||
Context: Google AI Overviews trigger on ~48% of searches; ChatGPT
|
||||
processes 2.5B queries/day; Gartner projects commercial organic
|
||||
search traffic will drop 25% by 2026. Monitoring is no longer optional.
|
||||
|
||||
## Commercial tools
|
||||
|
||||
|
||||
@@ -61,18 +61,9 @@ query. A one-sentence self-contained answer has the highest density.
|
||||
|
||||
### 4. Citations and statistics (strongest measured lever)
|
||||
|
||||
Aggarwal et al., 2024 ("GEO: Generative Engine Optimization", KDD 2024)
|
||||
report that their optimisation methods **collectively** boost visibility
|
||||
**by up to 40%** in generative-engine responses, and state the effect
|
||||
**varies across domains**. Citations/statistics/quotations are among those
|
||||
methods.
|
||||
|
||||
> **Attribute this correctly.** Until 2026-07-16 this section read "Adding
|
||||
> peer-cited statistics with clear sources increases AI visibility by up to
|
||||
> 40%" — pinning the paper's *aggregate* result on this *one* technique. The
|
||||
> paper publishes no separate figure per technique. When quoting it to a
|
||||
> client: "up to 40%, across the method set, domain-dependent" — never "+40%
|
||||
> if you add stats".
|
||||
Adding peer-cited statistics with clear sources increases AI visibility
|
||||
**by up to 40%** (Aggarwal et al., 2024 "GEO: Generative Engine
|
||||
Optimization").
|
||||
|
||||
Pattern: embed specific numbers with attribution.
|
||||
|
||||
@@ -109,20 +100,8 @@ Comparison tables are even stronger. Structure:
|
||||
|
||||
### 6. Freshness signals
|
||||
|
||||
Freshness is a real retrieval input: RAG systems fetch live and read
|
||||
timestamps, so a page updated this quarter carries a stronger recency
|
||||
signal than the same page last touched years ago. LLMrefs (a **vendor**,
|
||||
not peer review) reports cited content running **~25.7% fresher** than
|
||||
organic top-10 across ~17M citations. Substantive updates only — bumping a
|
||||
date string is not freshness.
|
||||
|
||||
> **The "3x" that lived here was grafted from another claim.** Until
|
||||
> 2026-07-16 this read "Pages not updated at least quarterly are 3x more
|
||||
> likely to lose AI citations (LLMRefs 2026 study)". LLMrefs' actual "3x"
|
||||
> says **brand mentions correlate ~3x more strongly with AI visibility than
|
||||
> backlinks** — a different subject entirely. No source supports a quarterly
|
||||
> decay multiplier. Recommend quarterly refresh on its merits; do not price
|
||||
> it with a borrowed number.
|
||||
Pages not updated at least quarterly are **3x more likely to lose AI
|
||||
citations** (LLMRefs 2026 study).
|
||||
|
||||
What to maintain:
|
||||
- Visible "Last updated: YYYY-MM-DD" at the top of content pages
|
||||
|
||||
@@ -21,20 +21,8 @@ existing instances. They no longer produce rich results.
|
||||
|
||||
### QAPage — single Q&A format
|
||||
|
||||
Use when the page is built around ONE primary question. Emitting the type
|
||||
that matches the content shape beats wrapping everything in a generic
|
||||
`Article`.
|
||||
|
||||
> **No lift figure here — the one that lived here was wrong.** Until
|
||||
> 2026-07-16 this read "Pages cited 58% more often by ChatGPT vs basic
|
||||
> Article schema", uncited. Nothing supports it. The nearest real number is
|
||||
> AccuraCast 2025 (~2,000 prompts across ChatGPT / AI Overviews /
|
||||
> Perplexity, ~9,000 cited sources): **`Person` schema appeared in 58.9%**
|
||||
> of cited sources — a *prevalence* count for a *different type* — while
|
||||
> **`FAQPage` appeared in 1.8%**, which points the opposite way to the claim
|
||||
> it was propping up. Q&A shape is still worth doing on genuinely
|
||||
> single-question pages; it is not worth a fabricated number. Do NOT quote a
|
||||
> QAPage lift % to a client — there isn't one.
|
||||
Pages cited 58% more often by ChatGPT vs basic Article schema.
|
||||
Use when the page is built around ONE primary question.
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -93,16 +81,8 @@ visible content.
|
||||
|
||||
### Speakable — voice + AI extraction marker
|
||||
|
||||
Speakable flags the passage best suited for voice readout and AI summary.
|
||||
|
||||
> **No voice-share figure — the one that lived here was a conflation.**
|
||||
> Until 2026-07-16 this read "62% of searches in 2026 involve voice",
|
||||
> uncited. No primary source carries it; 62% circulates as a *smart-speaker
|
||||
> adoption* number, not a share of searches. It is the same family as the
|
||||
> "50% of searches will be voice by 2020" myth — attributed to ComScore,
|
||||
> who **denied it**; the real origin is a 2014 Andrew Ng interview. Speakable
|
||||
> is cheap and harmless, so keep recommending it on TL;DR / summary blocks —
|
||||
> but justify it by extraction shape, never by a voice-share statistic.
|
||||
62% of searches in 2026 involve voice. Speakable flags the passage
|
||||
best suited for voice readout and AI summary.
|
||||
|
||||
```json
|
||||
{
|
||||
|
||||
+11
-425
@@ -81,17 +81,6 @@ hreflang, infer from detected URL structures.
|
||||
|
||||
## STEP 2 — DETECT TECHNICAL CONTEXT `[both]`
|
||||
|
||||
**FIRST — the CWD must BE the audited site.** You grep the current working
|
||||
directory; no dispatcher checks that it matches TARGET_URL. If a URL was
|
||||
supplied and the CWD shows no web project at all (no `package.json` /
|
||||
`composer.json` / `index.html` / `*.astro` / `*.php` / `.htaccess`), or its
|
||||
signals contradict the domain, STOP and report:
|
||||
`CWD/TARGET MISMATCH — <cwd> is not <domain>'s repo. Re-run from it, or
|
||||
confirm live-only audit (LOCAL findings will be N/A).`
|
||||
Never grep one codebase while curling another: the live half looks right,
|
||||
the code half is fiction, and the report reads as authoritative. `/harden`
|
||||
inherits this agent for its config axis, so the mismatch propagates there.
|
||||
|
||||
### Framework & rendering
|
||||
|
||||
```bash
|
||||
@@ -159,31 +148,13 @@ RECOMMENDATION : KEEP & CONFIGURE plugin | INSTALL <plugin> (P0 quick win) | M
|
||||
|
||||
### Infrastructure signals
|
||||
|
||||
**Origin vs edge — never infer the stack from `server:`.** That header names
|
||||
whatever answered: usually the EDGE (Cloudflare, Scaleway/OVH front, CDN,
|
||||
load balancer), not the origin. Apache behind an nginx front is a standard
|
||||
topology — TLS terminated upstream, the origin sees plain HTTP plus
|
||||
`X-Forwarded-Proto`.
|
||||
- Repo `.htaccess` + `server: nginx` = NOT drift, NOT dead config. Do not
|
||||
flag it, do not propose migrating it.
|
||||
- Never move headers into an `nginx.conf` absent from the repo. Server-side
|
||||
config you cannot read is a §14 gap, not a finding.
|
||||
- A header present live but in no repo config = "set upstream", never
|
||||
"missing".
|
||||
|
||||
`/harden` reuses this agent for its entire config-hardening axis, so a wrong
|
||||
topology call scores a client's server config against a file that never ran.
|
||||
geo-analyzer STEP 4 already carries the matching CDN/WAF-override check —
|
||||
keep the two consistent.
|
||||
|
||||
```bash
|
||||
# Server / hosting
|
||||
ls .htaccess nginx.conf netlify.toml vercel.json wrangler.toml 2>/dev/null
|
||||
# SEO files
|
||||
ls robots.txt sitemap.xml sitemap-index.xml sitemap-images.xml sitemap-videos.xml 2>/dev/null
|
||||
# Legal pages — source only (C1a: find ignores .gitignore, grep does not)
|
||||
mapfile -t FEXCL < <(bash ~/.claude/lib/source-scope.sh findargs)
|
||||
find . "${FEXCL[@]}" -maxdepth 3 \( -iname "*mention*" -o -iname "*legal*" -o -iname "*confidentialite*" -o -iname "*privacy*" -o -iname "*cgv*" -o -iname "*cgu*" \) 2>/dev/null | head -10
|
||||
# Legal pages
|
||||
find . -maxdepth 3 \( -iname "*mention*" -o -iname "*legal*" -o -iname "*confidentialite*" -o -iname "*privacy*" -o -iname "*cgv*" -o -iname "*cgu*" \) 2>/dev/null | head -10
|
||||
# Analytics / trackers
|
||||
grep -rl "gtag\|GTM-\|analytics\|matomo\|_paq\|plausible\|umami" --include="*.html" --include="*.js" --include="*.tsx" --include="*.astro" --include="*.php" . 2>/dev/null | head -10
|
||||
# Cookie consent / CMP
|
||||
@@ -245,21 +216,8 @@ anonymous PageSpeed lab data and STEP 4/STEP 11 emit the §11 user action
|
||||
|
||||
### HTTP headers & security
|
||||
|
||||
**Read them; score them only for `/harden` (I4).** This section stays — the
|
||||
raw headers are needed for `X-Robots-Tag`, canonical/redirect coherence, and
|
||||
the §14 observed-list. But under `/seo` the security headers themselves are
|
||||
out of scope for scoring: see the Technical axis note in STEP 9. Under
|
||||
`/harden` they are the entire job. Reading is not scoring.
|
||||
|
||||
**Guard the domain before it reaches a shell — mandatory, not optional.**
|
||||
Every curl below interpolates `$DOMAIN` inside double quotes, where `$` and
|
||||
backtick still execute. Run the guard FIRST and use only its output; if it
|
||||
exits non-zero, STOP this step and report the refusal — never "clean up" the
|
||||
value and retry.
|
||||
|
||||
```bash
|
||||
DOMAIN="$(bash ~/.claude/lib/url-guard.sh host "<production-domain>")" || {
|
||||
echo "STEP 4 aborted: domain refused by url-guard"; exit 2; }
|
||||
DOMAIN="<production-domain>"
|
||||
|
||||
# Headers
|
||||
curl -sI "https://$DOMAIN/" | head -30
|
||||
@@ -289,21 +247,8 @@ Evaluate each present/missing:
|
||||
- **LCP** (Largest Contentful Paint) — < 2.5s
|
||||
- **INP** (Interaction to Next Paint) — < 200ms (replaced FID in Mar 2024)
|
||||
- **CLS** (Cumulative Layout Shift) — < 0.1
|
||||
|
||||
**Core Web Vitals are exactly these three** (web.dev/articles/vitals,
|
||||
verified 2026-07-16). Google ships threshold changes with prior notice on a
|
||||
predictable annual cadence — a "new CWV" that only SEO blogs know about does
|
||||
not exist. Before adding a metric here, confirm it against a PRIMARY source:
|
||||
web.dev, the Chromium blog, or `developer.chrome.com/docs/crux/api` — that
|
||||
API metric list is decisive, because a metric CrUX cannot return is a metric
|
||||
we cannot score.
|
||||
|
||||
**WebSearch is not confirmation.** SEO blogs cross-cite each other into fake
|
||||
consensus. A "VSI (Visual Stability Index) — new 2026 signal, Core Web
|
||||
Vitals 2.0" line lived here until 2026-07-16 on exactly that basis: ten
|
||||
blogs asserted it, several claimed CrUX was already collecting it, and it is
|
||||
absent from both the CrUX API metric list and web.dev. Stated as fact, in a
|
||||
threshold list, in client-facing audits.
|
||||
- **VSI** (Visual Stability Index) — new 2026 signal, Google Core Web
|
||||
Vitals 2.0
|
||||
|
||||
When a GSC account+property were passed in context, fetch CrUX field
|
||||
data first (**tilde path mandatory** — this agent runs from the
|
||||
@@ -340,71 +285,13 @@ When STEP 0/STEP 1 recorded a GSC account+property (not "none"):
|
||||
```bash
|
||||
bash ~/.claude/lib/seo-data/fetch.sh queries --account "$GSC_ACCOUNT" --property "$GSC_PROPERTY" --days 90 --dim query
|
||||
bash ~/.claude/lib/seo-data/fetch.sh inspect --account "$GSC_ACCOUNT" --property "$GSC_PROPERTY" --url "https://$DOMAIN/"
|
||||
bash ~/.claude/lib/seo-data/fetch.sh cannibal --account "$GSC_ACCOUNT" --property "$GSC_PROPERTY" --days 90
|
||||
```
|
||||
|
||||
**`cannibal` — keyword cannibalisation, from Google's own data (C2).** Groups
|
||||
90 days of `query`+`page` rows and returns every query where 2+ of OUR pages
|
||||
compete, ranked by total impressions. The API always allowed multiple
|
||||
dimensions; this system only ever asked for one, so the conflict was invisible.
|
||||
|
||||
Read it:
|
||||
- `conflicts[]` → for each, the strongest page (most impressions) is listed
|
||||
first. That is usually the one to KEEP; the others either consolidate into
|
||||
it (301 + merge content) or get differentiated. Never "fix" this by deleting
|
||||
a page that has clicks — say what competes and let the user choose.
|
||||
- A conflict with a large impression total and every page beyond position 10
|
||||
is the real prize: Google can't decide which page to rank, so none rank.
|
||||
- `capped: true` → the row window was full; there are conflicts past the cut.
|
||||
Say so in §14 rather than presenting the list as exhaustive.
|
||||
- `status: degraded` → no GSC account. Cannibalisation is then **not
|
||||
auditable** — no substitute exists on-site. §14 line, do not guess it from
|
||||
title similarity.
|
||||
|
||||
**This is NOT the 30/70 rule, and do not merge the two.** Cannibalisation is
|
||||
a SERP fact Google measured. The 30/70 duplication rule is a content-similarity
|
||||
question with **no data source here**: measuring it properly needs main-content
|
||||
extraction (strip nav/header/footer), and without that a naive comparison of
|
||||
two same-template pages returns ~95% similar for every site, which is a
|
||||
confident false positive. So 30/70 stays an explicit LLM judgement over the
|
||||
≥3 same-family pages STEP 5 now samples for it — label it as judgement in the
|
||||
report, never as a measurement, and never quote a similarity percentage you
|
||||
did not compute.
|
||||
|
||||
Report: top queries; flag **QUICK WINS** = rows with position between 4
|
||||
and 10 AND high impressions (candidates to push onto page 1 with a
|
||||
title/meta/content tweak). Report index coverage from `inspect`. All
|
||||
emitted into SEO.md §2 (technical) and §8 (quick wins).
|
||||
|
||||
**`inspect` also returns `rich_results` — Google's own structured-data
|
||||
verdict on the live indexed URL.** It rides the same response (no extra
|
||||
call, no extra quota). This is the only programmatic JSON-LD validation in
|
||||
the system; everything else about schema is read by eye.
|
||||
|
||||
```
|
||||
rich_results.verdict : PASS | FAIL | NEUTRAL | VERDICT_UNSPECIFIED | ABSENT
|
||||
rich_results.types[] : {type, items, errors, warnings, issues[]}
|
||||
```
|
||||
|
||||
- `FAIL` + a type carrying `errors > 0` → that type **cannot show as a rich
|
||||
result**. Bundle item, cite the `issues[]` message verbatim — it is
|
||||
Google's wording, not ours, and geo-analyzer owns the JSON-LD fix
|
||||
(CROSS-AGENT NOTE).
|
||||
- `warnings` → recommended fields missing. Report, do not gate on them.
|
||||
- **`ABSENT` means Google detected no rich results on this URL** — the key
|
||||
is omitted upstream when nothing is found. It is NOT an error and NOT
|
||||
proof the markup is broken: a page with no structured data reads the same
|
||||
as one whose markup Google never parsed. Say "none detected", never
|
||||
"invalid".
|
||||
- `ABSENT` while the repo clearly ships JSON-LD → real finding: the markup
|
||||
is not reaching Google (SPA-rendered, blocked, or malformed). Cross-check
|
||||
before claiming it.
|
||||
|
||||
**Bound this honestly.** `index:inspect` is per-URL, quota'd, and works only
|
||||
on a GSC-verified property. It validates the URLs you sampled — not the
|
||||
site. Its reach is the STEP 9 COVERAGE ratio, and §14 must say so rather
|
||||
than let one PASS imply site-wide valid markup.
|
||||
|
||||
If `status=degraded` → note it in §2 and emit the §11 user action
|
||||
"Connecter GSC: `make seo-connect`".
|
||||
|
||||
@@ -472,119 +359,8 @@ Fetch rendered HTML. Extract and analyze:
|
||||
|
||||
## STEP 5 — ON-PAGE AUDIT `[both]`
|
||||
|
||||
### Rendering gate — run this BEFORE anything else in STEP 5 (R2)
|
||||
|
||||
```bash
|
||||
bash ~/.claude/lib/seo-data/fetch.sh rendercheck --url "https://$DOMAIN/"
|
||||
```
|
||||
|
||||
STEP 2 has always recorded `RENDERING: SSR/SSG/SPA/hybrid` and nothing ever
|
||||
acted on it. This is the rule that does. The verdict comes from what the
|
||||
server actually sent, not from reading package.json — a React SPA and a
|
||||
Next.js SSR app are indistinguishable there.
|
||||
|
||||
**`verdict: client-rendered` → REFUSE to score the On-page axis.** Do not
|
||||
score it low. Do not score it at all:
|
||||
- On-page → `N/A — content not in served HTML (client-rendered)`. Redistribute
|
||||
nothing; a missing axis is not a zero.
|
||||
- Every curl-based meta/H1/JSON-LD check would report "missing" against a site
|
||||
that may be perfectly correct once hydrated. Those are FALSE findings, and
|
||||
a bundle built on them would "fix" meta tags that already exist.
|
||||
- **No bundle item may come from a live on-page check on this site.** Source
|
||||
greps still apply — the JSX carries the tags — but you cannot tell which
|
||||
route renders what, so treat them as inventory, not as per-page findings.
|
||||
- `linkgraph` will refuse too (`no_links_in_html`) — the same blindness. Do
|
||||
not work around either refusal.
|
||||
|
||||
Still fully auditable, and worth saying so rather than returning an empty
|
||||
report: robots.txt, sitemap.xml, HTTP headers, redirects, `.htaccess` /
|
||||
framework config, CWV via CrUX (field data is real-user, hydration included),
|
||||
GSC queries + index coverage, legal pages, image weights.
|
||||
|
||||
**`verdict: partial`** → shell plus an SSR'd head, or a genuinely thin page.
|
||||
Score what is present, name what is not, and say which of the two you think
|
||||
it is.
|
||||
|
||||
**§0 line, mandatory when not server-rendered:**
|
||||
`Rendering: client-rendered — On-page NOT scored (content absent from served
|
||||
HTML). Global score excludes it. Fix: SSR/SSG (CLAUDE.md: public sites are
|
||||
never SPAs).`
|
||||
|
||||
This is the honest half of the R1/R2 call: we do not render JS (no Playwright,
|
||||
no Chromium), so we do not pretend to see what JS paints. Refusing is the
|
||||
finding.
|
||||
|
||||
**Record the denominator BEFORE sampling.** This step samples; the report
|
||||
says "audit". On a 500-page site a 12-page sample is 2.4% — the On-page score
|
||||
is an extrapolation from it, and the reader cannot know unless you print it.
|
||||
|
||||
```bash
|
||||
bash ~/.claude/lib/seo-data/fetch.sh sitemap --url "https://$DOMAIN/sitemap.xml"
|
||||
```
|
||||
|
||||
Returns `{count, urls[], index, dropped, ...}` — the coverage denominator and
|
||||
your sampling frame. It follows a `<sitemapindex>` one level, dedupes, strips
|
||||
whitespace, and handles `.xml.gz`. No auth, no venv, no Google.
|
||||
|
||||
Read it honestly:
|
||||
- `count` → the denominator for the STEP 9 COVERAGE line.
|
||||
- `dropped > 0` → entries that were not usable URLs. Worth a §14 line: a
|
||||
sitemap emitting junk is a tooling finding.
|
||||
- `children_failed > 0` or `children_skipped` → the frame is incomplete. Say
|
||||
so; do NOT present a partial denominator as the total.
|
||||
- `status: degraded` → denominator UNKNOWN. Print that, never let silence
|
||||
imply full coverage. `reason: unsafe_xml_dtd` is not a glitch — a sitemap
|
||||
carrying a DTD is broken tooling or a billion-laughs aimed at the auditor.
|
||||
Report it as a finding.
|
||||
|
||||
**Guard every URL before it reaches curl.** These come from the target's own
|
||||
server, not from the operator — the one place in this audit where a remote
|
||||
file's bytes flow into a shell:
|
||||
|
||||
```bash
|
||||
U="$(bash ~/.claude/lib/url-guard.sh url "$RAW_FROM_SITEMAP")" || continue
|
||||
```
|
||||
|
||||
The verb applies a garbage filter, not that guard; the guard belongs at the
|
||||
point of use (same contract as the sameAs check in geo-analyzer).
|
||||
|
||||
### Meta tags per page (sample 5-15 key pages)
|
||||
|
||||
**Group the sitemap URLs into families first** — a family is "pages one
|
||||
template renders". You do not need framework routing knowledge to see them,
|
||||
but you DO need to look at the actual URL shape, because it varies:
|
||||
|
||||
| Layout | Example | Family signal |
|
||||
|---|---|---|
|
||||
| Nested | `/creation-site-internet/essonne-91/`, `/creation-site-internet/seine-et-marne-77/` | **shared parent path** → 25 pages, 1 family |
|
||||
| **Flat** | `/lavage-auto-pomponne`, `/lavage-auto-torcy`, `/lavage-auto-chelles` | **shared slug prefix** → 8 pages, 1 family |
|
||||
|
||||
Both are real, measured on two live sites. First-path-segment alone handles
|
||||
the nested case and **fails the flat one**: those 8 city pages read as 8
|
||||
unrelated singletons, so the largest "family" becomes `/services` (5) and the
|
||||
doorway-page risk — the exact thing the 30/70 rule exists to catch — is
|
||||
invisible. Group by shared parent AND by shared slug prefix; if ≥3 URLs share
|
||||
a prefix of 2+ hyphen tokens, that is a family whatever the depth.
|
||||
|
||||
Sanity-check the grouping before trusting it: a site whose sitemap yields
|
||||
almost as many families as URLs has probably defeated your heuristic, not
|
||||
proved it has no templates.
|
||||
|
||||
**Sample by finding class, because the classes need opposite samples:**
|
||||
|
||||
| Looking for | Sample | Why |
|
||||
|---|---|---|
|
||||
| Code defects (canonical, OG, `<img>` dims, hreflang) | **1 per family** | one template renders the whole family — a missing canonical in `[dept]/index.astro` breaks all 25 identically. 1 per family ≈ 100% SOURCE coverage for ~8 fetches. |
|
||||
| **Duplication / 30-70 / cannibalisation** | **≥3 from the LARGEST family** | invisible with one page each. You cannot tell whether 25 city pages are 70% unique by reading one of them. |
|
||||
| Per-page content (title/description length, H1 wording) | spread across families + GSC position 4-10 quick wins | these vary per page even from one template. |
|
||||
|
||||
"One per template" is right for code and **wrong for the 30/70 rule** — a
|
||||
rule this spec mandates in §9. Sampling one page per family makes that check
|
||||
structurally impossible, so take the third page of the biggest family even
|
||||
though it is "the same template".
|
||||
|
||||
An un-sampled family is an un-audited family. Name the ones you skipped.
|
||||
|
||||
For each sampled page:
|
||||
```
|
||||
PAGE: <path>
|
||||
@@ -620,32 +396,10 @@ grep -rE '<img[^>]*>' --include="*.html" --include="*.astro" --include="*.tsx" -
|
||||
# Images missing dimensions (CLS risk)
|
||||
grep -rE '<img[^>]*>' --include="*.html" --include="*.astro" --include="*.tsx" --include="*.jsx" --include="*.php" . 2>/dev/null | grep -vE 'width=|height=' | head -30
|
||||
|
||||
# Check image asset sizes — source only, never build output (C1a)
|
||||
mapfile -t FEXCL < <(bash ~/.claude/lib/source-scope.sh findargs)
|
||||
find . "${FEXCL[@]}" -type f \( -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" -o -iname "*.gif" \) -printf "%s %p\n" 2>/dev/null | sort -rn | head -20
|
||||
# Check image asset sizes
|
||||
find . -type f \( -iname "*.jpg" -o -iname "*.jpeg" -o -iname "*.png" -o -iname "*.gif" \) ! -path "./node_modules/*" ! -path "./.git/*" -printf "%s %p\n" 2>/dev/null | sort -rn | head -20
|
||||
```
|
||||
|
||||
**Why the guard, and why `find` specifically (C1a).** `grep` and `find`
|
||||
disagree about this repo and you use both. Claude Code routes `grep` through
|
||||
ugrep with `--ignore-files`, so it honours `.gitignore` and never descends
|
||||
into a gitignored `dist/`. `find` honours nothing. Measured on a real Astro
|
||||
repo: this command returned **92 images, 45 of them under `dist/`** — every
|
||||
asset twice, source and generated copy, byte-identical. So "top 20 by size"
|
||||
was ~10 real images dressed as 20, and a batch-C item
|
||||
(`cwebp -q 80 <img> -o <img>.webp`) could target `dist/og-image.png`, whose
|
||||
`.webp` the dispatcher's own `npm run build` then erases. The fix lands,
|
||||
verification passes, nothing survives.
|
||||
|
||||
`FEXCL` MUST be consumed as a quoted array. `find . $FEXCL …` lets the shell
|
||||
glob `*/dist/*` against the CWD and hand the matches to find as search paths
|
||||
— that made the same run return 135 hits and kept every `dist/` file.
|
||||
|
||||
Do NOT add these exclusions to the `grep` lines: the shim already covers
|
||||
them, `public/` is deliberately kept (it is Astro/Vite/Next SOURCE and holds
|
||||
`favicon.ico`, `apple-touch-icon.png`, `robots.txt` — the very files STEP 4
|
||||
curls), and it is build output only for Hugo/Gatsby, which the script
|
||||
detects.
|
||||
|
||||
Flag images over 100 KB as compression candidates. WebP/AVIF preferred
|
||||
over JPEG/PNG.
|
||||
|
||||
@@ -666,34 +420,6 @@ Each embedded or self-hosted video should have:
|
||||
|
||||
### Internal linking + topic clusters (silos sémantiques)
|
||||
|
||||
```bash
|
||||
bash ~/.claude/lib/seo-data/fetch.sh linkgraph --url "https://$DOMAIN/sitemap.xml"
|
||||
```
|
||||
|
||||
**This answers the two questions below, which this spec has always asked and
|
||||
never had a command for (C3).** Crawls every sitemap URL once, extracts
|
||||
internal `<a href>`, and returns `orphans`, `beyond_3_clicks`, `unreachable`,
|
||||
`max_depth`. Measured cost: 24 pages in 2.7 s, 86 in 3.8 s — cheap enough to
|
||||
always run on FULL.
|
||||
|
||||
Read it honestly:
|
||||
- `orphans` present → real finding, act on it.
|
||||
- **`orphans_withheld: true` → there is NO orphan list, and you must not
|
||||
invent one.** It appears when the crawl was capped or any page failed. An
|
||||
orphan cannot be sampled: proving a page has no inbound link means having
|
||||
read every other page, so a partial crawl invents orphans. "Page X has no
|
||||
inbound links" when it does sends the client fixing what is not broken.
|
||||
§14 line, not a finding.
|
||||
- `reason: no_links_in_html` → **not a site with zero links; a site whose
|
||||
links are rendered by JS.** Every page would look orphaned — the worst false
|
||||
positive this tool could emit — so the verb refuses instead. Flag the SPA in
|
||||
§0 and stop; do not hand-roll a link audit around it.
|
||||
- `unreachable` ⊃ `orphans`: a page can have inbound links yet sit outside the
|
||||
homepage's reach (linked only from another unreachable page). Both matter,
|
||||
they are not the same finding.
|
||||
- `max_depth` > 3 → `beyond_3_clicks` names the pages. That is the ":613"
|
||||
check, now measured rather than asserted.
|
||||
|
||||
Sample critical pages. Check:
|
||||
- Every important page reachable within 3 clicks from homepage?
|
||||
- Navigation consistent?
|
||||
@@ -890,135 +616,29 @@ FIX: AUTO (<what agent will do>) | USER (<what user must do>)
|
||||
|
||||
| Axis | Weight (local B2C) | Weight (SaaS/national/content) | Score /20 |
|
||||
|---|---|---|---|
|
||||
| Technical (perf, CWV, indexability) | 20% | 30% | |
|
||||
| Technical (perf, CWV, security headers, indexability) | 20% | 30% | |
|
||||
| On-page (content, meta, headings, images, video, a11y, i18n) | 20% | 30% | |
|
||||
| SEO Local (NAP, GMB, citations) | 25% | 5% | |
|
||||
| Off-page (unlinked brand mentions — backlinks/authority NOT auditable, §14) | 10% | 15% | |
|
||||
| Off-page (backlinks, mentions, authority) | 10% | 15% | |
|
||||
| Social presence | 10% | 5% | |
|
||||
| Competitive position | 5% | 10% | |
|
||||
| Legal compliance | 10% | 5% | |
|
||||
|
||||
**Compute the scores, do not feel them (I7).** Emit your findings, then let
|
||||
the engine do the arithmetic:
|
||||
|
||||
```bash
|
||||
bash ~/.claude/lib/seo-data/fetch.sh score --findings /tmp/seo-findings.json
|
||||
```
|
||||
|
||||
```json
|
||||
{"depth":"FULL","profile":"local",
|
||||
"axes":{"technical":{"findings":[{"severity":"haute","affected":9,"sampled":12}]},
|
||||
"on-page":{"status":"na","reason":"client-rendered (R2)"},
|
||||
"off-page":{"status":"na","reason":"backlinks unauditable (I1)"}}}
|
||||
```
|
||||
|
||||
`profile`: `local` (B2C) | `national` (SaaS/national/content). Severities are
|
||||
`critique|haute|moyenne|basse` — `/harden`'s scale (-15/-8/-3/-1, clamp,
|
||||
then /5 into /20), so the whole skill family speaks one vocabulary.
|
||||
|
||||
**The split matters.** WHICH findings exist and how severe each is stays your
|
||||
judgement — irreducible. The addition is not: same findings in, same score
|
||||
out. Until now every axis was felt, so two runs over identical code could
|
||||
disagree, and `/client-handover` gates on 17/20.
|
||||
|
||||
- `affected`/`sampled` (optional) shift severity ONE step: ≥50% of the sample
|
||||
escalates, a single page de-escalates. A defect on 1 of 12 pages is not the
|
||||
defect on 12 of 12; pretending so is what made the old numbers wobble.
|
||||
- `status: "na"` → the axis is EXCLUDED and the remaining weights are
|
||||
renormalised for you. This is the R2 rule (client-rendered on-page) and the
|
||||
I1 rule (unauditable off-page), finally computed instead of done by hand.
|
||||
**N/A is not a zero** and the engine will not let it behave like one.
|
||||
- `status: "error"` → malformed findings. Fix them; never fall back to
|
||||
eyeballing a number.
|
||||
- Run it twice on the same file before publishing. If the output moved, your
|
||||
findings moved, and that is the thing to explain.
|
||||
|
||||
**Technical axis note:** CWV scored on CrUX field data (75th percentile,
|
||||
real users, from STEP 4) when available; otherwise lab PageSpeed
|
||||
Lighthouse run.
|
||||
|
||||
**Security headers are NOT scored here (I4).** `/harden` owns them and
|
||||
grades them out of 100 with three external validators — pricing them into
|
||||
this axis too was double-counting the same finding in two reports
|
||||
(`depth-matrix.md:29` already said drop; this spec contradicted it).
|
||||
- Dispatched from `/harden` (its prompt says NARROW-SCOPE): headers ARE the
|
||||
job — audit and score them per its brief, ignore this note.
|
||||
- Dispatched from `/seo`: do not score CSP, HSTS, X-Frame-Options,
|
||||
X-Content-Type-Options, Referrer-Policy, Permissions-Policy, COOP/CORP,
|
||||
cookie flags. STEP 4 still reads them — you need them for the one
|
||||
carve-out below — but they earn and lose no points here.
|
||||
|
||||
**Carve-out — `X-Robots-Tag` stays.** It is an indexing directive wearing a
|
||||
header's clothes: `noindex` served there deindexes the page as surely as a
|
||||
meta robots tag. Score it under indexability. That is what
|
||||
`depth-matrix.md:29` means by "unless it directly affects indexability" —
|
||||
it is the header that does, and the security headers above are not.
|
||||
|
||||
**Drop ≠ silence.** A user who never runs `/harden` must not read a clean
|
||||
Technical score as clean headers. Whenever depth=FULL, emit in §14:
|
||||
`Security headers (CSP, HSTS, X-Frame-Options…) — not scored here: /harden
|
||||
owns them (0-100 + Observatory/SecurityHeaders/SSL Labs). Run /harden
|
||||
<url>. Observed live this run: <present list | none observed>.`
|
||||
Name what you saw. An omission has to stay legible — the same reason
|
||||
COVERAGE is mandatory in STEP 9.
|
||||
|
||||
**On-page axis note (R2).** `rendercheck` verdict `client-rendered` → this
|
||||
axis is `N/A — content not in served HTML`, excluded from the weighted global,
|
||||
NOT scored zero. A zero says "your on-page is bad"; N/A says "we could not
|
||||
see it", and only one of those is true. Renormalise the remaining weights over
|
||||
the axes actually scored and say so on the SEO GLOBAL line. The code ceiling
|
||||
must state that no code fix raises an axis we did not measure — the unlock is
|
||||
SSR/SSG, and that is a user action, not a bundle item.
|
||||
|
||||
**Off-page axis note (I1).** Score ONLY the unlinked brand mentions
|
||||
gathered in STEP 6 (`web_search "<business-name>" -site:<domain>`).
|
||||
Backlink profile and domain authority have NO data source here — no index,
|
||||
no API, nothing. NEVER price them into the number: an unmeasured
|
||||
sub-component cannot be judged, and this axis carries 10-15% of a score
|
||||
that reaches a client via `/client-handover`. A low mention count is a low
|
||||
mention count — it is NOT evidence of a weak backlink profile.
|
||||
|
||||
Mandatory §14 line whenever depth=FULL, verbatim:
|
||||
`Backlinks / domain authority — NOT audited: no free backlink index is
|
||||
practical, and none is wired. Commercial: Ahrefs / Semrush / Majestic. The
|
||||
Off-page score above prices in brand mentions only.`
|
||||
|
||||
**This is the final state, not a placeholder (B1 killed, 2026-07-17.)** The
|
||||
free options were measured, not assumed:
|
||||
- **GSC has no links endpoint.** The Search Console API exposes exactly
|
||||
Search Analytics, Sitemaps, Sites, URL Inspection. The Links report is
|
||||
UI-only.
|
||||
- **Common Crawl's hyperlinkgraph is 17.3 GB gzipped** for the domain-edges
|
||||
file alone (+879 MB vertices, +2.3 GB ranks), measured live. Finding one
|
||||
domain's inbound links means scanning all of it, per audit. Not slow —
|
||||
non-viable, and abusive toward a nonprofit serving it free. The reference
|
||||
implementation everyone cites caps its download at 500 MiB, i.e. **2.9% of
|
||||
the edges file**, and reports whatever that arbitrary slice contained as a
|
||||
backlink profile. That is a random sample wearing a measurement's clothes,
|
||||
which is precisely what this axis note exists to prevent.
|
||||
- **Bing Webmaster's `GetUrlLinks` is the only free, viable source** — but it
|
||||
is first-party only (your verified properties), so it can never cover a
|
||||
competitor, and it needs the client's Bing account. See W2, deferred.
|
||||
|
||||
So: no number here beats a fabricated one. Weight deliberately unchanged —
|
||||
re-deriving it for an axis that is not going to widen would churn historical
|
||||
scores for nothing.
|
||||
|
||||
### LOCAL depth — 4 axes
|
||||
|
||||
| Axis | Weight (local B2C) | Weight (SaaS/national/content) | Score /20 |
|
||||
|---|---|---|---|
|
||||
| Technical (indexability, config) | 25% | 35% | |
|
||||
| Technical (security headers, indexability, config) | 25% | 35% | |
|
||||
| On-page (content, meta, headings, images, video, a11y, i18n) | 35% | 45% | |
|
||||
| SEO Local (markup, NAP in JSON-LD, legal) | 20% | 5% | |
|
||||
| Legal compliance (pages, CMP, mentions) | 20% | 15% | |
|
||||
|
||||
LOCAL axes not audited (Off-page, Social, Competitive) appear as
|
||||
`N/A — requires FULL audit` in the report. Off-page is the exception to
|
||||
that promise: FULL audits its brand-mentions share ONLY — backlinks and
|
||||
authority are unauditable at EVERY depth (see the Off-page axis note).
|
||||
Print `N/A — FULL audits brand mentions only` for it, never a bare
|
||||
"requires FULL audit" that FULL cannot keep.
|
||||
`N/A — requires FULL audit` in the report.
|
||||
|
||||
### Projected code-only score + trajectory to 17/20 (mandatory)
|
||||
|
||||
@@ -1056,9 +676,6 @@ misroutes the client-handover gate and the user's effort.
|
||||
|
||||
```
|
||||
SEO SCORING (<depth>)
|
||||
COVERAGE SOURCE: <N> of <M> page templates (<P>%) — skipped: <list|none>
|
||||
COVERAGE LIVE : <N> of <M> sitemap URLs (<P>%) — families: <fam N/M, …>
|
||||
| UNKNOWN (no sitemap / fetch degraded)
|
||||
Technical : XX/20 <justification>
|
||||
On-page : XX/20 <justification>
|
||||
SEO Local : XX/20 | N/A
|
||||
@@ -1070,28 +687,6 @@ Legal : XX/20 <justification>
|
||||
SEO GLOBAL (weighted): XX.X/20 (<depth>)
|
||||
```
|
||||
|
||||
**Both COVERAGE lines are mandatory, never omitted, never rounded up.** They
|
||||
are the honesty bound on every page-level axis: On-page and the on-page share
|
||||
of Technical are extrapolations from the sample, and `/client-handover` gates
|
||||
on these numbers.
|
||||
|
||||
**Report both, because they bound different findings — do not average them
|
||||
into one comforting number.**
|
||||
- **SOURCE** bounds CODE findings. One template renders its whole family, so
|
||||
1 page per family can legitimately reach 100% here. High SOURCE coverage is
|
||||
a real claim: the code paths were seen.
|
||||
- **LIVE** bounds CONTENT findings — title/description wording, thin pages,
|
||||
30/70 duplication. It stays low by design and that is fine, as long as it
|
||||
is printed. Measured on a real site: 12 of 86 URLs is 14% LIVE while the
|
||||
same 12 pages are 100% SOURCE. Reporting only the 14% understates the audit;
|
||||
reporting only the 100% oversells it. Both, or neither means anything.
|
||||
- LIVE < 25% → repeat in §0. A 17/20 for content drawn from 3% of a site is
|
||||
not a 17/20.
|
||||
- SOURCE < 100% → name the skipped templates in §0. That is not a sampling
|
||||
choice, it is code nobody read.
|
||||
- Denominator UNKNOWN (no sitemap, or `sitemap` degraded) → print UNKNOWN.
|
||||
Never let silence imply full coverage.
|
||||
|
||||
Per user instruction: this score represents **80% of the combined
|
||||
final score for local B2C (20% for GEO), or 75% for SaaS/national
|
||||
(25% for GEO)**. The `/seo` dispatcher combines SEO and GEO scores.
|
||||
@@ -1449,15 +1044,6 @@ PROCHAINE ETAPE : <highest-priority>
|
||||
`Write` on shared templates. `Write` is reserved for files you
|
||||
solely own: sitemap.xml, .htaccess, legal pages, new city/service
|
||||
pages. Full-template refactor → escalate as user action in §11.
|
||||
- **NEVER emit a bundle item targeting build output (C1a).** No path under
|
||||
`dist/ build/ .next/ .nuxt/ .output/ _site/ .astro/ .svelte-kit/ out/` —
|
||||
`bash ~/.claude/lib/source-scope.sh list` is the authoritative set. Those
|
||||
files are regenerated: the `npm run build` the dispatcher runs to VERIFY
|
||||
your fix is what erases it. The fix lands, verification passes, nothing
|
||||
survives, and the report claims it was applied. This bites batch C hardest
|
||||
(`cwebp -q 80 <img> -o <img>.webp` on a `dist/` asset writes a `.webp` the
|
||||
next build deletes). Fix the SOURCE that generates the artifact; if you
|
||||
cannot find it, that is a finding — say so, do not patch the artifact.
|
||||
- **Landing page protection.** Zero visible change except meta tags,
|
||||
footer links, JSON-LD, image optimization.
|
||||
- **Preserve existing valid SEO.** Don't rewrite correct tags.
|
||||
|
||||
+1
-239
@@ -80,247 +80,9 @@ fetch.sh queries --account client-a --property sc-domain:ex.com [--days 90] [--d
|
||||
→ {"status":"degraded","reason":"no_credentials"|"token_revoked"|"network_error"|"rate_limited"}
|
||||
|
||||
fetch.sh inspect --account client-a --property … --url https://ex.com/page
|
||||
→ {"status":"ok","source":"gsc","indexed":true,"coverage":"…","last_crawl":"…",
|
||||
"rich_results":{"verdict":"PASS|FAIL|NEUTRAL|VERDICT_UNSPECIFIED|ABSENT",
|
||||
"types":[{"type":"FAQ","items":2,"errors":2,"warnings":1,
|
||||
"issues":["Missing field 'acceptedAnswer'"]}]}}
|
||||
→ {"status":"ok","source":"gsc","indexed":true,"coverage":"…","last_crawl":"…"}
|
||||
→ {"status":"degraded","reason":"…"}
|
||||
|
||||
rich_results rides the SAME URL-Inspection response — Google already sends
|
||||
it, `inspect` used to discard it. No extra call, quota or OAuth scope.
|
||||
It is the only programmatic structured-data validation in the system.
|
||||
• verdict PARTIAL is never emitted — the API reserves it as unused.
|
||||
• verdict ABSENT is SYNTHETIC (not a Google enum): the API omits
|
||||
richResultsResult entirely when it detects no rich results. Surfaced
|
||||
as a value rather than a missing key, because a caller cannot tell an
|
||||
absent key apart from a check that never ran. ABSENT = "none
|
||||
detected", never "invalid".
|
||||
• errors/warnings count issue INSTANCES; issues[] is deduped — the same
|
||||
issueMessage repeats across every affected item.
|
||||
|
||||
fetch.sh cannibal --account client-a --property … [--days 90] [--rows 1000]
|
||||
→ {"status":"ok","source":"gsc","days":90,"rows_scanned":1000,"capped":true,
|
||||
"conflict_count":12,
|
||||
"conflicts":[{"query":"plombier paris","pages":3,"total_impressions":2400,
|
||||
"urls":[{"url":…,"clicks":…,"impressions":…,"position":…}]}]}
|
||||
→ {"status":"degraded","reason":"…"} # no account → NOT auditable
|
||||
|
||||
Keyword cannibalisation from Google's own data: queries where 2+ of OUR
|
||||
pages compete. Groups query+page rows; conflicts ranked by total
|
||||
impressions, and within each the strongest page first. `capped:true` means
|
||||
the row window was full — more conflicts exist past the cut, say so.
|
||||
Same auth, same quota family, no new scope: the API always accepted several
|
||||
dimensions at once, this engine only ever asked for one.
|
||||
• NOT the 30/70 duplication rule. This is a SERP fact Google measured.
|
||||
30/70 is content similarity, which has no data source here — doing it
|
||||
naively (compare two same-template pages without stripping nav/footer)
|
||||
returns ~95% similar for every site, a confident false positive. It stays
|
||||
an LLM judgement, labelled as one.
|
||||
• `queries` now takes `--dim query,page` (comma-separated) and `--rows`.
|
||||
Rows gained a `keys` list; `key` stays as keys[0], so the single-dim
|
||||
consumer is untouched.
|
||||
|
||||
safe_fetch.py — NOT a verb; the SSRF/DNS-rebinding-safe fetcher behind
|
||||
sitemap._fetch, so every network verb (sitemap, linkgraph, rendercheck,
|
||||
drift) inherits it. urlopen resolved then connected — two DNS lookups, a
|
||||
window a hostile authority uses to answer PUBLIC to validation and PRIVATE
|
||||
(169.254.169.254 metadata, 127.0.0.1, the LAN) to the connect. This resolves
|
||||
ONCE, validates every IP (ipaddress, dual-stack v4+v6), refuses if ANY is
|
||||
non-public (the multi-A vector), and connects to the exact validated IP with
|
||||
Host+SNI+cert for the real host — no second resolution to poison. Redirects
|
||||
are followed with each hop RE-VALIDATED (urlopen followed them blind).
|
||||
• Better than the source idea (claude-seo url_safety.py, MIT): dual-stack
|
||||
(theirs IPv4-only), no global monkeypatch so thread-safe by construction
|
||||
(theirs locks a patched socket.getaddrinfo), stdlib-only (no requests).
|
||||
• Refusal raises UnsafeTarget; callers already degrade → fail-open kept.
|
||||
• NOT covered, and said so: the shell `curl` in the agent specs runs in
|
||||
another process, unpinnable from here. Smaller surface (fixed set vs an
|
||||
operator-confirmed $DOMAIN); `curl --resolve` would close it, separate change.
|
||||
|
||||
fetch.sh sitemap --url https://ex.com/sitemap.xml
|
||||
→ {"status":"ok","source":"sitemap","index":false,"count":86,"dropped":0,
|
||||
"urls":["https://ex.com/", …]}
|
||||
→ {"status":"ok","index":true,"children_total":4,"children_read":4,
|
||||
"children_failed":0,"count":312,…} # <sitemapindex>, one level deep
|
||||
→ {"status":"degraded","reason":"fetch_failed"|"parse_failed"|"no_urls"
|
||||
|"unsafe_xml_dtd"}
|
||||
|
||||
No auth, no Google, no venv: stdlib only (urllib + xml.etree + gzip).
|
||||
Gives STEP 9's COVERAGE line the denominator it was told to print and never
|
||||
had, and STEP 5 a real sampling frame. Dedupes, strips whitespace, handles
|
||||
.xml.gz. Caps: 50 children of an index, 50k URLs, 20 MB read — each cut is
|
||||
REPORTED (children_skipped / truncated), never silent.
|
||||
|
||||
• NOT a security boundary. urllib fetches these, so nothing here reaches a
|
||||
shell. The CONSUMER interpolates them into curl, so seo-analyzer runs
|
||||
lib/url-guard.sh at the point of use — same contract as the sameAs check.
|
||||
A second copy of the guard here would only drift.
|
||||
• `unsafe_xml_dtd`: a sitemap NEVER has a DTD (sitemaps.org is <?xml?> then
|
||||
<urlset xmlns=>). Any doctype/entity is refused BEFORE parsing. xml.etree
|
||||
does not expand external entities, but it IS billion-laughs-vulnerable —
|
||||
1 KB expands to gigabytes, and the 20 MB read ceiling bounds the input,
|
||||
not the expansion. Refusing the construct beats depending on parser
|
||||
internals AND keeps this stdlib-only; defusedxml would drag in a venv for
|
||||
a document type that has no legitimate DTD.
|
||||
|
||||
fetch.sh rendercheck --url https://ex.com/
|
||||
→ {"status":"ok","verdict":"server-rendered"|"client-rendered"|"partial",
|
||||
"body_text_chars":7650,"h1_in_html":1,"jsonld_in_html":9,
|
||||
"meta_description_in_html":true,"html_bytes":132447,
|
||||
"warning":"…"} # warning only when not server-rendered
|
||||
|
||||
R2, the honest half of the SPA call. seo-analyzer has always recorded
|
||||
`RENDERING: SSR/SSG/SPA` and never acted on it; this is the signal it acts
|
||||
on. Verdict comes from what the server SENT — package.json cannot tell a
|
||||
React SPA from a Next.js SSR app.
|
||||
• client-rendered → the agent REFUSES to score On-page (N/A, not zero: a
|
||||
zero says "your on-page is bad", N/A says "we could not see it"). Every
|
||||
curl-based meta/H1/JSON-LD check would report "missing" against a site
|
||||
that is fine once hydrated — false findings, and a bundle that "fixes"
|
||||
tags which already exist.
|
||||
• Does NOT render JS. No Playwright, no Chromium, no venv. Refusing IS the
|
||||
finding.
|
||||
• Script/style text is not page text: measured 7 chars on a React shell
|
||||
whose inline window.__INITIAL_STATE__ is large. Without that, a 200 KB
|
||||
bundle reads as a rich page.
|
||||
• Measured 2026-07-17: zenquality 7650 chars/1 h1/9 jsonld and
|
||||
lavageangels356 13973/1/1 → server-rendered; a Vite shell → 7/0/0.
|
||||
|
||||
fetch.sh linkgraph --url https://ex.com/sitemap.xml [--max 500]
|
||||
→ {"status":"ok","source":"linkgraph","pages_crawled":86,"pages_failed":0,
|
||||
"total_internal_links":2015,"capped":false,"max_depth":2,
|
||||
"orphans":[…],"beyond_3_clicks":[…],"unreachable":[…]}
|
||||
→ {"status":"ok",…,"orphans_withheld":true,"reason_withheld":"crawl incomplete…"}
|
||||
→ {"status":"degraded","reason":"no_links_in_html"|"no_pages_fetched"|…}
|
||||
|
||||
Answers seo-analyzer.md:613 ("reachable within 3 clicks?") and :616 ("orphan
|
||||
pages?") — asked since forever, never computed. Stdlib only (urllib +
|
||||
html.parser + urljoin), no auth. Measured: 24 pages in 2.7s, 86 in 3.8s.
|
||||
• EXHAUSTIVE OR NOTHING. Orphans cannot be sampled: proving no inbound
|
||||
link means having read every other page. If the crawl is capped or any
|
||||
page failed, orphans are WITHHELD, never truncated — a false orphan
|
||||
sends a client fixing what is not broken.
|
||||
• no_links_in_html = a JS-rendered site, not a link-less one. Every page
|
||||
would read as orphaned, so it REFUSES rather than report that. Does not
|
||||
render JS by design (see the R1/R2 arbitration).
|
||||
• Filters what a link graph must never hold: assets (seen live:
|
||||
/css/main.css?v=1778157313), #anchors, mailto:/tel:/javascript:, other
|
||||
hosts. Normalises the trailing slash so /blog and /blog/ are one node
|
||||
rather than a phantom orphan pair.
|
||||
• Mock is pages.json ({url: html}), not a single page.html: one fixture
|
||||
cannot express a graph — every node would carry identical links.
|
||||
|
||||
fetch.sh score --findings <path.json | ->
|
||||
→ {"status":"ok","axes":{"technical":{"score_20":17.8,"weight":0.2,
|
||||
"weight_renormalised":0.2857,"findings":2}},
|
||||
"na":["off-page","on-page"],"weights_renormalised":true,"global_20":17.6}
|
||||
→ {"status":"error","reason":"unknown severity: 'bogus'"|"bad_findings_json"}
|
||||
|
||||
I7. /harden has a real scale (SKILL.md:435: -15/-8/-3/-1, clamp [0,100]);
|
||||
/seo had none, so every axis was FELT and two runs over identical code could
|
||||
disagree — while /client-handover gates on 17/20. Same scale here, /5 into
|
||||
/20, one vocabulary across the family.
|
||||
• The split: WHICH findings exist and how severe each is stays the LLM's
|
||||
judgement. The addition is not. Same findings in, same score out.
|
||||
• affected/sampled shift severity ONE step: >=50% of the sample escalates,
|
||||
a single page de-escalates. A defect on 1 of 12 pages is not the defect
|
||||
on 12 of 12.
|
||||
• status:"na" → axis EXCLUDED, remaining weights renormalised. This is
|
||||
R2's rule (client-rendered on-page) and I1's (unauditable off-page),
|
||||
computed rather than done by hand. N/A is not a zero, and the engine
|
||||
will not let it act like one.
|
||||
• Malformed input is an error, never a silently wrong number — unlike the
|
||||
fetch verbs, a degrade here would mean bad input, not a network fact.
|
||||
|
||||
fetch.sh schema_gen <reservation|order|discussion|profile> [flags] [--script-tag]
|
||||
→ {"status":"ok","source":"schema_gen","type":"<@type>","jsonld":{…}}
|
||||
→ {"status":"error","reason":"bad_usage"} # a REQUIRED flag omitted
|
||||
→ {"status":"degraded","reason":"…"} # a required flag given, empty
|
||||
|
||||
fetch.sh schema_gen reservation --provider "Marea NYC" \
|
||||
--start 2026-06-04T19:30:00-04:00 --party-size 4
|
||||
fetch.sh schema_gen order --merchant "Acme Pizza" --order-url https://acme.example/order
|
||||
fetch.sh schema_gen discussion --headline "…" --author "Sara Park" \
|
||||
--url https://forum.example.com/t/123 --date 2026-05-12T14:00:00Z
|
||||
fetch.sh schema_gen profile --name "Daniel Agrici" --url https://agricidaniel.com/about \
|
||||
--same-as https://github.com/AgriciDaniel --knows-about "SEO" "Schema markup"
|
||||
|
||||
Adapted from claude-seo's `schema_generate.py` (MIT) into this contract.
|
||||
Our system only AUDITS existing markup elsewhere; this is the one verb
|
||||
that GENERATES it — deterministic JSON-LD skeletons for the four v2
|
||||
high-leverage Schema.org types, so geo-analyzer's G2 batch stops
|
||||
hand-writing markup by hand. It only generates STRUCTURE: unknown field
|
||||
VALUES are the caller's job, `[À COMPLÉTER]` for anything unconfirmed —
|
||||
this verb never invents a sameAs, an email, or a business name.
|
||||
• Stdlib only, no network, no auth — runs even without the venv.
|
||||
• `--script-tag` wraps the cleaned jsonld in
|
||||
`<script type="application/ld+json">…</script>` under a `script` key,
|
||||
still inside the `ok` envelope. It must be given AFTER the type
|
||||
(`schema_gen reservation … --script-tag`, not before) — argparse
|
||||
subcommand flags only parse after their subcommand.
|
||||
• Never emits a JSON `null`: fields left unset are omitted from the
|
||||
`jsonld` object entirely rather than serialised as `null`.
|
||||
• A REQUIRED flag omitted → `{"status":"error","reason":"bad_usage"}`,
|
||||
exit 2 (bad usage, like every other verb). A required flag GIVEN but
|
||||
empty (argparse cannot catch that) → `{"status":"degraded",...}`,
|
||||
exit 0 — fail-open, never a traceback.
|
||||
|
||||
fetch.sh content_quality [--file <path.txt>] < text_on_stdin
|
||||
→ {"status":"ok","source":"content_quality","filler_score":0,"ai_pattern_score":0,
|
||||
"information_density":1.0,"overall_quality":90,"flags":[],
|
||||
"matches":{"filler":[],"ai_patterns":[]}}
|
||||
→ {"status":"degraded","reason":"empty_input"|"<file error>"}
|
||||
|
||||
fetch.sh content_quality --file article.txt
|
||||
printf '%s' "$BODY_TEXT" | fetch.sh content_quality
|
||||
|
||||
Adapted from claude-seo's `content_quality.py` (MIT) into this contract.
|
||||
100% deterministic — regex/word-lists (QRG §4.6 filler phrases + a
|
||||
Wikipedia "AI Cleanup" catalogue of LLM-typical phrasings, CC BY-SA 4.0),
|
||||
no LLM call, no network. Reads the text to score from `--file <path>` or,
|
||||
when `--file` is `-` or omitted, from stdin — the same idiom `score.py`
|
||||
uses for `--findings`.
|
||||
• **ADVISORY, NOT A VERDICT.** The output never claims "this text is
|
||||
AI-written" — modern generative tools can pass every heuristic here,
|
||||
and human writers use some of these phrases too. `flags` are
|
||||
candidates for HUMAN REVIEW, never an automatic finding. geo-analyzer
|
||||
STEP 8 (Content Shape for AI) treats `overall_quality`/`flags` as ONE
|
||||
measured input that INFORMS the axis; the axis itself stays an LLM
|
||||
judgement (30/70, Definition Lead), never replaced by this score.
|
||||
• `filler_score`/`ai_pattern_score` (0-100, higher = worse) count
|
||||
phrase-list hits scaled per 1000 tokens; `information_density`
|
||||
(0.0-1.0) is entities + numbers per 100 tokens; `overall_quality`
|
||||
(0-100, higher is better) is the weighted composite (also folds in a
|
||||
bigram-repetition penalty even though that score isn't itself a
|
||||
top-level field). `flags` fires at fixed thresholds: `filler`,
|
||||
`ai-patterns`, `low-density`, `repetitive`.
|
||||
• Stdlib only (argparse/json/re/sys/collections/typing) — runs even
|
||||
without the venv. Empty/whitespace-only input degrades rather than
|
||||
returning a false zero-value "ok": an empty analysis is not a result.
|
||||
• This is filler/AI-pattern SHAPE, not fact-checking — a text can be
|
||||
dense and well-cited yet still wrong; that stays a human/LLM call.
|
||||
|
||||
fetch.sh drift --url https://ex.com/sitemap.xml [--max 500]
|
||||
→ {"status":"ok","baseline":true,"captured":"…","pages":24,"store":"…"}
|
||||
→ {"status":"ok","baseline":false,"since":"…","gone":[…],"new":[…],
|
||||
"regressions":[{"url":…,"field":"canonical","was":"…","now":null}],
|
||||
"changes":[{"url":…,"field":"title","was":"…","now":"…"}]}
|
||||
|
||||
On-page drift between audits. seo-analyzer.md:1365 keeps only "date + score
|
||||
+ key changes" as PROSE the LLM writes about its own previous prose: lossy,
|
||||
unreproducible, machine-uncomparable. So "the redesign silently dropped 40
|
||||
canonicals" stays invisible. This snapshots title/description/canonical/
|
||||
robots/h1_count/jsonld_types per URL and diffs them.
|
||||
• NOT rank tracking (the common misread of this feature elsewhere).
|
||||
Positions come from GSC `queries`. This is regression detection.
|
||||
• Runs over the WHOLE sitemap, never a sample: a drift over a sample that
|
||||
changes between runs compares nothing.
|
||||
• LOSING a signal = regression. CHANGING one = change, possibly intended —
|
||||
the agent judges that, the engine only says which kind it is.
|
||||
• Store: ~/.claude/seo-data/drift/<host>.json, 0700, written via
|
||||
os.replace — never a half-written baseline. Corrupt store → treated as
|
||||
a first run rather than crashing the audit.
|
||||
|
||||
fetch.sh forget --label client-a
|
||||
→ {"status":"ok","removed":true|false} # false = label wasn't in the store
|
||||
|
||||
|
||||
@@ -1,242 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic filler / AI-slop content-quality scorer. Stdlib only.
|
||||
|
||||
Adapted from claude-seo (github.com/AgriciDaniel/claude-seo, MIT),
|
||||
content_quality.py — rewritten to the lib/seo-data fail-open contract.
|
||||
|
||||
Scores a block of text against three regex/word-list heuristics: padding
|
||||
"filler" phrases (QRG §4.6), LLM-typical phrasings ("AI-pattern" list),
|
||||
and a measured information density (entities + numbers per token). 100%
|
||||
deterministic — no LLM call, no network.
|
||||
|
||||
ADVISORY, NOT A VERDICT. This never claims "this text is AI-written" —
|
||||
modern generative tools can pass every heuristic here, and human writers
|
||||
use some of these phrases too. A low overall_quality or a filler/
|
||||
ai-patterns flag is a candidate for human review, nothing more. In
|
||||
geo-analyzer's STEP 8 (Content Shape for AI) it is ONE measured input
|
||||
that INFORMS the axis, which stays an LLM judgement (30/70, Definition
|
||||
Lead) — never a replacement for it, and never auto-filed as a finding on
|
||||
its own.
|
||||
|
||||
Attribution: the AI-pattern list draws from the Wikipedia "AI Cleanup"
|
||||
project's catalogue of LLM-typical phrasings (CC BY-SA 4.0), the same
|
||||
list claude-seo cites.
|
||||
|
||||
Envelope (see `_cli`)::
|
||||
|
||||
{"status": "ok", "source": "content_quality",
|
||||
"filler_score": 0..100, # higher = more filler-like
|
||||
"ai_pattern_score": 0..100, # higher = more AI-pattern hits
|
||||
"information_density": 0.0..1.0,
|
||||
"overall_quality": 0..100, # composite, higher is better
|
||||
"flags": ["filler", "ai-patterns", "low-density", "repetitive"],
|
||||
"matches": {"filler": [...], "ai_patterns": [...]}}
|
||||
{"status": "degraded", "reason": "empty_input" | "<why>"}
|
||||
"""
|
||||
import argparse, json, re, sys
|
||||
from collections import Counter
|
||||
from typing import Iterable
|
||||
|
||||
# Padding / filler phrases QRG §4.6 flags as "little-to-no value". The
|
||||
# lists are the value of this module — kept intact from the source, not
|
||||
# trimmed.
|
||||
_FILLER_PHRASES = (
|
||||
"it's important to note that",
|
||||
"in this article, we'll explore",
|
||||
"in this article we will explore",
|
||||
"in today's fast-paced world",
|
||||
"in today's digital age",
|
||||
"in today's competitive landscape",
|
||||
"needless to say",
|
||||
"at the end of the day",
|
||||
"when it comes to",
|
||||
"when all is said and done",
|
||||
"in the realm of",
|
||||
"in the world of",
|
||||
"the bottom line is",
|
||||
"without further ado",
|
||||
"first and foremost",
|
||||
"last but not least",
|
||||
"for what it's worth",
|
||||
"it goes without saying",
|
||||
"as we all know",
|
||||
"the truth is that",
|
||||
"the fact of the matter is",
|
||||
"more often than not",
|
||||
"let's dive in",
|
||||
"let's dive into",
|
||||
"let's take a closer look",
|
||||
"let's take a deeper look",
|
||||
)
|
||||
|
||||
# LLM-typical phrasings (Wikipedia AI Cleanup catalogue, CC BY-SA 4.0;
|
||||
# also used by claude-seo, MIT). Conservative: only phrases that
|
||||
# disproportionately appear in LLM output. Adding to this list should
|
||||
# require corpus evidence, not intuition.
|
||||
_AI_PATTERNS = (
|
||||
"delve into",
|
||||
"delve deeper into",
|
||||
"in the ever-evolving",
|
||||
"ever-evolving landscape",
|
||||
"ever-changing landscape",
|
||||
"in the dynamic landscape",
|
||||
"navigating the",
|
||||
"navigate the complexities",
|
||||
"tapestry of",
|
||||
"rich tapestry",
|
||||
"intricate tapestry",
|
||||
"embark on a journey",
|
||||
"embarking on this",
|
||||
"a testament to",
|
||||
"a beacon of",
|
||||
"the cornerstone of",
|
||||
"a cornerstone of",
|
||||
"at the heart of",
|
||||
"at its core",
|
||||
"in essence,",
|
||||
"in conclusion,",
|
||||
"ultimately,",
|
||||
"moreover,",
|
||||
"furthermore,",
|
||||
"however, it's worth noting",
|
||||
"it's worth noting that",
|
||||
"by leveraging",
|
||||
"leverage the power of",
|
||||
"leveraging the power of",
|
||||
"harness the power of",
|
||||
"unlock the potential",
|
||||
"unlock the full potential",
|
||||
"the realm of possibilities",
|
||||
"open up a world of",
|
||||
"a world of possibilities",
|
||||
"elevate your",
|
||||
"transform your",
|
||||
"revolutionize the way",
|
||||
"game-changer",
|
||||
"game-changing",
|
||||
"cutting-edge",
|
||||
"state-of-the-art",
|
||||
"in summary,",
|
||||
"to summarize,",
|
||||
"to put it simply,",
|
||||
"in a nutshell,",
|
||||
)
|
||||
|
||||
_TOKEN_RE = re.compile(r"[A-Za-z][A-Za-z'\-]*")
|
||||
_NUMBER_RE = re.compile(r"\b\d+(?:[.,]\d+)?(?:%|st|nd|rd|th)?\b")
|
||||
# Capitalised multi-word names: rough proper-noun heuristic. Two or more
|
||||
# capitalised tokens in a row count as one entity.
|
||||
_ENTITY_RE = re.compile(r"\b(?:[A-Z][a-z]+(?:\s+[A-Z][a-z]+)+)\b")
|
||||
|
||||
|
||||
def _count_phrase_hits(text: str, patterns: Iterable[str]) -> list:
|
||||
"""Patterns that appear at least once in text (case-insensitive)."""
|
||||
lowered = text.lower()
|
||||
return [p for p in patterns if p in lowered]
|
||||
|
||||
|
||||
def _repetition_score(tokens):
|
||||
"""Bigram repetition: fraction of bigrams that recur more than once."""
|
||||
if len(tokens) < 4:
|
||||
return 0.0
|
||||
bigrams = [tokens[i] + " " + tokens[i + 1] for i in range(len(tokens) - 1)]
|
||||
counts = Counter(bigrams)
|
||||
repeated = sum(1 for v in counts.values() if v > 1)
|
||||
return repeated / max(1, len(counts))
|
||||
|
||||
|
||||
def analyse(text):
|
||||
"""Score text against the filler / AI-pattern / density / repetition
|
||||
heuristics. Advisory only — see module docstring."""
|
||||
tokens = [t.lower() for t in _TOKEN_RE.findall(text)]
|
||||
n_tokens = len(tokens)
|
||||
|
||||
filler_hits = _count_phrase_hits(text, _FILLER_PHRASES)
|
||||
ai_hits = _count_phrase_hits(text, _AI_PATTERNS)
|
||||
|
||||
# Density: entities + numbers per 100 tokens. A high-density article
|
||||
# (case studies, data journalism) lands at ~5+; generic filler <2.
|
||||
entities = len(_ENTITY_RE.findall(text))
|
||||
numbers = len(_NUMBER_RE.findall(text))
|
||||
density_per_100 = (entities + numbers) * 100.0 / max(1, n_tokens)
|
||||
information_density = min(1.0, density_per_100 / 10.0)
|
||||
|
||||
rep_score = int(round(_repetition_score(tokens) * 100))
|
||||
|
||||
# Scale to per-1000 tokens so the score is comparable across lengths.
|
||||
scale = max(1.0, n_tokens / 1000.0)
|
||||
filler_score = min(100, int(round(len(filler_hits) / scale * 25)))
|
||||
ai_pattern_score = min(100, int(round(len(ai_hits) / scale * 15)))
|
||||
|
||||
flags = []
|
||||
if filler_score >= 50:
|
||||
flags.append("filler")
|
||||
if ai_pattern_score >= 40:
|
||||
flags.append("ai-patterns")
|
||||
if information_density < 0.20:
|
||||
flags.append("low-density")
|
||||
if rep_score >= 30:
|
||||
flags.append("repetitive")
|
||||
|
||||
# Composite: invert penalty signals, weight by impact. Same weights
|
||||
# as the source — the length bonus caps at 1000 tokens.
|
||||
overall = (
|
||||
(100 - filler_score) * 0.25
|
||||
+ (100 - ai_pattern_score) * 0.25
|
||||
+ information_density * 100 * 0.25
|
||||
+ (100 - rep_score) * 0.15
|
||||
+ min(100, n_tokens / 10.0) * 0.10
|
||||
)
|
||||
|
||||
return {
|
||||
"filler_score": filler_score,
|
||||
"ai_pattern_score": ai_pattern_score,
|
||||
"information_density": round(information_density, 3),
|
||||
"overall_quality": int(round(overall)),
|
||||
"flags": flags,
|
||||
"matches": {"filler": filler_hits, "ai_patterns": ai_hits},
|
||||
}
|
||||
|
||||
|
||||
def _build_parser():
|
||||
p = argparse.ArgumentParser(
|
||||
description="Deterministic filler / AI-slop content-quality scorer."
|
||||
)
|
||||
p.add_argument("--store", default=None) # accepted+ignored (dispatch)
|
||||
p.add_argument(
|
||||
"--file", default="-",
|
||||
help="Path to a text file, or - for stdin (default -).",
|
||||
)
|
||||
return p
|
||||
|
||||
|
||||
def _read_input(path):
|
||||
"""Read the analysis target from stdin ('-'/omitted) or a plain file.
|
||||
Plain `open()` only — no pathlib, to stay stdlib-minimal per contract."""
|
||||
if path in (None, "-"):
|
||||
return sys.stdin.read()
|
||||
return open(path, encoding="utf-8", errors="replace").read()
|
||||
|
||||
|
||||
def _cli():
|
||||
try:
|
||||
args = _build_parser().parse_args()
|
||||
text = _read_input(args.file)
|
||||
if not text or not text.strip():
|
||||
print(json.dumps({"status": "degraded", "reason": "empty_input"}))
|
||||
return
|
||||
envelope = {"status": "ok", "source": "content_quality"}
|
||||
envelope.update(analyse(text))
|
||||
print(json.dumps(envelope, indent=2))
|
||||
except SystemExit as e:
|
||||
if e.code not in (0, None):
|
||||
print(json.dumps({"status": "error", "reason": "bad_usage"}))
|
||||
raise
|
||||
except Exception as e:
|
||||
# Fail-open: a missing --file, an unreadable/binary file, or any
|
||||
# other unexpected error degrades rather than crashing the caller.
|
||||
print(json.dumps({"status": "degraded", "reason": str(e)}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
_cli()
|
||||
@@ -1,184 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""On-page drift between audits. Stdlib only.
|
||||
|
||||
seo-analyzer.md:1365 says "on re-run, move current content to Historique
|
||||
(summary: date + score + key changes)". That is prose the LLM writes about its
|
||||
own previous prose: lossy, unreproducible, and machine-uncomparable. So "the
|
||||
redesign silently dropped 40 canonicals" is invisible unless someone happens
|
||||
to notice.
|
||||
|
||||
This snapshots the machine-readable signals per URL and diffs them.
|
||||
|
||||
NOT rank tracking — a common misread of the same feature elsewhere. Positions
|
||||
come from GSC (`queries`). This is on-page regression detection: what the site
|
||||
said last time vs now.
|
||||
|
||||
Runs over the WHOLE sitemap, never a sample: a drift over a sample that
|
||||
changes between runs compares nothing.
|
||||
"""
|
||||
import argparse, json, os, re, time
|
||||
from html.parser import HTMLParser
|
||||
|
||||
import sitemap as sm
|
||||
|
||||
STORE_DIR = os.path.expanduser("~/.claude/seo-data/drift")
|
||||
MAX_PAGES = 500
|
||||
# Losing a signal is a regression. Changing one may be intentional — the agent
|
||||
# judges that, we only report which kind it is.
|
||||
TRACKED = ("title", "description", "canonical", "robots", "h1_count", "jsonld_types")
|
||||
|
||||
class _Signals(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__(convert_charrefs=True)
|
||||
self.title, self.description, self.canonical, self.robots = None, None, None, None
|
||||
self.h1_count, self.jsonld_types = 0, []
|
||||
self._in_title, self._in_ld = False, False
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
a = dict(attrs)
|
||||
if tag == "title":
|
||||
self._in_title = True
|
||||
elif tag == "h1":
|
||||
self.h1_count += 1
|
||||
elif tag == "meta":
|
||||
n = (a.get("name") or "").lower()
|
||||
if n == "description":
|
||||
self.description = (a.get("content") or "").strip() or None
|
||||
elif n == "robots":
|
||||
self.robots = (a.get("content") or "").strip() or None
|
||||
elif tag == "link" and "canonical" in (a.get("rel") or "").lower():
|
||||
self.canonical = (a.get("href") or "").strip() or None
|
||||
elif tag == "script" and a.get("type") == "application/ld+json":
|
||||
self._in_ld = True
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
if tag == "title":
|
||||
self._in_title = False
|
||||
elif tag == "script":
|
||||
self._in_ld = False
|
||||
|
||||
def handle_data(self, data):
|
||||
if self._in_title and data.strip():
|
||||
self.title = re.sub(r"\s+", " ", data.strip())
|
||||
elif self._in_ld:
|
||||
self.jsonld_types.extend(re.findall(r'"@type"\s*:\s*"([^"]+)"', data))
|
||||
|
||||
def _signals(html):
|
||||
p = _Signals()
|
||||
try:
|
||||
p.feed(html)
|
||||
except Exception:
|
||||
pass
|
||||
return {"title": p.title, "description": p.description,
|
||||
"canonical": p.canonical, "robots": p.robots,
|
||||
"h1_count": p.h1_count, "jsonld_types": sorted(set(p.jsonld_types))}
|
||||
|
||||
def _mock_pages():
|
||||
"""{url: html}, same convention as linkgraph: a single page.html fixture
|
||||
cannot express a multi-page snapshot — every URL would look identical."""
|
||||
raw = sm._mock("pages.json")
|
||||
return json.loads(raw.decode("utf-8")) if raw else None
|
||||
|
||||
def _capture(urls):
|
||||
pages = _mock_pages()
|
||||
snap, failed = {}, 0
|
||||
for u in urls:
|
||||
if pages is not None:
|
||||
html = pages.get(u)
|
||||
if html is None:
|
||||
failed += 1
|
||||
continue
|
||||
else:
|
||||
try:
|
||||
html = sm._fetch(u).decode("utf-8", "replace")
|
||||
except Exception:
|
||||
failed += 1
|
||||
continue
|
||||
snap[u] = _signals(html)
|
||||
return snap, failed
|
||||
|
||||
def _store_path(sitemap_url):
|
||||
from urllib.parse import urlparse
|
||||
host = urlparse(sitemap_url).netloc.lower()
|
||||
safe = re.sub(r"[^a-z0-9.-]", "_", host) or "unknown"
|
||||
return os.path.join(STORE_DIR, safe + ".json")
|
||||
|
||||
def _load(path):
|
||||
if not os.path.exists(path):
|
||||
return None
|
||||
try:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
except Exception:
|
||||
return None # corrupt store -> treat as first run
|
||||
|
||||
def _save(path, snap, stamp):
|
||||
os.makedirs(os.path.dirname(path), mode=0o700, exist_ok=True)
|
||||
tmp = path + ".tmp"
|
||||
with open(tmp, "w", encoding="utf-8") as f:
|
||||
json.dump({"captured": stamp, "pages": snap}, f)
|
||||
os.replace(tmp, path) # atomic: never a half-written baseline
|
||||
|
||||
def _classify(old, new):
|
||||
"""LOST a signal = regression. Changed it = change. Only the first is
|
||||
unambiguous; the agent judges the rest."""
|
||||
regressions, changes = [], []
|
||||
for f in TRACKED:
|
||||
o, n = old.get(f), new.get(f)
|
||||
if o == n:
|
||||
continue
|
||||
row = {"field": f, "was": o, "now": n}
|
||||
# Covers every tracked field uniformly: "Titre" -> None, 1 -> 0,
|
||||
# ["Article"] -> []. Had the value, lost the value.
|
||||
(regressions if (o and not n) else changes).append(row)
|
||||
return regressions, changes
|
||||
|
||||
def drift(sitemap_url, max_pages=MAX_PAGES):
|
||||
sm_res = sm.sitemap(sitemap_url)
|
||||
if sm_res.get("status") != "ok":
|
||||
return sm_res
|
||||
urls = sm_res["urls"][:max_pages]
|
||||
snap, failed = _capture(urls)
|
||||
if not snap:
|
||||
return {"status": "degraded", "reason": "no_pages_fetched"}
|
||||
stamp = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
path = _store_path(sitemap_url)
|
||||
prev = _load(path)
|
||||
_save(path, snap, stamp)
|
||||
if prev is None:
|
||||
return {"status": "ok", "baseline": True, "captured": stamp,
|
||||
"pages": len(snap), "pages_failed": failed, "store": path}
|
||||
old = prev.get("pages", {})
|
||||
regressions, changes = [], []
|
||||
for u, new in snap.items():
|
||||
if u not in old:
|
||||
continue
|
||||
r, c = _classify(old[u], new)
|
||||
for row in r:
|
||||
regressions.append(dict(row, url=u))
|
||||
for row in c:
|
||||
changes.append(dict(row, url=u))
|
||||
return {"status": "ok", "baseline": False,
|
||||
"since": prev.get("captured"), "captured": stamp,
|
||||
"pages": len(snap), "pages_failed": failed,
|
||||
"gone": sorted(set(old) - set(snap)),
|
||||
"new": sorted(set(snap) - set(old)),
|
||||
"regressions": regressions, "changes": changes, "store": path}
|
||||
|
||||
def _cli():
|
||||
try:
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--url", required=True, help="sitemap URL")
|
||||
p.add_argument("--max", type=int, default=MAX_PAGES)
|
||||
p.add_argument("--store", default=None) # accepted+ignored
|
||||
args = p.parse_args()
|
||||
print(json.dumps(drift(args.url, args.max), indent=2))
|
||||
except SystemExit as e:
|
||||
if e.code not in (0, None):
|
||||
print(json.dumps({"status": "error", "reason": "bad_usage"}))
|
||||
raise
|
||||
except Exception:
|
||||
print(json.dumps({"status": "degraded", "reason": "unexpected_error"}))
|
||||
|
||||
if __name__ == "__main__":
|
||||
_cli()
|
||||
+2
-17
@@ -27,23 +27,8 @@ _label_safe() ( LC_ALL=C; case "$1" in ''|[!A-Za-z0-9]*|*[!A-Za-z0-9._-]*) exit
|
||||
cmd="${1:-}"; shift || true
|
||||
case "$cmd" in
|
||||
accounts) exec "$PY" "$HERE/tokenstore.py" list --file "$STORE" ;;
|
||||
crux|queries|inspect|cannibal)
|
||||
crux|queries|inspect)
|
||||
exec "$PY" "$HERE/google_seo.py" "$cmd" --store "$STORE" "$@" ;;
|
||||
# No auth, no Google: stdlib-only, runs even without the venv.
|
||||
sitemap)
|
||||
exec "$PY" "$HERE/sitemap.py" --store "$STORE" "$@" ;;
|
||||
score)
|
||||
exec "$PY" "$HERE/score.py" --store "$STORE" "$@" ;;
|
||||
schema_gen)
|
||||
exec "$PY" "$HERE/schema_gen.py" --store "$STORE" "$@" ;;
|
||||
content_quality)
|
||||
exec "$PY" "$HERE/content_quality.py" --store "$STORE" "$@" ;;
|
||||
drift)
|
||||
exec "$PY" "$HERE/drift.py" --store "$STORE" "$@" ;;
|
||||
rendercheck)
|
||||
exec "$PY" "$HERE/render_check.py" --store "$STORE" "$@" ;;
|
||||
linkgraph)
|
||||
exec "$PY" "$HERE/linkgraph.py" --store "$STORE" "$@" ;;
|
||||
forget)
|
||||
# forget --label <label> → drop one account; forget --all → empty the store.
|
||||
# Local removal only — does NOT revoke the grant at Google's end.
|
||||
@@ -56,6 +41,6 @@ case "$cmd" in
|
||||
fi
|
||||
echo '{"status":"error","reason":"usage: fetch.sh forget {--label <label>|--all} (label charset: A-Za-z0-9._-)"}'
|
||||
exit 2 ;;
|
||||
*) echo '{"status":"error","reason":"usage: fetch.sh {accounts|crux|queries|inspect|cannibal|sitemap|rendercheck|linkgraph|drift|score|schema_gen|content_quality|forget} [flags]"}'
|
||||
*) echo '{"status":"error","reason":"usage: fetch.sh {accounts|crux|queries|inspect|forget} [flags]"}'
|
||||
exit 2 ;;
|
||||
esac
|
||||
|
||||
@@ -1,7 +0,0 @@
|
||||
{"rows":[
|
||||
{"keys":["plombier paris","https://ex.com/plombier"],"clicks":40,"impressions":900,"ctr":0.044,"position":6.3},
|
||||
{"keys":["plombier paris","https://ex.com/services/plomberie"],"clicks":3,"impressions":300,"ctr":0.010,"position":14.1},
|
||||
{"keys":["urgence fuite","https://ex.com/urgence"],"clicks":5,"impressions":1200,"ctr":0.004,"position":8.9},
|
||||
{"keys":["urgence fuite","https://ex.com/blog/fuite-que-faire"],"clicks":2,"impressions":800,"ctr":0.003,"position":11.4},
|
||||
{"keys":["urgence fuite","https://ex.com/services/depannage"],"clicks":1,"impressions":400,"ctr":0.002,"position":19.2},
|
||||
{"keys":["devis plomberie","https://ex.com/devis"],"clicks":9,"impressions":150,"ctr":0.060,"position":4.1}]}
|
||||
@@ -1,5 +0,0 @@
|
||||
{
|
||||
"https://ex.com/": "<html><head><title>Accueil</title><meta name='description' content='desc'><link rel='canonical' href='https://ex.com/'><script type='application/ld+json'>{\"@type\":\"LocalBusiness\"}</script></head><body><h1>Accueil</h1></body></html>",
|
||||
"https://ex.com/a": "<html><head><title>Page A</title><link rel='canonical' href='https://ex.com/a'></head><body><h1>A</h1></body></html>",
|
||||
"https://ex.com/gone": "<html><head><title>Bientot supprimee</title></head><body><h1>G</h1></body></html>"
|
||||
}
|
||||
@@ -1,6 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
|
||||
<url><loc>https://ex.com/</loc></url>
|
||||
<url><loc>https://ex.com/a</loc></url>
|
||||
<url><loc>https://ex.com/gone</loc></url>
|
||||
</urlset>
|
||||
@@ -1,5 +0,0 @@
|
||||
{
|
||||
"https://ex.com/": "<html><head><title>Accueil refondue</title><meta name='description' content='desc'><link rel='canonical' href='https://ex.com/'></head><body><p>plus de h1, plus de jsonld</p></body></html>",
|
||||
"https://ex.com/a": "<html><head><title>Page A</title></head><body><h1>A</h1></body></html>",
|
||||
"https://ex.com/neuve": "<html><head><title>Neuve</title></head><body><h1>N</h1></body></html>"
|
||||
}
|
||||
@@ -1,6 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
|
||||
<url><loc>https://ex.com/</loc></url>
|
||||
<url><loc>https://ex.com/a</loc></url>
|
||||
<url><loc>https://ex.com/neuve</loc></url>
|
||||
</urlset>
|
||||
@@ -1,9 +0,0 @@
|
||||
{
|
||||
"https://ex.com/": "<html><body><a href='/a'>a</a> <a href='/b/'>b trailing slash</a> <a href='#top'>anchor</a> <a href='/css/main.css?v=9'>asset</a> <a href='mailto:x@ex.com'>mail</a> <a href='tel:+33'>tel</a> <a href='https://other.com/x'>external</a> <a href='/img/logo.png'>img</a></body></html>",
|
||||
"https://ex.com/a": "<html><body><a href='/'>home</a> <a href='/deep'>deep</a></body></html>",
|
||||
"https://ex.com/b": "<html><body><a href='/'>home</a></body></html>",
|
||||
"https://ex.com/deep": "<html><body><a href='https://ex.com/deeper'>deeper absolute</a></body></html>",
|
||||
"https://ex.com/deeper": "<html><body><a href='deepest'>relative</a></body></html>",
|
||||
"https://ex.com/deepest": "<html><body><a href='/'>home</a></body></html>",
|
||||
"https://ex.com/orphan": "<html><body><a href='/'>home — links out, nobody links in</a></body></html>"
|
||||
}
|
||||
@@ -1,10 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
|
||||
<url><loc>https://ex.com/</loc></url>
|
||||
<url><loc>https://ex.com/a</loc></url>
|
||||
<url><loc>https://ex.com/b</loc></url>
|
||||
<url><loc>https://ex.com/deep</loc></url>
|
||||
<url><loc>https://ex.com/deeper</loc></url>
|
||||
<url><loc>https://ex.com/deepest</loc></url>
|
||||
<url><loc>https://ex.com/orphan</loc></url>
|
||||
</urlset>
|
||||
@@ -1,2 +0,0 @@
|
||||
{"inspectionResult":{"indexStatusResult":{
|
||||
"verdict":"PASS","coverageState":"Submitted and indexed","lastCrawlTime":"2026-07-01T10:00:00Z"}}}
|
||||
@@ -1,10 +0,0 @@
|
||||
<?xml version="1.0"?>
|
||||
<!DOCTYPE urlset [
|
||||
<!ENTITY lol "lol">
|
||||
<!ENTITY lol2 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
|
||||
<!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
|
||||
<!ENTITY lol4 "&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;">
|
||||
]>
|
||||
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
|
||||
<url><loc>https://ex.com/&lol4;</loc></url>
|
||||
</urlset>
|
||||
@@ -1,5 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
|
||||
<sitemap><loc>https://ex.com/sitemap-pages.xml</loc></sitemap>
|
||||
<sitemap><loc>https://ex.com/sitemap-blog.xml</loc></sitemap>
|
||||
</sitemapindex>
|
||||
@@ -1,5 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
|
||||
<url><loc>https://ex.com/child-a</loc></url>
|
||||
<url><loc>https://ex.com/child-b</loc></url>
|
||||
</urlset>
|
||||
@@ -1,8 +0,0 @@
|
||||
<!DOCTYPE html><html lang="fr"><head>
|
||||
<title>Mon App</title>
|
||||
<script type="module" crossorigin src="/assets/index-a1b2c3.js"></script>
|
||||
<link rel="stylesheet" href="/assets/index-d4e5f6.css">
|
||||
</head><body>
|
||||
<div id="root"></div>
|
||||
<script>window.__INITIAL_STATE__={"user":null,"routes":["/","/about","/contact"],"config":{"apiUrl":"https://api.example.com","features":["a","b","c"]}};</script>
|
||||
</body></html>
|
||||
@@ -1,8 +0,0 @@
|
||||
<!DOCTYPE html><html lang="fr"><head>
|
||||
<title>Lavage auto</title>
|
||||
<meta name="description" content="Lavage auto à la main en Seine-et-Marne.">
|
||||
<script type="application/ld+json">{"@context":"https://schema.org","@type":"LocalBusiness","name":"X"}</script>
|
||||
</head><body>
|
||||
<h1>Lavage auto à la main</h1>
|
||||
<p>Lavage automobile à la main à Lagny-sur-Marne, detailing et protection céramique. Lavage automobile à la main à Lagny-sur-Marne, detailing et protection céramique. Lavage automobile à la main à Lagny-sur-Marne, detailing et protection céramique. Lavage automobile à la main à Lagny-sur-Marne, detailing et protection céramique. Lavage automobile à la main à Lagny-sur-Marne, detailing et protection céramique. Lavage automobile à la main à Lagny-sur-Marne, detailing et protection céramique. Lavage automobile à la main à Lagny-sur-Marne, detailing et protection céramique. Lavage automobile à la main à Lagny-sur-Marne, detailing et protection céramique.</p>
|
||||
</body></html>
|
||||
@@ -1,11 +1,2 @@
|
||||
{"inspectionResult":{
|
||||
"indexStatusResult":{
|
||||
"verdict":"PASS","coverageState":"Submitted and indexed","lastCrawlTime":"2026-07-01T10:00:00Z"},
|
||||
"richResultsResult":{"verdict":"FAIL","detectedItems":[
|
||||
{"richResultType":"Breadcrumbs","items":[{"name":"Unnamed item","issues":[]}]},
|
||||
{"richResultType":"FAQ","items":[
|
||||
{"name":"Q1","issues":[
|
||||
{"issueMessage":"Missing field 'acceptedAnswer'","severity":"ERROR"}]},
|
||||
{"name":"Q2","issues":[
|
||||
{"issueMessage":"Missing field 'acceptedAnswer'","severity":"ERROR"},
|
||||
{"issueMessage":"Unspecified image","severity":"WARNING"}]}]}]}}}
|
||||
{"inspectionResult":{"indexStatusResult":{
|
||||
"verdict":"PASS","coverageState":"Submitted and indexed","lastCrawlTime":"2026-07-01T10:00:00Z"}}}
|
||||
|
||||
@@ -1,25 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
|
||||
xmlns:xhtml="http://www.w3.org/1999/xhtml"
|
||||
xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
|
||||
<!-- image:loc also ends with }loc — it must NOT be counted as a page -->
|
||||
<url>
|
||||
<loc>https://ex.com/</loc>
|
||||
<changefreq>weekly</changefreq>
|
||||
<xhtml:link rel="alternate" hreflang="en" href="https://ex.com/en/" />
|
||||
<image:image>
|
||||
<image:loc>https://ex.com/img/logo.png</image:loc>
|
||||
<image:title>Logo</image:title>
|
||||
</image:image>
|
||||
<image:image>
|
||||
<image:loc>https://ex.com/img/hero.jpeg</image:loc>
|
||||
</image:image>
|
||||
</url>
|
||||
<url><loc>https://ex.com/services</loc></url>
|
||||
<url><loc>https://ex.com/blog</loc></url>
|
||||
<url><loc>https://ex.com/blog</loc></url>
|
||||
<url><loc> https://ex.com/spaced </loc></url>
|
||||
<url><loc>ftp://ex.com/nope</loc></url>
|
||||
<url><loc>https://ex.com/bad"quote</loc></url>
|
||||
<url><loc></loc></url>
|
||||
</urlset>
|
||||
+7
-100
@@ -89,16 +89,13 @@ def _gsc_session(store_path, account):
|
||||
return AuthorizedSession(creds)
|
||||
|
||||
def _norm_queries(raw, dim):
|
||||
# `keys` is the list the API actually returns (one entry per requested
|
||||
# dimension); `key` stays as keys[0] so the single-dim consumer that reads
|
||||
# it keeps working. Additive — nothing to migrate.
|
||||
return {"status": "ok", "source": "gsc", "dimension": dim, "rows": [
|
||||
{"key": r["keys"][0], "keys": r["keys"], "clicks": r.get("clicks", 0),
|
||||
{"key": r["keys"][0], "clicks": r.get("clicks", 0),
|
||||
"impressions": r.get("impressions", 0), "ctr": r.get("ctr", 0),
|
||||
"position": r.get("position")}
|
||||
for r in raw.get("rows", [])]}
|
||||
|
||||
def queries(store_path, account, property, days=90, dim="query", rows=100):
|
||||
def queries(store_path, account, property, days=90, dim="query"):
|
||||
raw = _mock("gsc_queries.json")
|
||||
if raw is None:
|
||||
sess = _gsc_session(store_path, account)
|
||||
@@ -109,89 +106,14 @@ def queries(store_path, account, property, days=90, dim="query", rows=100):
|
||||
import urllib.parse
|
||||
url = ("https://searchconsole.googleapis.com/webmasters/v3/sites/"
|
||||
+ urllib.parse.quote(property, safe="") + "/searchAnalytics/query")
|
||||
# dim accepts a comma-separated list: the API groups by several
|
||||
# dimensions at once ("no limit... but you cannot group by the same
|
||||
# dimension twice"), and query+page is what exposes cannibalisation.
|
||||
dims = [d.strip() for d in dim.split(",") if d.strip()]
|
||||
r = sess.post(url, json={"startDate": start.isoformat(), "endDate": end.isoformat(),
|
||||
"dimensions": dims, "rowLimit": rows}, timeout=30)
|
||||
"dimensions": [dim], "rowLimit": 100}, timeout=30)
|
||||
if r.status_code == 429:
|
||||
return {"status": "degraded", "reason": "rate_limited"}
|
||||
r.raise_for_status()
|
||||
raw = r.json()
|
||||
return _norm_queries(raw, dim)
|
||||
|
||||
def _rollup_issues(items):
|
||||
"""Count issue instances by severity; dedupe messages (they repeat per item)."""
|
||||
errors = warnings = 0
|
||||
msgs = []
|
||||
for item in items:
|
||||
for iss in item.get("issues", []):
|
||||
sev = iss.get("severity")
|
||||
if sev == "ERROR":
|
||||
errors += 1
|
||||
elif sev == "WARNING":
|
||||
warnings += 1
|
||||
msg = iss.get("issueMessage")
|
||||
if msg and msg not in msgs:
|
||||
msgs.append(msg)
|
||||
return errors, warnings, msgs
|
||||
|
||||
def _norm_rich(ir):
|
||||
"""richResultsResult → verdict + per-type rollup. Google OMITS the key when
|
||||
it detects no rich results, so absence is data, not an error: surfaced as the
|
||||
synthetic verdict ABSENT (not a Google enum) rather than a missing key, which
|
||||
a caller cannot tell apart from a check that never ran. PARTIAL is never
|
||||
emitted — the API reserves it as unused."""
|
||||
rr = ir.get("richResultsResult")
|
||||
if rr is None:
|
||||
return {"verdict": "ABSENT", "types": []}
|
||||
types = []
|
||||
for det in rr.get("detectedItems", []):
|
||||
errors, warnings, msgs = _rollup_issues(det.get("items", []))
|
||||
types.append({"type": det.get("richResultType"),
|
||||
"items": len(det.get("items", [])),
|
||||
"errors": errors, "warnings": warnings, "issues": msgs})
|
||||
return {"verdict": rr.get("verdict"), "types": types}
|
||||
|
||||
def _group_by_query(rows):
|
||||
"""query+page rows -> {query: [row, …]}. Deterministic aggregation, not
|
||||
judgement: the agent must not be asked to group 1000 rows by eye."""
|
||||
by_q = {}
|
||||
for r in rows:
|
||||
keys = r.get("keys") or []
|
||||
if len(keys) < 2:
|
||||
continue
|
||||
by_q.setdefault(keys[0], []).append(
|
||||
{"url": keys[1], "clicks": r["clicks"],
|
||||
"impressions": r["impressions"], "position": r["position"]})
|
||||
return by_q
|
||||
|
||||
def cannibal(store_path, account, property, days=90, rows=1000):
|
||||
"""Queries where 2+ of our own pages compete for the same term.
|
||||
|
||||
Google's own data says it; nothing in this system asked. Cannibalisation
|
||||
is a SERP fact, not a content-similarity guess — do not confuse it with
|
||||
the 30/70 duplication rule, which has no data source here."""
|
||||
res = queries(store_path, account, property, days, "query,page", rows)
|
||||
if res.get("status") != "ok":
|
||||
return res
|
||||
conflicts = []
|
||||
for q, pages in _group_by_query(res["rows"]).items():
|
||||
if len(pages) < 2:
|
||||
continue
|
||||
pages.sort(key=lambda p: p["impressions"], reverse=True)
|
||||
conflicts.append({"query": q, "pages": len(pages),
|
||||
"total_impressions": sum(p["impressions"] for p in pages),
|
||||
"urls": pages})
|
||||
conflicts.sort(key=lambda c: c["total_impressions"], reverse=True)
|
||||
return {"status": "ok", "source": "gsc", "days": days,
|
||||
"rows_scanned": len(res["rows"]),
|
||||
# rows_scanned == rows means the window was FULL: there may be more
|
||||
# conflicts past the cut. Reported, never silently truncated.
|
||||
"capped": len(res["rows"]) >= rows,
|
||||
"conflict_count": len(conflicts), "conflicts": conflicts}
|
||||
|
||||
def inspect(store_path, account, property, url):
|
||||
raw = _mock("gsc_inspect.json")
|
||||
if raw is None:
|
||||
@@ -204,15 +126,11 @@ def inspect(store_path, account, property, url):
|
||||
return {"status": "degraded", "reason": "rate_limited"}
|
||||
r.raise_for_status()
|
||||
raw = r.json()
|
||||
ir = raw["inspectionResult"]
|
||||
isr = ir["indexStatusResult"]
|
||||
# rich_results rides the SAME response — Google already sent it and this
|
||||
# function used to discard it. No extra call, no extra quota, no new scope.
|
||||
isr = raw["inspectionResult"]["indexStatusResult"]
|
||||
return {"status": "ok", "source": "gsc",
|
||||
"indexed": isr.get("verdict") == "PASS",
|
||||
"coverage": isr.get("coverageState"),
|
||||
"last_crawl": isr.get("lastCrawlTime"),
|
||||
"rich_results": _norm_rich(ir)}
|
||||
"last_crawl": isr.get("lastCrawlTime")}
|
||||
|
||||
def _cli():
|
||||
try:
|
||||
@@ -227,15 +145,7 @@ def _cli():
|
||||
pq.add_argument("--account", required=True)
|
||||
pq.add_argument("--property", required=True)
|
||||
pq.add_argument("--days", type=int, default=90)
|
||||
pq.add_argument("--dim", default="query",
|
||||
help="one dimension, or a comma-separated list (query,page)")
|
||||
pq.add_argument("--rows", type=int, default=100)
|
||||
pn = sub.add_parser("cannibal")
|
||||
pn.add_argument("--store", required=True)
|
||||
pn.add_argument("--account", required=True)
|
||||
pn.add_argument("--property", required=True)
|
||||
pn.add_argument("--days", type=int, default=90)
|
||||
pn.add_argument("--rows", type=int, default=1000)
|
||||
pq.add_argument("--dim", default="query")
|
||||
pi = sub.add_parser("inspect")
|
||||
pi.add_argument("--store", required=True)
|
||||
pi.add_argument("--account", required=True)
|
||||
@@ -246,10 +156,7 @@ def _cli():
|
||||
print(json.dumps(crux(args.url, args.strategy), indent=2))
|
||||
elif args.cmd == "queries":
|
||||
print(json.dumps(queries(args.store, args.account, args.property,
|
||||
args.days, args.dim, args.rows), indent=2))
|
||||
elif args.cmd == "cannibal":
|
||||
print(json.dumps(cannibal(args.store, args.account, args.property,
|
||||
args.days, args.rows), indent=2))
|
||||
args.days, args.dim), indent=2))
|
||||
elif args.cmd == "inspect":
|
||||
print(json.dumps(inspect(args.store, args.account, args.property,
|
||||
args.url), indent=2))
|
||||
|
||||
@@ -1,170 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Internal link graph -> orphans + click depth. Stdlib only.
|
||||
|
||||
seo-analyzer.md asks "Every important page reachable within 3 clicks?" (:613)
|
||||
and "Orphan pages (no inbound internal links)?" (:616) and has never had a
|
||||
command that answers either. This is that command.
|
||||
|
||||
EXHAUSTIVE OR NOTHING. You cannot sample orphans: proving a page has no
|
||||
inbound link means having read every other page. A partial crawl invents
|
||||
orphans, and "page X has no inbound links" when it does is the worst finding
|
||||
this tool could emit — it sends a client fixing what is not broken. So when
|
||||
the cap bites, orphans are WITHHELD, not truncated.
|
||||
|
||||
Does NOT render JS. On a client-side-rendered SPA the links are not in the
|
||||
HTML, every page looks orphaned, and that is a catastrophic false positive —
|
||||
so an empty link graph is REFUSED (no_links_in_html), never reported.
|
||||
"""
|
||||
import argparse, json
|
||||
from html.parser import HTMLParser
|
||||
from urllib.parse import urljoin, urlparse, urldefrag
|
||||
|
||||
import sitemap as sm # sibling module: fetch + parse
|
||||
|
||||
MAX_PAGES = 500
|
||||
# Extensions that are assets, not pages. Seen live: /css/main.css?v=1778157313
|
||||
ASSET_EXT = (".css", ".js", ".mjs", ".png", ".jpg", ".jpeg", ".gif", ".webp",
|
||||
".avif", ".svg", ".ico", ".woff", ".woff2", ".ttf", ".eot",
|
||||
".pdf", ".zip", ".mp4", ".webm", ".xml", ".json", ".txt", ".rss")
|
||||
|
||||
class _Links(HTMLParser):
|
||||
def __init__(self):
|
||||
super().__init__(convert_charrefs=True)
|
||||
self.hrefs = []
|
||||
def handle_starttag(self, tag, attrs):
|
||||
if tag != "a":
|
||||
return
|
||||
for k, v in attrs:
|
||||
if k == "href" and v:
|
||||
self.hrefs.append(v)
|
||||
|
||||
def _norm(u):
|
||||
"""Canonical form for graph identity. Drops the fragment, keeps the query
|
||||
(?p=2 IS a different page), and unifies the trailing slash so /blog and
|
||||
/blog/ are one node rather than a phantom orphan pair."""
|
||||
u = urldefrag(u)[0]
|
||||
p = urlparse(u)
|
||||
path = p.path or "/"
|
||||
if len(path) > 1 and path.endswith("/"):
|
||||
path = path[:-1]
|
||||
out = "%s://%s%s" % (p.scheme, p.netloc.lower(), path)
|
||||
return out + ("?" + p.query if p.query else "")
|
||||
|
||||
def _page_links(base, html, host):
|
||||
"""Internal page links from one document. Filters what a link graph must
|
||||
never contain: assets, #anchors, mailto:/tel:, and other hosts."""
|
||||
p = _Links()
|
||||
try:
|
||||
p.feed(html)
|
||||
except Exception:
|
||||
pass # tolerate malformed markup
|
||||
out = set()
|
||||
for h in p.hrefs:
|
||||
h = h.strip()
|
||||
if not h or h.startswith(("#", "mailto:", "tel:", "javascript:", "data:")):
|
||||
continue
|
||||
absu = urljoin(base, h)
|
||||
pr = urlparse(absu)
|
||||
if pr.scheme not in ("http", "https") or pr.netloc.lower() != host:
|
||||
continue
|
||||
if pr.path.lower().endswith(ASSET_EXT):
|
||||
continue
|
||||
out.add(_norm(absu))
|
||||
return out
|
||||
|
||||
def _mock_pages():
|
||||
"""{url: html} for tests. A single page.html fixture cannot express a
|
||||
GRAPH — every node would carry identical links — so the mock is a map."""
|
||||
raw = sm._mock("pages.json")
|
||||
return json.loads(raw.decode("utf-8")) if raw else None
|
||||
|
||||
def _crawl(urls, host):
|
||||
"""Fetch each page once; return {page: {links}} plus a failure count."""
|
||||
pages = _mock_pages()
|
||||
graph, failed = {}, 0
|
||||
for u in urls:
|
||||
if pages is not None:
|
||||
html = pages.get(u)
|
||||
if html is None:
|
||||
failed += 1
|
||||
continue
|
||||
else:
|
||||
try:
|
||||
html = sm._fetch(u).decode("utf-8", "replace")
|
||||
except Exception:
|
||||
failed += 1
|
||||
continue
|
||||
graph[_norm(u)] = _page_links(u, html, host)
|
||||
return graph, failed
|
||||
|
||||
def _depths(graph, root):
|
||||
"""BFS click-depth from the homepage. Absent = unreachable by links."""
|
||||
seen, frontier, d = {root: 0}, [root], 0
|
||||
while frontier:
|
||||
d += 1
|
||||
nxt = []
|
||||
for node in frontier:
|
||||
for tgt in graph.get(node, ()):
|
||||
if tgt not in seen:
|
||||
seen[tgt] = d
|
||||
nxt.append(tgt)
|
||||
frontier = nxt
|
||||
return seen
|
||||
|
||||
def linkgraph(sitemap_url, max_pages=MAX_PAGES):
|
||||
sm_res = sm.sitemap(sitemap_url)
|
||||
if sm_res.get("status") != "ok":
|
||||
return sm_res # propagate the sitemap's own degrade
|
||||
urls = sm_res["urls"]
|
||||
capped = len(urls) > max_pages
|
||||
host = urlparse(urls[0]).netloc.lower()
|
||||
graph, failed = _crawl(urls[:max_pages], host)
|
||||
if not graph:
|
||||
return {"status": "degraded", "reason": "no_pages_fetched"}
|
||||
total_links = sum(len(v) for v in graph.values())
|
||||
if total_links == 0:
|
||||
# Every page orphaned is never the truth — it is a JS-rendered site.
|
||||
return {"status": "degraded", "reason": "no_links_in_html",
|
||||
"pages_crawled": len(graph),
|
||||
"hint": "links absent from served HTML (SPA?) — see R1/R2"}
|
||||
inbound = {n: 0 for n in graph}
|
||||
for src, tgts in graph.items():
|
||||
for t in tgts:
|
||||
if t in inbound and t != src:
|
||||
inbound[t] += 1
|
||||
root = _norm("%s://%s/" % (urlparse(urls[0]).scheme, host))
|
||||
depth = _depths(graph, root)
|
||||
out = {"status": "ok", "source": "linkgraph",
|
||||
"pages_crawled": len(graph), "pages_failed": failed,
|
||||
"total_internal_links": total_links, "capped": capped,
|
||||
"max_depth": max(depth.values()) if depth else 0,
|
||||
"beyond_3_clicks": sorted(n for n, d in depth.items() if d > 3),
|
||||
"unreachable": sorted(n for n in graph if n not in depth)}
|
||||
if capped or failed:
|
||||
# A page can only be called orphaned if EVERY other page was read.
|
||||
out["orphans_withheld"] = True
|
||||
out["reason_withheld"] = ("crawl incomplete (capped=%s, failed=%d) — "
|
||||
"an orphan from a partial crawl is a false "
|
||||
"orphan" % (capped, failed))
|
||||
else:
|
||||
out["orphans"] = sorted(n for n, c in inbound.items()
|
||||
if c == 0 and n != root)
|
||||
return out
|
||||
|
||||
def _cli():
|
||||
try:
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--url", required=True, help="sitemap URL")
|
||||
p.add_argument("--max", type=int, default=MAX_PAGES)
|
||||
p.add_argument("--store", default=None) # accepted+ignored
|
||||
args = p.parse_args()
|
||||
print(json.dumps(linkgraph(args.url, args.max), indent=2))
|
||||
except SystemExit as e:
|
||||
if e.code not in (0, None):
|
||||
print(json.dumps({"status": "error", "reason": "bad_usage"}))
|
||||
raise
|
||||
except Exception:
|
||||
print(json.dumps({"status": "degraded", "reason": "unexpected_error"}))
|
||||
|
||||
if __name__ == "__main__":
|
||||
_cli()
|
||||
@@ -1,106 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Is the content in the served HTML, or painted by JS? Stdlib only.
|
||||
|
||||
seo-analyzer records `RENDERING: SSR/SSG/SPA/hybrid` and then does nothing
|
||||
with it. That is the gap this closes. On a client-rendered site `curl` returns
|
||||
an empty shell, so every meta/H1/JSON-LD check reports "missing" and the audit
|
||||
emits a page of false findings against a site that may be perfectly fine.
|
||||
|
||||
The verdict is taken from what the server actually sent — not from guessing at
|
||||
package.json, where a React SPA and a Next.js SSR app look identical.
|
||||
|
||||
R2, not R1: this REPORTS blindness so the agent can refuse to score. It does
|
||||
not render JS. No Playwright, no Chromium, no venv.
|
||||
"""
|
||||
import argparse, json, re
|
||||
from html.parser import HTMLParser
|
||||
|
||||
import sitemap as sm # sibling: _fetch / _mock
|
||||
|
||||
# A shell can still carry a title + a couple of nav words. These thresholds
|
||||
# separate "shell" from "page" on the two real sites measured 2026-07-17
|
||||
# (server-rendered: 1 h1, thousands of body chars) and on a hydration stub.
|
||||
MIN_TEXT = 400
|
||||
MIN_H1 = 1
|
||||
|
||||
class _Doc(HTMLParser):
|
||||
"""Collect body text and the tags an SEO audit reads. Script/style content
|
||||
is NOT text: a 200 KB React bundle would otherwise look like a rich page."""
|
||||
SKIP = ("script", "style", "noscript", "template", "svg")
|
||||
|
||||
def __init__(self):
|
||||
super().__init__(convert_charrefs=True)
|
||||
self.text, self.h1, self.jsonld, self.meta_desc = [], 0, 0, False
|
||||
self._skip = 0
|
||||
self._ld = False
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
a = dict(attrs)
|
||||
if tag in self.SKIP:
|
||||
self._skip += 1
|
||||
self._ld = tag == "script" and a.get("type") == "application/ld+json"
|
||||
elif tag == "h1":
|
||||
self.h1 += 1
|
||||
elif tag == "meta" and a.get("name", "").lower() == "description":
|
||||
self.meta_desc = bool((a.get("content") or "").strip())
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
if tag in self.SKIP and self._skip:
|
||||
self._skip -= 1
|
||||
self._ld = False
|
||||
|
||||
def handle_data(self, data):
|
||||
if self._ld:
|
||||
self.jsonld += 1
|
||||
elif not self._skip:
|
||||
s = data.strip()
|
||||
if s:
|
||||
self.text.append(s)
|
||||
|
||||
def _verdict(text_chars, h1, jsonld):
|
||||
if text_chars >= MIN_TEXT and h1 >= MIN_H1:
|
||||
return "server-rendered"
|
||||
if text_chars < MIN_TEXT and h1 == 0 and jsonld == 0:
|
||||
return "client-rendered"
|
||||
return "partial" # shell + some SSR'd head, or thin page
|
||||
|
||||
def render_check(url):
|
||||
raw = sm._mock("page.html")
|
||||
if raw is None:
|
||||
try:
|
||||
raw = sm._fetch(url)
|
||||
except Exception:
|
||||
return {"status": "degraded", "reason": "fetch_failed"}
|
||||
html = raw.decode("utf-8", "replace")
|
||||
d = _Doc()
|
||||
try:
|
||||
d.feed(html)
|
||||
except Exception:
|
||||
pass # tolerate malformed markup
|
||||
text = re.sub(r"\s+", " ", " ".join(d.text)).strip()
|
||||
verdict = _verdict(len(text), d.h1, d.jsonld)
|
||||
out = {"status": "ok", "source": "render_check", "verdict": verdict,
|
||||
"body_text_chars": len(text), "h1_in_html": d.h1,
|
||||
"jsonld_in_html": d.jsonld, "meta_description_in_html": d.meta_desc,
|
||||
"html_bytes": len(raw)}
|
||||
if verdict != "server-rendered":
|
||||
out["warning"] = ("content is not in the served HTML — curl-based "
|
||||
"on-page checks will report false 'missing' findings")
|
||||
return out
|
||||
|
||||
def _cli():
|
||||
try:
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--url", required=True)
|
||||
p.add_argument("--store", default=None) # accepted+ignored
|
||||
args = p.parse_args()
|
||||
print(json.dumps(render_check(args.url), indent=2))
|
||||
except SystemExit as e:
|
||||
if e.code not in (0, None):
|
||||
print(json.dumps({"status": "error", "reason": "bad_usage"}))
|
||||
raise
|
||||
except Exception:
|
||||
print(json.dumps({"status": "degraded", "reason": "unexpected_error"}))
|
||||
|
||||
if __name__ == "__main__":
|
||||
_cli()
|
||||
@@ -1,164 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""SSRF- and DNS-rebinding-safe HTTP(S) fetch. Stdlib only.
|
||||
|
||||
The verbs that fetch remote content (sitemap, linkgraph, render_check, drift)
|
||||
all route through sitemap._fetch, which used urllib.request.urlopen. urlopen
|
||||
resolves the host, then connects — two DNS lookups with a window between them.
|
||||
A hostile authority can answer PUBLIC to the validation lookup and a PRIVATE
|
||||
address (169.254.169.254 cloud metadata, 127.0.0.1, the LAN) to the connect
|
||||
lookup. That is DNS rebinding, and a name-level guard cannot see it.
|
||||
|
||||
This collapses the two lookups into one: resolve ONCE, validate every returned
|
||||
IP, then connect to the exact validated IP while preserving the Host header,
|
||||
TLS SNI, and certificate validation for the real hostname. There is no second
|
||||
resolution to poison.
|
||||
|
||||
Better than the reference implementation this idea came from (claude-seo
|
||||
url_safety.py, MIT) on three axes, all verified before writing:
|
||||
- dual-stack: validates IPv4 AND IPv6 (theirs is IPv4-only);
|
||||
- no global state: each connection pins its own socket, so it is thread-safe
|
||||
by construction (theirs monkeypatches socket.getaddrinfo behind a global
|
||||
lock);
|
||||
- stdlib only: http.client + ssl + ipaddress, no `requests`.
|
||||
|
||||
NOT covered, stated rather than left silent: the shell `curl` calls in the
|
||||
agent specs (seo-analyzer/geo-analyzer STEP 4, the sameAs loop) run in a
|
||||
separate process and cannot be pinned from here. Their surface is smaller
|
||||
(a fixed set against an operator-typed/confirmed $DOMAIN). Closing them needs
|
||||
`curl --resolve` and is a separate change.
|
||||
"""
|
||||
import gzip
|
||||
import http.client
|
||||
import ipaddress
|
||||
import socket
|
||||
import ssl
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
DEFAULT_TIMEOUT = 20
|
||||
DEFAULT_MAX_BYTES = 20 * 1024 * 1024
|
||||
MAX_REDIRECTS = 5
|
||||
|
||||
|
||||
class UnsafeTarget(Exception):
|
||||
"""A URL resolved to a non-public address, or a redirect did. Raised BEFORE
|
||||
any connection to that address. Callers already wrap _fetch in try/except
|
||||
and degrade, so the fail-open contract is preserved."""
|
||||
|
||||
|
||||
# Special-use ranges that `is_global` reports as public but are not legitimate
|
||||
# fetch targets. 192.88.99.0/24 = RFC 3068 6to4-relay anycast (a security
|
||||
# review flagged it 2026-07-17). Grows if more surface.
|
||||
_EXTRA_DENY = (ipaddress.ip_network("192.88.99.0/24"),)
|
||||
|
||||
|
||||
def _ip_is_public(ip_str):
|
||||
"""A globally routable unicast address, dual-stack. `is_global` is the
|
||||
decisive gate — it alone rejects CGNAT (100.64/10) that the per-flag checks
|
||||
miss — with the explicit flags plus an extra special-use deny list as
|
||||
defence in depth."""
|
||||
ip = ipaddress.ip_address(ip_str)
|
||||
if not ip.is_global:
|
||||
return False
|
||||
if any(ip in net for net in _EXTRA_DENY):
|
||||
return False
|
||||
return not (ip.is_private or ip.is_loopback or ip.is_link_local
|
||||
or ip.is_reserved or ip.is_multicast or ip.is_unspecified)
|
||||
|
||||
|
||||
def _resolve_pinned(host, port, resolver=socket.getaddrinfo):
|
||||
"""Resolve host ONCE and return [(family, ip)] for connecting. Refuse if
|
||||
ANY resolved address is non-public — a name advertising both public and
|
||||
private A records is exactly the multi-answer rebinding vector, and a
|
||||
legitimate public site does not do it. `resolver` is injected in tests to
|
||||
plant a private address and prove the refusal."""
|
||||
try:
|
||||
infos = resolver(host, port, type=socket.SOCK_STREAM)
|
||||
except socket.gaierror as e:
|
||||
raise UnsafeTarget("cannot resolve %r: %s" % (host, e))
|
||||
pinned = []
|
||||
for family, _type, _proto, _canon, sockaddr in infos:
|
||||
ip = sockaddr[0]
|
||||
if not _ip_is_public(ip):
|
||||
raise UnsafeTarget("%s resolves to non-public %s" % (host, ip))
|
||||
pinned.append((family, ip))
|
||||
if not pinned:
|
||||
raise UnsafeTarget("%s resolved to nothing" % host)
|
||||
return pinned
|
||||
|
||||
|
||||
class _PinnedHTTPSConnection(http.client.HTTPSConnection):
|
||||
"""HTTPS to a pinned IP, with SNI + cert validation for the real host."""
|
||||
def __init__(self, host, pinned_ip, family, **kw):
|
||||
super().__init__(host, **kw) # host → Host header + SNI
|
||||
self._pinned_ip = pinned_ip
|
||||
self._family = family
|
||||
|
||||
def connect(self):
|
||||
sock = socket.create_connection((self._pinned_ip, self.port),
|
||||
timeout=self.timeout)
|
||||
# server_hostname = the real host → SNI + hostname check both use it,
|
||||
# never the IP.
|
||||
self.sock = self._context.wrap_socket(sock, server_hostname=self.host)
|
||||
|
||||
|
||||
class _PinnedHTTPConnection(http.client.HTTPConnection):
|
||||
"""Plain HTTP to a pinned IP (Host header stays the real host)."""
|
||||
def __init__(self, host, pinned_ip, family, **kw):
|
||||
super().__init__(host, **kw)
|
||||
self._pinned_ip = pinned_ip
|
||||
self._family = family
|
||||
|
||||
def connect(self):
|
||||
self.sock = socket.create_connection((self._pinned_ip, self.port),
|
||||
timeout=self.timeout)
|
||||
|
||||
|
||||
def _one_request(url, timeout, max_bytes, resolver):
|
||||
"""One hop: resolve+pin the host, connect, return (status, headers, body)."""
|
||||
p = urlparse(url)
|
||||
if p.scheme not in ("http", "https"):
|
||||
raise UnsafeTarget("scheme must be http/https: %r" % url)
|
||||
host = p.hostname
|
||||
if not host:
|
||||
raise UnsafeTarget("no host in %r" % url)
|
||||
port = p.port or (443 if p.scheme == "https" else 80)
|
||||
family, ip = _resolve_pinned(host, port, resolver)[0] # any is public here
|
||||
ctx = ssl.create_default_context() if p.scheme == "https" else None
|
||||
if p.scheme == "https":
|
||||
conn = _PinnedHTTPSConnection(host, ip, family, port=port,
|
||||
timeout=timeout, context=ctx)
|
||||
else:
|
||||
conn = _PinnedHTTPConnection(host, ip, family, port=port,
|
||||
timeout=timeout)
|
||||
try:
|
||||
path = p.path or "/"
|
||||
if p.query:
|
||||
path += "?" + p.query
|
||||
# No Accept-Encoding: keep HTTP bodies un-gzipped; the .xml.gz
|
||||
# content-level case is handled by the caller's magic-byte check.
|
||||
conn.request("GET", path, headers={"Host": host,
|
||||
"User-Agent": "claude-seo-data/1.0"})
|
||||
r = conn.getresponse()
|
||||
body = r.read(max_bytes)
|
||||
return r.status, {k.lower(): v for k, v in r.getheaders()}, body
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def safe_fetch(url, timeout=DEFAULT_TIMEOUT, max_bytes=DEFAULT_MAX_BYTES,
|
||||
max_redirects=MAX_REDIRECTS, resolver=socket.getaddrinfo):
|
||||
"""Fetch url with resolve-then-pin, following redirects and RE-VALIDATING
|
||||
each hop — urlopen followed redirects to whatever address the Location
|
||||
named, re-opening the rebinding window on every hop. Returns the raw body
|
||||
bytes (the caller handles content-level gzip)."""
|
||||
seen = 0
|
||||
current = url
|
||||
while True:
|
||||
status, headers, body = _one_request(current, timeout, max_bytes, resolver)
|
||||
if status in (301, 302, 303, 307, 308) and "location" in headers:
|
||||
seen += 1
|
||||
if seen > max_redirects:
|
||||
raise UnsafeTarget("too many redirects from %r" % url)
|
||||
current = urljoin(current, headers["location"]) # re-validated next loop
|
||||
continue
|
||||
return body
|
||||
@@ -1,301 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic JSON-LD generators for four Schema.org types. Stdlib only.
|
||||
|
||||
Adapted from claude-seo (github.com/AgriciDaniel/claude-seo, MIT),
|
||||
schema_generate.py — rewritten to the lib/seo-data fail-open contract.
|
||||
|
||||
Everywhere else in this repo we AUDIT existing markup (google_seo.py
|
||||
`inspect`, geo-analyzer's JSON-LD rules); this is the one verb that
|
||||
GENERATES it. Reservation + potentialAction matter now that AI Mode
|
||||
executes restaurant reservations; DiscussionForumPosting is a live SERP
|
||||
feature; ProfilePage with sameAs/knowsAbout is the cheapest entity-graph
|
||||
builder for AI citation correlation. geo-analyzer's G2 batch calls this
|
||||
instead of hand-writing the markup — it only generates STRUCTURE, unknown
|
||||
field VALUES stay the caller's `[À COMPLÉTER]` placeholder, never invented
|
||||
here.
|
||||
"""
|
||||
import argparse, json
|
||||
|
||||
|
||||
def reservation(provider, start, *, end=None, party_size=None,
|
||||
reservation_id=None, reservation_for_name=None,
|
||||
customer_name=None, customer_email=None,
|
||||
kind="FoodEstablishmentReservation"):
|
||||
"""Reservation JSON-LD block. Defaults to FoodEstablishment."""
|
||||
payload = {
|
||||
"@context": "https://schema.org",
|
||||
"@type": kind,
|
||||
"reservationStatus": "https://schema.org/ReservationConfirmed",
|
||||
"provider": {"@type": "Organization", "name": provider},
|
||||
"reservationFor": {
|
||||
"@type": "FoodEstablishment"
|
||||
if kind == "FoodEstablishmentReservation" else "Place",
|
||||
"name": reservation_for_name or provider,
|
||||
},
|
||||
"startTime": start,
|
||||
"endTime": end,
|
||||
"partySize": party_size,
|
||||
"reservationId": reservation_id,
|
||||
}
|
||||
if customer_name or customer_email:
|
||||
payload["underName"] = {"@type": "Person", "name": customer_name,
|
||||
"email": customer_email}
|
||||
return payload
|
||||
|
||||
|
||||
def order_action(merchant, *, order_url, name="Order online",
|
||||
accepted_payment_method=None, delivery_method=None):
|
||||
"""OrderAction potentialAction block. Attach to a Product/Service via
|
||||
{"@type": "Product", "potentialAction": <this dict>}."""
|
||||
payload = {
|
||||
"@context": "https://schema.org",
|
||||
"@type": "OrderAction",
|
||||
"name": name,
|
||||
"target": {
|
||||
"@type": "EntryPoint",
|
||||
"urlTemplate": order_url,
|
||||
"inLanguage": "en-US",
|
||||
"actionPlatform": [
|
||||
"https://schema.org/DesktopWebPlatform",
|
||||
"https://schema.org/MobileWebPlatform",
|
||||
],
|
||||
},
|
||||
"deliveryMethod": delivery_method or [
|
||||
"https://schema.org/OnSitePickup",
|
||||
"https://schema.org/ParcelService",
|
||||
],
|
||||
"priceSpecification": {
|
||||
"@type": "PriceSpecification",
|
||||
"eligibleTransactionVolume": {
|
||||
"@type": "PriceSpecification",
|
||||
"minPrice": 0,
|
||||
"priceCurrency": "USD",
|
||||
},
|
||||
},
|
||||
"merchant": {"@type": "Organization", "name": merchant},
|
||||
}
|
||||
if accepted_payment_method:
|
||||
payload["acceptedPaymentMethod"] = [
|
||||
{"@type": "PaymentMethod", "name": m}
|
||||
for m in accepted_payment_method
|
||||
]
|
||||
return payload
|
||||
|
||||
|
||||
def discussion(headline, author, *, url, date_published, text=None,
|
||||
date_modified=None, interaction_count=None,
|
||||
comment_count=None):
|
||||
"""DiscussionForumPosting JSON-LD block."""
|
||||
payload = {
|
||||
"@context": "https://schema.org",
|
||||
"@type": "DiscussionForumPosting",
|
||||
"headline": headline,
|
||||
"author": {"@type": "Person", "name": author},
|
||||
"datePublished": date_published,
|
||||
"dateModified": date_modified,
|
||||
"url": url,
|
||||
"mainEntityOfPage": {"@type": "WebPage", "@id": url},
|
||||
"text": text,
|
||||
"commentCount": comment_count,
|
||||
}
|
||||
if interaction_count:
|
||||
payload["interactionStatistic"] = [
|
||||
{"@type": "InteractionCounter",
|
||||
"interactionType": "https://schema.org/%s" % k,
|
||||
"userInteractionCount": v}
|
||||
for k, v in interaction_count.items()
|
||||
]
|
||||
return payload
|
||||
|
||||
|
||||
def profile(name, *, url, description=None, same_as=None, knows_about=None,
|
||||
works_for=None, image=None, job_title=None):
|
||||
"""ProfilePage JSON-LD block. sameAs + knowsAbout is the entity-graph
|
||||
helper for AI citation correlation — Wikipedia/GitHub/LinkedIn/ORCID
|
||||
URLs in sameAs disambiguate the person across knowledge graphs."""
|
||||
person = {
|
||||
"@type": "Person",
|
||||
"name": name,
|
||||
"url": url,
|
||||
"description": description,
|
||||
"sameAs": list(same_as) if same_as else None,
|
||||
"knowsAbout": list(knows_about) if knows_about else None,
|
||||
"worksFor": {"@type": "Organization", "name": works_for}
|
||||
if works_for else None,
|
||||
"image": image,
|
||||
"jobTitle": job_title,
|
||||
}
|
||||
return {"@context": "https://schema.org", "@type": "ProfilePage",
|
||||
"mainEntity": person, "url": url}
|
||||
|
||||
|
||||
def _strip_nones(value):
|
||||
"""Recursively drop dict keys AND list elements whose value is None —
|
||||
the emitted JSON-LD must never contain a null."""
|
||||
if isinstance(value, dict):
|
||||
return {k: _strip_nones(v) for k, v in value.items() if v is not None}
|
||||
if isinstance(value, list):
|
||||
return [_strip_nones(v) for v in value if v is not None]
|
||||
return value
|
||||
|
||||
|
||||
def _need(value, field):
|
||||
"""Raise on a schema-required field that is present but empty — the
|
||||
case argparse's `required=True` cannot catch (an empty string is a
|
||||
given flag, not a missing one)."""
|
||||
if value is None or not str(value).strip():
|
||||
raise ValueError("missing required field: %s" % field)
|
||||
return value
|
||||
|
||||
|
||||
def _generate(kind, args):
|
||||
"""Route to the matching generator, enforcing schema-required fields."""
|
||||
if kind == "reservation":
|
||||
return reservation(
|
||||
_need(args.provider, "provider"), _need(args.start, "start"),
|
||||
end=args.end, party_size=args.party_size,
|
||||
reservation_id=args.reservation_id,
|
||||
reservation_for_name=args.reservation_for_name,
|
||||
customer_name=args.customer_name,
|
||||
customer_email=args.customer_email, kind=args.reservation_kind,
|
||||
)
|
||||
if kind == "order":
|
||||
return order_action(
|
||||
_need(args.merchant, "merchant"),
|
||||
order_url=_need(args.order_url, "order_url"), name=args.name,
|
||||
accepted_payment_method=args.accepted_payment_method,
|
||||
delivery_method=args.delivery_method,
|
||||
)
|
||||
if kind == "discussion":
|
||||
interaction = {"LikeAction": args.likes} if args.likes else None
|
||||
return discussion(
|
||||
_need(args.headline, "headline"), _need(args.author, "author"),
|
||||
url=_need(args.url, "url"),
|
||||
date_published=_need(args.date_published, "date_published"),
|
||||
text=args.text, date_modified=args.date_modified,
|
||||
interaction_count=interaction, comment_count=args.comment_count,
|
||||
)
|
||||
if kind == "profile":
|
||||
return profile(
|
||||
_need(args.name, "name"), url=_need(args.url, "url"),
|
||||
description=args.description, same_as=args.same_as,
|
||||
knows_about=args.knows_about, works_for=args.works_for,
|
||||
image=args.image, job_title=args.job_title,
|
||||
)
|
||||
raise ValueError("unknown kind: %r" % kind) # pragma: no cover — argparse
|
||||
|
||||
|
||||
def _envelope(payload, script_tag):
|
||||
cleaned = _strip_nones(payload)
|
||||
out = {"status": "ok", "source": "schema_gen",
|
||||
"type": cleaned.get("@type"), "jsonld": cleaned}
|
||||
if script_tag:
|
||||
pretty = json.dumps(cleaned, indent=2, ensure_ascii=False)
|
||||
out["script"] = ('<script type="application/ld+json">\n%s\n</script>'
|
||||
% pretty)
|
||||
return out
|
||||
|
||||
|
||||
def _script_tag_parent():
|
||||
"""`--script-tag` as a shared parent parser, so it is valid on every
|
||||
subcommand — `fetch.sh schema_gen <type> [flags]` puts the type FIRST,
|
||||
and argparse only accepts a flag after a subcommand token if that flag
|
||||
was declared on the subparser, not the top-level one."""
|
||||
parent = argparse.ArgumentParser(add_help=False)
|
||||
parent.add_argument(
|
||||
"--script-tag", action="store_true",
|
||||
help="Wrap jsonld in <script type=application/ld+json>.",
|
||||
)
|
||||
return parent
|
||||
|
||||
|
||||
def _add_reservation_args(sub, parents):
|
||||
p = sub.add_parser("reservation", parents=parents,
|
||||
help="FoodEstablishmentReservation et al.")
|
||||
p.add_argument("--provider", required=True)
|
||||
p.add_argument("--start", required=True, help="ISO 8601 startTime.")
|
||||
p.add_argument("--end")
|
||||
p.add_argument("--party-size", type=int)
|
||||
p.add_argument("--reservation-id")
|
||||
p.add_argument("--reservation-for-name")
|
||||
p.add_argument("--customer-name")
|
||||
p.add_argument("--customer-email")
|
||||
p.add_argument(
|
||||
"--reservation-kind", dest="reservation_kind",
|
||||
default="FoodEstablishmentReservation",
|
||||
choices=(
|
||||
"FoodEstablishmentReservation", "LodgingReservation",
|
||||
"RentalCarReservation", "TaxiReservation", "EventReservation",
|
||||
"TrainReservation", "FlightReservation",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _add_order_args(sub, parents):
|
||||
p = sub.add_parser("order", parents=parents,
|
||||
help="OrderAction (potentialAction).")
|
||||
p.add_argument("--merchant", required=True)
|
||||
p.add_argument("--order-url", required=True)
|
||||
p.add_argument("--name", default="Order online")
|
||||
p.add_argument("--accepted-payment-method", nargs="*", default=None)
|
||||
p.add_argument("--delivery-method", nargs="*", default=None)
|
||||
|
||||
|
||||
def _add_discussion_args(sub, parents):
|
||||
p = sub.add_parser("discussion", parents=parents,
|
||||
help="DiscussionForumPosting.")
|
||||
p.add_argument("--headline", required=True)
|
||||
p.add_argument("--author", required=True)
|
||||
p.add_argument("--url", required=True)
|
||||
p.add_argument("--date", dest="date_published", required=True)
|
||||
p.add_argument("--text")
|
||||
p.add_argument("--date-modified")
|
||||
p.add_argument("--comment-count", type=int)
|
||||
p.add_argument("--likes", type=int, default=None,
|
||||
help="LikeAction count (interactionStatistic).")
|
||||
|
||||
|
||||
def _add_profile_args(sub, parents):
|
||||
p = sub.add_parser("profile", parents=parents,
|
||||
help="ProfilePage with sameAs / knowsAbout.")
|
||||
p.add_argument("--name", required=True)
|
||||
p.add_argument("--url", required=True)
|
||||
p.add_argument("--description")
|
||||
p.add_argument("--same-as", nargs="*", default=None)
|
||||
p.add_argument("--knows-about", nargs="*", default=None)
|
||||
p.add_argument("--works-for")
|
||||
p.add_argument("--image")
|
||||
p.add_argument("--job-title")
|
||||
|
||||
|
||||
def _build_parser():
|
||||
p = argparse.ArgumentParser(
|
||||
description="Schema.org JSON-LD generators (stdlib, deterministic)."
|
||||
)
|
||||
p.add_argument("--store", default=None) # accepted+ignored (dispatch)
|
||||
sub = p.add_subparsers(dest="kind", required=True)
|
||||
parents = [_script_tag_parent()]
|
||||
_add_reservation_args(sub, parents)
|
||||
_add_order_args(sub, parents)
|
||||
_add_discussion_args(sub, parents)
|
||||
_add_profile_args(sub, parents)
|
||||
return p
|
||||
|
||||
|
||||
def _cli():
|
||||
try:
|
||||
args = _build_parser().parse_args()
|
||||
payload = _generate(args.kind, args)
|
||||
print(json.dumps(_envelope(payload, args.script_tag), indent=2))
|
||||
except SystemExit as e:
|
||||
if e.code not in (0, None):
|
||||
print(json.dumps({"status": "error", "reason": "bad_usage"}))
|
||||
raise
|
||||
except Exception as e:
|
||||
# Fail-open: a missing required field or any other unexpected error
|
||||
# is a normal outcome here, never a traceback or empty stdout.
|
||||
print(json.dumps({"status": "degraded", "reason": str(e)}))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
_cli()
|
||||
@@ -1,113 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Deterministic /20 scoring from a findings list. Stdlib only.
|
||||
|
||||
/harden has a real scale (SKILL.md:435 — Critique -15, Haute -8, Moyenne -3,
|
||||
Basse -1, clamp [0,100]). /seo has none: every axis is felt, not computed, so
|
||||
two runs over identical code can produce different scores. That is a
|
||||
credibility problem on its own, and /client-handover gates on 17/20 — a
|
||||
wobbling number makes the gate arbitrary. H2 sharpens it further: now that
|
||||
drift reports what actually changed, a score moving on its own is visibly
|
||||
noise.
|
||||
|
||||
The split is the point. The LLM keeps the irreducible judgement — WHICH
|
||||
findings exist and how severe each is. The arithmetic stops being judgement:
|
||||
same findings in, same score out. Same principle as grouping cannibalisation
|
||||
rows in the engine rather than asking a model to add up 1000 of them.
|
||||
|
||||
Scale is /harden's, /5 into /20, so the whole skill family speaks one
|
||||
vocabulary.
|
||||
"""
|
||||
import argparse, json, sys
|
||||
|
||||
PENALTY = {"critique": 15, "haute": 8, "moyenne": 3, "basse": 1}
|
||||
|
||||
# STEP 9 weights. FULL = 7 axes, LOCAL = 4 (off-page/social/competitive are
|
||||
# not audited at that depth).
|
||||
WEIGHTS = {
|
||||
("FULL", "local"): {"technical": .20, "on-page": .20, "seo-local": .25,
|
||||
"off-page": .10, "social": .10, "competitive": .05,
|
||||
"legal": .10},
|
||||
("FULL", "national"): {"technical": .30, "on-page": .30, "seo-local": .05,
|
||||
"off-page": .15, "social": .05, "competitive": .10,
|
||||
"legal": .05},
|
||||
("LOCAL", "local"): {"technical": .25, "on-page": .35, "seo-local": .20,
|
||||
"legal": .20},
|
||||
("LOCAL", "national"):{"technical": .35, "on-page": .45, "seo-local": .05,
|
||||
"legal": .15},
|
||||
}
|
||||
|
||||
def _axis_score(findings):
|
||||
"""100 - Σ penalties, clamped, then /5 → /20. Prevalence shifts severity
|
||||
ONE step, never invents one: a finding on 1 of 12 sampled pages is not the
|
||||
same defect as one on 12 of 12, and pretending otherwise is what made the
|
||||
old scores unreproducible."""
|
||||
total = 0
|
||||
for f in findings:
|
||||
sev = str(f.get("severity", "")).lower()
|
||||
if sev not in PENALTY:
|
||||
raise ValueError("unknown severity: %r" % f.get("severity"))
|
||||
order = ["basse", "moyenne", "haute", "critique"]
|
||||
i = order.index(sev)
|
||||
aff, samp = f.get("affected"), f.get("sampled")
|
||||
if isinstance(aff, int) and isinstance(samp, int) and samp > 0:
|
||||
ratio = aff / samp
|
||||
if ratio >= 0.5:
|
||||
i = min(i + 1, len(order) - 1) # widespread → escalate
|
||||
elif aff <= 1:
|
||||
i = max(i - 1, 0) # isolated → de-escalate
|
||||
total += PENALTY[order[i]]
|
||||
return round(max(0, 100 - total) / 5.0, 1)
|
||||
|
||||
def score(payload):
|
||||
depth = str(payload.get("depth", "FULL")).upper()
|
||||
profile = str(payload.get("profile", "local")).lower()
|
||||
key = (depth, profile)
|
||||
if key not in WEIGHTS:
|
||||
return {"status": "error", "reason": "unknown depth/profile: %s/%s"
|
||||
% (depth, profile)}
|
||||
weights, axes_in = WEIGHTS[key], payload.get("axes", {})
|
||||
scored, na = {}, []
|
||||
for axis, w in weights.items():
|
||||
a = axes_in.get(axis)
|
||||
if a is None or str(a.get("status", "")).lower() == "na":
|
||||
na.append(axis) # N/A is not a zero
|
||||
continue
|
||||
try:
|
||||
s = _axis_score(a.get("findings", []))
|
||||
except ValueError as e:
|
||||
return {"status": "error", "reason": str(e)}
|
||||
scored[axis] = {"score_20": s, "weight": w,
|
||||
"findings": len(a.get("findings", []))}
|
||||
if not scored:
|
||||
return {"status": "degraded", "reason": "no_axis_scored"}
|
||||
# Renormalise over what was actually measured. R2 mandates this for a
|
||||
# client-rendered on-page axis and left it to the model to do by hand.
|
||||
live = sum(v["weight"] for v in scored.values())
|
||||
for v in scored.values():
|
||||
v["weight_renormalised"] = round(v["weight"] / live, 4)
|
||||
glob = sum(v["score_20"] * v["weight"] / live for v in scored.values())
|
||||
return {"status": "ok", "source": "score", "depth": depth,
|
||||
"profile": profile, "axes": scored, "na": sorted(na),
|
||||
"weights_renormalised": round(live, 4) != 1.0,
|
||||
"global_20": round(glob, 1)}
|
||||
|
||||
def _cli():
|
||||
try:
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--findings", default="-", help="JSON path, or - for stdin")
|
||||
p.add_argument("--store", default=None) # accepted+ignored
|
||||
args = p.parse_args()
|
||||
raw = sys.stdin.read() if args.findings == "-" else \
|
||||
open(args.findings, encoding="utf-8").read()
|
||||
print(json.dumps(score(json.loads(raw)), indent=2))
|
||||
except SystemExit as e:
|
||||
if e.code not in (0, None):
|
||||
print(json.dumps({"status": "error", "reason": "bad_usage"}))
|
||||
raise
|
||||
except Exception:
|
||||
# Unlike the fetch verbs this is pure arithmetic: a degrade here means
|
||||
# malformed input, never a network fact.
|
||||
print(json.dumps({"status": "error", "reason": "bad_findings_json"}))
|
||||
|
||||
if __name__ == "__main__":
|
||||
_cli()
|
||||
@@ -57,363 +57,12 @@ has "queries position field" "$Q" '"position": 6.3'
|
||||
I="$(SEO_DATA_MOCK_DIR="$MOCK" python3 "$SD/google_seo.py" inspect \
|
||||
--store "$S2" --account client-a --property sc-domain:ex.com --url https://ex.com/x)"
|
||||
has "inspect indexed true" "$I" '"indexed": true'
|
||||
# rich_results rides the same URL-Inspection response (no extra call/quota)
|
||||
has "rich verdict surfaced" "$I" '"verdict": "FAIL"'
|
||||
has "rich type breadcrumbs" "$I" '"type": "Breadcrumbs"'
|
||||
has "rich type faq" "$I" '"type": "FAQ"'
|
||||
has "rich counts error severity" "$I" '"errors": 2'
|
||||
has "rich counts warn severity" "$I" '"warnings": 1'
|
||||
has "rich keeps issue message" "$I" "Missing field 'acceptedAnswer'"
|
||||
# same issueMessage repeats across items — the rollup must collapse it to one
|
||||
NMSG="$(printf '%s' "$I" | grep -cF "Missing field 'acceptedAnswer'")"
|
||||
[ "$NMSG" = "1" ] && ok "rich dedupes issue messages" \
|
||||
|| no "rich dedupes issue messages" "got $NMSG occurrences"
|
||||
# Google OMITS richResultsResult when it detects none — absence is data, and
|
||||
# must not KeyError nor vanish into a missing key
|
||||
NR="$(SEO_DATA_MOCK_DIR="$SD/fixtures-norich" python3 "$SD/google_seo.py" inspect \
|
||||
--store "$S2" --account client-a --property sc-domain:ex.com --url https://ex.com/x)"
|
||||
has "no-rich → synthetic ABSENT" "$NR" '"verdict": "ABSENT"'
|
||||
has "no-rich keeps index status" "$NR" '"indexed": true'
|
||||
hasnt "no-rich emits no PARTIAL" "$NR" 'PARTIAL'
|
||||
DEG="$(env -u SEO_DATA_MOCK_DIR python3 "$SD/google_seo.py" queries \
|
||||
--store "$TMP2/none.json" --account nobody --property sc-domain:ex.com)"
|
||||
has "gsc degrades w/o creds" "$DEG" '"status": "degraded"'
|
||||
has "gsc degrade reason" "$DEG" 'no_credentials'
|
||||
rm -rf "$TMP2"
|
||||
|
||||
echo "── cannibalisation ──"
|
||||
# `keys` is additive: the single-dim consumer that reads `key` must not break
|
||||
has "queries keeps key (compat)" "$Q" '"key": "plombier paris"'
|
||||
has "queries adds keys list" "$Q" '"keys"'
|
||||
CAN="$(SEO_DATA_MOCK_DIR="$SD/fixtures-cannibal" python3 "$SD/google_seo.py" cannibal \
|
||||
--store "$S2" --account client-a --property sc-domain:ex.com)"
|
||||
has "cannibal ok" "$CAN" '"status": "ok"'
|
||||
# fixture: 3 pages on "urgence fuite", 2 on "plombier paris", 1 on "devis"
|
||||
has "cannibal finds 2 conflicts" "$CAN" '"conflict_count": 2'
|
||||
has "cannibal counts pages" "$CAN" '"pages": 3'
|
||||
has "cannibal sums impressions" "$CAN" '"total_impressions": 2400'
|
||||
hasnt "single-page query is not a conflict" "$CAN" 'devis plomberie'
|
||||
# biggest conflict first, and inside it the strongest page first
|
||||
CAN_FIRST="$(printf '%s' "$CAN" | python3 -c 'import sys,json; d=json.load(sys.stdin); print(d["conflicts"][0]["query"], d["conflicts"][0]["urls"][0]["url"])')"
|
||||
check_first() { [ "$1" = "$2" ] && ok "$3" || no "$3" "got[$1]"; }
|
||||
check_first "$CAN_FIRST" "urgence fuite https://ex.com/urgence" "cannibal ranks by impact"
|
||||
has "cannibal reports the cap" "$CAN" '"capped": false'
|
||||
|
||||
echo "── safe_fetch (DNS-rebinding / SSRF) ──"
|
||||
# Inject a hostile resolver: the name is public, the address is internal. This
|
||||
# is the rebinding vector a name-level guard cannot see — prove it is refused
|
||||
# BEFORE any connection. Deterministic + offline via the injected resolver.
|
||||
sfpy() { PYTHONPATH="$SD" python3 -c "$1" 2>&1; }
|
||||
REBIND="$(sfpy '
|
||||
import socket, safe_fetch as sf
|
||||
def meta(h,p,**k): return [(socket.AF_INET,socket.SOCK_STREAM,6,"",("169.254.169.254",p))]
|
||||
try: sf.safe_fetch("https://evil.example/", resolver=meta); print("CONNECTED")
|
||||
except sf.UnsafeTarget as e: print("REFUSED", e)')"
|
||||
has "rebind to metadata refused" "$REBIND" 'REFUSED'
|
||||
has "refusal names the ip" "$REBIND" '169.254.169.254'
|
||||
hasnt "never connected" "$REBIND" 'CONNECTED'
|
||||
MIXED="$(sfpy '
|
||||
import socket, safe_fetch as sf
|
||||
def mix(h,p,**k): return [(socket.AF_INET,socket.SOCK_STREAM,6,"",("93.184.216.34",p)),
|
||||
(socket.AF_INET,socket.SOCK_STREAM,6,"",("127.0.0.1",p))]
|
||||
try: sf.safe_fetch("https://evil.example/", resolver=mix); print("CONNECTED")
|
||||
except sf.UnsafeTarget as e: print("REFUSED")')"
|
||||
has "multi-A public+private refused" "$MIXED" 'REFUSED'
|
||||
# classification, dual-stack — is_global catches CGNAT the per-flags miss
|
||||
CLS="$(sfpy '
|
||||
import safe_fetch as sf
|
||||
pub=[c for c in ["8.8.8.8","2606:2800:220:1:248:1893:25c8:1946"] if sf._ip_is_public(c)]
|
||||
bad=[c for c in ["169.254.169.254","127.0.0.1","10.0.0.1","192.168.1.1","100.64.1.1","::1","fe80::1","0.0.0.0"] if sf._ip_is_public(c)]
|
||||
print("PUB",len(pub),"BADPASS",len(bad))')"
|
||||
has "public v4+v6 pass" "$CLS" 'PUB 2'
|
||||
has "no internal ip passes" "$CLS" 'BADPASS 0'
|
||||
# security review 2026-07-17: 6to4-relay anycast passes is_global — extra deny
|
||||
SIXTOFOUR="$(sfpy 'import safe_fetch as sf; print("6TO4", sf._ip_is_public("192.88.99.1"))')"
|
||||
has "6to4 relay anycast refused" "$SIXTOFOUR" '6TO4 False'
|
||||
# scheme + stdlib
|
||||
SCHEME="$(sfpy '
|
||||
import safe_fetch as sf
|
||||
try: sf.safe_fetch("file:///etc/passwd"); print("OK")
|
||||
except sf.UnsafeTarget: print("REFUSED")')"
|
||||
has "non-http scheme refused" "$SCHEME" 'REFUSED'
|
||||
IMP="$(/bin/grep -E "^(import|from) " "$SD/safe_fetch.py" | /bin/grep -cvE "gzip|http\.client|ipaddress|socket|ssl|urllib\.parse")"
|
||||
[ "$IMP" = "0" ] && ok "safe_fetch is stdlib-only" || no "safe_fetch is stdlib-only" "$IMP non-stdlib imports"
|
||||
hasnt "no requests dependency" "$(cat "$SD/safe_fetch.py")" 'import requests'
|
||||
|
||||
echo "── sitemap ──"
|
||||
SM="$(SEO_DATA_MOCK_DIR="$MOCK" python3 "$SD/sitemap.py" --url https://ex.com/sitemap.xml)"
|
||||
has "sitemap ok" "$SM" '"status": "ok"'
|
||||
has "sitemap not an index" "$SM" '"index": false'
|
||||
# fixture holds 8 <loc>: 1 empty, blog twice, ftp:// and a quoted one to drop
|
||||
has "sitemap dedupes" "$SM" '"count": 4'
|
||||
has "sitemap counts drops" "$SM" '"dropped": 2'
|
||||
has "sitemap strips whitespace" "$SM" '"https://ex.com/spaced"'
|
||||
hasnt "sitemap drops non-http" "$SM" 'ftp://'
|
||||
hasnt "sitemap drops shell-meta" "$SM" 'bad"quote'
|
||||
# namespace-agnostic: real sitemaps carry sitemaps.org xmlns (+ xhtml here)
|
||||
has "sitemap reads namespaced" "$SM" '"https://ex.com/services"'
|
||||
# REGRESSION: <image:loc> also ends with '}loc'. An endswith test counted image
|
||||
# sitemap entries as pages — a real native site returned 27 for 24 <url>, and
|
||||
# img/logo.png was about to be sampled and audited as a page.
|
||||
hasnt "image:loc is not a page" "$SM" '/img/logo.png'
|
||||
hasnt "image:loc jpeg not a page" "$SM" '/img/hero.jpeg'
|
||||
has "image ns does not inflate count" "$SM" '"count": 4'
|
||||
|
||||
IDX="$(SEO_DATA_MOCK_DIR="$SD/fixtures-sitemap-index" python3 "$SD/sitemap.py" \
|
||||
--url https://ex.com/sitemap.xml)"
|
||||
has "sitemapindex detected" "$IDX" '"index": true'
|
||||
has "sitemapindex fans out" "$IDX" '"children_read": 2'
|
||||
has "sitemapindex no child fail" "$IDX" '"children_failed": 0'
|
||||
has "sitemapindex yields urls" "$IDX" '"https://ex.com/child-a"'
|
||||
|
||||
# A sitemap NEVER has a DTD. Refused at the door: xml.etree does not expand
|
||||
# external entities but IS billion-laughs-vulnerable, and the 20MB read ceiling
|
||||
# bounds the input, not the expansion. Refusing beats depending on the parser,
|
||||
# and keeps this module stdlib-only (no defusedxml, no venv).
|
||||
DTD="$(SEO_DATA_MOCK_DIR="$SD/fixtures-sitemap-dtd" python3 "$SD/sitemap.py" \
|
||||
--url https://ex.com/sitemap.xml)"
|
||||
has "billion-laughs refused" "$DTD" '"status": "degraded"'
|
||||
has "dtd reason is distinct" "$DTD" 'unsafe_xml_dtd'
|
||||
hasnt "dtd never parsed" "$DTD" '"count"'
|
||||
# security review 2026-07-17: a >4KB leading comment pushed <!DOCTYPE past the
|
||||
# old raw[:4096] scan while ET still parsed+expanded it. Now the whole doc is
|
||||
# scanned. Prove a padded DTD is refused and the entity never expands.
|
||||
PADDED="$(python3 -c '
|
||||
import sys; sys.path.insert(0,"'"$SD"'"); import sitemap as sm
|
||||
bomb=("<?xml version=\"1.0\"?>\n<!-- "+("x"*5000)+" -->\n"
|
||||
"<!DOCTYPE d [ <!ENTITY lol \"lol\"> ]>\n<urlset><url><loc>https://x/&lol;</loc></url></urlset>").encode()
|
||||
try: sm._refuse_dtd(bomb); print("PARSED")
|
||||
except sm.UnsafeXML: print("REFUSED")')"
|
||||
has "padded DTD refused (full scan)" "$PADDED" 'REFUSED'
|
||||
|
||||
echo "── render_check (R2) ──"
|
||||
SPA="$(SEO_DATA_MOCK_DIR="$SD/fixtures-spa" python3 "$SD/render_check.py" \
|
||||
--url https://spa.example/)"
|
||||
has "spa → client-rendered" "$SPA" '"verdict": "client-rendered"'
|
||||
has "spa has no h1 in html" "$SPA" '"h1_in_html": 0'
|
||||
has "spa warns about false negs" "$SPA" 'false'
|
||||
# the shell carries a fat window.__INITIAL_STATE__ script: script text is NOT
|
||||
# page text, or a 200KB React bundle would read as a rich page
|
||||
has "script text is not content" "$SPA" '"body_text_chars": 7'
|
||||
SSR="$(SEO_DATA_MOCK_DIR="$SD/fixtures-ssr" python3 "$SD/render_check.py" \
|
||||
--url https://ssr.example/)"
|
||||
has "ssr → server-rendered" "$SSR" '"verdict": "server-rendered"'
|
||||
has "ssr counts jsonld" "$SSR" '"jsonld_in_html": 1'
|
||||
has "ssr sees meta description" "$SSR" '"meta_description_in_html": true'
|
||||
hasnt "ssr emits no warning" "$SSR" 'warning'
|
||||
|
||||
echo "── linkgraph ──"
|
||||
LG="$(SEO_DATA_MOCK_DIR="$SD/fixtures-linkgraph" python3 "$SD/linkgraph.py" \
|
||||
--url https://ex.com/sitemap.xml)"
|
||||
has "linkgraph ok" "$LG" '"status": "ok"'
|
||||
has "linkgraph crawls all" "$LG" '"pages_crawled": 7'
|
||||
# THE test: a planted page nobody links to must be found. Two live sites both
|
||||
# returned zero orphans; without this, "always returns []" looks identical.
|
||||
has "finds the planted orphan" "$LG" '"https://ex.com/orphan"'
|
||||
has "orphan is also unreachable" "$LG" '"unreachable"'
|
||||
has "depth chain measured" "$LG" '"max_depth": 4'
|
||||
has "flags >3 clicks" "$LG" '"https://ex.com/deepest"'
|
||||
# home links: /a and /b only. anchor, .css?v=, mailto:, tel:, external, .png
|
||||
# are not page links — 9 total across the 7 pages.
|
||||
has "filters non-page links" "$LG" '"total_internal_links": 9'
|
||||
hasnt "no external host" "$LG" 'other.com'
|
||||
hasnt "no asset link" "$LG" 'main.css'
|
||||
hasnt "no image link" "$LG" 'logo.png'
|
||||
# /b/ in the markup vs /b in the sitemap must be ONE node, not a phantom orphan
|
||||
hasnt "trailing slash unified" "$LG" '"https://ex.com/b/"'
|
||||
|
||||
# An orphan from a partial crawl is a false orphan: withhold, do not truncate.
|
||||
CAP="$(SEO_DATA_MOCK_DIR="$SD/fixtures-linkgraph" python3 "$SD/linkgraph.py" \
|
||||
--url https://ex.com/sitemap.xml --max 3)"
|
||||
has "cap is reported" "$CAP" '"capped": true'
|
||||
has "capped withholds orphans" "$CAP" '"orphans_withheld": true'
|
||||
hasnt "capped emits no orphans" "$CAP" '"orphans":'
|
||||
|
||||
echo "── score (I7) ──"
|
||||
sc() { printf '%s' "$1" | python3 "$SD/score.py" --findings -; }
|
||||
# technical: haute(-8) + moyenne(-3) = 100-11 = 89 → 17.8
|
||||
B='{"depth":"FULL","profile":"local","axes":{"technical":{"findings":[{"severity":"haute"},{"severity":"moyenne"}]},"seo-local":{"findings":[]},"off-page":{"findings":[]},"social":{"findings":[]},"competitive":{"findings":[]},"legal":{"findings":[]},"on-page":{"findings":[]}}}'
|
||||
R="$(sc "$B")"
|
||||
has "harden scale, /5 into /20" "$R" '"score_20": 17.8'
|
||||
has "no findings = 20" "$R" '"score_20": 20.0'
|
||||
has "nothing renormalised" "$R" '"weights_renormalised": false'
|
||||
# THE point of I7: same findings in, same score out
|
||||
A1="$(sc "$B" | python3 -c 'import sys,json;print(json.load(sys.stdin)["global_20"])')"
|
||||
A2="$(sc "$B" | python3 -c 'import sys,json;print(json.load(sys.stdin)["global_20"])')"
|
||||
[ "$A1" = "$A2" ] && ok "score is reproducible" || no "score is reproducible" "$A1 vs $A2"
|
||||
# N/A is not a zero, and R2 mandated renormalising by hand — now computed
|
||||
NA='{"depth":"FULL","profile":"local","axes":{"technical":{"findings":[]},"on-page":{"status":"na"},"seo-local":{"findings":[]},"off-page":{"status":"na"},"social":{"findings":[]},"competitive":{"findings":[]},"legal":{"findings":[]}}}'
|
||||
RN="$(sc "$NA")"
|
||||
has "na axes listed" "$RN" '"on-page"'
|
||||
has "renormalisation flagged" "$RN" '"weights_renormalised": true'
|
||||
# all axes 20 → global must stay 20: N/A must not drag the mean down
|
||||
has "na is not a zero" "$RN" '"global_20": 20.0'
|
||||
# prevalence shifts severity ONE step, both ways
|
||||
WIDE='{"depth":"LOCAL","profile":"local","axes":{"technical":{"findings":[{"severity":"moyenne","affected":10,"sampled":12}]},"on-page":{"findings":[]},"seo-local":{"findings":[]},"legal":{"findings":[]}}}'
|
||||
ONE='{"depth":"LOCAL","profile":"local","axes":{"technical":{"findings":[{"severity":"moyenne","affected":1,"sampled":12}]},"on-page":{"findings":[]},"seo-local":{"findings":[]},"legal":{"findings":[]}}}'
|
||||
has "widespread escalates (-8)" "$(sc "$WIDE")" '"score_20": 18.4'
|
||||
has "isolated de-escalates (-1)" "$(sc "$ONE")" '"score_20": 19.8'
|
||||
# malformed input is an error, never a silently wrong number
|
||||
has "unknown severity rejected" "$(sc '{"depth":"FULL","profile":"local","axes":{"technical":{"findings":[{"severity":"bogus"}]}}}')" '"status": "error"'
|
||||
has "unknown profile rejected" "$(sc '{"depth":"FULL","profile":"martian","axes":{}}')" '"status": "error"'
|
||||
has "garbage json is an error" "$(sc 'not json')" '"status": "error"'
|
||||
|
||||
echo "── drift (H2) ──"
|
||||
DH="$(mktemp -d)"
|
||||
D1="$(HOME="$DH" SEO_DATA_MOCK_DIR="$SD/fixtures-drift-v1" python3 "$SD/drift.py" \
|
||||
--url https://ex.com/sitemap.xml)"
|
||||
has "first run is a baseline" "$D1" '"baseline": true'
|
||||
has "baseline captures pages" "$D1" '"pages": 3'
|
||||
hasnt "baseline diffs nothing" "$D1" '"regressions"'
|
||||
# v2: canonical lost on /a, h1+jsonld lost on /, title reworded, /gone removed,
|
||||
# /neuve added. Losses are regressions; a reworded title is not.
|
||||
D2="$(HOME="$DH" SEO_DATA_MOCK_DIR="$SD/fixtures-drift-v2" python3 "$SD/drift.py" \
|
||||
--url https://ex.com/sitemap.xml)"
|
||||
has "second run diffs" "$D2" '"baseline": false'
|
||||
has "detects removed url" "$D2" '"https://ex.com/gone"'
|
||||
has "detects added url" "$D2" '"https://ex.com/neuve"'
|
||||
has "lost canonical = regression" "$D2" '"canonical"'
|
||||
has "lost h1 = regression" "$D2" '"h1_count"'
|
||||
has "lost jsonld = regression" "$D2" '"jsonld_types"'
|
||||
# the classification IS the feature: losing a signal != changing one
|
||||
NREG="$(printf '%s' "$D2" | python3 -c 'import sys,json; print(len(json.load(sys.stdin)["regressions"]))')"
|
||||
NCHG="$(printf '%s' "$D2" | python3 -c 'import sys,json; print(len(json.load(sys.stdin)["changes"]))')"
|
||||
[ "$NREG" = "3" ] && ok "3 losses classed as regressions" \
|
||||
|| no "3 losses classed as regressions" "got $NREG"
|
||||
[ "$NCHG" = "1" ] && ok "reworded title is a change, not a regression" \
|
||||
|| no "reworded title is a change, not a regression" "got $NCHG"
|
||||
rm -rf "$DH"
|
||||
|
||||
echo "── schema_gen ──"
|
||||
SG() { python3 "$SD/schema_gen.py" "$@"; }
|
||||
RES="$(SG reservation --provider "Chez X" --start "2026-08-01T19:00")"
|
||||
has "reservation ok" "$RES" '"status": "ok"'
|
||||
has "reservation type surfaced" "$RES" '"type": "FoodEstablishmentReservation"'
|
||||
has "jsonld has @context" "$RES" '"@context": "https://schema.org"'
|
||||
has "reservation keeps provider" "$RES" 'Chez X'
|
||||
has "reservation keeps start" "$RES" '2026-08-01T19:00'
|
||||
PROF="$(SG profile --name "Jane Doe" --url https://ex.com/about)"
|
||||
has "profile ok" "$PROF" '"status": "ok"'
|
||||
has "profile type surfaced" "$PROF" '"type": "ProfilePage"'
|
||||
ORD="$(SG order --merchant "Acme" --order-url https://ex.com/order)"
|
||||
has "order ok" "$ORD" '"status": "ok"'
|
||||
has "order type surfaced" "$ORD" '"type": "OrderAction"'
|
||||
DISC="$(SG discussion --headline "Q" --author "Jo" --url https://ex.com/t/1 \
|
||||
--date 2026-05-01T00:00:00Z)"
|
||||
has "discussion ok" "$DISC" '"status": "ok"'
|
||||
has "discussion type surfaced" "$DISC" '"type": "DiscussionForumPosting"'
|
||||
# argparse required=True catches an OMITTED flag → bad usage, exit 2
|
||||
BADRES="$(SG reservation --start 2026-08-01T19:00 2>/dev/null)"; BADRC=$?
|
||||
hasnt "missing --provider is not ok" "$BADRES" '"status": "ok"'
|
||||
[ "$BADRC" = "2" ] && ok "missing --provider exit 2" \
|
||||
|| no "missing --provider exit 2" "got $BADRC"
|
||||
# a required field argparse ALLOWS through (flag given, value empty) must
|
||||
# still fail open — degraded, not a crash, exit 0
|
||||
EMPTYRES="$(SG reservation --provider "" --start 2026-08-01T19:00)"; EMPTYRC=$?
|
||||
hasnt "empty --provider is not ok" "$EMPTYRES" '"status": "ok"'
|
||||
has "empty --provider degrades" "$EMPTYRES" '"status": "degraded"'
|
||||
[ "$EMPTYRC" = "0" ] && ok "empty --provider exit 0" \
|
||||
|| no "empty --provider exit 0" "got $EMPTYRC"
|
||||
# --script-tag must work AFTER the type, matching `fetch.sh schema_gen
|
||||
# <type> [flags]` — the shape the dispatcher actually calls it with. The
|
||||
# envelope is JSON, so the `script` field's own quotes are backslash-escaped
|
||||
# in the raw stdout — decode it to check the LITERAL wrapper string.
|
||||
SCRIPT="$(SG profile --name "Jane Doe" --url https://ex.com/about --script-tag)"
|
||||
SCRIPT_TAG="$(printf '%s' "$SCRIPT" | \
|
||||
python3 -c 'import sys,json; print(json.load(sys.stdin)["script"])')"
|
||||
has "script-tag wraps output" "$SCRIPT_TAG" '<script type="application/ld+json">'
|
||||
# an omitted optional field must never surface as a JSON null
|
||||
hasnt "no null ever emitted" "$RES" 'null'
|
||||
# stdlib ONLY — no requests/httpx/bs4/any third-party import
|
||||
IMPORTS="$(grep -E '^(import|from) ' "$SD/schema_gen.py")"
|
||||
if printf '%s' "$IMPORTS" | grep -qiE 'requests|httpx|bs4'; then
|
||||
no "schema_gen stdlib only" "third-party import found: $IMPORTS"
|
||||
else
|
||||
ok "schema_gen stdlib only"
|
||||
fi
|
||||
# dispatch wiring: --store precedes the type (fetch.sh's own convention),
|
||||
# --script-tag comes after it (the caller's convention) — both must work
|
||||
# through the real fetch.sh entrypoint, not just the bare script
|
||||
FSG="$(SEO_DATA_ENV_FILE=/dev/null SEO_DATA_STORE=/nonexistent bash "$SD/fetch.sh" \
|
||||
schema_gen reservation --provider "Chez X" --start 2026-08-01T19:00 --script-tag)"
|
||||
has "fetch dispatches schema_gen" "$FSG" '"status": "ok"'
|
||||
FSG_TAG="$(printf '%s' "$FSG" | \
|
||||
python3 -c 'import sys,json; print(json.load(sys.stdin)["script"])')"
|
||||
has "fetch schema_gen script-tag" "$FSG_TAG" '<script type="application/ld+json">'
|
||||
|
||||
echo "── content_quality ──"
|
||||
CQ() { python3 "$SD/content_quality.py" "$@"; }
|
||||
# feed the phrase list's OWN entries so the match is exact, not paraphrased —
|
||||
# a detector proven only on the maintainer's paraphrase proves nothing
|
||||
FILLER_TXT="In today's fast-paced world, it's important to note that this \
|
||||
article will delve into the ever-evolving landscape of technology. Let's \
|
||||
dive in and navigate the complexities together, leveraging the power of \
|
||||
innovation to unlock the potential of your business. Ultimately, this \
|
||||
cutting-edge, state-of-the-art approach is a testament to progress. \
|
||||
Moreover, furthermore, in conclusion, transform your outcomes today."
|
||||
CLEAN_TXT="The 2024 ADEME report found French households spent 2,137 EUR \
|
||||
on heating, up 12% from 2021."
|
||||
FILLER_OUT="$(printf '%s' "$FILLER_TXT" | CQ)"
|
||||
CLEAN_OUT="$(printf '%s' "$CLEAN_TXT" | CQ)"
|
||||
has "filler text is ok" "$FILLER_OUT" '"status": "ok"'
|
||||
has "clean text is ok" "$CLEAN_OUT" '"status": "ok"'
|
||||
# flags is a JSON array — extract it in isolation so the check can't be
|
||||
# fooled by the always-present "matches": {"filler": [...]} key sharing
|
||||
# the same quoted word
|
||||
FILLER_FLAGS="$(printf '%s' "$FILLER_OUT" | \
|
||||
python3 -c 'import sys,json; print(",".join(json.load(sys.stdin)["flags"]))')"
|
||||
CLEAN_FLAGS="$(printf '%s' "$CLEAN_OUT" | \
|
||||
python3 -c 'import sys,json; print(",".join(json.load(sys.stdin)["flags"]))')"
|
||||
case "$FILLER_FLAGS" in
|
||||
*filler*|*ai-patterns*) ok "filler-heavy text is flagged" ;;
|
||||
*) no "filler-heavy text is flagged" "flags: $FILLER_FLAGS" ;;
|
||||
esac
|
||||
hasnt "clean text is not flagged filler" "$CLEAN_FLAGS" 'filler'
|
||||
hasnt "clean text is not flagged ai-patterns" "$CLEAN_FLAGS" 'ai-patterns'
|
||||
# proves BOTH directions: an always-flag or a never-flag detector is useless
|
||||
FILLER_Q="$(printf '%s' "$FILLER_OUT" | \
|
||||
python3 -c 'import sys,json; print(json.load(sys.stdin)["overall_quality"])')"
|
||||
CLEAN_Q="$(printf '%s' "$CLEAN_OUT" | \
|
||||
python3 -c 'import sys,json; print(json.load(sys.stdin)["overall_quality"])')"
|
||||
[ "$FILLER_Q" -lt 50 ] && ok "filler-heavy text scores LOW overall_quality" \
|
||||
|| no "filler-heavy text scores LOW overall_quality" "got $FILLER_Q"
|
||||
[ "$CLEAN_Q" -gt "$FILLER_Q" ] && ok "clean dense text scores higher" \
|
||||
|| no "clean dense text scores higher" "$CLEAN_Q vs $FILLER_Q"
|
||||
# empty / whitespace-only input never crashes and never claims a result
|
||||
EMPTY_OUT="$(printf '' | CQ)"
|
||||
has "empty input degrades" "$EMPTY_OUT" '"status": "degraded"'
|
||||
has "empty input reason" "$EMPTY_OUT" 'empty_input'
|
||||
WS_OUT="$(printf ' \n\t ' | CQ)"
|
||||
has "whitespace-only degrades" "$WS_OUT" '"status": "degraded"'
|
||||
# --file path works, no fixture committed — mktemp + rm
|
||||
CQTMP="$(mktemp)"; printf '%s' "$CLEAN_TXT" > "$CQTMP"
|
||||
FILE_OUT="$(CQ --file "$CQTMP")"
|
||||
has "file input is ok" "$FILE_OUT" '"status": "ok"'
|
||||
rm -f "$CQTMP"
|
||||
# a missing --file degrades, never a traceback
|
||||
MISSING_OUT="$(CQ --file /nonexistent/path/content-quality-test.txt)"
|
||||
has "missing --file degrades" "$MISSING_OUT" '"status": "degraded"'
|
||||
# stdlib ONLY — asserted, not assumed
|
||||
CQ_IMPORTS="$(grep -E '^(import|from) ' "$SD/content_quality.py")"
|
||||
if printf '%s' "$CQ_IMPORTS" | grep -qivE '^(import argparse, json, re, sys|from collections import counter|from typing import iterable)$'; then
|
||||
no "content_quality stdlib only" "unexpected import: $CQ_IMPORTS"
|
||||
else
|
||||
ok "content_quality stdlib only"
|
||||
fi
|
||||
# ADVISORY HONESTY (LRN-131/133): a heuristic signal, never a verdict
|
||||
hasnt "never claims ai-written" "$FILLER_OUT" 'ai-written'
|
||||
hasnt "never claims is AI verdict" "$FILLER_OUT" 'is AI'
|
||||
# dispatch wiring: --store precedes the verb (fetch.sh's own convention);
|
||||
# both stdin AND --file must work through the real entrypoint
|
||||
FCQ_STDIN="$(printf '%s' "$CLEAN_TXT" | \
|
||||
SEO_DATA_ENV_FILE=/dev/null SEO_DATA_STORE=/nonexistent bash "$SD/fetch.sh" content_quality)"
|
||||
has "fetch dispatches content_quality (stdin)" "$FCQ_STDIN" '"status": "ok"'
|
||||
CQTMP2="$(mktemp)"; printf '%s' "$CLEAN_TXT" > "$CQTMP2"
|
||||
FCQ_FILE="$(SEO_DATA_ENV_FILE=/dev/null SEO_DATA_STORE=/nonexistent bash "$SD/fetch.sh" \
|
||||
content_quality --file "$CQTMP2")"
|
||||
has "fetch dispatches content_quality (--file)" "$FCQ_FILE" '"status": "ok"'
|
||||
rm -f "$CQTMP2"
|
||||
|
||||
echo "── fetch.sh ──"
|
||||
FETCH="$SD/fetch.sh"
|
||||
# SEO_DATA_ENV_FILE=/dev/null: tests must NEVER source the real ~/.claude/.env —
|
||||
@@ -539,8 +188,6 @@ tf "analyzer calls fetch crux" "$REPO/agents/seo-analyzer.md" "fetch.sh crux"
|
||||
tf "analyzer calls fetch queries" "$REPO/agents/seo-analyzer.md" "fetch.sh queries"
|
||||
tf "analyzer gsc subsection" "$REPO/agents/seo-analyzer.md" "Performance GSC"
|
||||
tf "catalog gsc oauth entry" "$REPO/agents/resources/automation-catalog.md" "make seo-connect"
|
||||
tf "geo-analyzer wires schema_gen" "$REPO/agents/geo-analyzer.md" "fetch.sh schema_gen"
|
||||
tf "geo-analyzer wires content_quality" "$REPO/agents/geo-analyzer.md" "fetch.sh content_quality"
|
||||
|
||||
echo "── account-mgmt locks ──"
|
||||
tf "skill routes account verbs" "$REPO/skills/seo/SKILL.md" "forget --all"
|
||||
@@ -553,9 +200,6 @@ tf "readme documents fetch.sh" "$REPO/lib/seo-data/README.md" "fetch.sh"
|
||||
tf "readme documents seo-connect" "$REPO/lib/seo-data/README.md" "make seo-connect"
|
||||
tf "readme documents forget" "$REPO/lib/seo-data/README.md" "forget --all"
|
||||
tf "readme revocation note" "$REPO/lib/seo-data/README.md" "myaccount.google.com/permissions"
|
||||
tf "readme documents schema_gen" "$REPO/lib/seo-data/README.md" "schema_gen"
|
||||
tf "readme documents content_quality" "$REPO/lib/seo-data/README.md" "content_quality"
|
||||
tf "readme states advisory caveat" "$REPO/lib/seo-data/README.md" "ADVISORY, NOT A VERDICT"
|
||||
|
||||
echo ""
|
||||
echo "seo-data engine: $PASS pass, $FAIL fail"
|
||||
|
||||
@@ -1,191 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Sitemap discovery -> normalized JSON. Stdlib only: no venv, no requests, no
|
||||
auth. Gives STEP 9 COVERAGE the denominator it was told to report and never
|
||||
had, and STEP 5 a real sampling frame instead of "5-15 key pages" chosen by
|
||||
eye.
|
||||
|
||||
Deliberately NOT a security boundary. urllib fetches these URLs, so nothing
|
||||
here reaches a shell and there is no injection surface to guard. The consumer
|
||||
is different: seo-analyzer interpolates URLs into curl, so IT must run
|
||||
lib/url-guard.sh at the point of use (same pattern as the sameAs check).
|
||||
Duplicating the guard here would just add a second copy to drift. `_sane`
|
||||
below is a cheap garbage filter, not that guard.
|
||||
"""
|
||||
import argparse, gzip, json, os
|
||||
from urllib.parse import urlparse
|
||||
|
||||
MAX_URLS = 50000 # sitemaps.org caps one file at 50k
|
||||
MAX_CHILDREN = 50 # sitemapindex fan-out cap: bound the work, report the cut
|
||||
TIMEOUT = 20
|
||||
|
||||
def _mock(name):
|
||||
d = os.environ.get("SEO_DATA_MOCK_DIR")
|
||||
if not d:
|
||||
return None
|
||||
path = os.path.join(d, name)
|
||||
if not os.path.exists(path):
|
||||
return None
|
||||
with open(path, "rb") as f:
|
||||
return f.read()
|
||||
|
||||
def _fetch(url):
|
||||
# SSRF + DNS-rebinding safe: resolve-then-pin, redirects re-validated.
|
||||
# This is the single seam for ALL network egress — linkgraph/render_check/
|
||||
# drift all call sitemap._fetch — so pinning here covers every verb.
|
||||
import safe_fetch # sibling, lazy
|
||||
raw = safe_fetch.safe_fetch(url, timeout=TIMEOUT, max_bytes=20 * 1024 * 1024)
|
||||
if raw[:2] == b"\x1f\x8b": # sitemap.xml.gz is common
|
||||
raw = gzip.decompress(raw)
|
||||
return raw
|
||||
|
||||
class UnsafeXML(Exception):
|
||||
"""A DTD reached the parser. Refused before parsing, not mitigated after."""
|
||||
|
||||
def _refuse_dtd(raw):
|
||||
"""A sitemap NEVER has a DTD: sitemaps.org is <?xml?> then <urlset xmlns=>.
|
||||
So refuse any doctype/entity outright, at the door.
|
||||
|
||||
This is the reason we do not pull in defusedxml. The stdlib parser is not
|
||||
the problem for XXE — xml.etree.ElementTree does not expand external
|
||||
entities, it raises on them — but it IS vulnerable to billion-laughs, where
|
||||
a 1 KB document expands to gigabytes in RAM. The 20 MB read ceiling bounds
|
||||
the input, not the expansion. Rejecting the construct beats depending on
|
||||
the parser's internals, and keeps this module stdlib-only: no venv, same as
|
||||
google_seo.py's mock/degrade paths. A sitemap with a DTD is not a sitemap
|
||||
we want anyway.
|
||||
"""
|
||||
# Scan the WHOLE document, not a prefix. A security review (2026-07-17)
|
||||
# showed a >4 KB leading comment pushed <!DOCTYPE past the old raw[:4096]
|
||||
# window while ET.fromstring still parsed and EXPANDED the entities —
|
||||
# billion-laughs reopened. A legitimate sitemap contains neither construct
|
||||
# anywhere, so a full case-insensitive scan is correct; over ≤20 MB it is a
|
||||
# single re.search, microseconds, no 20 MB uppercased copy.
|
||||
import re # stdlib, lazy
|
||||
if re.search(rb"(?i)<!\s*(DOCTYPE|ENTITY)", raw):
|
||||
raise UnsafeXML("DTD in sitemap")
|
||||
|
||||
SITEMAP_NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
|
||||
|
||||
def _is_page_loc(tag):
|
||||
"""A PAGE <loc>: sitemaps.org namespace, or namespace-less.
|
||||
|
||||
NOT <image:loc> or <video:loc>. Those live in Google's extension
|
||||
namespaces and name an ASSET inside a <url>, not a page of its own. An
|
||||
endswith('}loc') test matches them too — that shipped, and a real site
|
||||
caught it: 24 <url> + 3 <image:loc> came back as a count of 27, so the
|
||||
COVERAGE denominator was 12.5% too high and img/logo.png was about to be
|
||||
sampled and audited as a page.
|
||||
"""
|
||||
return tag == SITEMAP_NS + "loc" or tag == "loc"
|
||||
|
||||
def _locs(raw):
|
||||
"""(page <loc> texts, is_sitemapindex).
|
||||
|
||||
Walks the DIRECT children of each <url>/<sitemap> rather than root.iter():
|
||||
that alone excludes <image:image><image:loc>, and the namespace test above
|
||||
is the second lock. XML comments iterate as elements with no children, so
|
||||
they fall through harmlessly.
|
||||
"""
|
||||
import xml.etree.ElementTree as ET # stdlib, lazy
|
||||
_refuse_dtd(raw)
|
||||
root = ET.fromstring(raw)
|
||||
is_index = root.tag.endswith("sitemapindex")
|
||||
out = []
|
||||
for entry in root: # <url> | <sitemap>
|
||||
for child in entry: # direct children only
|
||||
if _is_page_loc(child.tag):
|
||||
text = (child.text or "").strip()
|
||||
if text:
|
||||
out.append(text)
|
||||
break # one <loc> per entry
|
||||
return out, is_index
|
||||
|
||||
def _sane(u):
|
||||
"""Cheap garbage filter — NOT lib/url-guard.sh. Drops what could never be a
|
||||
real page URL; the consumer still guards before curling."""
|
||||
if not u or len(u) > 2048:
|
||||
return False
|
||||
if any(c in u for c in '\n\r\t "\'\\`$<>{}|^'):
|
||||
return False
|
||||
return urlparse(u).scheme in ("http", "https")
|
||||
|
||||
def _expand(children):
|
||||
"""Fetch each child sitemap of an index. A child that fails is skipped and
|
||||
counted, never fatal: one dead child must not lose the other 49."""
|
||||
urls, ok, failed = [], 0, 0
|
||||
for c in children:
|
||||
raw = _mock("sitemap_child.xml")
|
||||
if raw is None:
|
||||
try:
|
||||
raw = _fetch(c)
|
||||
except Exception:
|
||||
failed += 1
|
||||
continue
|
||||
try:
|
||||
sub, _ = _locs(raw)
|
||||
except Exception:
|
||||
failed += 1
|
||||
continue
|
||||
urls.extend(sub)
|
||||
ok += 1
|
||||
return urls, ok, failed
|
||||
|
||||
def sitemap(url):
|
||||
raw = _mock("sitemap.xml")
|
||||
if raw is None:
|
||||
try:
|
||||
raw = _fetch(url)
|
||||
except Exception:
|
||||
return {"status": "degraded", "reason": "fetch_failed"}
|
||||
try:
|
||||
locs, is_index = _locs(raw)
|
||||
except UnsafeXML:
|
||||
# Distinct from parse_failed on purpose: this one is a finding, not a
|
||||
# glitch. A sitemap carrying a DTD is either broken tooling or someone
|
||||
# aiming a billion-laughs at the auditor.
|
||||
return {"status": "degraded", "reason": "unsafe_xml_dtd"}
|
||||
except Exception:
|
||||
return {"status": "degraded", "reason": "parse_failed"}
|
||||
out = {"status": "ok", "source": "sitemap", "index": is_index}
|
||||
if is_index:
|
||||
out["children_total"] = len(locs)
|
||||
kids, ok, failed = _expand(locs[:MAX_CHILDREN])
|
||||
out["children_read"], out["children_failed"] = ok, failed
|
||||
if len(locs) > MAX_CHILDREN: # say what was cut
|
||||
out["children_skipped"] = len(locs) - MAX_CHILDREN
|
||||
locs = kids
|
||||
seen, urls, dropped = set(), [], 0
|
||||
for u in locs:
|
||||
if not _sane(u):
|
||||
dropped += 1
|
||||
continue
|
||||
if u in seen:
|
||||
continue
|
||||
seen.add(u)
|
||||
urls.append(u)
|
||||
if len(urls) > MAX_URLS:
|
||||
out["truncated"] = len(urls) - MAX_URLS
|
||||
urls = urls[:MAX_URLS]
|
||||
out["count"], out["dropped"], out["urls"] = len(urls), dropped, urls
|
||||
if not urls:
|
||||
return {"status": "degraded", "reason": "no_urls"}
|
||||
return out
|
||||
|
||||
def _cli():
|
||||
try:
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--url", required=True)
|
||||
p.add_argument("--store", default=None) # accepted+ignored: uniform dispatch
|
||||
args = p.parse_args()
|
||||
print(json.dumps(sitemap(args.url), indent=2))
|
||||
except SystemExit as e:
|
||||
if e.code not in (0, None):
|
||||
print(json.dumps({"status": "error", "reason": "bad_usage"}))
|
||||
raise
|
||||
except Exception:
|
||||
# Same fail-open contract as google_seo.py: never a traceback, never
|
||||
# empty stdout, exit 0 so the audit degrades instead of dying.
|
||||
print(json.dumps({"status": "degraded", "reason": "unexpected_error"}))
|
||||
|
||||
if __name__ == "__main__":
|
||||
_cli()
|
||||
@@ -1,79 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Emit the directory exclusions that separate SOURCE from BUILD OUTPUT.
|
||||
#
|
||||
# EXCL="$(bash ~/.claude/lib/source-scope.sh grep)"
|
||||
# grep -rl "gtag" $EXCL --include="*.html" . # note: $EXCL unquoted
|
||||
#
|
||||
# mapfile -t FEXCL < <(bash ~/.claude/lib/source-scope.sh findargs)
|
||||
# find . "${FEXCL[@]}" -iname '*.jpg' -printf '%s %p\n' # quoted array!
|
||||
#
|
||||
# findargs emits ONE TOKEN PER LINE and MUST be consumed through a quoted
|
||||
# array. A flat string does not work: `find . $FEXCL ...` lets the shell glob
|
||||
# `*/dist/*` against the CWD before find ever sees it, and the matches are then
|
||||
# passed as search PATHS. Measured on zenquality: that turned 90 hits into 135
|
||||
# and kept every dist/ file. The array form passes each token literally.
|
||||
#
|
||||
# WHY: grep and find disagree about what is in the repo, and seo-analyzer uses
|
||||
# both.
|
||||
#
|
||||
# grep → Claude Code installs a shell function routing grep to ugrep with
|
||||
# `--ignore-files`, i.e. .gitignore-aware. A gitignored dist/ is
|
||||
# invisible to it when recursing from `.`. Verified 2026-07-17.
|
||||
# find → knows nothing about .gitignore. It sees everything.
|
||||
#
|
||||
# So on zenquality (Astro, dist/ gitignored, built locally) the spec's image
|
||||
# audit at seo-analyzer.md:497 returns 92 images of which 45 live in dist/ —
|
||||
# every asset listed twice, source and generated copy, identical bytes. Two
|
||||
# real consequences:
|
||||
# 1. "top 20 by size" is half generated duplicates: ~10 real images audited
|
||||
# while 20 are claimed.
|
||||
# 2. Batch C (`cwebp -q 80 <img> -o <img>.webp`) can target dist/og-image.png.
|
||||
# The .webp lands in dist/ and the `npm run build` that /seo runs to VERIFY
|
||||
# the fix erases it. The fix lands, verification passes, nothing survives.
|
||||
#
|
||||
# The grep side is already safe by accident — do NOT "fix" it to match find.
|
||||
# `grep` mode below is defence in depth for the cases the shim misses: a repo
|
||||
# that COMMITS its build output (no .gitignore entry to honour), or a directory
|
||||
# that is not a git repo at all.
|
||||
#
|
||||
# `public/` is deliberately NOT in the always-list: it is SOURCE for
|
||||
# Astro/Vite/Next and holds the very files this audit checks — favicon.ico,
|
||||
# apple-touch-icon.png, robots.txt, OG images. It is build OUTPUT only for
|
||||
# Hugo and Gatsby, detected below. Blanket-excluding it would blind the audit
|
||||
# to its own resource checks.
|
||||
#
|
||||
# Exclusions are by NAME, not path, so a monorepo's frontend/dist is caught
|
||||
# exactly like a root ./dist.
|
||||
set -uo pipefail
|
||||
|
||||
_die() { echo "source-scope: $1" >&2; exit 2; }
|
||||
|
||||
# Build output + tool caches. Never source.
|
||||
ALWAYS=(node_modules .git dist build .next .nuxt .output _site .astro
|
||||
.svelte-kit .cache out coverage .vercel .netlify .turbo)
|
||||
|
||||
# public/ is output for exactly these two generators.
|
||||
_public_is_output() {
|
||||
find . -maxdepth 3 \( -name "gatsby-config.js" -o -name "gatsby-config.ts" \
|
||||
-o -name "gatsby-config.mjs" -o -name "hugo.toml" -o -name "hugo.yaml" \
|
||||
-o -name "hugo.json" \) 2>/dev/null | read -r _ && return 0
|
||||
# Hugo's legacy config.toml is ambiguous on its own — pair it with archetypes/
|
||||
[ -d ./archetypes ] && [ -f ./config.toml ] && return 0
|
||||
return 1
|
||||
}
|
||||
|
||||
_list() {
|
||||
printf '%s\n' "${ALWAYS[@]}"
|
||||
_public_is_output && printf 'public\n'
|
||||
return 0
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
list) _list ;;
|
||||
# Safe unquoted: --exclude-dir=NAME carries no glob character.
|
||||
grep) _list | while read -r d; do printf -- '--exclude-dir=%s ' "$d"; done; echo ;;
|
||||
# One token per line — consume with mapfile + a QUOTED array, never a flat
|
||||
# string (see header: the shell would glob */dist/* against the CWD).
|
||||
findargs) _list | while read -r d; do printf '!\n-path\n*/%s/*\n' "$d"; done ;;
|
||||
*) _die "usage: source-scope.sh {list|grep|findargs}" ;;
|
||||
esac
|
||||
@@ -1,76 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# lib/tests/source-scope.test.sh
|
||||
set -u
|
||||
S="$(cd "$(dirname "$0")/../.." && pwd)/lib/source-scope.sh"
|
||||
pass=0; fail=0
|
||||
check() { if [ "$2" = "$3" ]; then pass=$((pass+1)); else fail=$((fail+1));
|
||||
printf 'FAIL %s: got[%s] want[%s]\n' "$1" "$2" "$3"; fi; }
|
||||
# does `list` (run inside dir $1) contain the name $2?
|
||||
listed() { ( cd "$1" && bash "$S" list 2>/dev/null | grep -qxF "$2" ) \
|
||||
&& echo yes || echo no; }
|
||||
|
||||
TMP="$(mktemp -d)"
|
||||
|
||||
# --- always-excluded build output + caches ---
|
||||
mkdir -p "$TMP/plain"
|
||||
for d in node_modules .git dist build .next .nuxt .output _site .astro \
|
||||
.svelte-kit .cache out coverage .vercel .netlify .turbo; do
|
||||
check "A-$d-listed" "$(listed "$TMP/plain" "$d")" yes
|
||||
done
|
||||
|
||||
# --- public/ is SOURCE by default: Astro/Vite/Next keep favicon.ico,
|
||||
# apple-touch-icon.png and robots.txt there, and the audit checks them ---
|
||||
check B1-public-kept-by-default "$(listed "$TMP/plain" public)" no
|
||||
|
||||
# --- public/ is OUTPUT for Gatsby and Hugo only ---
|
||||
mkdir -p "$TMP/gatsby"; : > "$TMP/gatsby/gatsby-config.js"
|
||||
check B2-gatsby-js "$(listed "$TMP/gatsby" public)" yes
|
||||
mkdir -p "$TMP/gatsby2"; : > "$TMP/gatsby2/gatsby-config.ts"
|
||||
check B3-gatsby-ts "$(listed "$TMP/gatsby2" public)" yes
|
||||
mkdir -p "$TMP/hugo"; : > "$TMP/hugo/hugo.toml"
|
||||
check B4-hugo-toml "$(listed "$TMP/hugo" public)" yes
|
||||
mkdir -p "$TMP/hugo2"; : > "$TMP/hugo2/hugo.yaml"
|
||||
check B5-hugo-yaml "$(listed "$TMP/hugo2" public)" yes
|
||||
# legacy config.toml alone is ambiguous (many tools use it) — needs archetypes/
|
||||
mkdir -p "$TMP/amb"; : > "$TMP/amb/config.toml"
|
||||
check B6-config-toml-alone-is-ambiguous "$(listed "$TMP/amb" public)" no
|
||||
mkdir -p "$TMP/hugo3/archetypes"; : > "$TMP/hugo3/config.toml"
|
||||
check B7-config-toml-plus-archetypes "$(listed "$TMP/hugo3" public)" yes
|
||||
|
||||
# --- grep mode: flags, and no glob character (safe unquoted) ---
|
||||
G="$(cd "$TMP/plain" && bash "$S" grep)"
|
||||
case "$G" in *--exclude-dir=dist*) check C1-grep-has-dist ok ok ;;
|
||||
*) check C1-grep-has-dist "missing" ok ;; esac
|
||||
case "$G" in *"*"*) check C2-grep-has-no-glob "has-glob" ok ;;
|
||||
*) check C2-grep-has-no-glob ok ok ;; esac
|
||||
|
||||
# --- findargs: one token per line, 3 tokens per dir ---
|
||||
N="$(cd "$TMP/plain" && bash "$S" findargs | wc -l)"
|
||||
D="$(cd "$TMP/plain" && bash "$S" list | wc -l)"
|
||||
check D1-findargs-3-tokens-per-dir "$N" "$((D * 3))"
|
||||
check D2-findargs-first-token "$(cd "$TMP/plain" && bash "$S" findargs | head -1)" '!'
|
||||
|
||||
# --- FUNCTIONAL: the array form actually excludes build output ---
|
||||
# A flat unquoted string does NOT work here: the shell globs */dist/* against
|
||||
# the CWD and passes the matches to find as search paths. Measured on a real
|
||||
# repo, that turned 90 hits into 135 and kept every dist/ file.
|
||||
W="$TMP/work"; mkdir -p "$W/src" "$W/dist" "$W/public" "$W/node_modules"
|
||||
: > "$W/src/a.png"; : > "$W/dist/a.png"; : > "$W/public/favicon.ico"
|
||||
: > "$W/node_modules/dep.png"
|
||||
cd "$W" || exit 1
|
||||
mapfile -t FEXCL < <(bash "$S" findargs)
|
||||
check E1-excludes-dist "$(find . "${FEXCL[@]}" -name 'a.png' | grep -c '/dist/')" 0
|
||||
check E2-keeps-src "$(find . "${FEXCL[@]}" -name 'a.png' | grep -c '/src/')" 1
|
||||
check E3-excludes-nodem "$(find . "${FEXCL[@]}" -name '*.png' | grep -c 'node_modules')" 0
|
||||
# public/ survives: the audit's own resource checks live there
|
||||
check E4-keeps-public "$(find . "${FEXCL[@]}" -name 'favicon.ico' | wc -l)" 1
|
||||
cd / || exit 1
|
||||
|
||||
# --- usage ---
|
||||
bash "$S" >/dev/null 2>&1; check X1-no-args "$?" 2
|
||||
bash "$S" bogus >/dev/null 2>&1; check X2-bad-verb "$?" 2
|
||||
# `find` was renamed to `findargs` when the flat-string form proved unsafe
|
||||
bash "$S" find >/dev/null 2>&1; check X3-old-find-verb-gone "$?" 2
|
||||
|
||||
rm -rf "$TMP"
|
||||
printf 'PASS=%s FAIL=%s\n' "$pass" "$fail"; [ "$fail" -eq 0 ]
|
||||
@@ -1,74 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# lib/tests/url-guard.test.sh
|
||||
set -u
|
||||
G="$(cd "$(dirname "$0")/../.." && pwd)/lib/url-guard.sh"
|
||||
pass=0; fail=0
|
||||
check() { if [ "$2" = "$3" ]; then pass=$((pass+1)); else fail=$((fail+1));
|
||||
printf 'FAIL %s: got[%s] want[%s]\n' "$1" "$2" "$3"; fi; }
|
||||
# rc of a guard call, output discarded
|
||||
rc() { bash "$G" "$1" "$2" >/dev/null 2>&1; return $?; }
|
||||
# stdout of a guard call (empty on refusal)
|
||||
out() { bash "$G" "$1" "$2" 2>/dev/null; }
|
||||
|
||||
# --- hosts that must pass, echoing back unchanged ---
|
||||
rc host "example.com"; check H1-plain "$?" 0
|
||||
rc host "www.sub.example.co.uk"; check H2-subdomains "$?" 0
|
||||
rc host "my-site.fr"; check H3-hyphen "$?" 0
|
||||
check H4-echoes-input "$(out host example.com)" "example.com"
|
||||
|
||||
# --- shell metacharacters: the reason this guard exists ---
|
||||
# Inside the double quotes seo-analyzer.md:257 uses, $ ` \ " break out.
|
||||
rc host 'x$(id)'; check H5-cmdsubst "$?" 2
|
||||
rc host 'x`id`'; check H6-backtick "$?" 2
|
||||
rc host 'x;id'; check H7-semicolon "$?" 2
|
||||
rc host 'x|id'; check H8-pipe "$?" 2
|
||||
rc host 'x&id'; check H9-ampersand "$?" 2
|
||||
rc host 'x"'; check H10-dquote "$?" 2
|
||||
rc host "x'"; check H11-squote "$?" 2
|
||||
rc host 'x\y'; check H12-backslash "$?" 2
|
||||
rc host 'x y'; check H13-space "$?" 2
|
||||
rc host 'a
|
||||
b'; check H14-newline "$?" 2
|
||||
# the real payload: read the OAuth vault into a request
|
||||
rc host 'x$(cat ${HOME}/.claude/.env)'; check H15-env-exfil "$?" 2
|
||||
check H16-refusal-is-silent "$(out host 'x$(id)')" ""
|
||||
|
||||
# --- literal local / private / metadata targets ---
|
||||
rc host "localhost"; check L1-localhost "$?" 2
|
||||
rc host "LOCALHOST"; check L2-case-folded "$?" 2
|
||||
rc host "127.0.0.1"; check L3-loopback "$?" 2
|
||||
rc host "10.1.2.3"; check L4-private-10 "$?" 2
|
||||
rc host "192.168.1.1"; check L5-private-192 "$?" 2
|
||||
rc host "172.16.0.1"; check L6-private-172-lo "$?" 2
|
||||
rc host "172.31.255.254"; check L7-private-172-hi "$?" 2
|
||||
rc host "172.32.0.1"; check L8-172-32-is-public "$?" 0
|
||||
rc host "169.254.169.254"; check L9-link-local "$?" 2
|
||||
rc host "metadata.google.internal"; check L10-gcp-metadata "$?" 2
|
||||
rc host "0.0.0.0"; check L11-any-addr "$?" 2
|
||||
rc host "printer.local"; check L12-mdns "$?" 2
|
||||
|
||||
# --- urls ---
|
||||
rc url "https://example.com/"; check U1-https "$?" 0
|
||||
rc url "http://example.com/a/b?x=1&y=2"; check U2-query "$?" 0
|
||||
rc url "https://example.com:8443/p"; check U3-port "$?" 0
|
||||
rc url "https://example.com/a%20b#frag"; check U4-pct-and-frag "$?" 0
|
||||
check U5-echoes-input "$(out url https://example.com/x)" "https://example.com/x"
|
||||
rc url "ftp://example.com/"; check U6-ftp "$?" 2
|
||||
rc url "file:///etc/passwd"; check U7-file "$?" 2
|
||||
rc url "gopher://example.com/"; check U8-gopher "$?" 2
|
||||
rc url "example.com"; check U9-no-scheme "$?" 2
|
||||
rc url 'https://example.com/$(id)'; check U10-cmdsubst "$?" 2
|
||||
rc url 'https://example.com/`id`'; check U11-backtick "$?" 2
|
||||
rc url "https://localhost/x"; check U12-local "$?" 2
|
||||
rc url "https://127.0.0.1:8080/admin"; check U13-loopback "$?" 2
|
||||
# authority confusion: the real host is after the @, not before it
|
||||
rc url "https://trusted.com@127.0.0.1/"; check U14-userinfo-local "$?" 2
|
||||
rc url "https://trusted.com@evil.com/"; check U15-userinfo-any "$?" 2
|
||||
|
||||
# --- usage ---
|
||||
rc host ""; check X1-host-empty "$?" 2
|
||||
bash "$G" >/dev/null 2>&1; check X2-no-args "$?" 2
|
||||
bash "$G" bogus x >/dev/null 2>&1; check X3-bad-verb "$?" 2
|
||||
bash "$G" host a b >/dev/null 2>&1; check X4-extra-args "$?" 2
|
||||
|
||||
printf 'PASS=%s FAIL=%s\n' "$pass" "$fail"; [ "$fail" -eq 0 ]
|
||||
@@ -1,83 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Validate a host or URL BEFORE it reaches a shell command or curl.
|
||||
# Echoes the value on stdout when safe; exits 2 with a reason on stderr.
|
||||
#
|
||||
# HOST="$(bash ~/.claude/lib/url-guard.sh host "$RAW")" || exit 2
|
||||
# URL="$(bash ~/.claude/lib/url-guard.sh url "$RAW")" || exit 2
|
||||
#
|
||||
# WHY: /seo and /geo interpolate externally-supplied strings into ~10 curl
|
||||
# commands (seo-analyzer.md:254+, geo-analyzer.md:248+). Today $DOMAIN is typed
|
||||
# by the operator, so the risk is self-inflicted. The sitemap crawl (C1) changes
|
||||
# that: URLs then come from the TARGET'S OWN SERVER — a remote file whose bytes
|
||||
# reach a shell. Inside the double quotes those curls use, the characters that
|
||||
# break out are $ ` \ " — so a <loc> of
|
||||
# https://x/$(cat ${HOME}/.claude/.env)
|
||||
# would read GOOGLE_OAUTH_CLIENT_SECRET and CRUX_API_KEY straight out of the
|
||||
# vault and into a request. Allowlist, per CLAUDE.md: explicit allowlist beats
|
||||
# implicit denylist.
|
||||
#
|
||||
# DNS-level SSRF (a public hostname that RESOLVES to a private address, or
|
||||
# rebinds between check and connect): this NAME-level guard does not catch it —
|
||||
# closing it needs resolve-then-pin at the HTTP layer. That is now DONE for the
|
||||
# Python egress: lib/seo-data/safe_fetch.py pins every fetch (sitemap, linkgraph,
|
||||
# rendercheck, drift). It is NOT done for shell `curl`, which cannot pin without
|
||||
# `curl --resolve`; those paths keep this literal-local check only. Stated, not
|
||||
# silent — see lib/seo-data/README.md (safe_fetch).
|
||||
set -uo pipefail
|
||||
|
||||
_die() { echo "url-guard: $1" >&2; exit 2; }
|
||||
|
||||
# Whole-string charset guards: C locale + POSIX `case`, the same shape as
|
||||
# fetch.sh:25 _label_safe. Newline-proof and locale-independent, unlike a
|
||||
# per-line grep. No `$` or backtick inside the patterns, so nothing expands.
|
||||
_host_charset_ok() ( LC_ALL=C; case "$1" in
|
||||
''|[!A-Za-z0-9]*|*[!A-Za-z0-9.-]*) exit 1 ;; esac )
|
||||
|
||||
# Authority + path + query. Excludes $ ` \ " ' ; | ( ) * ! space and newline —
|
||||
# none of which a real sitemap URL needs, all of which a shell reads.
|
||||
_rest_charset_ok() ( LC_ALL=C; case "$1" in
|
||||
''|*[!A-Za-z0-9._~:/?#@=\&%+,-]*) exit 1 ;; esac )
|
||||
|
||||
# Literal local/private/metadata targets. This is a LITERAL check, not a DNS
|
||||
# one: it stops the obvious, not a hostname that resolves inward.
|
||||
_host_is_local() ( LC_ALL=C
|
||||
# ${1,,} not tr: no fork, and no SC2018/SC2019 noise. Safe because the
|
||||
# charset guard has already run — the string is [A-Za-z0-9.-] by here.
|
||||
case "${1,,}" in
|
||||
localhost|*.localhost|*.local|0.0.0.0|broadcasthost) exit 0 ;;
|
||||
127.*|10.*|169.254.*|192.168.*) exit 0 ;;
|
||||
172.1[6-9].*|172.2[0-9].*|172.3[01].*) exit 0 ;;
|
||||
metadata.google.internal|metadata) exit 0 ;;
|
||||
*) exit 1 ;;
|
||||
esac )
|
||||
|
||||
_reject_local() { _host_is_local "$1" && _die "local/private target refused: '$1'"; return 0; }
|
||||
|
||||
check_host() {
|
||||
_host_charset_ok "$1" || _die "host charset (allowed A-Za-z0-9.-): '$1'"
|
||||
_reject_local "$1"
|
||||
printf '%s\n' "$1"
|
||||
}
|
||||
|
||||
check_url() {
|
||||
local rest host
|
||||
case "$1" in
|
||||
https://*) rest="${1#https://}" ;;
|
||||
http://*) rest="${1#http://}" ;;
|
||||
*) _die "scheme must be http or https: '$1'" ;;
|
||||
esac
|
||||
_rest_charset_ok "$rest" || _die "url charset: '$1'"
|
||||
host="${rest%%/*}"; host="${host%%\?*}"; host="${host%%#*}"
|
||||
# user@host hides the real target: https://trusted.com@127.0.0.1/ hits .0.0.1
|
||||
case "$host" in *@*) _die "userinfo in authority (confusion vector): '$1'" ;; esac
|
||||
host="${host%%:*}" # drop :port before validating the host
|
||||
_host_charset_ok "$host" || _die "host charset: '$host'"
|
||||
_reject_local "$host"
|
||||
printf '%s\n' "$1"
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
host) [ $# -eq 2 ] || _die "usage: url-guard.sh host <hostname>"; check_host "$2" ;;
|
||||
url) [ $# -eq 2 ] || _die "usage: url-guard.sh url <url>"; check_url "$2" ;;
|
||||
*) _die "usage: url-guard.sh {host|url} <value>" ;;
|
||||
esac
|
||||
+6
-20
@@ -134,25 +134,11 @@
|
||||
"Read(**/credentials.json)",
|
||||
"Read(**/.aws/credentials)",
|
||||
"Read(**/.azure/**)",
|
||||
"Edit(**/.env)",
|
||||
"Edit(**/.env.*)",
|
||||
"Edit(**/secrets/**)",
|
||||
"Edit(**/*.pem)",
|
||||
"Edit(**/*.key)",
|
||||
"Edit(**/*.p12)",
|
||||
"Edit(**/*.pfx)",
|
||||
"Edit(**/id_rsa*)",
|
||||
"Edit(**/id_ed25519*)",
|
||||
"Edit(**/.ssh/**)",
|
||||
"Edit(**/credentials)",
|
||||
"Edit(**/credentials.json)",
|
||||
"Edit(**/.aws/credentials)",
|
||||
"Edit(**/.azure/**)",
|
||||
"Edit(**/*.lock)",
|
||||
"Edit(**/package-lock.json)",
|
||||
"Edit(**/pnpm-lock.yaml)",
|
||||
"Edit(**/go.sum)",
|
||||
"Edit(**/node_modules/**)",
|
||||
"Write(**/.env)",
|
||||
"Write(**/.env.*)",
|
||||
"Write(**/secrets/**)",
|
||||
"Write(**/*.pem)",
|
||||
"Write(**/*.key)",
|
||||
"Bash(eval *)",
|
||||
"Bash(exec *)",
|
||||
"Bash(find * -delete*)",
|
||||
@@ -250,7 +236,7 @@
|
||||
"disableBypassPermissionsMode": "disable",
|
||||
"additionalDirectories": []
|
||||
},
|
||||
"model": "opus[1m]",
|
||||
"model": "claude-fable-5[1m]",
|
||||
"hooks": {
|
||||
"SessionStart": [
|
||||
{
|
||||
|
||||
@@ -1 +1 @@
|
||||
0.9.15
|
||||
0.9.6
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: graphify
|
||||
description: "Use for any question about a codebase, its architecture, file relationships, or project content — especially when graphify-out/ exists, where the question should be treated as a graphify query first. Turns any input (code, docs, papers, images, videos) into a persistent knowledge graph with god nodes, community detection, and query/path/explain tools."
|
||||
description: "Use when graphify-out/ exists (or the user asks to build a knowledge graph): questions about the codebase, its architecture, file relationships, or project content are then treated as graphify queries first. Turns any input (code, docs, papers, images, videos) into a persistent knowledge graph with god nodes, community detection, and query/path/explain tools."
|
||||
---
|
||||
|
||||
# /graphify
|
||||
@@ -10,7 +10,7 @@ Turn any folder of files into a navigable knowledge graph with community detecti
|
||||
## Usage
|
||||
|
||||
```
|
||||
/graphify # full pipeline on current directory (HTML viz; add --obsidian for a vault)
|
||||
/graphify # full pipeline on current directory → Obsidian vault
|
||||
/graphify <path> # full pipeline on specific path
|
||||
/graphify https://github.com/<owner>/<repo> # clone repo then run full pipeline on it
|
||||
/graphify https://github.com/<owner>/<repo> --branch <branch> # clone a specific branch
|
||||
@@ -70,7 +70,7 @@ PYTHON=""
|
||||
GRAPHIFY_BIN=$(which graphify 2>/dev/null)
|
||||
# 1. uv tool installs — most reliable on modern Mac/Linux
|
||||
if [ -z "$PYTHON" ] && command -v uv >/dev/null 2>&1; then
|
||||
_UV_PY=$(uv tool run --from graphifyy python -c "import sys; print(sys.executable)" 2>/dev/null)
|
||||
_UV_PY=$(uv tool run graphifyy python -c "import sys; print(sys.executable)" 2>/dev/null)
|
||||
if [ -n "$_UV_PY" ]; then PYTHON="$_UV_PY"; fi
|
||||
fi
|
||||
# 2. Read shebang from graphify binary (pipx and direct pip installs)
|
||||
@@ -86,7 +86,7 @@ if [ -z "$PYTHON" ]; then PYTHON="python3"; fi
|
||||
if ! "$PYTHON" -c "import graphify" 2>/dev/null; then
|
||||
if command -v uv >/dev/null 2>&1; then
|
||||
uv tool install --upgrade graphifyy -q 2>&1 | tail -3
|
||||
_UV_PY=$(uv tool run --from graphifyy python -c "import sys; print(sys.executable)" 2>/dev/null)
|
||||
_UV_PY=$(uv tool run graphifyy python -c "import sys; print(sys.executable)" 2>/dev/null)
|
||||
if [ -n "$_UV_PY" ]; then PYTHON="$_UV_PY"; fi
|
||||
else
|
||||
"$PYTHON" -m pip install graphifyy -q 2>/dev/null \
|
||||
@@ -313,8 +313,7 @@ from graphify.cache import save_semantic_cache
|
||||
from pathlib import Path
|
||||
|
||||
new = json.loads(Path('graphify-out/.graphify_semantic_new.json').read_text(encoding=\"utf-8\")) if Path('graphify-out/.graphify_semantic_new.json').exists() else {'nodes':[],'edges':[],'hyperedges':[]}
|
||||
uncached = [line for line in Path('graphify-out/.graphify_uncached.txt').read_text(encoding=\"utf-8\").splitlines() if line]
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH', allowed_source_files=uncached)
|
||||
saved = save_semantic_cache(new.get('nodes', []), new.get('edges', []), new.get('hyperedges', []), root='INPUT_PATH')
|
||||
print(f'Cached {saved} files')
|
||||
"
|
||||
```
|
||||
|
||||
+3
-12
@@ -204,9 +204,9 @@ Ask ONCE before dispatching the agents:
|
||||
```
|
||||
RAPPORT EXTERNE (optionnel) — un autre regard sur le site :
|
||||
|
||||
1. Fichier — donnez le chemin de l'export (PDF/MD/TXT), où qu'il soit
|
||||
(ex. `~/Téléchargements/sorank-2026-07-16.pdf`). Rangement conseillé
|
||||
mais optionnel : `.claude/audits/external/`.
|
||||
1. Fichier — déposez l'export (PDF/MD/TXT) dans
|
||||
`.claude/audits/external/` (ex. `sorank-YYYY-MM-DD.pdf`),
|
||||
donnez le nom du fichier. (`mkdir -p .claude/audits/external`)
|
||||
2. Collé — collez ici le contenu du PDF ou le "prompt pour IA"
|
||||
que l'outil suggère.
|
||||
3. Ignorer — continuer sans. Le rapport final recommandera
|
||||
@@ -348,15 +348,6 @@ audit GEO/AI signals (llms.txt, AI crawlers, QAPage/Speakable schemas,
|
||||
entity SEO, content shape for AI, AI visibility) — the geo-analyzer
|
||||
agent runs in parallel and owns those.
|
||||
|
||||
Do NOT score security headers either (CSP, HSTS, X-Frame-Options,
|
||||
X-Content-Type-Options, Referrer-Policy, Permissions-Policy, COOP/CORP,
|
||||
cookie flags) — `/harden` owns them and grades them 0-100 against three
|
||||
external validators (`depth-matrix.md:29`). Read them, keep
|
||||
`X-Robots-Tag` under indexability (it is an indexing directive, not a
|
||||
security header), and declare the rest in §14 with a "run /harden" pointer
|
||||
plus what you observed live. Dropping them from the score must not make
|
||||
them silent.
|
||||
|
||||
FILE OWNERSHIP (authoritative, prevents parallel-edit conflicts):
|
||||
- YOU OWN (read+write): sitemap.xml, image/video sitemaps, .htaccess,
|
||||
meta tags (title, description, OG, Twitter, canonical, robots meta),
|
||||
|
||||
@@ -10,17 +10,13 @@
|
||||
"Bash(curl * | bash)" // pipe pattern — block code injection
|
||||
```
|
||||
|
||||
### Read / Edit — gitignore syntax
|
||||
### Read / Write / Edit — gitignore syntax
|
||||
```json
|
||||
"Read(**/.env)" // any .env in any subdirectory
|
||||
"Read(**/secrets/**)" // anything inside secrets/
|
||||
"Read(src/**/*.ts)" // all .ts under src/
|
||||
"Edit(**/*.key)" // deny writing any .key file — Edit covers
|
||||
// Write/Edit/MultiEdit/NotebookEdit
|
||||
"Write(**/*.key)" // deny writing any .key file
|
||||
```
|
||||
`Write(path)` rules are **inert**: file permission checks only match
|
||||
`Edit(path)`. Claude Code warns at startup for every `Write(glob)` rule.
|
||||
Always write the file-write ban as `Edit(...)`.
|
||||
|
||||
### WebFetch / WebSearch
|
||||
```json
|
||||
|
||||
Reference in New Issue
Block a user