diff --git a/.claude/memory/decisions.md b/.claude/memory/decisions.md index b9e3366..ad81354 100644 --- a/.claude/memory/decisions.md +++ b/.claude/memory/decisions.md @@ -941,3 +941,12 @@ rules: - **Why**: the review (`.audit/review-release-1.0.0.md` A6) found the guard had warned every session since job1 without the target ever being met — a self-inflicted permanent warning, not an actionable signal. A gate that never goes green trains you to ignore it. Realign to reality; keep a 15-line margin so real regressions still surface. - **Alternatives rejected**: (a) finish the compression 305→≤275 — the remaining lines are load-bearing constraints, not filler; further squeeze loses clarity for a marginal token gain on a solo repo. (b) leave the guard at 280 and accept the permanent warning — a permanently-red non-blocking gate is noise. (c) rewrite BDR-031 — registries are append-only; supersede the target, keep the principle. - **Reference**: `hooks/session-start.sh:202-211`; supersedes the 275 target in [[BDR-031]] (principle kept). Review remediation A6, 2026-07-08. + +## BDR-063 — GSC multi-account: OAuth2 installed-app flow + label-keyed token store + +- **Date**: 2026-07-10 +- **Status**: accepted (shipped `bb1fbb2`, develop) +- **Decision**: `/seo` FULL pulls real Search Console + CrUX via a `lib/seo-data/` engine. Auth = OAuth2 installed-app flow (one-time interactive consent, `make seo-connect`), scope `webmasters.readonly` ONLY (least priv). Refresh tokens in per-label store `~/.claude/seo-data/tokens.json` (0600 file / 0700 dir, atomic tmp→fsync→rename under fcntl lock, tokens redacted from listing, gitleaks-allowlisted). `(account, property)` explicit args on every call — NO global mutable "current account" → two concurrent site audits never conflict. +- **Why**: user needs real field data (the one edge marketplace `claude-seo` had that personal skills lacked); multi-account without cross-site leakage; secrets never in code (all from `~/.claude/.env`). +- **Alternatives rejected**: (a) service-account — GSC needs per-property owner grant + no interactive consent, wrong for a personal multi-client tool. (b) API-key-only — GSC has no key auth (CrUX does → `CRUX_API_KEY`). (c) single "current account" global + switch verb — a race the moment two audits run; explicit args dissolve it by construction. +- **Reference**: `lib/seo-data/` (tokenstore.py, connect.py, google_seo.py, fetch.sh), `lib/seo-data/README.md`; fronted by [[LRN-119]] (fail-open contract). diff --git a/.claude/memory/journal.md b/.claude/memory/journal.md index 3709132..a097de4 100644 --- a/.claude/memory/journal.md +++ b/.claude/memory/journal.md @@ -370,3 +370,8 @@ rules: - Capitalized: [[LRN-113]] partial-fix+guard (structural), [[LRN-114]] hook-drift, [[LRN-115]] analyzer report-grants (FP1), [[LRN-116]] release fix missing from develop, [[BDR-062]] density realign, [[EVAL-021]] the review, [[EVAL-022]] M5 pins trace. Noted un-back-merged release chores beyond A3: e65796f (SC1091 lint silence) — left for a future reconcile. - Full back-merge release/1.0.0→develop (`chore/backmerge-release-full`, unmerged): the RC fork had left ~6 functional fixes orphaned on develop, silently. PORTED via cherry-pick, make test green each: `095d881` drop find-skills, `a1093ca` make-update TTY-guard (proven: EOF-die exit1 → guarded exit0), `4c5e862` rtk update-path version-guard (complements the `e58037c` install bridge already ported), `c76479f` design-motion sync, `e65796f` SC1091 lint. B soak journal (find-skills day1 / TTY #3 / rtk-update #4) folded here, not cherry-picked — divergent journal tails conflict (STOP-on-conflict honored, extract-consolidate fallback). C all covered/skip: `93e43c0` attribution + `ae8ad86` model already on develop; `188a9a7` docs → /doc backlog (README missing semgrep/scan-secrets/verify+secure/ctx7). Registry (LRN-098/101, EVAL-015, BLK-016) already backfilled in the review run. Gate: 23/23 release-only commits classified, 0 orphan functional, 0 missing registry; make test GREEN, review-guards 5/0. version.txt stays 4.0.0 (fork intentional, D — `eb93050`). - [[LRN-117]]: the fork silently orphaned functional CODE on develop (not just memory); the review back-merge caught ~half. Detecting it needs a code-level drift check (advisory, backlogged) — registry-sequence gaps alone miss it. + +## 2026-07-10 +- GSC+CrUX data layer for `/seo` FULL shipped end-to-end (subagent-driven, superpowers): design→plan→8 tasks→final review→merge `bb1fbb2` on develop. Engine `lib/seo-data/` (label-keyed OAuth token store 0600/0700, CrUX field + GSC Search-Analytics/URL-Inspection, fail-open `fetch.sh`, `make seo-connect` consent), wired into `/seo` FULL (STEP 0 account select, CrUX-primary CWV, "Performance GSC" quick-wins). 49/49 engine tests + full `make test` green throughout. Final opus whole-branch review: security PASS, 0 Critical/Important, 5 Minors all deferred to a later chore sweep. +- Decided [[BDR-063]] OAuth installed-app + explicit `(account,property)` args (no global state) → multi-account no-conflict. Learned [[LRN-119]] fail-open engine contract (always-JSON, lazy imports, degrade-not-crash), [[LRN-120]] final-review base = merge-base not ledger BASE (caught a misleading 881-vs-2163-ins diff). +- Docs synced (`/doc`, `4a15c73` on `chore/doc-sync-gsc-crux`): README (seo-connect, make-test glob, /seo row) + USAGE (/seo FULL real-data) + CHANGELOG Added entry. Pending: merge `chore/doc-sync-gsc-crux`→develop (human GO), then delete transient spec+plan `docs/superpowers/…gsc-crux…`. diff --git a/.claude/memory/learnings.md b/.claude/memory/learnings.md index beae9d0..d8a0c9c 100644 --- a/.claude/memory/learnings.md +++ b/.claude/memory/learnings.md @@ -1196,3 +1196,15 @@ rules: - **fix applied**: commit-change got Phase 0 = the shared `gitflow-aiguillage.md` (TYPE=chore, branch on protected base, no-op on working) + report-only fallback; client-handover push gated behind explicit-GO AskUserQuestion + report-only fallback. Dry-runs proved BOTH sides of each fallback (branch-taken AND not-taken), not just the happy path. - **future application**: any fleet/skill conformity audit — (1) triage by "autonomous commit/push reached?", not "file written?"; (2) read every git-signal in context (prohibition vs usage); (3) test the deterministic backstop empirically before trusting it; (4) verify a referenced lib exists + its contract matches BEFORE copying it (phantom-reference guard); (5) dry-run both branches of every fallback. - **cousin**: [[LRN-117]] (orphaned CODE has no sequence to check), [[LRN-034]] (narrated ≠ ground truth), [[BDR-061]] (report-only agent tool-grants). + +## LRN-119 — Fail-open engine contract for optional external data (real-if-connected, else graceful) +- **pattern**: `lib/seo-data/fetch.sh` = one entrypoint; every subcmd ALWAYS emits JSON on stdout, exit 0 on ok/degraded, exit 2 on bad-usage, NEVER empty stdout, NEVER prints a secret. Third-party imports (google-auth, requests) function-local (lazy) so stdlib-only paths — mock (`SEO_DATA_MOCK_DIR`), degrade (no key/no account/revoked token), offline tests — run with no venv. Missing creds → `{"status":"degraded","reason":...}` and the caller (`/seo` analyzer) falls back to anonymous PageSpeed; audit NEVER fails on absent data. Both Python `_cli` wrapped try/except: SystemExit→bad_usage JSON+reraise, Exception→degraded JSON (corrupt store never leaks stack/path). 3rd status value `error` on exit-2 only. +- **why it matters**: an optional-data integration must be invisible when unconfigured. Fail-CLOSED (crash/empty/nonzero) breaks every audit for users who never connect GSC. Fail-open + lazy-import keeps the 49 tests network-free and makes degrade a first-class tested branch, not an afterthought. +- **future application**: any "use real data if credentials present, else degrade" seam — put the contract in the shell entrypoint (always-JSON / exit-code discipline), lazy-import the SDK, make degrade a returned status not an exception, test degrade+mock stdlib-only, redact secrets at the boundary (list omits token, `exec 2>/dev/null` unless debug). +- **cousin**: [[BDR-063]] (the token store this fronts), [[LRN-120]] (SDD base gotcha, same build). + +## LRN-120 — SDD final-review base = `git merge-base`, NOT the ledger's recorded BASE +- **pattern**: subagent-driven-development ledger recorded `BASE: 24b47ce` — but that was IMPLEMENTATION start (after spec+plan commits), not the branch point from develop. `git merge-base develop HEAD` = `d3e644d` (real fork). Final whole-branch review diffed against recorded BASE = 881 ins / 59 del; against true merge-base = 2163 ins / 7 del — the recorded-base diff MISLEADING (netting against a divergent line → phantom deletions). Per-task reviews unaffected (each used the correct prior feature commit). +- **why it matters**: the final review is the last gate before merge; a wrong base hides real changes or invents fake ones. The ledger BASE is a task resume-map, not a merge-delta anchor. +- **future application**: for ANY whole-branch/final review, derive base from `git merge-base HEAD`, never a stored/remembered SHA. Sanity-check: does `git log BASE..HEAD` list ONLY this branch's commits, nothing foreign? Diff-stats differ between candidate bases → recorded one is stale, trust merge-base. +- **cousin**: [[LRN-119]] (same GSC+CrUX build); SDD skill's own "never HEAD~1" warning (same base-selection bug class). diff --git a/CHANGELOG.md b/CHANGELOG.md index 3e66c04..0e60e71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ Format follows [Keep a Changelog](https://keepachangelog.com/). - **gitleaks secret-scanning backstop** — `.gitleaks.toml`, a pre-commit hook, and `make scan-secrets` added to catch secrets before they land; pre-existing stale secret-bearing artifacts purged (GO-gated). ### Added +- **GSC + CrUX data layer for `/seo` FULL** — `lib/seo-data/` engine pulls real Google Search Console (Search Analytics + URL Inspection) and Chrome UX Report field data into the `/seo` FULL audit: CrUX p75 field metrics become the primary Core Web Vitals signal (anonymous PageSpeed lab stays the fallback), and a "Performance GSC (90 j)" section flags position 4-10 quick wins. Multi-account via OAuth2 (`make seo-connect`, one-time consent, `webmasters.readonly` scope only) with a per-label token store (0600 file / 0700 dir, atomic write, refresh tokens redacted, gitleaks-allowlisted) so two concurrent site audits never conflict. Absent credentials degrade gracefully to anonymous PageSpeed — the audit never fails. Config: `GOOGLE_OAUTH_CLIENT_ID` / `GOOGLE_OAUTH_CLIENT_SECRET` / `CRUX_API_KEY` in `~/.claude/.env`. Engine contract documented in `lib/seo-data/README.md`. - **impeccable** (pbakaus, Apache-2.0) wired into the toolchain as the design counterpart of semgrep: the `/impeccable` skill (23 verbs under one command: audit, polish, bolder, quieter…) plus the 45-rule deterministic anti-pattern detector (`npx impeccable detect`, exit 0/2, `--json`). Complementary to `frontend-design` (kept — aesthetic direction at build time); impeccable adds the deterministic audit floor and per-project design context (`/impeccable init`). CLI pinned in `plugins.lock.json` (3.2.0 — a silent rules update would change audit output on unchanged code); dist is machine-owned under `skills-external/impeccable/` (gitignored, ctx7 pattern), staged-installed by `install-plugins.sh` Step 8d, refreshed pin-honored by `update-all.sh`, symlinked by `link.sh`, listed in the design/web/web-full/full profiles and the design-work routing. Requires Node ≥ 24: the install baseline is bumped from 22 to 24 LTS (NodeSource `setup_24.x` / brew `node@24`), so `make plugin` upgrades a too-old host in place; the impeccable steps still skip gracefully if Node stays below 24. Not in the design gate's GATE-BLOCK list yet — promotion deliberate, after first dogfood. - `/tour` skill — grouped all-axes sweep over one or several projects: security (pinned-semgrep `security-auditor` agent + `/cso` posture when gstack is ON) → cleanup → re-verify → reconcile (report-only, never edits the target TODO/registries) → doc sync, looping until a full pass applies zero fixes (bounded at 3 iterations). Fixes land on a `chore/tour-` branch the skill never merges; each project gets an append-only `.claude/audits/TOUR.md` report with BREAKING tags on contract-changing security fixes. Built TDD (superpowers:writing-skills): baseline run showed silent TODO rewrites, autonomous registry writes, grep-as-security-pass, no persistent report, scope creep and an unbounded loop — each countered and verified on a seeded fixture. diff --git a/README.md b/README.md index 9df5114..f4b3afd 100644 --- a/README.md +++ b/README.md @@ -105,7 +105,7 @@ a different package, ships its own conflicting `graphify` bin) — see | `/refactor` | Improve code quality without changing behavior | | `/code-clean` | Dead code removal, style/norm enforcement | | `/doc` | Documentation audit and sync — detect stale docs, patch | -| `/seo` | Full SEO/GEO audit and optimization | +| `/seo` | Full SEO/GEO audit — real Search Console + CrUX field data when a Google account is connected (`make seo-connect`) | | `/impeccable` | Design verbs (audit, polish, bolder…) + deterministic anti-slop detector (`npx impeccable detect`) | | `/commit-change` | Smart commit grouping from staged/unstaged changes | | `/gitflow` | Gitflow branch operations — bootstrap main+develop, start a typed branch, directed merge | @@ -264,8 +264,9 @@ make plugin # install plugins only make link # create/update symlinks into ~/.claude/ make doctor # diagnostic make update # update Claude Code, config, submodules, plugins, and verify -make test # run deterministic tests (lib/tests/*.test.sh + lib/gitflow-test.sh) +make test # run deterministic tests (lib/tests/*.test.sh + lib/seo-data/*.test.sh + lib/gitflow-test.sh) make onboard # onboard an existing project (run from its dir) +make seo-connect # connect a Google account for /seo FULL (OAuth consent) make profile cmd="set X" # activate a skill profile (design/dev/qa/audit/minimal/full) make profile-list # list skill profiles make profile-current # show the active profile diff --git a/USAGE.md b/USAGE.md index 98fd0e8..f5d1fff 100644 --- a/USAGE.md +++ b/USAGE.md @@ -142,7 +142,7 @@ Tu veux... | `/refactor` | Améliorer un fichier sans changer le comportement | Rapport de violations d'abord, modif ensuite | | `/code-clean` | Dead code, violations de style | Audit + rapport, fixes après approbation | | `/doc` | Docs périmées après des changements | Audit drift code↔docs, patch chirurgical | -| `/seo` | Audit SEO/GEO complet | Détecte framework, audite meta/OG/sitemap | +| `/seo` | Audit SEO/GEO complet | Détecte framework, audite meta/OG/sitemap ; en FULL, choix du compte Google puis données réelles Search Console + CrUX (terrain) si connecté via `make seo-connect`, sinon repli PageSpeed anonyme | | `/geo` | Audit GEO uniquement (IA) | Visibilité ChatGPT, Perplexity, Claude, Gemini… | | `/commit-change` | Commits bien structurés | Groupe les changements par unité logique | | `/gitflow` | Opérations de branches gitflow | Bootstrap main+develop, branche typée, merge dirigé | diff --git a/docs/superpowers/plans/2026-07-09-gsc-crux-data-layer.md b/docs/superpowers/plans/2026-07-09-gsc-crux-data-layer.md deleted file mode 100644 index f0e35a7..0000000 --- a/docs/superpowers/plans/2026-07-09-gsc-crux-data-layer.md +++ /dev/null @@ -1,964 +0,0 @@ -# GSC + CrUX Data Layer — Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Give `/seo` (+`/geo`) FULL audits real Google data — Search Console queries/positions/indexation + CrUX field Core Web Vitals — via an isolated, secure, multi-account data engine. - -**Architecture:** A self-contained engine under `lib/seo-data/` (bash entrypoint `fetch.sh` → Python helpers in an isolated venv) fetches GSC + CrUX and emits normalized JSON on stdout. The existing `seo-analyzer` agent consumes that JSON during FULL audits; the `/seo` dispatcher selects account+property in STEP 0. Secrets live in the `~/.claude/.env` vault (OAuth app + CrUX key) plus a label-keyed token store `~/.claude/seo-data/tokens.json`. - -**Tech Stack:** Bash (entrypoint, tests), Python 3.14 (`google-auth`, `google-auth-oauthlib`, `requests` — pinned, in a dedicated venv), GSC Search Console API v3 + URL Inspection, CrUX API. - -**Spec:** `docs/superpowers/specs/2026-07-09-gsc-crux-data-layer-design.md` (transient — delete after ship+doc+capitalize). - -## Global Constraints - -Every task's requirements implicitly include these (verbatim from the spec): - -- **Security first.** Secrets never in git, files `0600` / dirs `0700`, OAuth scope **exactly** `https://www.googleapis.com/auth/webmasters.readonly`, no secret ever printed to stdout/stderr/report. -- **Offline-testable.** Third-party imports (`google.*`, `requests`) are **lazy** — imported only inside real OAuth/HTTP code paths. The `accounts`, mock (`SEO_DATA_MOCK_DIR` set), and degraded paths run on **stdlib only**, no venv, no network. `make test` never hits the network. -- **Graceful degradation (fail-open audit).** Missing creds / missing venv / revoked token / HTTP 429 → JSON `{"status":"degraded","reason":"…"}` on stdout with **exit 0**. Bad CLI usage → exit 2. -- **Multi-account, no shared state.** Account + property are **explicit arguments** on every `fetch.sh` call. No "current account" global. Store writes only happen during `connect` (atomic `tmp`→`fsync`→`rename` under `fcntl` lock); audits are read-only. -- **Store keyed by user label**, not email (keeps scope minimal). Properties discovered via `sites.list` (already in scope). -- **Canonical env path.** Read secrets from `~/.claude/.env` (canonical), never `$REPO/.env` (symlink may be absent on a fresh machine). -- **Repo test convention.** Bash test following the repo idiom (helpers `tf`/`tr_`/`tn` + `PASS`/`FAIL` counters, final line `[ "$FAIL" -eq 0 ]`). The engine test lives at `lib/seo-data/seo-data.test.sh` — co-located with the engine, **deliberately NOT under `lib/tests/`** which the `config-protection.sh` hook gates as a guardrail dir. Task 6 extends the `make test` target to also discover `lib/seo-data/*.test.sh`. During TDD, run it directly: `bash lib/seo-data/seo-data.test.sh`. -- **No commit attribution trailers** (no `Co-Authored-By`, no `Claude-Session`). -- **Branch:** all commits on `feature/gsc-crux-data-layer` (already created). - ---- - -## File Structure - -**Engine (created):** -- `lib/seo-data/tokenstore.py` — label-keyed token store I/O (atomic + locked). Stdlib only. -- `lib/seo-data/google_seo.py` — CrUX + GSC calls, OAuth refresh (lazy), mock mode, normalization → JSON. -- `lib/seo-data/connect.py` — one-time OAuth consent + `sites.list` discovery + persist to store. -- `lib/seo-data/fetch.sh` — bash entrypoint: source env, pick python, dispatch, degrade, redact. -- `lib/seo-data/requirements.txt` — pinned deps. -- `lib/seo-data/README.md` — usage contract. - -**Tests (created):** -- `lib/seo-data/seo-data.test.sh` — deterministic bash test (drives CLIs against fixtures, checks locks). -- `lib/seo-data/fixtures/*.json` — synthetic API responses (no real secret/PII). - -**Wiring (modified):** -- `.env.example`, `install.sh`, `Makefile`, `doctor.sh`, `.gitleaks.toml`, `.gitignore`. - -**Integration (modified):** -- `agents/seo-analyzer.md`, `skills/seo/SKILL.md`, `agents/resources/automation-catalog.md`. - -**Interface contract (used across tasks):** -``` -tokenstore.py (module + CLI: python3 tokenstore.py {list|set} --file PATH …) - load(path) -> dict - list_accounts(path) -> list[dict] # [{label, properties, granted_at}] NO refresh_token - get_refresh_token(path, label) -> str | None - save_account(path, label, refresh_token, scopes: list[str], properties: list[str]) -> None - -google_seo.py (module + CLI: python3 google_seo.py {crux|queries|inspect} …) - crux(url, strategy='mobile') -> dict - queries(store_path, account, property, days=90, dim='query') -> dict - inspect(store_path, account, property, url) -> dict - # all return {"status":"ok"|"degraded", ...} - -fetch.sh {accounts|crux|queries|inspect} [flags] -> JSON on stdout - -connect.py (CLI: python3 connect.py --label LABEL) - run_consent(client_id, client_secret, scopes) -> str # refresh_token - discover_properties(refresh_token, client_id, client_secret) -> list[str] - persist(store_path, label, refresh_token, scopes, properties) -> None -``` - ---- - -## Task 1: Token store (`tokenstore.py`) - -Label-keyed, atomic, locked store. Foundation for everything; stdlib only so it tests without a venv. - -**Files:** -- Create: `lib/seo-data/tokenstore.py` -- Create: `lib/seo-data/seo-data.test.sh` - -**Interfaces:** -- Consumes: nothing. -- Produces: `load`, `list_accounts`, `get_refresh_token`, `save_account` (signatures in File Structure) + CLI `list`/`set`. - -- [ ] **Step 1: Write the failing test** — create `lib/seo-data/seo-data.test.sh`: - -```bash -#!/usr/bin/env bash -# Deterministic tests for the seo-data engine (no network, no venv). -set -u -REPO="$(cd "$(dirname "$0")/../.." && pwd)" -SD="$REPO/lib/seo-data" -PASS=0; FAIL=0 -ok() { echo " PASS $1"; PASS=$((PASS+1)); } -no() { echo " FAIL $1 — $2"; FAIL=$((FAIL+1)); } -# assert stdout of a command contains / omits a fixed string -has() { if printf '%s' "$2" | grep -qF -- "$3"; then ok "$1"; else no "$1" "missing: $3"; fi; } -hasnt(){ if printf '%s' "$2" | grep -qF -- "$3"; then no "$1" "forbidden: $3"; else ok "$1"; fi; } - -echo "── tokenstore ──" -TMP="$(mktemp -d)"; STORE="$TMP/tokens.json" -python3 "$SD/tokenstore.py" set --file "$STORE" --label client-a \ - --refresh-token RT_AAA --scopes https://www.googleapis.com/auth/webmasters.readonly \ - --properties sc-domain:a.com,https://www.a.com/ >/dev/null -python3 "$SD/tokenstore.py" set --file "$STORE" --label client-b \ - --refresh-token RT_BBB --scopes https://www.googleapis.com/auth/webmasters.readonly \ - --properties sc-domain:b.com >/dev/null -LIST="$(python3 "$SD/tokenstore.py" list --file "$STORE")" -has "list shows client-a" "$LIST" '"client-a"' -has "list shows client-b" "$LIST" '"client-b"' -has "list shows a property" "$LIST" 'sc-domain:a.com' -hasnt "list redacts refresh tokens" "$LIST" 'RT_AAA' -PERM="$(stat -c '%a' "$STORE")" -[ "$PERM" = "600" ] && ok "store file is 0600" || no "store file 0600" "got $PERM" -rm -rf "$TMP" - -echo "" -echo "seo-data engine: $PASS pass, $FAIL fail" -[ "$FAIL" -eq 0 ] -``` - -- [ ] **Step 2: Run it, verify red** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: FAIL — `tokenstore.py` does not exist (`python3: can't open file`). - -- [ ] **Step 3: Implement `lib/seo-data/tokenstore.py`** (stdlib only): - -```python -#!/usr/bin/env python3 -"""Label-keyed OAuth refresh-token store. Atomic writes under an fcntl lock. -No third-party deps — must run without the venv (used by the offline test path).""" -import argparse, fcntl, json, os, sys, tempfile -from datetime import datetime, timezone - -def load(path): - if not os.path.exists(path): - return {"version": 1, "accounts": {}} - with open(path, "r", encoding="utf-8") as f: - return json.load(f) - -def list_accounts(path): - data = load(path) - return [ - {"label": lbl, "properties": a.get("properties", []), - "granted_at": a.get("granted_at")} - for lbl, a in data.get("accounts", {}).items() - ] # refresh_token intentionally omitted (redaction) - -def get_refresh_token(path, label): - return load(path).get("accounts", {}).get(label, {}).get("refresh_token") - -def save_account(path, label, refresh_token, scopes, properties): - os.makedirs(os.path.dirname(path), mode=0o700, exist_ok=True) - lock_path = path + ".lock" - with open(lock_path, "w") as lock: - fcntl.flock(lock, fcntl.LOCK_EX) # serialize concurrent connects - data = load(path) - data.setdefault("version", 1) - data.setdefault("accounts", {}) - data["accounts"][label] = { - "refresh_token": refresh_token, - "scopes": scopes, - "granted_at": datetime.now(timezone.utc).isoformat(), - "properties": properties, - } - fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path), suffix=".tmp") - try: - with os.fdopen(fd, "w", encoding="utf-8") as f: - json.dump(data, f, indent=2) - f.flush(); os.fsync(f.fileno()) - os.chmod(tmp, 0o600) - os.replace(tmp, path) # atomic - finally: - if os.path.exists(tmp): - os.unlink(tmp) - -def _cli(): - p = argparse.ArgumentParser() - sub = p.add_subparsers(dest="cmd", required=True) - pl = sub.add_parser("list"); pl.add_argument("--file", required=True) - ps = sub.add_parser("set") - for flag in ("--file", "--label", "--refresh-token"): - ps.add_argument(flag, required=True) - ps.add_argument("--scopes", default="") - ps.add_argument("--properties", default="") - args = p.parse_args() - if args.cmd == "list": - print(json.dumps({"status": "ok", "accounts": list_accounts(args.file)})) - else: - save_account(args.file, args.label, getattr(args, "refresh_token"), - [s for s in args.scopes.split(",") if s], - [x for x in args.properties.split(",") if x]) - print(json.dumps({"status": "ok"})) - -if __name__ == "__main__": - _cli() -``` - -- [ ] **Step 4: Run it, verify green** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: PASS (5 tokenstore checks pass). - -- [ ] **Step 5: Commit** - -```bash -git add lib/seo-data/tokenstore.py lib/seo-data/seo-data.test.sh -git commit -m "feat(seo-data): label-keyed atomic OAuth token store" -``` - ---- - -## Task 2: CrUX fetch (`google_seo.py` — CrUX path) - -Simplest data path (API key, no OAuth). Establishes the mock-mode + degrade + normalization pattern. - -**Files:** -- Create: `lib/seo-data/google_seo.py` -- Create: `lib/seo-data/fixtures/crux_mobile.json` -- Modify: `lib/seo-data/seo-data.test.sh` (append CrUX section) - -**Interfaces:** -- Consumes: env `CRUX_API_KEY`, env `SEO_DATA_MOCK_DIR`. -- Produces: `crux(url, strategy='mobile') -> dict`; CLI `python3 google_seo.py crux --url … [--strategy …]`. - -- [ ] **Step 1: Write the fixture** — `lib/seo-data/fixtures/crux_mobile.json` (shape of the CrUX API `record.metrics`): - -```json -{"record":{"key":{"formFactor":"PHONE"},"metrics":{ - "largest_contentful_paint":{"percentiles":{"p75":2100}}, - "interaction_to_next_paint":{"percentiles":{"p75":180}}, - "cumulative_layout_shift":{"percentiles":{"p75":"0.08"}}}}} -``` - -- [ ] **Step 2: Write the failing test** — append to `lib/seo-data/seo-data.test.sh` before the final summary: - -```bash -echo "── crux (mock) ──" -CRUX_OK="$(SEO_DATA_MOCK_DIR="$REPO/lib/seo-data/fixtures" \ - python3 "$SD/google_seo.py" crux --url https://ex.com --strategy mobile)" -has "crux status ok" "$CRUX_OK" '"status": "ok"' -has "crux lcp p75 mapped" "$CRUX_OK" '"lcp_p75_ms": 2100' -has "crux inp p75 mapped" "$CRUX_OK" '"inp_p75_ms": 180' -has "crux cls p75 mapped" "$CRUX_OK" '"cls_p75": 0.08' -CRUX_DEG="$(env -u CRUX_API_KEY -u SEO_DATA_MOCK_DIR \ - python3 "$SD/google_seo.py" crux --url https://ex.com)" -has "crux degrades w/o key" "$CRUX_DEG" '"status": "degraded"' -has "crux degrade reason" "$CRUX_DEG" 'no_crux_key' -``` - -- [ ] **Step 3: Run it, verify red** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: FAIL — `google_seo.py` missing. - -- [ ] **Step 4: Implement the CrUX path** — create `lib/seo-data/google_seo.py` (lazy `requests` import; mock reads the fixture and runs the REAL normalizer): - -```python -#!/usr/bin/env python3 -"""CrUX + GSC fetch → normalized JSON. Third-party imports are LAZY so mock and -degraded paths run stdlib-only (no venv, no network).""" -import argparse, json, os, sys - -def _mock(name): - d = os.environ.get("SEO_DATA_MOCK_DIR") - if not d: - return None - path = os.path.join(d, name) - if not os.path.exists(path): - return None - with open(path, encoding="utf-8") as f: - return json.load(f) - -def _norm_crux(raw): - m = raw["record"]["metrics"] - def p75(metric): - return m.get(metric, {}).get("percentiles", {}).get("p75") - out = {"status": "ok", "source": "crux"} - lcp = p75("largest_contentful_paint") - inp = p75("interaction_to_next_paint") - cls = p75("cumulative_layout_shift") - # Low-traffic origins often miss a metric (INP notably) — omit, don't crash. - if lcp is not None: - out["lcp_p75_ms"] = int(lcp) - if inp is not None: - out["inp_p75_ms"] = int(inp) - if cls is not None: - out["cls_p75"] = float(cls) - if len(out) == 2: # no metric at all - return {"status": "degraded", "reason": "no_field_data"} - return out - -def _crux_query(key, body): - import requests # lazy - return requests.post( - "https://chromeuxreport.googleapis.com/v1/records:queryRecord?key=" + key, - json=body, timeout=20) - -def crux(url, strategy="mobile"): - raw = _mock("crux_%s.json" % strategy) - if raw is None: - key = os.environ.get("CRUX_API_KEY") - if not key: - return {"status": "degraded", "reason": "no_crux_key"} - ff = "PHONE" if strategy == "mobile" else "DESKTOP" - r = _crux_query(key, {"url": url, "formFactor": ff}) - if r.status_code == 404: # no page-level data → try origin-level - r = _crux_query(key, {"origin": url.rstrip("/"), "formFactor": ff}) - if r.status_code == 404: - return {"status": "degraded", "reason": "no_field_data"} - if r.status_code == 429: - return {"status": "degraded", "reason": "rate_limited"} - r.raise_for_status() - raw = r.json() - return _norm_crux(raw) - -def _cli(): - p = argparse.ArgumentParser() - sub = p.add_subparsers(dest="cmd", required=True) - pc = sub.add_parser("crux") - pc.add_argument("--url", required=True) - pc.add_argument("--strategy", default="mobile", choices=["mobile", "desktop"]) - pc.add_argument("--store", default=None) # accepted+ignored: uniform fetch.sh dispatch - args = p.parse_args() - try: - if args.cmd == "crux": - print(json.dumps(crux(args.url, args.strategy), indent=2)) - except Exception: - # Fail-open data contract: ANY unexpected error (HTTP 403/5xx, DNS, - # timeout) degrades with exit 0 — never a traceback, never empty stdout. - print(json.dumps({"status": "degraded", "reason": "unexpected_error"})) - -if __name__ == "__main__": - _cli() -``` - -- [ ] **Step 5: Run it, verify green** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: PASS (tokenstore + 6 CrUX checks). - -- [ ] **Step 6: Commit** - -```bash -git add lib/seo-data/google_seo.py lib/seo-data/fixtures/crux_mobile.json lib/seo-data/seo-data.test.sh -git commit -m "feat(seo-data): CrUX field-data fetch with mock mode and graceful degrade" -``` - ---- - -## Task 3: GSC fetch (`google_seo.py` — queries + inspect) - -Adds Search Analytics + URL Inspection with OAuth refresh (lazy) reusing `tokenstore`. - -**Files:** -- Modify: `lib/seo-data/google_seo.py` (add `queries`, `inspect`, `_gsc_session`, extend CLI) -- Create: `lib/seo-data/fixtures/gsc_queries.json`, `lib/seo-data/fixtures/gsc_inspect.json` -- Modify: `lib/seo-data/seo-data.test.sh` (append GSC section) - -**Interfaces:** -- Consumes: `tokenstore.get_refresh_token`, env `GOOGLE_OAUTH_CLIENT_ID/SECRET`, `SEO_DATA_MOCK_DIR`. -- Produces: `queries(store_path, account, property, days=90, dim='query')`, `inspect(store_path, account, property, url)`; CLI `queries`/`inspect`. - -- [ ] **Step 1: Write fixtures** - -`lib/seo-data/fixtures/gsc_queries.json` (Search Analytics `rows` shape): -```json -{"rows":[ - {"keys":["plombier paris"],"clicks":40,"impressions":900,"ctr":0.044,"position":6.3}, - {"keys":["urgence fuite"],"clicks":5,"impressions":1200,"ctr":0.004,"position":8.9}]} -``` -`lib/seo-data/fixtures/gsc_inspect.json` (URL Inspection shape): -```json -{"inspectionResult":{"indexStatusResult":{ - "verdict":"PASS","coverageState":"Submitted and indexed","lastCrawlTime":"2026-07-01T10:00:00Z"}}} -``` - -- [ ] **Step 2: Write the failing test** — append before the summary: - -```bash -echo "── gsc (mock) ──" -MOCK="$REPO/lib/seo-data/fixtures" -TMP2="$(mktemp -d)"; S2="$TMP2/tokens.json" -python3 "$SD/tokenstore.py" set --file "$S2" --label client-a --refresh-token RT \ - --scopes https://www.googleapis.com/auth/webmasters.readonly --properties sc-domain:ex.com >/dev/null -Q="$(SEO_DATA_MOCK_DIR="$MOCK" python3 "$SD/google_seo.py" queries \ - --store "$S2" --account client-a --property sc-domain:ex.com --days 90)" -has "queries ok" "$Q" '"status": "ok"' -has "queries row key" "$Q" 'plombier paris' -has "queries position field" "$Q" '"position": 6.3' -I="$(SEO_DATA_MOCK_DIR="$MOCK" python3 "$SD/google_seo.py" inspect \ - --store "$S2" --account client-a --property sc-domain:ex.com --url https://ex.com/x)" -has "inspect indexed true" "$I" '"indexed": true' -DEG="$(env -u SEO_DATA_MOCK_DIR python3 "$SD/google_seo.py" queries \ - --store "$TMP2/none.json" --account nobody --property sc-domain:ex.com)" -has "gsc degrades w/o creds" "$DEG" '"status": "degraded"' -has "gsc degrade reason" "$DEG" 'no_credentials' -rm -rf "$TMP2" -``` - -- [ ] **Step 3: Run it, verify red** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: FAIL — `queries`/`inspect` not implemented (argparse error / AttributeError). - -- [ ] **Step 4: Implement** — add to `google_seo.py`: - -```python -def _gsc_session(store_path, account): - """Return an authorized requests.Session or a degrade dict. Lazy imports.""" - rt = None - if store_path and account: - import tokenstore # local module, stdlib - rt = tokenstore.get_refresh_token(store_path, account) - cid = os.environ.get("GOOGLE_OAUTH_CLIENT_ID") - csec = os.environ.get("GOOGLE_OAUTH_CLIENT_SECRET") - if not (rt and cid and csec): - return {"status": "degraded", "reason": "no_credentials"} - from google.oauth2.credentials import Credentials # lazy - from google.auth.transport.requests import AuthorizedSession, Request - creds = Credentials(None, refresh_token=rt, client_id=cid, client_secret=csec, - token_uri="https://oauth2.googleapis.com/token", - scopes=["https://www.googleapis.com/auth/webmasters.readonly"]) - try: - creds.refresh(Request()) - except Exception as e: - # Only a real RefreshError means re-consent; a network blip must NOT - # send the user back through OAuth. - from google.auth.exceptions import RefreshError # lazy - reason = "token_revoked" if isinstance(e, RefreshError) else "network_error" - return {"status": "degraded", "reason": reason} - return AuthorizedSession(creds) - -def _norm_queries(raw, dim): - return {"status": "ok", "source": "gsc", "dimension": dim, "rows": [ - {"key": r["keys"][0], "clicks": r.get("clicks", 0), - "impressions": r.get("impressions", 0), "ctr": r.get("ctr", 0), - "position": r.get("position")} - for r in raw.get("rows", [])]} - -def queries(store_path, account, property, days=90, dim="query"): - raw = _mock("gsc_queries.json") - if raw is None: - sess = _gsc_session(store_path, account) - if isinstance(sess, dict): - return sess - import datetime as _dt - end = _dt.date.today(); start = end - _dt.timedelta(days=days) - import urllib.parse - url = ("https://searchconsole.googleapis.com/webmasters/v3/sites/" - + urllib.parse.quote(property, safe="") + "/searchAnalytics/query") - r = sess.post(url, json={"startDate": start.isoformat(), "endDate": end.isoformat(), - "dimensions": [dim], "rowLimit": 100}, timeout=30) - if r.status_code == 429: - return {"status": "degraded", "reason": "rate_limited"} - r.raise_for_status() - raw = r.json() - return _norm_queries(raw, dim) - -def inspect(store_path, account, property, url): - raw = _mock("gsc_inspect.json") - if raw is None: - sess = _gsc_session(store_path, account) - if isinstance(sess, dict): - return sess - r = sess.post("https://searchconsole.googleapis.com/v1/urlInspection/index:inspect", - json={"inspectionUrl": url, "siteUrl": property}, timeout=30) - if r.status_code == 429: - return {"status": "degraded", "reason": "rate_limited"} - r.raise_for_status() - raw = r.json() - isr = raw["inspectionResult"]["indexStatusResult"] - return {"status": "ok", "source": "gsc", - "indexed": isr.get("verdict") == "PASS", - "coverage": isr.get("coverageState"), - "last_crawl": isr.get("lastCrawlTime")} -``` -Extend `_cli()` (add subparsers `queries` and `inspect`, each with `--store --account --property`, plus `--days`/`--dim` for queries and `--url` for inspect; dispatch to the functions and `print(json.dumps(..., indent=2))` **inside the existing top-level `try/except`** from Task 2 — the fail-open contract covers every subcommand). Ensure the script's dir is importable for `import tokenstore` (add `sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))` at top). - -- [ ] **Step 5: Run it, verify green** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: PASS (tokenstore + CrUX + 7 GSC checks). - -- [ ] **Step 6: Commit** - -```bash -git add lib/seo-data/google_seo.py lib/seo-data/fixtures/gsc_queries.json lib/seo-data/fixtures/gsc_inspect.json lib/seo-data/seo-data.test.sh -git commit -m "feat(seo-data): GSC Search Analytics + URL Inspection with lazy OAuth refresh" -``` - ---- - -## Task 4: Bash entrypoint (`fetch.sh`) - -The stable CLI the analyzers call. Sources env, picks python (venv else system), dispatches, guarantees degrade-exit-0 and redaction. - -**Files:** -- Create: `lib/seo-data/fetch.sh` -- Modify: `lib/seo-data/seo-data.test.sh` (append fetch.sh section) - -**Interfaces:** -- Consumes: `~/.claude/.env` (canonical), `google_seo.py`, `tokenstore.py`, optional `~/.claude/.venv-seo-data/`. -- Produces: `fetch.sh {accounts|crux|queries|inspect} [flags]` → JSON stdout, exit 0 on ok/degrade, exit 2 on bad usage. - -- [ ] **Step 1: Write the failing test** — append before the summary: - -```bash -echo "── fetch.sh ──" -FETCH="$SD/fetch.sh" -# SEO_DATA_ENV_FILE=/dev/null: tests must NEVER source the real ~/.claude/.env — -# on a machine with a live CRUX_API_KEY the degrade tests would hit the network. -NOENV=/dev/null -ACC="$(SEO_DATA_ENV_FILE=$NOENV SEO_DATA_STORE=/nonexistent/tokens.json bash "$FETCH" accounts)" -has "accounts empty is ok json" "$ACC" '"accounts": []' -CR="$(SEO_DATA_ENV_FILE=$NOENV SEO_DATA_MOCK_DIR="$MOCK" bash "$FETCH" crux --url https://ex.com)" -has "fetch crux ok" "$CR" '"status": "ok"' -SEO_DATA_ENV_FILE=$NOENV bash "$FETCH" bogus-subcmd >/dev/null 2>&1; RC=$? -[ "$RC" = "2" ] && ok "bad subcmd exit 2" || no "bad subcmd exit 2" "got $RC" -DG="$(SEO_DATA_ENV_FILE=$NOENV env -u SEO_DATA_MOCK_DIR -u CRUX_API_KEY bash "$FETCH" crux --url https://ex.com)"; RC=$? -has "degrade json" "$DG" '"status": "degraded"' -[ "$RC" = "0" ] && ok "degrade exit 0" || no "degrade exit 0" "got $RC" -hasnt "no secret echoed" "$DG" 'RT_' -``` - -- [ ] **Step 2: Run it, verify red** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: FAIL — `fetch.sh` missing. - -- [ ] **Step 3: Implement `lib/seo-data/fetch.sh`:** - -```bash -#!/usr/bin/env bash -# Stable entrypoint for the seo-data engine. JSON on stdout; exit 0 on ok/degrade, -# exit 2 on bad usage. Never prints secrets. -set -uo pipefail -HERE="$(cd "$(dirname "$0")" && pwd)" -ENV_FILE="${SEO_DATA_ENV_FILE:-${HOME}/.claude/.env}" # canonical; tests override to /dev/null -STORE="${SEO_DATA_STORE:-${HOME}/.claude/seo-data/tokens.json}" -VENV_PY="${HOME}/.claude/.venv-seo-data/bin/python3" - -# Library stderr must never leak a secret into agent context — suppress it -# globally unless explicitly debugging (SEO_DATA_DEBUG=1 restores it). -[ -n "${SEO_DATA_DEBUG:-}" ] || exec 2>/dev/null - -# Load secrets quietly (sourced, never echoed). -if [ -f "$ENV_FILE" ]; then - set -a; # shellcheck source=/dev/null - . "$ENV_FILE"; set +a -fi -# Prefer the isolated venv (has google-auth); fall back to system python3 for -# stdlib-only paths (accounts / mock / degrade). -PY="python3"; [ -x "$VENV_PY" ] && PY="$VENV_PY" - -cmd="${1:-}"; shift || true -case "$cmd" in - accounts) exec "$PY" "$HERE/tokenstore.py" list --file "$STORE" ;; - crux|queries|inspect) - exec "$PY" "$HERE/google_seo.py" "$cmd" --store "$STORE" "$@" ;; - *) echo '{"status":"error","reason":"usage: fetch.sh {accounts|crux|queries|inspect} [flags]"}' - exit 2 ;; -esac -``` -Notes: `crux` accepts and ignores `--store` (already wired in Task 2's `_cli`) so `fetch.sh` dispatches uniformly. The global `exec 2>/dev/null` (unless `SEO_DATA_DEBUG=1`) plus `google_seo.py`'s top-level `try/except` together guarantee: JSON always on stdout, never a traceback, never a leaked secret, exit 0 on every data-path outcome. - -- [ ] **Step 4: Run it, verify green** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: PASS (all prior + 6 fetch.sh checks). - -- [ ] **Step 5: Commit** - -```bash -git add lib/seo-data/fetch.sh lib/seo-data/google_seo.py lib/seo-data/seo-data.test.sh -git commit -m "feat(seo-data): fetch.sh entrypoint with venv/system fallback and redaction" -``` - ---- - -## Task 5: OAuth consent (`connect.py`) + pinned deps - -One-time interactive consent + `sites.list` discovery + persist. Browser flow is manually verified; the persist + label logic is unit-tested. - -**Files:** -- Create: `lib/seo-data/connect.py` -- Create: `lib/seo-data/requirements.txt` -- Modify: `lib/seo-data/seo-data.test.sh` (append persist test) - -**Interfaces:** -- Consumes: env `GOOGLE_OAUTH_CLIENT_ID/SECRET`, `tokenstore.save_account`. -- Produces: `run_consent`, `discover_properties`, `persist`; CLI `python3 connect.py --label LABEL`. - -- [ ] **Step 1: Write `requirements.txt`** (pinned; versions current as of 2026-07 — the implementer verifies latest patch at execution): - -``` -google-auth==2.40.0 -google-auth-oauthlib==1.2.2 -requests==2.32.4 -``` - -- [ ] **Step 2: Write the failing test** — append before the summary (tests only the offline-safe `persist`, via the tokenstore it wraps): - -```bash -echo "── connect (persist, offline) ──" -TMP3="$(mktemp -d)"; S3="$TMP3/tokens.json" -python3 -c "import sys; sys.path.insert(0,'$SD'); import connect; \ -connect.persist('$S3','client-x','RT_X',['https://www.googleapis.com/auth/webmasters.readonly'],['sc-domain:x.com'])" -L3="$(python3 "$SD/tokenstore.py" list --file "$S3")" -has "connect.persist wrote label" "$L3" '"client-x"' -has "connect.persist wrote prop" "$L3" 'sc-domain:x.com' -hasnt "connect.persist redacts" "$L3" 'RT_X' -rm -rf "$TMP3" -``` - -- [ ] **Step 3: Run it, verify red** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: FAIL — `connect` module / `persist` missing. - -- [ ] **Step 4: Implement `lib/seo-data/connect.py`:** - -```python -#!/usr/bin/env python3 -"""One-time OAuth consent + GSC property discovery + persist. Third-party imports -are lazy so `persist` is testable stdlib-only.""" -import argparse, os, sys -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -import tokenstore - -SCOPES = ["https://www.googleapis.com/auth/webmasters.readonly"] - -def run_consent(client_id, client_secret): - from google_auth_oauthlib.flow import InstalledAppFlow # lazy - cfg = {"installed": {"client_id": client_id, "client_secret": client_secret, - "auth_uri": "https://accounts.google.com/o/oauth2/auth", - "token_uri": "https://oauth2.googleapis.com/token", - "redirect_uris": ["http://localhost"]}} - flow = InstalledAppFlow.from_client_config(cfg, scopes=SCOPES) - creds = flow.run_local_server(port=0) # opens browser, one-time consent - if not creds.refresh_token: - raise SystemExit("No refresh token returned. Revoke prior grant and retry.") - return creds.refresh_token - -def discover_properties(refresh_token, client_id, client_secret): - from google.oauth2.credentials import Credentials - from google.auth.transport.requests import AuthorizedSession, Request - creds = Credentials(None, refresh_token=refresh_token, client_id=client_id, - client_secret=client_secret, - token_uri="https://oauth2.googleapis.com/token", scopes=SCOPES) - creds.refresh(Request()) - r = AuthorizedSession(creds).get( - "https://searchconsole.googleapis.com/webmasters/v3/sites", timeout=30) - r.raise_for_status() - return [e["siteUrl"] for e in r.json().get("siteEntry", [])] - -def persist(store_path, label, refresh_token, scopes, properties): - tokenstore.save_account(store_path, label, refresh_token, scopes, properties) - -def _cli(): - p = argparse.ArgumentParser() - p.add_argument("--label", required=True) - p.add_argument("--store", default=os.path.expanduser("~/.claude/seo-data/tokens.json")) - args = p.parse_args() - cid = os.environ.get("GOOGLE_OAUTH_CLIENT_ID") - csec = os.environ.get("GOOGLE_OAUTH_CLIENT_SECRET") - if not (cid and csec): - raise SystemExit("Set GOOGLE_OAUTH_CLIENT_ID/SECRET in ~/.claude/.env first.") - existing = {a["label"] for a in tokenstore.list_accounts(args.store)} - if args.label in existing: - ans = input("Label '%s' exists. Overwrite? [y/N] " % args.label).strip().lower() - if ans != "y": - raise SystemExit("Aborted.") - rt = run_consent(cid, csec) - props = discover_properties(rt, cid, csec) - persist(args.store, args.label, rt, SCOPES, props) - print("Connected '%s'. Properties: %s" % (args.label, ", ".join(props) or "(none)")) - -if __name__ == "__main__": - _cli() -``` - -- [ ] **Step 5: Run it, verify green** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: PASS (all prior + 3 persist checks). - -- [ ] **Step 6: Manual verification (documented, not automated)** — after Task 6 wires `make seo-connect`: run it once against a real GCP OAuth client, confirm the browser consent completes, the store gains the label with discovered properties, and a second `fetch.sh queries` runs non-interactively. - -- [ ] **Step 7: Commit** - -```bash -git add lib/seo-data/connect.py lib/seo-data/requirements.txt lib/seo-data/seo-data.test.sh -git commit -m "feat(seo-data): OAuth consent + property discovery + pinned deps" -``` - ---- - -## Task 6: Install / deploy wiring - -`.env.example`, `Makefile seo-connect`, `install.sh` step, `doctor.sh` check, gitleaks allowlist, gitignore. All content-locked by the bash test. - -**Files:** -- Modify: `.env.example`, `Makefile`, `install.sh`, `doctor.sh`, `.gitleaks.toml`, `.gitignore` -- Modify: `lib/seo-data/seo-data.test.sh` (append wiring locks) - -**Interfaces:** -- Consumes: `lib/seo-data/{connect.py,requirements.txt}`. -- Produces: `make seo-connect`; doctor check; allowlisted store path. - -- [ ] **Step 1: Write the failing test** — append before the summary: - -```bash -echo "── wiring locks ──" -tf() { if grep -qF -- "$3" "$2" 2>/dev/null; then ok "$1"; else no "$1" "missing: $3"; fi; } -tf "env.example client id" "$REPO/.env.example" "GOOGLE_OAUTH_CLIENT_ID=" -tf "env.example crux key" "$REPO/.env.example" "CRUX_API_KEY=" -tf "makefile seo-connect" "$REPO/Makefile" "seo-connect:" -tf "makefile discovers test" "$REPO/Makefile" "lib/seo-data/*.test.sh" -tf "install prompts connect" "$REPO/install.sh" "make seo-connect" -tf "doctor checks seo-data" "$REPO/doctor.sh" "seo-data" -tf "gitleaks allowlist store" "$REPO/.gitleaks.toml" "seo-data/tokens.json" -tf "gitignore venv" "$REPO/.gitignore" ".venv-seo-data" -``` - -- [ ] **Step 2: Run it, verify red** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: FAIL — none of the 8 locks present yet. - -- [ ] **Step 3: Apply the wiring edits** - -`.env.example` — append: -``` -# ── Google SEO data layer (lib/seo-data) — used by /seo FULL ── -# OAuth Desktop client: GCP console → APIs & Services → Credentials → OAuth client (Desktop). -# Scope requested at consent: webmasters.readonly. One-time setup: make seo-connect -GOOGLE_OAUTH_CLIENT_ID= -GOOGLE_OAUTH_CLIENT_SECRET= -# CrUX + PageSpeed API key (GCP console → Credentials → API key, restricted to those APIs). -# Get it: https://developer.chrome.com/docs/crux/api -CRUX_API_KEY= -``` - -`Makefile` — add target + `.PHONY`: -```make -seo-connect: ## Connect a Google account for /seo FULL (creates venv, OAuth consent) - @python3 -m venv "$$HOME/.claude/.venv-seo-data" - @"$$HOME/.claude/.venv-seo-data/bin/pip" install -q -r lib/seo-data/requirements.txt - @bash -c 'read -r -p "Label for this account (e.g. client-a): " label; \ - "$$HOME/.claude/.venv-seo-data/bin/python3" lib/seo-data/connect.py --label "$$label"' -``` -(Add `seo-connect` to the `.PHONY:` line at the top of the Makefile.) - -Also extend the existing `test:` target so `make test` discovers the engine test. Change its loop line: -```make - @fail=0; for t in lib/tests/*.test.sh lib/gitflow-test.sh lib/tests/run-*.sh; do \ -``` -to add `lib/seo-data/*.test.sh`: -```make - @fail=0; for t in lib/tests/*.test.sh lib/seo-data/*.test.sh lib/gitflow-test.sh lib/tests/run-*.sh; do \ -``` -(Editing the Makefile is allowed — it is not a config-protection-gated file. This *adds* test discovery; it does not weaken any gate.) - -`install.sh` — after the `bash "$REPO/link.sh"` block (§5, ~line 107), before plugins: -```bash -# ── 5b. Optional: connect a Google account for /seo FULL ── -echo "" -if [ -f "$HOME/.claude/seo-data/tokens.json" ]; then - ok "seo-data: a Google account is already connected" -else - info "SEO data layer (GSC + CrUX) is optional. To enable real Search Console" - info "data in /seo FULL, add GOOGLE_OAUTH_* + CRUX_API_KEY to ~/.claude/.env," - info "then run: make seo-connect" -fi -``` - -`doctor.sh` — add a check block (non-fatal, canonical env, mirrors existing WARN style): -```bash -echo "── seo-data (GSC/CrUX) ──" -if grep -qE '^[[:space:]]*(export[[:space:]]+)?CRUX_API_KEY=.' "$HOME/.claude/.env" 2>/dev/null; then - ok "CRUX_API_KEY present" -else - warn "CRUX_API_KEY absent in ~/.claude/.env — /seo FULL falls back to lab PageSpeed" -fi -if [ -f "$HOME/.claude/seo-data/tokens.json" ]; then - ok "seo-data: $(python3 "$REPO/lib/seo-data/tokenstore.py" list --file "$HOME/.claude/seo-data/tokens.json" | grep -o '"label"' | wc -l) account(s) connected" -else - warn "seo-data: no Google account connected (run: make seo-connect) — GSC data disabled" -fi -``` -(Use the same `ok`/`warn` helpers doctor.sh already defines; if they differ, match its local names.) - -`.gitleaks.toml` — add to `[allowlist].paths` (after the `.env` entry): -```toml - # seo-data OAuth token store — legitimate local secret (like ~/.claude/.env), - # 0600, outside git. Allowlisted so `make scan-secrets` doesn't flag the vault. - '''(^|/)\.claude/seo-data/tokens\.json$''', -``` - -`.gitignore` — after the `.env*` block (~line 114): -``` -# seo-data engine local artifacts (live under ~/.claude, never committed) -.venv-seo-data/ -seo-data/tokens.json -``` - -- [ ] **Step 4: Run it, verify green** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: PASS (all prior + 8 wiring locks). Then `make test` — the whole suite still green (now including the seo-data engine test via the new glob). - -- [ ] **Step 5: Commit** - -```bash -git add .env.example Makefile install.sh doctor.sh .gitleaks.toml .gitignore lib/seo-data/seo-data.test.sh -git commit -m "chore(seo-data): install/make/doctor wiring + gitleaks allowlist for token store" -``` - ---- - -## Task 7: Analyzer + skill integration - -Make `/seo` FULL actually consume the engine: account selection in STEP 0, CrUX field data + a GSC performance subsection in the analyzer, automation-catalog entry. - -**Files:** -- Modify: `skills/seo/SKILL.md` (STEP 0 — account/property selection, FULL only) -- Modify: `agents/seo-analyzer.md` (STEP 4 CWV terrain via `fetch.sh crux`; new "Performance GSC" subsection via `fetch.sh queries`/`inspect`; STEP 9 Technical axis note) -- Modify: `agents/resources/automation-catalog.md` (GSC OAuth connection entry) -- Modify: `lib/seo-data/seo-data.test.sh` (append integration locks) - -**Interfaces:** -- Consumes: `lib/seo-data/fetch.sh` CLI contract. -- Produces: analyzer output enriched with real GSC/CrUX; passes `(account, property)` explicitly. - -- [ ] **Step 1: Write the failing test** — append before the summary: - -```bash -echo "── integration locks ──" -tf "skill step0 account select" "$REPO/skills/seo/SKILL.md" "COMPTE GOOGLE" -tf "analyzer calls fetch crux" "$REPO/agents/seo-analyzer.md" "fetch.sh crux" -tf "analyzer calls fetch queries" "$REPO/agents/seo-analyzer.md" "fetch.sh queries" -tf "analyzer gsc subsection" "$REPO/agents/seo-analyzer.md" "Performance GSC" -tf "catalog gsc oauth entry" "$REPO/agents/resources/automation-catalog.md" "make seo-connect" -``` - -- [ ] **Step 2: Run it, verify red** - -Run: `bash lib/seo-data/seo-data.test.sh` -Expected: FAIL — 5 integration locks absent. - -- [ ] **Step 3: Apply the integration edits** (concrete anchors from the /analyze report): - -`skills/seo/SKILL.md` — in **STEP 0**, after the "Audit depth" block, add a FULL-only account-selection block (main loop, interactive) exactly as specified in spec §6, opening with the line `COMPTE GOOGLE pour cet audit FULL :`, listing connected accounts from `bash ~/.claude/lib/seo-data/fetch.sh accounts` (**tilde path mandatory** — skills run from the audited project's directory, not the claude-config repo), an option to run `make seo-connect`, and an "Ignore" option. Record the chosen `(account, property)` in the shared context block and pass it into **both** analyzer dispatch prompts (STEP 1) under `BUSINESS CONTEXT` as `GSC account: