feat(seo-data): content_quality verb — deterministic filler/AI-slop signal

Cherry-picked from claude-seo (github.com/AgriciDaniel/claude-seo, MIT)
content_quality.py, rewritten to the lib/seo-data contract per BDR-070. The
Content Shape axis was 100% LLM judgement; this gives it a measured input.

fetch.sh content_quality (stdin or --file) → {filler_score, ai_pattern_score,
information_density, overall_quality, flags[], matches{}}. 100% deterministic:
QRG §4.6 filler list (26 phrases) + AI-pattern list (46) kept intact, regex
matching, no LLM. Stdlib only (argparse/json/re/sys/collections/typing).

Advisory, NOT a verdict — the point of the wiring. It never claims a page "is
AI-written" (LRN-131/133); flags are candidates for human review. geo-analyzer
STEP 8 Check 10 makes it a deterministic input that INFORMS checks 1-9, never
replaces them, never scored on its own. A low number is not an automatic
finding.

Detection proven both directions (a detector that always- or never-flags is
useless): filler+slop text → flags [filler, low-density], overall 34-49; clean
dense factual text (dates/EUR/percentages) → no flags, overall 90. Empty input →
degraded/empty_input, never zeros-as-a-result.

Verified: GATE 1 verifier CONFORME 10/10 (both directions exercised live, lists
diffed intact vs source, advisory language confirmed); GATE 2 self-scan clean
(only sink is read-only open() for --file); seo-data 190 → 210 pass, 0 fail;
full suite green; shellcheck + py_compile clean.

agents/geo-analyzer.md

@@ -551,6 +551,13 @@ sample of a 300-page site says nothing about the other 294.
    pronouns?
 8. **Lists/tables vs prose** — structured where possible?
 9. **30/70 rule** (if city/service variants exist) — ≥70% unique?
+10. **Filler/AI-slop signal (deterministic)** — feed each sampled page's
+    body text to `fetch.sh content_quality`. It is a DETERMINISTIC input
+    that INFORMS checks 1-9 (word-list/density heuristics, no LLM call);
+    it never replaces your read of them. A low `overall_quality` or a
+    `filler`/`ai-patterns` flag is a candidate for human review, not an
+    automatic finding — do not let the number become the verdict, and do
+    not claim a page "is AI-written" from it.
 
 ### Sampling command
 
@@ -561,6 +568,11 @@ for f in index.html $(find . "${FEXCL[@]}" -maxdepth 3 \( -name "*.astro" -o -na
   echo "=== $f ==="
   grep -oE '<(h1|h2|h3)[^>]*>[^<]+</(h1|h2|h3)>|^#{1,3} .+' "$f" 2>/dev/null | head -20
 done
+
+# Filler/AI-slop signal (Check 10) — strip markup to plain body text, then
+# score it. Advisory only: pair the number with your own read of Checks 1-9.
+sed -e 's/<[^>]*>//g' index.html | \
+  bash ~/.claude/lib/seo-data/fetch.sh content_quality
 ```
 
 ### Findings
@@ -576,6 +588,9 @@ CITED STATISTICS    : <avg per page>
 FRESHNESS VISIBLE   : <n/N pages>
 PRONOUN-HEAVY       : <n/N pages flagged>
 30/70 RULE          : pass | fail | N/A
+FILLER/AI-SLOP SIGNAL : <avg overall_quality>/100, flags: <n/N pages flagged>
+                       (deterministic, advisory — informs checks 1-9, never
+                       a verdict, never scored on its own)
 PRIORITY ACTIONS    : <top 5>
 ```
 

lib/seo-data/README.md

@@ -248,6 +248,41 @@ fetch.sh schema_gen <reservation|order|discussion|profile> [flags] [--script-tag
       empty (argparse cannot catch that) → `{"status":"degraded",...}`,
       exit 0 — fail-open, never a traceback.
 
+fetch.sh content_quality [--file <path.txt>] < text_on_stdin
+  → {"status":"ok","source":"content_quality","filler_score":0,"ai_pattern_score":0,
+     "information_density":1.0,"overall_quality":90,"flags":[],
+     "matches":{"filler":[],"ai_patterns":[]}}
+  → {"status":"degraded","reason":"empty_input"|"<file error>"}
+
+  fetch.sh content_quality --file article.txt
+  printf '%s' "$BODY_TEXT" | fetch.sh content_quality
+
+  Adapted from claude-seo's `content_quality.py` (MIT) into this contract.
+  100% deterministic — regex/word-lists (QRG §4.6 filler phrases + a
+  Wikipedia "AI Cleanup" catalogue of LLM-typical phrasings, CC BY-SA 4.0),
+  no LLM call, no network. Reads the text to score from `--file <path>` or,
+  when `--file` is `-` or omitted, from stdin — the same idiom `score.py`
+  uses for `--findings`.
+    • **ADVISORY, NOT A VERDICT.** The output never claims "this text is
+      AI-written" — modern generative tools can pass every heuristic here,
+      and human writers use some of these phrases too. `flags` are
+      candidates for HUMAN REVIEW, never an automatic finding. geo-analyzer
+      STEP 8 (Content Shape for AI) treats `overall_quality`/`flags` as ONE
+      measured input that INFORMS the axis; the axis itself stays an LLM
+      judgement (30/70, Definition Lead), never replaced by this score.
+    • `filler_score`/`ai_pattern_score` (0-100, higher = worse) count
+      phrase-list hits scaled per 1000 tokens; `information_density`
+      (0.0-1.0) is entities + numbers per 100 tokens; `overall_quality`
+      (0-100, higher is better) is the weighted composite (also folds in a
+      bigram-repetition penalty even though that score isn't itself a
+      top-level field). `flags` fires at fixed thresholds: `filler`,
+      `ai-patterns`, `low-density`, `repetitive`.
+    • Stdlib only (argparse/json/re/sys/collections/typing) — runs even
+      without the venv. Empty/whitespace-only input degrades rather than
+      returning a false zero-value "ok": an empty analysis is not a result.
+    • This is filler/AI-pattern SHAPE, not fact-checking — a text can be
+      dense and well-cited yet still wrong; that stays a human/LLM call.
+
 fetch.sh drift --url https://ex.com/sitemap.xml [--max 500]
   → {"status":"ok","baseline":true,"captured":"…","pages":24,"store":"…"}
   → {"status":"ok","baseline":false,"since":"…","gone":[…],"new":[…],

lib/seo-data/content_quality.py

@@ -0,0 +1,242 @@
+#!/usr/bin/env python3
+"""Deterministic filler / AI-slop content-quality scorer. Stdlib only.
+
+Adapted from claude-seo (github.com/AgriciDaniel/claude-seo, MIT),
+content_quality.py — rewritten to the lib/seo-data fail-open contract.
+
+Scores a block of text against three regex/word-list heuristics: padding
+"filler" phrases (QRG §4.6), LLM-typical phrasings ("AI-pattern" list),
+and a measured information density (entities + numbers per token). 100%
+deterministic — no LLM call, no network.
+
+ADVISORY, NOT A VERDICT. This never claims "this text is AI-written" —
+modern generative tools can pass every heuristic here, and human writers
+use some of these phrases too. A low overall_quality or a filler/
+ai-patterns flag is a candidate for human review, nothing more. In
+geo-analyzer's STEP 8 (Content Shape for AI) it is ONE measured input
+that INFORMS the axis, which stays an LLM judgement (30/70, Definition
+Lead) — never a replacement for it, and never auto-filed as a finding on
+its own.
+
+Attribution: the AI-pattern list draws from the Wikipedia "AI Cleanup"
+project's catalogue of LLM-typical phrasings (CC BY-SA 4.0), the same
+list claude-seo cites.
+
+Envelope (see `_cli`)::
+
+    {"status": "ok", "source": "content_quality",
+     "filler_score": 0..100,        # higher = more filler-like
+     "ai_pattern_score": 0..100,    # higher = more AI-pattern hits
+     "information_density": 0.0..1.0,
+     "overall_quality": 0..100,     # composite, higher is better
+     "flags": ["filler", "ai-patterns", "low-density", "repetitive"],
+     "matches": {"filler": [...], "ai_patterns": [...]}}
+    {"status": "degraded", "reason": "empty_input" | "<why>"}
+"""
+import argparse, json, re, sys
+from collections import Counter
+from typing import Iterable
+
+# Padding / filler phrases QRG §4.6 flags as "little-to-no value". The
+# lists are the value of this module — kept intact from the source, not
+# trimmed.
+_FILLER_PHRASES = (
+    "it's important to note that",
+    "in this article, we'll explore",
+    "in this article we will explore",
+    "in today's fast-paced world",
+    "in today's digital age",
+    "in today's competitive landscape",
+    "needless to say",
+    "at the end of the day",
+    "when it comes to",
+    "when all is said and done",
+    "in the realm of",
+    "in the world of",
+    "the bottom line is",
+    "without further ado",
+    "first and foremost",
+    "last but not least",
+    "for what it's worth",
+    "it goes without saying",
+    "as we all know",
+    "the truth is that",
+    "the fact of the matter is",
+    "more often than not",
+    "let's dive in",
+    "let's dive into",
+    "let's take a closer look",
+    "let's take a deeper look",
+)
+
+# LLM-typical phrasings (Wikipedia AI Cleanup catalogue, CC BY-SA 4.0;
+# also used by claude-seo, MIT). Conservative: only phrases that
+# disproportionately appear in LLM output. Adding to this list should
+# require corpus evidence, not intuition.
+_AI_PATTERNS = (
+    "delve into",
+    "delve deeper into",
+    "in the ever-evolving",
+    "ever-evolving landscape",
+    "ever-changing landscape",
+    "in the dynamic landscape",
+    "navigating the",
+    "navigate the complexities",
+    "tapestry of",
+    "rich tapestry",
+    "intricate tapestry",
+    "embark on a journey",
+    "embarking on this",
+    "a testament to",
+    "a beacon of",
+    "the cornerstone of",
+    "a cornerstone of",
+    "at the heart of",
+    "at its core",
+    "in essence,",
+    "in conclusion,",
+    "ultimately,",
+    "moreover,",
+    "furthermore,",
+    "however, it's worth noting",
+    "it's worth noting that",
+    "by leveraging",
+    "leverage the power of",
+    "leveraging the power of",
+    "harness the power of",
+    "unlock the potential",
+    "unlock the full potential",
+    "the realm of possibilities",
+    "open up a world of",
+    "a world of possibilities",
+    "elevate your",
+    "transform your",
+    "revolutionize the way",
+    "game-changer",
+    "game-changing",
+    "cutting-edge",
+    "state-of-the-art",
+    "in summary,",
+    "to summarize,",
+    "to put it simply,",
+    "in a nutshell,",
+)
+
+_TOKEN_RE = re.compile(r"[A-Za-z][A-Za-z'\-]*")
+_NUMBER_RE = re.compile(r"\b\d+(?:[.,]\d+)?(?:%|st|nd|rd|th)?\b")
+# Capitalised multi-word names: rough proper-noun heuristic. Two or more
+# capitalised tokens in a row count as one entity.
+_ENTITY_RE = re.compile(r"\b(?:[A-Z][a-z]+(?:\s+[A-Z][a-z]+)+)\b")
+
+
+def _count_phrase_hits(text: str, patterns: Iterable[str]) -> list:
+    """Patterns that appear at least once in text (case-insensitive)."""
+    lowered = text.lower()
+    return [p for p in patterns if p in lowered]
+
+
+def _repetition_score(tokens):
+    """Bigram repetition: fraction of bigrams that recur more than once."""
+    if len(tokens) < 4:
+        return 0.0
+    bigrams = [tokens[i] + " " + tokens[i + 1] for i in range(len(tokens) - 1)]
+    counts = Counter(bigrams)
+    repeated = sum(1 for v in counts.values() if v > 1)
+    return repeated / max(1, len(counts))
+
+
+def analyse(text):
+    """Score text against the filler / AI-pattern / density / repetition
+    heuristics. Advisory only — see module docstring."""
+    tokens = [t.lower() for t in _TOKEN_RE.findall(text)]
+    n_tokens = len(tokens)
+
+    filler_hits = _count_phrase_hits(text, _FILLER_PHRASES)
+    ai_hits = _count_phrase_hits(text, _AI_PATTERNS)
+
+    # Density: entities + numbers per 100 tokens. A high-density article
+    # (case studies, data journalism) lands at ~5+; generic filler <2.
+    entities = len(_ENTITY_RE.findall(text))
+    numbers = len(_NUMBER_RE.findall(text))
+    density_per_100 = (entities + numbers) * 100.0 / max(1, n_tokens)
+    information_density = min(1.0, density_per_100 / 10.0)
+
+    rep_score = int(round(_repetition_score(tokens) * 100))
+
+    # Scale to per-1000 tokens so the score is comparable across lengths.
+    scale = max(1.0, n_tokens / 1000.0)
+    filler_score = min(100, int(round(len(filler_hits) / scale * 25)))
+    ai_pattern_score = min(100, int(round(len(ai_hits) / scale * 15)))
+
+    flags = []
+    if filler_score >= 50:
+        flags.append("filler")
+    if ai_pattern_score >= 40:
+        flags.append("ai-patterns")
+    if information_density < 0.20:
+        flags.append("low-density")
+    if rep_score >= 30:
+        flags.append("repetitive")
+
+    # Composite: invert penalty signals, weight by impact. Same weights
+    # as the source — the length bonus caps at 1000 tokens.
+    overall = (
+        (100 - filler_score) * 0.25
+        + (100 - ai_pattern_score) * 0.25
+        + information_density * 100 * 0.25
+        + (100 - rep_score) * 0.15
+        + min(100, n_tokens / 10.0) * 0.10
+    )
+
+    return {
+        "filler_score": filler_score,
+        "ai_pattern_score": ai_pattern_score,
+        "information_density": round(information_density, 3),
+        "overall_quality": int(round(overall)),
+        "flags": flags,
+        "matches": {"filler": filler_hits, "ai_patterns": ai_hits},
+    }
+
+
+def _build_parser():
+    p = argparse.ArgumentParser(
+        description="Deterministic filler / AI-slop content-quality scorer."
+    )
+    p.add_argument("--store", default=None)   # accepted+ignored (dispatch)
+    p.add_argument(
+        "--file", default="-",
+        help="Path to a text file, or - for stdin (default -).",
+    )
+    return p
+
+
+def _read_input(path):
+    """Read the analysis target from stdin ('-'/omitted) or a plain file.
+    Plain `open()` only — no pathlib, to stay stdlib-minimal per contract."""
+    if path in (None, "-"):
+        return sys.stdin.read()
+    return open(path, encoding="utf-8", errors="replace").read()
+
+
+def _cli():
+    try:
+        args = _build_parser().parse_args()
+        text = _read_input(args.file)
+        if not text or not text.strip():
+            print(json.dumps({"status": "degraded", "reason": "empty_input"}))
+            return
+        envelope = {"status": "ok", "source": "content_quality"}
+        envelope.update(analyse(text))
+        print(json.dumps(envelope, indent=2))
+    except SystemExit as e:
+        if e.code not in (0, None):
+            print(json.dumps({"status": "error", "reason": "bad_usage"}))
+        raise
+    except Exception as e:
+        # Fail-open: a missing --file, an unreadable/binary file, or any
+        # other unexpected error degrades rather than crashing the caller.
+        print(json.dumps({"status": "degraded", "reason": str(e)}))
+
+
+if __name__ == "__main__":
+    _cli()

lib/seo-data/fetch.sh

@@ -36,6 +36,8 @@ case "$cmd" in
     exec "$PY" "$HERE/score.py" --store "$STORE" "$@" ;;
   schema_gen)
     exec "$PY" "$HERE/schema_gen.py" --store "$STORE" "$@" ;;
+  content_quality)
+    exec "$PY" "$HERE/content_quality.py" --store "$STORE" "$@" ;;
   drift)
     exec "$PY" "$HERE/drift.py" --store "$STORE" "$@" ;;
   rendercheck)
@@ -54,6 +56,6 @@ case "$cmd" in
     fi
     echo '{"status":"error","reason":"usage: fetch.sh forget {--label <label>|--all} (label charset: A-Za-z0-9._-)"}'
     exit 2 ;;
-  *) echo '{"status":"error","reason":"usage: fetch.sh {accounts|crux|queries|inspect|cannibal|sitemap|rendercheck|linkgraph|drift|score|schema_gen|forget} [flags]"}'
+  *) echo '{"status":"error","reason":"usage: fetch.sh {accounts|crux|queries|inspect|cannibal|sitemap|rendercheck|linkgraph|drift|score|schema_gen|content_quality|forget} [flags]"}'
      exit 2 ;;
 esac

lib/seo-data/seo-data.test.sh

@@ -290,6 +290,79 @@ FSG_TAG="$(printf '%s' "$FSG" | \
   python3 -c 'import sys,json; print(json.load(sys.stdin)["script"])')"
 has "fetch schema_gen script-tag" "$FSG_TAG" '<script type="application/ld+json">'
 
+echo "── content_quality ──"
+CQ() { python3 "$SD/content_quality.py" "$@"; }
+# feed the phrase list's OWN entries so the match is exact, not paraphrased —
+# a detector proven only on the maintainer's paraphrase proves nothing
+FILLER_TXT="In today's fast-paced world, it's important to note that this \
+article will delve into the ever-evolving landscape of technology. Let's \
+dive in and navigate the complexities together, leveraging the power of \
+innovation to unlock the potential of your business. Ultimately, this \
+cutting-edge, state-of-the-art approach is a testament to progress. \
+Moreover, furthermore, in conclusion, transform your outcomes today."
+CLEAN_TXT="The 2024 ADEME report found French households spent 2,137 EUR \
+on heating, up 12% from 2021."
+FILLER_OUT="$(printf '%s' "$FILLER_TXT" | CQ)"
+CLEAN_OUT="$(printf '%s' "$CLEAN_TXT" | CQ)"
+has "filler text is ok"          "$FILLER_OUT" '"status": "ok"'
+has "clean text is ok"           "$CLEAN_OUT"  '"status": "ok"'
+# flags is a JSON array — extract it in isolation so the check can't be
+# fooled by the always-present "matches": {"filler": [...]} key sharing
+# the same quoted word
+FILLER_FLAGS="$(printf '%s' "$FILLER_OUT" | \
+  python3 -c 'import sys,json; print(",".join(json.load(sys.stdin)["flags"]))')"
+CLEAN_FLAGS="$(printf '%s' "$CLEAN_OUT" | \
+  python3 -c 'import sys,json; print(",".join(json.load(sys.stdin)["flags"]))')"
+case "$FILLER_FLAGS" in
+  *filler*|*ai-patterns*) ok "filler-heavy text is flagged" ;;
+  *) no "filler-heavy text is flagged" "flags: $FILLER_FLAGS" ;;
+esac
+hasnt "clean text is not flagged filler"      "$CLEAN_FLAGS" 'filler'
+hasnt "clean text is not flagged ai-patterns" "$CLEAN_FLAGS" 'ai-patterns'
+# proves BOTH directions: an always-flag or a never-flag detector is useless
+FILLER_Q="$(printf '%s' "$FILLER_OUT" | \
+  python3 -c 'import sys,json; print(json.load(sys.stdin)["overall_quality"])')"
+CLEAN_Q="$(printf '%s' "$CLEAN_OUT" | \
+  python3 -c 'import sys,json; print(json.load(sys.stdin)["overall_quality"])')"
+[ "$FILLER_Q" -lt 50 ] && ok "filler-heavy text scores LOW overall_quality" \
+                        || no "filler-heavy text scores LOW overall_quality" "got $FILLER_Q"
+[ "$CLEAN_Q" -gt "$FILLER_Q" ] && ok "clean dense text scores higher" \
+                                || no "clean dense text scores higher" "$CLEAN_Q vs $FILLER_Q"
+# empty / whitespace-only input never crashes and never claims a result
+EMPTY_OUT="$(printf '' | CQ)"
+has "empty input degrades"       "$EMPTY_OUT" '"status": "degraded"'
+has "empty input reason"         "$EMPTY_OUT" 'empty_input'
+WS_OUT="$(printf '   \n\t ' | CQ)"
+has "whitespace-only degrades"   "$WS_OUT" '"status": "degraded"'
+# --file path works, no fixture committed — mktemp + rm
+CQTMP="$(mktemp)"; printf '%s' "$CLEAN_TXT" > "$CQTMP"
+FILE_OUT="$(CQ --file "$CQTMP")"
+has "file input is ok"           "$FILE_OUT" '"status": "ok"'
+rm -f "$CQTMP"
+# a missing --file degrades, never a traceback
+MISSING_OUT="$(CQ --file /nonexistent/path/content-quality-test.txt)"
+has "missing --file degrades"    "$MISSING_OUT" '"status": "degraded"'
+# stdlib ONLY — asserted, not assumed
+CQ_IMPORTS="$(grep -E '^(import|from) ' "$SD/content_quality.py")"
+if printf '%s' "$CQ_IMPORTS" | grep -qivE '^(import argparse, json, re, sys|from collections import counter|from typing import iterable)$'; then
+  no "content_quality stdlib only" "unexpected import: $CQ_IMPORTS"
+else
+  ok "content_quality stdlib only"
+fi
+# ADVISORY HONESTY (LRN-131/133): a heuristic signal, never a verdict
+hasnt "never claims ai-written"  "$FILLER_OUT" 'ai-written'
+hasnt "never claims is AI verdict" "$FILLER_OUT" 'is AI'
+# dispatch wiring: --store precedes the verb (fetch.sh's own convention);
+# both stdin AND --file must work through the real entrypoint
+FCQ_STDIN="$(printf '%s' "$CLEAN_TXT" | \
+  SEO_DATA_ENV_FILE=/dev/null SEO_DATA_STORE=/nonexistent bash "$SD/fetch.sh" content_quality)"
+has "fetch dispatches content_quality (stdin)" "$FCQ_STDIN" '"status": "ok"'
+CQTMP2="$(mktemp)"; printf '%s' "$CLEAN_TXT" > "$CQTMP2"
+FCQ_FILE="$(SEO_DATA_ENV_FILE=/dev/null SEO_DATA_STORE=/nonexistent bash "$SD/fetch.sh" \
+  content_quality --file "$CQTMP2")"
+has "fetch dispatches content_quality (--file)" "$FCQ_FILE" '"status": "ok"'
+rm -f "$CQTMP2"
+
 echo "── fetch.sh ──"
 FETCH="$SD/fetch.sh"
 # SEO_DATA_ENV_FILE=/dev/null: tests must NEVER source the real ~/.claude/.env —
@@ -416,6 +489,7 @@ tf "analyzer calls fetch queries" "$REPO/agents/seo-analyzer.md" "fetch.sh queri
 tf "analyzer gsc subsection"    "$REPO/agents/seo-analyzer.md"   "Performance GSC"
 tf "catalog gsc oauth entry"    "$REPO/agents/resources/automation-catalog.md" "make seo-connect"
 tf "geo-analyzer wires schema_gen" "$REPO/agents/geo-analyzer.md" "fetch.sh schema_gen"
+tf "geo-analyzer wires content_quality" "$REPO/agents/geo-analyzer.md" "fetch.sh content_quality"
 
 echo "── account-mgmt locks ──"
 tf "skill routes account verbs" "$REPO/skills/seo/SKILL.md" "forget --all"
@@ -429,6 +503,8 @@ tf "readme documents seo-connect" "$REPO/lib/seo-data/README.md" "make seo-conne
 tf "readme documents forget"   "$REPO/lib/seo-data/README.md" "forget --all"
 tf "readme revocation note"    "$REPO/lib/seo-data/README.md" "myaccount.google.com/permissions"
 tf "readme documents schema_gen" "$REPO/lib/seo-data/README.md" "schema_gen"
+tf "readme documents content_quality" "$REPO/lib/seo-data/README.md" "content_quality"
+tf "readme states advisory caveat" "$REPO/lib/seo-data/README.md" "ADVISORY, NOT A VERDICT"
 
 echo ""
 echo "seo-data engine: $PASS pass, $FAIL fail"