docs(docs): R5.4 fourth gate round declared, guarantee restated exactly, audits

This commit is contained in:
bchanot committed 2026-10-23 13:02:22 +02:00
1 parent dcef2eb79c
commit aff3a96d55
3 files changed
+196 -17

No files matched your search

@@ -0,0 +1,166 @@
# Security gate R5.4, round 4 (fresh): feature/rust-r5-pipeline-core @ 6d4fdd6
Scope: `git diff c6e0d85..HEAD` (36 existing files), focus on 124d954 (`curation/report_guard.rs`,
`curation/markdown.rs`, `draft/{pipeline,fields,ops}.rs`). Product files read whole: report_guard.rs,
markdown.rs, changelog.rs, review_report.rs, overlay.rs, drawing.rs, app_review.rs, svg_compare.rs,
labels.rs, stats.rs, bin/import_equine.rs, plus the R5.4 hunks of fields.rs, ops.rs, pipeline.rs, mod.rs.
```
SECURITY — VERDICT: BLOCK(3)
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
SCOPE: 36 files
BLOCKING:
1. [HIGH] markdown-guard/character-reference — crates/core/src/curation/report_guard.rs:43 — a character reference without its `;`, or with a name over 32 characters, passes the guard and Prettier rewrites it: `a &#42 b` → `a \* b`, `&#x2A` → `\*`, `&#95` → `\_`, `a&#95b` → `a_b`, `&#000…042;` (35-character name) → `\*`; a non-terminated reference followed later by a bare `&` before `,` or `"` makes Prettier overwrite that `&`: `&lt x &` → `&lt x <`, `&#65 x &` → `&#65 x A` — hint: outside code spans refuse `&#` whatever follows, and `&` followed by a legacy (non-terminated) entity name with or without `;`; keep `&` before other text admitted (the equine REVIEW-REPORT holds `&pid=S1019` in a URL)
2. [HIGH] markdown-guard/image-reference — crates/core/src/curation/report_guard.rs:33 — `![…]` is not among the refused constructs; Prettier parses it as an image reference and prints the alt from its decoded, unescaped value: `![a\"b]` → `![a"b]`, `![a\|b]` → `![a|b]`, `![x|y]` in a CHANGELOG cell, `![&amp]` → `![&]`, `![&lt]` → `![<]`, `![&#42]` → `![*]`; the `![` and its `]` can come from two insertions on one list line (title, rationale, question) — hint: refuse `![` outside code spans (the equine reports hold none)
3. [HIGH] markdown-guard/wiki-link — crates/core/src/curation/report_guard.rs:34 — the padded-wiki check splits on `[[`, so the text between a `[[` and the first `]]` is never examined when another `[[` sits in between; Prettier's tokenizer (`/^\[\[(?<linkContents>.+?)\]\]/su`, contents trimmed) matches across it: `[[ a [[b]]` → `[[a [[b]]`, also `[[ é [[中]]`, `[[ 1 [[2]]`, `[[a]] [[ b [[c]]`, `[[ x [ [[y]]` — hint: for every `[[` occurrence (overlapping `[[[` included) take the text up to the first following `]]` and refuse it when it starts or ends with whitespace, or refuse `[[` followed by whitespace and whitespace followed by `]]`
REPORTED (non-blocking):
- [MEDIUM] declaration-wrong (guarantee overstated) — docs/development/RUST.md:3164, 3169, 3184, 3214
- [LOW] resource/quadratic-underscore-check — crates/core/src/curation/report_guard.rs:88
- [LOW] lint-allow in test files (convention, not a scanner suppression) — crates/core/tests/curation_review.rs:6, crates/core/tests/curation_markdown.rs:6
PROOF: semgrep 128 rules on 36 files → 0 findings; checklist 13 checks → 6 findings
```
## Severity basis
The three blocking findings break the owner requirement this guard exists for: a report is byte for
byte what the TypeScript+Prettier pipeline wrote, or it is refused loudly. Each one is a silent
difference reachable through the normal authoring channel (a title, rationale or question in
`changes.json`, a citation in `references.json`, a label set by an op), with no refusal and a clean
exit. There is no confidentiality or availability impact and no actor beyond the author of the
curation files; the classification rests on the explicit integrity requirement and on the fact that
RUST.md states the opposite (finding M1). In two families the TypeScript output is the one that alters
the author's text (`&lt x &` became `&lt x <`), so the fix is a refusal, never an imitation of
Prettier.
## Method and evidence (differential, all in the scratchpad)
- TypeScript side: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/ts/sweep.mts`,
the sweep generator's exact `changesFile(probe)` on the mini sheet, run through `packages/anatomy/src`
(`applyChangeset`, `renderChangelog`, `renderReviewReport`) and `formatMarkdown` (Prettier 3.8.3,
options of `packages/anatomy/src/format.ts`).
- Rust side: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/rs/src/main.rs`,
a throwaway binary linking `faunosteo-core` (`curation`), same changeset through `apply_changeset`,
`render_changelog`, `render_review_report`, `format_report`. Raw texts matched the TypeScript in every
case (no RAWDIFF), so every difference is the guard admitting what Prettier rewrites.
- Batches: 79 hand-picked probes (20 silent), 2,500 seeded random strings over the admitted alphabet
plus `&#`, `&amp`, `&lt`, `![`, `[[` atoms (151 silent, every one holding `&#`, a legacy entity or
`![`), 3,000 seeded random strings without `&`, `![` or `[[` (0 silent), 55 targeted edges (7 silent),
13 entity variants (2 silent). Total 5,647 probes, 180 silent differences, all in the three families.
Probe lists and results: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/out/`.
- Prettier internals confirmed in `prettier@3.8.3/plugins/markdown.js`: wiki tokenizer
`/^\[\[(?<linkContents>.+?)\]\]/su` with `.trim()`, placed before `link`; the unescape restore keeps the
source of a one-character text node except `*` and `_` (`Tf=/^\\?.$/su`), which is why `&#42` and
`&#95` come out as `\*` and `\_`; `imageReference` prints `node.alt`, the decoded value.
Sample (review list item, probe in the title):
```
rust: - `CHG-001` Titre a &#42 b (niveau C, …
ts: - `CHG-001` Titre a \* b (niveau C, …
rust: - `CHG-001` Titre ![a\"b] (niveau C, …
ts: - `CHG-001` Titre ![a"b] (niveau C, …
rust: - `CHG-001` Titre [[ a [[b]] (niveau C, …
ts: - `CHG-001` Titre [[a [[b]] (niveau C, …
rust: … `probe-ref` Citation &lt x &, cité par 1 nœud(s).
ts: … `probe-ref` Citation &lt x <, cité par 1 nœud(s).
```
In the CHANGELOG the same probes also shift the table alignment (Prettier pads on the rewritten text).
## Findings in detail
### H1 [HIGH] character references not terminated by `;` (report_guard.rs:43-53)
`reference` requires `after.contains(';')` and `name.len() <= 32`. The parser Prettier uses decodes
non-terminated numeric references (`&#42`, `&#x2A`, `&#0042`) and the legacy named ones (`&lt`, `&amp`,
`&copy`…), and numeric references take any number of leading zeros, so the 32 cap lets
`&#00000000000000000000000000000000042;` through. A reference decoding to `*` or `_` is escaped
(`\*`, `\_`) or merged (`a&#95b` → `a_b`); a non-terminated reference followed later by a bare `&` (seen
before `,` and `"`, i.e. at the end of a citation or a JSON value) gets that `&` replaced by the decoded
character. Fix direction: refuse `&#` outside code spans unconditionally, and `&` followed by a legacy
entity name (the non-terminated set) regardless of `;`; keep the existing `&name;` rule; keep `&` before
anything else admitted, since the equine REVIEW-REPORT carries `&pid=S1019` inside a URL. Add the
reproducers above to `r5.markdown-sweep-oracles.test.ts`.
### H2 [HIGH] image references (report_guard.rs:33-55)
`rewritten_construct` refuses `](` and `][` but not a shortcut image reference `![…]`. Its alt is printed
from the decoded, unescaped value, so any `\"`, `\|` (including the `\|` that `cell()` adds in the
CHANGELOG) or reference inside it is rewritten. `![a]` with plain text stays identical, which is why the
sweep's `![img](x.png)` (refused through `](`) never showed it. Fix direction: refuse `![` outside code
spans.
### H3 [HIGH] wiki links whose contents hold a `[[` (report_guard.rs:34-41)
`plain.split("[[")` yields the segments between occurrences, not the text the regex matches: for
`[[ a [[b]]` the first segment ` a ` holds no `]]` and is skipped, the second `b]]` is unpadded, so the
line is admitted while Prettier matches ` a [[b` and trims it. Fix direction: emulate the lazy match
from each `[[` position (a `]]` at offset 1 or more), or refuse `[[` followed by whitespace and
whitespace followed by `]]`.
### M1 [MEDIUM] declarations overstated (docs/development/RUST.md)
- Line 3164: "a `[[ … ]]` padded inside" is refused: false for H3.
- Line 3169: "so an unknown form is refused by default": false; the allowlist is per character, and
constructs built from admitted characters (`&`, `#`, `!`, `[`, `]`) are admitted.
- Line 3184: "the constructs refused whole (links, HTML tags, autolinks, character references)": false for
non-terminated or long references (H1), and image references are not refused at all (H2).
- Line 3214: the residual risk says an uncovered combination "could still differ" and that the last rounds
"found none": three families with 180 reproducers were found in one round. The declaration must be
rewritten with the fixes, and the sweep extended with these families.
### L1 [LOW] quadratic underscore check (report_guard.rs:88)
For each `_`, `chars[i + 1..].iter().collect::<String>()` copies the rest of the text to test the URL
starts: O(n²) per authored text. Measured with the probe binary: 10 KB of `a_` 14 ms, 40 KB 221 ms, 80 KB
0.91 s (x4 per doubling), so about 2.5 min per occurrence at 1 MB and hours near the 16 MiB file cap; a
title or rationale is checked in up to four places. Owner-authored input of a dev tool, hence LOW. Fix
direction: compare the slice in place (`chars[i + 1..].starts_with(&['h','t','t','p',…])` or byte offsets
on the `&str`).
### L2 [LOW] lint allows in the new test files (context)
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` heads
`curation_review.rs` and `curation_markdown.rs`, the same header as the pre-existing `curation_draft.rs`:
test-only file access and unwraps, not a security-scanner suppression. No `nosemgrep`, `nosec`,
`eslint-disable`, `ts-ignore` or new allow in library code.
## Checklist (13 checks)
| # | Check | Result |
|---|---|---|
| 1 | Hardcoded secret, token, key, auth URL (added lines, incl. docs and oracle JSON) | none |
| 2 | SQL built by concatenation | none in the diff (the seed SQL writer is R5.3, unchanged) |
| 3 | Unsanitized render (innerHTML, eval, raw) | none; SVG text goes through the R3 serializer (`& < > "` and tab/newline escaped in attributes) after `first_unwritable` (XML 1.0 code points, CR refused) |
| 4 | Sensitive endpoint without authz | not applicable (offline dev tool, no endpoint) |
| 5 | Stack trace or internal path to a user | not applicable (CLI stderr for the developer) |
| 6 | Secret or PII in a log | none (stdout and stderr carry file names, counts, refusals) |
| 7 | Tracked `.env` or credential file | only `.env.example`; nothing in the diff |
| 8 | Guardrails scanner (`--domain api,supabase`) | 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`) outside the diff, filtered |
| 9 | New suppressions (anti-gaming) | none in library code; L2 |
| 10 | Markdown guard differential (5,647 probes) | H1, H2, H3, M1 |
| 11 | Strict SVG comparison | holds (below) |
| 12 | Resource use of the new code | L1; renderers indexed, document bound unchanged |
| 13 | `--check` / `--out` behaviour | holds (below) |
## What holds
- Channels: both draft reports pass every raw line through `format_report` (pipeline.rs:273-274); no
authored text reaches a heading; list-item fields with a control character are refused
(`assert_list_items_single_line`); ids that reach code spans are stable ids or region codes, checked at
the op (ops.rs `assert_stable_id`), field segments refuse controls (NUL included) and backticks; a line
with an odd backtick count is refused, so no code span can hide a line break. The R5.2 IMPORT-REPORT
stays on `format_markdown` with legacy sources only. View ids in file names are stable ids, and
`write_exclusive` refuses any non-normal path component.
- Strict SVG comparison (`svg_compare.rs`), 11 crafted cases against the committed `app-main.svg`: the
legitimate trigger-name difference is the only acceptance; an extra comment, trailing text, a prolog
change, another English name, a renamed attribute, a DOCTYPE, an empty or 100,000-level produced file,
a 100,000-level committed file and a stylesheet PI in the committed file are all refused, none panics.
- `--check` (binary built in the scratchpad, read-only): `legacy-import: 6 files, 0 differences`,
`draft: declared: review/app-main.svg`, `draft: 12 files, 0 differences`, exit 0.
- `--out` into a fresh `mktemp -d` under the scratchpad: 18 files written; a second run refused
`not empty, nothing written` (exit 2); legacy identical to the seed, draft identical except
`app-main.svg`. Working tree clean afterwards.
- Panics: none found in the new code (slices after an ASCII `**` match, `chars[i + 1..]` with `i < len`,
`compact_value` cut bounded by the length test, overlay NaN paths printed as `String(number)`).
- semgrep: 128 rules (p/security-audit, p/secrets, p/owasp-top-ten) on 36 files, 0 findings, 0 errors.
@@ -31,3 +31,8 @@ Fresh verifier and security on the fixed commit: below.
## Round 4
Fresh verifier and security on the fixed commit: below.
## Round 4 on 6d4fdd6: security BLOCK(3) (verifier pending at fix time)
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-4.md`): HIGH non-terminated / long character references (`&#42 b` → `\*`, `&lt x &` → `&lt x <`), HIGH image references `![…]` (alt reprinted decoded), HIGH nested padded wiki links (`[[ a [[b]]`); MEDIUM §11.14/§11.15 overstated ("unknown form refused by default", "refused whole", "found none"); LOW quadratic underscore check. Control: 3,000 random strings without `&`, `![`, `[[`: 0 silent.
- Fixed: `&` admitted only before a space or as a URL query separator `&name=` (name not a legacy-reference prefix, the 106 HTML legacy names); `![` refused; padded `[[`/`]]` refused when the matching half exists; URL checks without allocation; sweep 782 probes (new families, random alphabet with `&#`, `&amp`, `&lt`, `![`, `[[`, `]]`): 279 identical, 503 refused, 0 silent; docs restated exactly.
+25 -17
View File
@@ -3157,24 +3157,30 @@ also rewrites Markdown in authored text (escapes, emphasis, whitespace, links);
reproduces only the table alignment. After a denylist that leaked round after round, the guard of
the draft's two reports (`curation/report_guard.rs`, `format_report`; the R5.2 import report keeps
`format_markdown`, verified at R5.2) is an allowlist: outside code spans, letters and digits,
single spaces (none at a line end), the punctuation `.,;:!?'"«»()[]{}/-–—…%°≠=+#&<>’‘“”±→×≤≥|`, an
underscore between two ASCII letters or digits and not glued to a URL (`http://`, `https://`,
`www.`), a backslash before `"` or `|` (a JSON quote, the cell escape), `**word**` with one run of
letters between spaces or line edges (the import report's form), and no inline construct (an
inline link, image or reference `](`, `][`; a `[[ … ]]` padded inside; `<` before a letter, `/`,
`!` or `?`: an HTML tag or an autolink; a character reference `&name;`, `&#n;`); inside code spans
(generated ids and targets), anything but a carriage return or a tab, never empty, no outer space;
a table row with another cell count than its header and a control character in a list-item field
(title, rationale, question, citation, URL, DOI, vet-flag value) are refused too. Everything else
is refused with `UnsupportedMarkdown`, so an unknown form is refused by default.
single spaces (none at a line end), the punctuation `.,;:!?'"«»()[]{}/-–—…%°≠=+#<>’‘“”±→×≤≥|`, an
`&` before a space or as a URL's query separator (`&name=` in a bare URL, the name not starting
with one of HTML's 106 legacy references, which Prettier decodes without a `;`), an underscore
between two ASCII letters or digits and not glued to a URL (`http://`, `https://`, `www.`), a
backslash before `"` or `|` (a JSON quote, the cell escape), `**word**` with one run of letters
between spaces or line edges (the import report's form), and no inline construct (an inline link,
image or reference `](`, `][`; an image reference `![`; a `[[` before a space with a `]]` after
it, or a `]]` after a space with a `[[` before it; `<` before a letter, `/`, `!` or `?`: an HTML
tag or an autolink; a character reference `&name;`, `&#n;`); inside code spans (generated ids and
targets), anything but a carriage return or a tab, never empty, no outer space; a table row with
another cell count than its header and a control character in a list-item field (title,
rationale, question, citation, URL, DOI, vet-flag value) are refused too. Everything else is
refused with `UnsupportedMarkdown`: a character outside the set is refused by default; the forms
built from admitted characters are ruled one by one, so a form no rule names and no probe covers
could still pass.
The guarantee is empirical, not a proof of Prettier's behaviour: the TypeScript sweep
(`oracles/r5/markdown-sweep.json`, generator `packages/anatomy/tests/r5.markdown-sweep-oracles.test.ts`,
digests of the raw and formatted reports) holds 762 probes: 48 characters in six positions,
whitespace patterns, Markdown constructs, the families each gate round found, and 400 strings from a
seeded generator, each placed in a title, a rationale, a question, a citation, a label after a
digests of the raw and formatted reports) holds 782 probes: 48 characters in six positions,
whitespace patterns, Markdown constructs, the families each gate round found (strong pairs, URLs,
non-terminated references, image references, nested padded wiki links), and 400 strings from a
seeded generator over an alphabet that includes `&#`, `&amp`, `&lt`, `![`, `[[` and `]]`, each placed in a title, a rationale, a question, a citation, a label after a
word and a label alone (a cell start). Result (test `every_probe_is_formatted_as_the_typescript_or_refused`):
304 probes byte for byte with the TypeScript, 458 refused, 0 silent difference; a probe made only
279 probes byte for byte with the TypeScript, 503 refused, 0 silent difference; a probe made only
of what the equine reports use, outside the constructs, is never refused (a polygon's
`[[10,10],…]`, the DOI's `<0697:toih>` and the URLs' inner underscores pass). The equine reports
pass the guard unchanged.
@@ -3210,9 +3216,11 @@ Residual risks: the reports admit an allowlist of characters and forms and refus
including text Prettier would have kept (links, HTML, character references, symbols, combining
marks, non-ASCII spaces, most uses of `*` and `_`): a future changeset using them stops the
pipeline, visibly, until the allowlist is widened from new TypeScript probes or the port grows a
Markdown printer; the allowlist itself is proven by probes, not by a model of Prettier, so a
combination of admitted characters no probe covered could still differ (the sweep's random part
and the three gate differentials found none in their last rounds); the comparison of the application's drawings accepts
Markdown printer; the allowlist is proven by probes, not by a model of Prettier: each gate round
of R5.4 found a family the previous rules missed (Markdown syntax, constructs, strong pairs and
URLs, then non-terminated references, image references and nested padded wiki links), each now
ruled and frozen in the sweep, so a combination of admitted characters no probe covers could still
differ; the comparison of the application's drawings accepts
exactly the trigger-name rule, byte for byte; the frozen
`horse.svg` is read, never regenerated, so a future change of the drawing needs its own
decision; the residual risks of §11.13 stand.