docs(docs): R5.4 second gate round declared, contract clarification (markdown-sweep oracle), audits

This commit is contained in:
bchanot committed 2026-10-23 12:10:36 +02:00
1 parent df82424107
commit 077d91bdff
4 files changed
+235 -12

No files matched your search

@@ -0,0 +1,195 @@
```
SECURITY — VERDICT: PASS
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
SCOPE: 32 files
BLOCKING:
(none)
REPORTED (non-blocking):
- [MEDIUM] markdown-guard-bypass-via-ids — crates/core/src/curation/draft/review_report.rs:367 (also markdown.rs:149, ops.rs:286, ops.rs:197, changelog.rs:81)
- [MEDIUM] uncapped-recursive-parse — crates/core/src/curation/draft/svg_compare.rs:108 (called at :129)
- [MEDIUM] silent-prettier-difference-and-wrong-declaration — crates/core/src/curation/markdown.rs:115 (docs/development/RUST.md §11.14, §11.15)
- [LOW] overlay-non-finite-number — crates/core/src/curation/draft/overlay.rs:163
- [LOW] new-lint-allow-in-tests (context) — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
PROOF: semgrep 128 rules on 32 files → 0 findings; checklist 9 checks (8 defaults + anti-gaming) → 1 finding (LOW); targeted probes (68 authored changesets through the Rust renderers, 136 report comparisons against Prettier 3.8.3; compare_drawing at depths 60 to 100000; one read-only --check) → 4 findings
```
Gate on `git diff c6e0d85..HEAD` (HEAD 68dec25, branch feature/rust-r5-pipeline-core), focus on fix commit 94f8b35.
No HIGH or CRITICAL. Three MEDIUMs: one makes a declared guard leak, one regresses a depth cap
the project documents as an invariant, one shows the RUST.md parity declarations are wrong. None is
reachable from outside the repository: every input is authored in the owner's own seed folder.
## Method
- semgrep 1.168.0 (`EIO_BACKEND=posix`, `--metrics=off`, the three pinned packs) on the 32 diff
files that still exist: 128 rules, 0 findings, 0 errors.
- Checklist (CLAUDE.md defaults) over the diff: secrets, SQL concatenation, unsanitized render,
authz, error exposure, secrets in logs, tracked `.env`, guardrails scanner; anti-gaming suppressions.
- Guardrails `scan.py --domain api,supabase`: 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`,
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177`). That file is not in the diff, so the WARN is filtered out.
- I read whole: `markdown.rs`, `review_report.rs`, `svg_compare.rs`, `drawing.rs`, `overlay.rs`,
`app_review.rs`, `changelog.rs`, `labels.rs`, `stats.rs`, `pipeline.rs`, `bin/import_equine.rs`.
I also read the parts of `ops.rs`, `apply.rs`, `schema.rs`, `targets.rs`, `fields.rs`,
`anatomy/svg/{serialize,tree}.rs` and roxmltree 0.21.1 `tokenizer.rs` that these files rely on.
- Probes, all in the scratchpad (`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/`):
a throwaway crate linking `faunosteo-core` (feature `curation`). It feeds 68 authored changesets
on the mini sheet of `oracles/r5/markdown-sweep.json` through `apply_changeset`,
`render_changelog`, `render_review_report` and `format_markdown`. A Node script then runs the
repository's Prettier 3.8.3 (the options of `packages/anatomy/src/format.ts`) on the same raw
texts and compares (136 comparisons: 118 identical, 4 refused, 14 silent differences). A second
entry point calls `compare_drawing` on nested input. Finally `import-equine --check` ran once
(target dir in the scratchpad, `--locked`): `legacy-import: 6 files, 0 differences`,
`draft: declared: review/app-main.svg`, `draft: 12 files, 0 differences`. The working tree was
clean before and after. `--out` was never run.
## Findings
### 1. [MEDIUM] Markdown guard bypassed through ids: forged headings, sections and tables, raw CR in table rows
Where: `review_report.rs:367` (`assert_list_items_single_line`), `markdown.rs:149` (code parts
checked only for emptiness and outer spaces), `ops.rs:286` (`add` takes any string id),
`ops.rs:197` (`rename` takes any string id), `fields.rs:115` (only the first field segment is
checked), `changelog.rs:81` (`cell()` turns `\n` into a space but leaves `\r`).
The single-line check covers titles, rationales, questions, citations, URLs, DOIs and vet-flag
values. It misses the other authored texts the review report writes into list items. These are:
- the record subjects (op target plus field), at lines 97, 167 and 181;
- the change `reference_ids`, at lines 98 and 339.
Ids are only checked as stable ids at the end of the changeset (`apply.rs:210`,
`curated_issues`). So an item added with an arbitrary id and then removed, or renamed away, leaves
its raw id in the records while the final document stays valid. The same holds for a field path
`name.<anything>` set on such an item. A reference added and removed the same way can be cited in
`reference_ids`, because `assert_declared` runs before the ops.
`guard_line` checks line by line. A newline inside a code span, with one backtick on each side,
therefore yields lines that each pass. Measured, all passing the whole guard (`format_markdown`
Ok on both reports):
- Add then remove annotation `x`␊`## 11. Section forgée`␊`y` (written with a backtick after `x`
and before `y`). REVIEW-REPORT §2 and §3 then read:
```
- `CHG-001` Titre (niveau A, Correction, confirmation requise : non). Raison. Objets : `annotation:x`
## 11. Section forgée
`y`. Références : aucune.
```
The report gets an eleventh section. A `kind: translation` change forges an H1 in §7. A
reference id forges a heading in the `Références :` part. Prettier puts blank lines around the
heading, so the Rust file also differs silently from the TypeScript's.
- An id carrying the lines `| Forgé | Tableau |`, `| --- | --- |`, `| a | b |` produces a table
inside REVIEW-REPORT, which `format_markdown` aligns. Prettier keeps it indented inside the list
item, so this is a silent difference again.
- Id `x\ry`. CR is refused in plain text but not inside a code span. The CHANGELOG row
`| `CHG-001` Titre | `annotation:x\ry` | ...` keeps a raw CR. GFM (goldmark on Gitea) reads it
as a line ending, which breaks the table. Prettier normalizes it to LF, splitting the row in its
own output. The same raw CR shows up in the REVIEW-REPORT list item.
Impact: the reports, the CHANGELOG and the review document the osteopath signs off on, can carry
structure that never came from the generator. The owner's contract ("refused, never a report that
differs silently") does not hold for this channel. It takes a deliberately crafted changeset in
the repository (add then remove, or a rename chain). No accidental path was found.
Fix direction:
- At the root: check ids as stable ids when the op runs. That covers the added item's `id` (and
the key field), `rename.id`, and every target id segment and field segment. The final document
requires stable ids anyway, so refusing earlier loses nothing that could be published.
- Defence in depth: `guard_text` refuses any control character inside a code part, and
`assert_list_items_single_line` also checks record subjects and change `reference_ids`.
- Add these probes to the sweep, then declare the refusal in RUST.md.
### 2. [MEDIUM] Uncapped recursive XML parse of the committed drawing, ahead of the prescan
Where: `svg_compare.rs:108`. `outside_root` calls `roxmltree::Document::parse` with default
options. Its caller is line 129, before `parse_svg`, which is the function that runs
`prescan(SVG_CAPS)`. This ordering is new in 94f8b35.
roxmltree's `parse_element` → `parse_content` → `parse_element` recurses once per nesting level.
The workspace manifest itself says the core prescan must cap depth before any parse. Measured with
`compare_drawing(produced, committed, Some(rule))`: depths 60, 70 and 1000 are refused cleanly.
At depth 100000 (a 300 KB committed text, far below the 16 MiB read cap) the process dies with
`thread 'main' has overflowed its stack`, exit 134. In `--check` this happens whenever a
`review/app-*.svg` differs from the produced bytes.
Impact: an abort, not a pass. It fails closed, writes nothing and exits non-zero. Still, it breaks
the "depth and resource caps" requirement and the project's own prescan-first invariant.
`drawing.rs` orders the same two calls correctly (`read_svg` first).
Fix direction: call `parse_svg(committed)` before `outside_root`, or run `prescan(text, &SVG_CAPS)`
at the top of `outside_root`. Add a nested-input case to `curation_review.rs`.
### 3. [MEDIUM] Authored plain text that passes the guard but which Prettier rewrites; RUST.md §11.14 and §11.15 overstate the guarantee
Where: `markdown.rs:115` (the `_` rule tests `is_alphanumeric` on both neighbours), and the guard
having no rule for link syntax.
Measured, each passing the guard, then differing from Prettier with no refusal:
- `中_中`, `ア_ア`, `한_한` in a title, rationale or changed label. Prettier's text splitter makes
each CJK, kana or Hangul character its own word, so the `_` sits on a word boundary. In tables
it becomes `中\_中`. In a review list item that holds two such texts, Prettier reads
`_中 … 中_` as emphasis and prints `中*中 … 中*中`. Rust writes the raw text.
- `[a]( https://x.org )` in a changed label: Prettier prints `[a](https://x.org)`, Rust keeps the
spaces.
These do not break the Rust output's structure. They break parity: TypeScript-era files and Rust
files differ without a refusal. Two declarations are therefore wrong as written:
- §11.14 says the port "refuses ... every text Prettier would rewrite, never writing a report that
differs silently". That holds for the 220 sweep probes only.
- §11.15 says "links in authored text are kept as the TypeScript kept them". The TypeScript output
was Prettier's, which normalizes this link form.
Fix direction:
- Refuse `_` when either neighbour is a character Prettier's splitter isolates (CJK, kana,
Hangul), or more simply whenever a neighbour is not an ASCII or Latin letter or digit.
- Refuse a link destination with surrounding whitespace, or refuse `](` outright, since the equine
reports use none.
- Add these probes to the sweep and correct both declarations.
### 4. [LOW] Overlay prints an overflowed coordinate as `null`
Where: `overlay.rs:163`. `text_of` special-cases NaN only. `js_number` prints a non-finite value
as `null`, where `String(Infinity)` printed `Infinity`. It is reachable with finite but extreme
document values (a view box `min_y` near `f64::MAX`, or polygon points around `1e308`, whose sum
overflows). The XML stays well-formed and nothing is injected. It is a silent parity gap only.
Fix: print `Infinity` / `-Infinity` in `text_of`.
### 5. [LOW] New lint allows in the two new test files (context, anti-gaming)
Where: `curation_markdown.rs:6` and `curation_review.rs:6`, both
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. This is the
pattern `clippy.toml` documents for tests that do their own I/O; 34 existing test files carry it.
It is not a security suppression. There is no `nosemgrep`, `nosec`, `eslint-disable`,
`ts-ignore` or new allow in library code.
## Checked and clean
- Strict SVG comparison logic (apart from finding 2). The fallback accepts only when
`produced == prolog + serialize(rule(parse(committed))) + epilog` and the committed file
rebuilds byte for byte from its own tree. So the produced file can differ from the committed one
only where the rule changed the tree. The rule's values come from the produced document's view
names, and `document.json` is compared byte for byte. A changed French name makes the rule
change nothing, which is refused. A DTD is refused by roxmltree's default options. Comments,
PIs, entity forms and other encodings fail the rebuild check. I found no way to get an
unauthorized difference accepted.
- Markup injection in the drawings: the serializer escapes `& < > "`, plus `\t \n \r` in
attributes and `& < > \r` in text. Element and attribute names are fixed by code.
`drawing_text` refuses non-XML characters and CR before writing.
- Table cells: `|` is always escaped. An authored `\|`, a trailing backslash and a backslash before
a space are refused. Pipes and a trailing backslash inside code spans format exactly like
Prettier. The per-cell backtick check and the cell-count check in `align` hold.
- Control characters (NUL, FF, VT, NEL, U+2028), NBSP, BOM, U+3000, zero-width space, `#`, `-`,
`+`, `1.`, `>`, `---`, `~x~`, `~~x~~`, backslash escapes before punctuation, entities, autolinks,
images, footnote syntax and raw HTML at the start of a cell: all identical to Prettier.
- Panics on authored input: none found in the new library code. There is no `unwrap`, `expect` or
unchecked index, and the slices fall on ASCII or roxmltree boundaries. Tree recursion
(`first_difference`, `TriggerNames::apply`, `serialize`) runs on trees from `parse_svg`, which
is depth-capped at 64. `first_unwritable` uses an explicit stack.
- Resources: the guard and the alignment are linear per line. Renderers are indexed (subjects
grouped, citation counts in one walk, hashed previous markers). Record memory is a declared
residual (§11.13).
- `--check`: read-only, as declared. Reads go through the canonical-path, regular-file and
16 MiB checks. A file that is missing, not produced or different fails. `declared:` is printed
only after the strict match. `--out` claim logic: the only change is that an existing
destination is now checked against the seed by identity. Sound.
- Secrets, SQL built from strings, authz, error exposure, logging of secrets or PII, tracked
`.env`: nothing in the diff (only `.env.example` is tracked).
@@ -11,3 +11,13 @@ Contract `.claude/tasks/contracts/2026-10-23-rust-r5-review.md`. Fresh verifier,
## Round 2
Fresh verifier and security on the fixed commit: below.
## Round 2 on 68dec25: verifier ECARTS(2), security PASS (3 MEDIUM, 2 LOW)
- Verifier: C1, C3 to C6, C8, C9 MET. C2: silent Prettier rewrites the guard missed (inline link destinations and titles, autolinks with an entity, `[[ a ]]`, character references decoding to `*`/`_`), shown end to end on the equine pipeline with one appended change; §11.15 sentence on links wrong. C7: `markdown-sweep.json` not excepted by the contract. Differential: 2,430 targeted + 4,500 random cases, 0 raw-text or drawing differences.
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-2.md`): MEDIUM ids with line breaks (added then removed, renamed, nested, field segments) forging report sections through code spans, raw CR in code spans; MEDIUM uncapped roxmltree parse of the committed drawing before the prescan (stack overflow at depth 100,000); MEDIUM `中_中` / `[a]( x )` silent and §11.14-11.15 overstated; LOW Infinity printed `null`; LOW test allows (context).
- Fixed: construct rules (`](`, `][`, padded `[[ ]]`, autolinks, character references) from new sweep probes; underscore rule ASCII-only (the sweep showed `é_é` rewritten too); CR/TAB refused anywhere in a line; ids validated at the op (add, nested, rename) and field segments without control characters or backticks; core reader before roxmltree in the comparison; `Infinity`; contract clarification for the second oracle; docs (counts 247/187/60, declared rows, §11.15). Tests: forged ids, field segments, 100,000-level drawing.
## Round 3
Fresh verifier and security on the fixed commit: below.
@@ -50,3 +50,11 @@ identity-checked, cite membership hashed, a failed seed walk named as such, the
C9. **Gates.** `pnpm rust:check` and `pnpm check:all` green; security auditor PASS on the diff; this contract CONFORME by a
fresh verifier; RUST.md §11.14 (R5.4) and §11.15 (checkpoint), CHANGELOG, CLAUDE.md. No merge, no R5.5, no fingerprint
switch, no release, no deploy.
## Clarifications
- 2026-10-23 (gate round 2, C7): `crates/core/tests/oracles/r5/markdown-sweep.json` is a second new oracle of this slice,
written by the TypeScript generator `packages/anatomy/tests/r5.markdown-sweep-oracles.test.ts` (`MAKE_R5_ORACLES=1`). It
freezes what Prettier does to authored Markdown, from which the port's `UnsupportedMarkdown` guard is derived (C2 read as
"byte for byte with the TypeScript, or a loud refusal declared in RUST.md §11.14"). It is added, never a frozen file
changed; C7 excepts it with `review-synthetic.json`.
+22 -12
View File
@@ -3116,7 +3116,7 @@ LOWs only, §11.12). Contract `.claude/tasks/contracts/2026-10-23-rust-r5-review
| `pipeline.rs` | `render_draft_artifacts` complete, in the TypeScript order: document, assets, sheet-ref, migration map, the overlays and the application's drawings per view, CHANGELOG, REVIEW-REPORT, seed SQL |
| `bin/import_equine.rs` | `--check` compares the 6 + 12 files with explicit criteria (byte identity; the application's drawings may fall back on parsed trees under the declared rule, printed as `declared:`); a file missing, not produced or different fails; the pending list and its `shortcut:` are gone; `--out` writes the 18 files |
Parity, every expectation written by the TypeScript (`tests/curation_review.rs`, 6 tests):
Parity, every expectation written by the TypeScript (`tests/curation_review.rs`, 8 tests):
- `CHANGELOG.md` (1,114,533 bytes) and `REVIEW-REPORT.md` of the committed draft byte for byte;
their raw texts equal the ledger's pre-Prettier digests, the aligned texts its digests;
@@ -3141,8 +3141,8 @@ Parity, every expectation written by the TypeScript (`tests/curation_review.rs`,
without the rule.
`pnpm r5:import-equine --check`: `legacy-import: 6 files, 0 differences`; `draft: declared:
review/app-main.svg (trees equal under the R3 trigger-name rule)`; `draft: 12 files, 0
differences`. `--out` then `diff -r`: legacy identical; draft identical but `app-main.svg`.
review/app-main.svg (byte-identical once the R3 trigger-name rule is applied)`; `draft: 12
files, 0 differences`. `--out` then `diff -r`: legacy identical; draft identical but `app-main.svg`.
R5.3 carry-over (final gate LOWs, not applied to the verified state): an existing destination
folder is itself identity-checked against the seed's folders; a failed seed walk reads `the seed
@@ -3159,20 +3159,27 @@ would rewrite, never writing a report that differs silently. The rules come from
by the TypeScript (`oracles/r5/markdown-sweep.json`, generator
`packages/anatomy/tests/r5.markdown-sweep-oracles.test.ts`: 31 characters in six positions,
whitespace patterns and Markdown constructs, each placed in a title, a rationale, a question, a
citation and a changed label; 220 probes): outside code spans, `*` outside a well-formed
`**strong**` span, an underscore not between two letters or digits, a backslash before a space or
citation and a changed label; 247 probes): a carriage return or a tab anywhere, code spans
included; outside code spans, an inline link, image or
reference (`](`, `][`), a `[[ … ]]` with inner spaces, an autolink (`<scheme:…>`, `<…@…>`), a
character reference (`&name;`, `&#n;`, `&#xh;`; the last three refused whole although Prettier
keeps some of them), `*` outside a well-formed
`**strong**` span, an underscore not between two ASCII letters or digits (Prettier escapes one
next to an accented letter or a CJK, kana or Hangul character), a backslash before a space or
at the end or an authored one before the cell escape `\|`, two spaces in a row, a tab, a carriage
return, an odd number of backticks, an empty code span or one with an outer space; a table row
whose cell count differs from its header; a control character (a line break included) in a
title, rationale, question, citation, URL, DOI or vet-flag value written in a list item. Result
(test `every_probe_is_formatted_as_the_typescript_or_refused`): 187 probes byte for byte with the
TypeScript, 33 refused, 0 silent difference; a probe made only of what the equine reports use is
never refused. The equine reports and the R5.2 import report pass the guard unchanged.
TypeScript, 60 refused, 0 silent difference; a probe made only of what the equine reports use,
outside the constructs, is never refused (a polygon's `[[10,10],…]` and the DOI's
`<0697:toih>` pass). The equine reports and the R5.2 import report pass the guard unchanged.
| Input | TypeScript | Rust |
|---|---|---|
| authored text with Markdown Prettier rewrites (`*x*`, `_x`, `a b`, a tab, a line break in a list item, `` `` ``, `a\|b`…) | written, then rewritten by Prettier (escapes, `_emphasis_`, collapsed spaces, indented continuation lines; a row with an extra cell) | refused (`UnsupportedMarkdown`) |
| authored text with Markdown Prettier rewrites (`*x*`, `_x`, `a b`, a tab, a line break in a list item, `` `` ``, `a\|b`, `[a](<b>)`, `[a](b "t")`, `[[ a ]]`, `&#42;`…) or one of the constructs refused whole (links, autolinks, character references) | written, then rewritten by Prettier (escapes, `_emphasis_`, collapsed spaces, indented continuation lines; a row with an extra cell) | refused (`UnsupportedMarkdown`) |
| a pictograph in a table cell (an emoji, and ©, ®, ™, ✓, ★, ⚠ and the other characters `text::is_emoji_candidate` lists) | measured with Prettier's tables | refused (`UnsupportedWidth`, R5.2): the set is wider than the emoji, the refusal stays loud |
| an id an `add` writes (the item's, and those nested in its `children` or `categories`) or a `rename` id that is not a stable id (a region code for a region), or a field segment holding a control character or a backtick | accepted when a later op removed or renamed the item: the raw id stayed in the records and reached the reports' code spans (a line break there forged a section, security gate R5.4 round 2) | refused at the op, the check the final document applies anyway (test `ids_and_field_segments_that_could_forge_report_structure_are_refused`) |
| a label, id or path holding a code point XML 1.0 refuses, or a carriage return, in a review drawing | written (a non-well-formed file; a raw CR read back as a line feed) | refused: the drawing is not written |
The drawing comparison hides nothing (security gate, MEDIUM): the rule fallback of `--check`
@@ -3181,8 +3188,10 @@ instruction, encoding or entity form can escape the comparison), the rule to cha
and the rebuilt text after the rule to equal the produced one byte for byte; `declared:` is
printed only then (test `the_rule_fallback_refuses_what_the_tree_does_not_carry`: a stylesheet
processing instruction, comments, a trailing comment and another declared encoding refused).
The renderers are indexed (citation counts in one walk, previous markers and kept ids hashed,
subjects grouped per change).
The committed drawing is read by the core's reader before any other parse, so its depth cap
holds (a 100,000-level file is refused, never a stack overflow). The overlay prints a coordinate
that overflowed as `Infinity`, as `String(number)` did. The renderers are indexed (citation
counts in one walk, previous markers and kept ids hashed, subjects grouped per change).
Tests: `cargo test -p faunosteo-core --features curation` (review 8, Markdown 1, draft 16,
importer 5, binary 3, formatting 6, plus the crate's suites); anatomy 144 (two generators added).
@@ -3196,8 +3205,9 @@ at R3 and identical to the editor canvas.
Residual risks: the reports refuse what Prettier would rewrite (emphasis, escapes, whitespace
runs, line breaks in list items) and the pictographs the width table does not reproduce: a
future changeset using them stops the pipeline, visibly, until the port grows a Markdown printer
or the text changes; raw HTML and links in authored text are kept as the TypeScript kept them
(GFM renders them; Gitea strips raw HTML); the comparison of the application's drawings accepts
or the text changes; inline links, autolinks and character references are refused as well, even
where Prettier would keep them; raw HTML in authored text (`<b>…</b>`) is kept as the TypeScript
kept it, byte for byte (GFM renders it; Gitea strips it); the comparison of the application's drawings accepts
exactly the trigger-name rule, byte for byte; the frozen
`horse.svg` is read, never regenerated, so a future change of the drawing needs its own
decision; the residual risks of §11.13 stand.