diff --git a/.claude/audits/2026-10-23-rust-r5-review-security-9.md b/.claude/audits/2026-10-23-rust-r5-review-security-9.md new file mode 100644 index 00000000..19824402 --- /dev/null +++ b/.claude/audits/2026-10-23-rust-r5-review-security-9.md @@ -0,0 +1,88 @@ +SECURITY — VERDICT: BLOCK(1) +TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten +SCOPE: 29 files (`git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`, HEAD 8bdb079) +BLOCKING: + 1. [HIGH] resource-bound / records weight undercount — crates/core/src/curation/draft/records.rs:176-177 (`parts_weight`) — `change.reference_ids` costs 1 unit per byte, but every record clones the whole `ChangeSummary`, and each 1-char id clone takes about 56 bytes (24-byte String header plus the smallest malloc chunk). The 16 Mi-unit bound therefore lets roughly 0.9 GB of cloned ids through before it trips. — hint: charge each id at 64 + len, as `op_weight` already does for `Cite`, or stop cloning the summary into every record (share it, or index the change) +REPORTED (non-blocking): + - [MEDIUM] resource-bound / parse amplification (pre-existing R5.3 code, outside the diff) — crates/core/src/curation/draft/schema.rs:405-412 (`parse_changeset`: Value tree, `value.clone()`, typed value, `assert_same_shape` serialising it back), plus the Change clones in pipeline.rs:147-159 + - [MEDIUM] guardrails SB-POLICY-NOT-OWNER (outside the diff) — supabase/migrations/20261008110000_anatomical_sheets.sql:1177 + - [LOW] new lint allowance (clippy, not a security suppression; same convention as the 20 existing integration test files) — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6 +PROOF: semgrep 162 rules on 29 files → 0 findings; checklist 8 checks → 4 findings (1 HIGH, 2 MEDIUM, 1 LOW) + +## Detail + +### 1. [HIGH] The records bound does not bound memory for change reference ids + +The bound comes from R5.4 (`records::charge`, `MAX_RECORDED_WEIGHT` = 16 Mi units). It is +documented (records.rs:36-41, RUST.md §11.14) as keeping the records, and the reports built +from them, small. `parts_weight` adds `change.reference_ids` as the sum of their byte lengths. +Each record clones the change's summary, so it clones the whole id vector too. Duplicate ids +are accepted (`non_empty_strings` does no dedup), and one declared 1-char reference id is enough. + +Measured with a scratch harness that runs `build_equine_draft` + `render_draft_artifacts` on +the seed's legacy sources and frozen drawing (read-only), a crafted `changes.json`, and +`references.json` plus one reference `r`. One level A change with `reference_ids` = `["r"] × N` +and M `set` ops on one finding (`/usr/bin/time -f %M`): + +| changes.json | N | M | outcome | peak RSS | +|---|---|---|---|---| +| equine (committed) | — | — | 11 files written | 17.8 MB | +| 1.0 MB | 200,000 | 100 | refused at op 83 by the records bound | 948 MB | +| 2.0 MB | 400,000 | 60 | refused at op 41 by the records bound | 971 MB | +| 8.0 MB | 2,000,000 | 8 | draft built, refused at the table bound | 1.30 GB | +| 16.4 MB (under the 16 MiB cap) | 4,100,000 | 4 | draft built, refused at the table bound | 1.67 GB | + +The bound trips only after about 0.9 GB is already allocated: a 1 MB input reaches 948 MB. +That breaks the brief's criterion (peak memory well below the GB range for any input under +the 16 MiB caps), and it breaks the R5.4 claim that the records bound keeps the reports' +inputs small. Other record parts are charged closely enough: JSON values cost 64 units for +roughly 32 to 110 bytes, string bytes cost 1 unit for 1 byte. + +Fix direction: charge 64 + len per change reference id, or keep a single copy of each +change's summary and stop cloning it into every record. Add a test with a duplicated +reference-id vector next to `the_records_bound_holds_inside_one_op_and_the_marking`. + +### 2. [MEDIUM] Parse-phase amplification (outside the diff) + +`parse_changeset` on the 16.4 MB / 4.1 M-id file peaks at 1.075 GB by itself (scratch +`parseonly` binary). That is about 65 times the input: the parsed Value tree, a clone of it +passed to `from_value`, the typed value, and `assert_same_shape` serialising the typed value +back. After parsing, `build_equine_draft` clones the changes again, `apply_change` builds a +summary, and `DraftResult.changes` holds more summaries, each about 230 MB for that input. +This code is R5.3 and does not appear in c6e0d85..HEAD, so it is reported, not blocking. +It falls under the same "any input under the 16 MiB caps" criterion and is worth handling +in the same fix (avoid the clone before `from_value`; compare shapes without re-serialising +everything). + +### Checked with no finding + +- `--out` / `--check` file safety (import_equine.rs), by reading: refuses an empty path or + `..`; the parent is canonicalised; a destination inside the seed is refused by prefix and by + (dev, inode) of every seed folder; a symlinked or non-empty destination is refused; writes + use `create_dir` + `create_new`, never overwrite, never delete; generated names must be + plain `Normal` components (an absolute path or `..` is refused); reads are canonicalised + into the seed, regular files only, 16 MiB cap; `--check` never lists through a link. The + TOCTOU on a single-user machine is the declared residual (RUST.md §11.13). +- Markdown structure and parity: the scratch differential fuzz of `format_report` against + Prettier 3.8.3 (repo options) ran 33,000 random reports built from admitted and + construct-prone tokens in list items, bibliography lines and table cells. 8,146 were + admitted, 0 differed. Hand-written constructs (`[a] (b)`, `{{ }}`, `[[a]]`, `<`, `|`, + `\"`, `\|`, setext and list markers mid-line) gave no difference. Every authored field is + mid-line behind a generated prefix; line breaks and controls are refused in list items and + turned into spaces in cells as the TypeScript did; code spans only carry stable ids, + targets and field segments limited to `[A-Za-z0-9_-]`. +- Table alignment: every Alphabetic or Numeric code point from U+00A0 to U+3FFFF in a cell + (149,044) is aligned as Prettier aligns it, or refused (emoji candidates, `UnsupportedWidth`). +- SVG well-formedness: attributes and text are escaped (`& < > "`, `\t \n \r` as references), + XML-invalid code points and CR are refused before writing (`drawing::first_unwritable`), + element names come from the reader's allowlist or are constants, and ids are stable ids. +- Existing suites: `cargo test -p faunosteo-core --features curation --release --test + curation_review --test curation_markdown --test curation_draft` → 30 passed. +- Secrets, SQL concatenation, eval/innerHTML, sensitive logging, tracked `.env`: none in the + diff (the only `println!` added prints a generated file name; `.env.example` is the + placeholder file). + +### Not covered by this run + +I did not finish the bound analysis of the review drawings' markers and labels +(`overlay.rs`, `app_review.rs` through the R3 layer). Nothing is concluded about them here. diff --git a/.claude/audits/2026-10-23-rust-r5-review-verify.md b/.claude/audits/2026-10-23-rust-r5-review-verify.md index b0f1e79f..db9f283c 100644 --- a/.claude/audits/2026-10-23-rust-r5-review-verify.md +++ b/.claude/audits/2026-10-23-rust-r5-review-verify.md @@ -65,6 +65,11 @@ Fresh verifier and security on the fixed commit: below. - Security (`.claude/audits/2026-10-23-rust-r5-review-security-8.md`): HIGH records bound checked only after a whole op and after the B/C marking: one op records once per item, each record cloning the change texts and the op value (B add of 10,000 children + 128 KiB rationale → 1.37 GB; subtree set 64 KiB → 2.0 GB; subtree cite 4,000 refs → 2.1 GB); RUST.md "tens of MB at most" false. MEDIUM review drawings per view (≈273 KB each view, 2,000 views → 547 MB). LOW `record_weight` ignored the op for cite/rename/remove/move; LOW `TriggerNames::apply` linear view search; LOW test allows, bin allow (context). - Fixed: the records weight charged at each record before the change and the op are cloned (`records::charge`, `RecordParts`; `Recorder::record` and `mark_to_confirm` return the refusal; op weight for every kind; equine 1,195,000), test `the_records_bound_holds_inside_one_op_and_the_marking`; document bound test from a padded baseline (the records bound now stops a growing changeset first); review drawings projected and bounded at 64 MiB before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`), test `review_drawings_past_their_bound_are_refused_before_rendering`; RUST.md rows and residual risks rewritten exactly. -## Round 9 +## Round 9 on 8bdb079: security BLOCK(1) (verifier pending at fix time); `pnpm check:all` green 15:24:59 + +- Security (`.claude/audits/2026-10-23-rust-r5-review-security-9.md`): HIGH records weight undercount: the change's reference ids at one unit per byte while each cloned id costs ≈56 bytes (200,000 one-character ids × 100 set ops: 948 MB before the bound; 16.4 MB file: 1.67 GB). MEDIUM `parse_changeset` ≈65× the input (1.075 GB at 16.4 MB; R5.3, outside the diff); MEDIUM guardrails WARN in a migration outside the diff; LOW test allows. Not covered: markers and labels bound of the drawings. +- Fixed: every cloned string (texts, ids, op strings, object keys) weighs 64 + its bytes (`records::text_weight`; equine 1,965,944, bound 8.5× above); ids case added to `the_records_bound_holds_inside_one_op_and_the_marking`; parse amplification declared in §11.15 (owner decision on a lower input cap); RUST.md row updated. + +## Round 10 Fresh verifier and security on the fixed commit: below. diff --git a/crates/core/src/curation/draft/records.rs b/crates/core/src/curation/draft/records.rs index 4504fc7f..061ae005 100644 --- a/crates/core/src/curation/draft/records.rs +++ b/crates/core/src/curation/draft/records.rs @@ -147,9 +147,10 @@ impl<'a> Recorder<'a> { } } -/// What a record holds, in weight units: 64 per JSON value (a `Value` node costs tens of -/// bytes) plus one per byte of every string: the values, and the change and the op, which each -/// record clones. A rename or a remove records the whole item and one op can record once per +/// What a record holds, in weight units, about its bytes in memory: 64 per JSON value and per +/// string (a `Value` node or a `String` header and its allocation cost tens of bytes, so a run of +/// one-character ids is not free) plus one per byte of text: the values, and the change and the +/// op, which each record clones. A rename or a remove records the whole item and one op can record once per /// item it reaches, so the records can grow far past the document: `charge` bounds their total. pub fn record_weight(record: &ChangeRecord) -> usize { parts_weight(&RecordParts { @@ -173,22 +174,31 @@ fn parts_weight(parts: &RecordParts<'_>) -> usize { parts.target, parts.field.unwrap_or_default(), ]; - let text: usize = texts.iter().map(|text| text.len()).sum::() - + change.reference_ids.iter().map(String::len).sum::(); + let text: usize = texts.iter().map(|text| text_weight(text)).sum::() + + change + .reference_ids + .iter() + .map(|id| text_weight(id)) + .sum::(); text + op_weight(parts.op) + value_weight(parts.before) + value_weight(parts.after) } /// The op a record clones: its target, its value or its ids. fn op_weight(op: &Op) -> usize { let carried = match op { - Op::Set(op) => op.field.len() + value_weight(&op.value), + Op::Set(op) => text_weight(&op.field) + value_weight(&op.value), Op::Add(op) => value_weight(&op.value), - Op::Rename(op) => op.id.len(), - Op::Move(op) => op.to.len(), - Op::Cite(op) => op.reference_ids.iter().map(|id| 64 + id.len()).sum(), + Op::Rename(op) => text_weight(&op.id), + Op::Move(op) => text_weight(&op.to), + Op::Cite(op) => op.reference_ids.iter().map(|id| text_weight(id)).sum(), Op::Remove(_) => 0, }; - 64 + op.target().len() + carried + 64 + text_weight(op.target()) + carried +} + +/// A cloned string: its header and allocation, then its bytes. +fn text_weight(text: &str) -> usize { + 64 + text.len() } /// The weight of a value, walked with an explicit stack (no recursion). @@ -200,7 +210,7 @@ fn value_weight(value: &Value) -> usize { match value { Value::String(text) => weight += text.len(), Value::Object(map) => { - weight += map.keys().map(String::len).sum::(); + weight += map.keys().map(|key| text_weight(key)).sum::(); pending.extend(map.values()); } Value::Array(items) => pending.extend(items), diff --git a/crates/core/tests/curation_draft.rs b/crates/core/tests/curation_draft.rs index aba5c0cc..31c9ffcc 100644 --- a/crates/core/tests/curation_draft.rs +++ b/crates/core/tests/curation_draft.rs @@ -833,11 +833,24 @@ fn the_records_bound_holds_inside_one_op_and_the_marking() { ("B", "r".repeat(8_192), json!([add])), ("A", "Essai".to_owned(), json!([add, subtree])), ]; + // A run of one-character reference ids weighs its headers too (security gate R5.4 round 9: + // 200,000 ids cloned into each record reached 948 MB at one unit per byte). + let mut cases = cases.to_vec(); + let sets: Vec = (0..100) + .map(|i| json!({ "op": "set", "target": MOBILITY, "field": "name.en", "value": format!("v{i}") })) + .collect(); + cases.push(("ids", "Essai".to_owned(), Value::from(sets))); for (level, rationale, ops) in cases { + let ids = if level == "ids" { + vec!["atlas"; 200_000] + } else { + Vec::new() + }; + let level = if level == "ids" { "A" } else { level }; let file = json!({ "changes": [{ "id": "CHG-001", "level": level, "kind": "technical", "title": { "fr": "Essai", "en": "Test" }, "rationale": rationale, - "reference_ids": [], "ops": ops, + "reference_ids": ids, "ops": ops, }]}); let changeset = parse_changeset(&to_js_compact(&file).unwrap()).unwrap(); let error = apply_changeset(&frozen["baseline"], &references, &changeset.changes) diff --git a/docs/development/RUST.md b/docs/development/RUST.md index b4892980..4d5c6ebc 100644 --- a/docs/development/RUST.md +++ b/docs/development/RUST.md @@ -3204,7 +3204,7 @@ pass the guard unchanged. | authored text outside the allowlist: Markdown Prettier rewrites (`*x*`, `_x`, `x_**a**`, `**a****b**`, `Voir_https://…`, `a b`, a non-ASCII space, a tab, a line break in a list item, `` `` ``, `a\|b`, `[a]()`, `[[ a ]]`, `*`…), the constructs refused whole (links, HTML tags, autolinks, character references) and any character outside the admitted set (pictographs, symbols, combining marks without the Alphabetic property; an alphabetic mark such as U+0345 is a letter and Prettier aligns it alike) | written, then rewritten by Prettier (escapes, `_emphasis_`, collapsed spaces, indented continuation lines, normalised links; a row with an extra cell), or kept | refused (`UnsupportedMarkdown`), even where Prettier would have kept the text | | a pictograph in a table cell (an emoji, and ©, ®, ™, ✓, ★, ⚠ and the other characters `text::is_emoji_candidate` lists) | measured with Prettier's tables | refused (`UnsupportedWidth`, R5.2): the set is wider than the emoji, the refusal stays loud | | an aligned report table past 32 Mi display columns in all (rows × padded row width; the equine CHANGELOG's widest table is about 0.5 MB): one long authored cell pads every row of its level, so an 80 KB rationale over 14,000 records asked for 1 GB (security gate R5.4 round 6) | Prettier aligned it (the same amplification) | refused (`Guard`): the CHANGELOG's tables are projected from their cells (bytes bound the display width) before the raw table is joined (test `a_changelog_whose_alignment_would_explode_is_refused_before_it_is_built`, through the pipeline), every other table in the aligner before any padded row is built (test `an_aligned_table_past_the_bound_is_refused_before_it_is_built`), import report included | -| records weighing more than 16 Mi units in all (`records::record_weight`: 64 per JSON value plus one per string byte of the values, of the change and of the op, which each record clones; the equine records weigh 1,195,000): a rename or a remove records the whole item and one op, or the B/C marking, records once per item it reaches, so 400 renames of a 5,000-child item asked for 4.3 GB (round 7), a level B add of 10,000 children with a 128 KiB rationale 1.37 GB inside one marking, a 64 KiB value set over a subtree 2.0 GB (round 8) | recorded (the same copies) | refused at the record that passes the bound, charged before the change and the op are cloned into it (`records::charge`, tests `the_records_cannot_outgrow_their_bound`, `the_records_bound_holds_inside_one_op_and_the_marking`); the raw report rows, built from the records, are bounded with them; the 200,000-value document bound now holds the baseline (test from a padded baseline) | +| records weighing more than 16 Mi units in all (`records::record_weight`, about the bytes in memory: 64 per JSON value and per string, plus one per byte of text, for the values, the change and the op, which each record clones; the equine records weigh 1,965,944): a rename or a remove records the whole item and one op, or the B/C marking, records once per item it reaches, so 400 renames of a 5,000-child item asked for 4.3 GB (round 7), a level B add of 10,000 children with a 128 KiB rationale 1.37 GB inside one marking, a 64 KiB value set over a subtree 2.0 GB (round 8), 200,000 one-character reference ids cloned into each record 948 MB at one unit per byte (round 9) | recorded (the same copies) | refused at the record that passes the bound, charged before the change and the op are cloned into it (`records::charge`, tests `the_records_cannot_outgrow_their_bound`, `the_records_bound_holds_inside_one_op_and_the_marking`); the raw report rows, built from the records, are bounded with them; the 200,000-value document bound now holds the baseline (test from a padded baseline) | | review drawings past 64 MiB in all, projected before any is rendered (two per view, each repeating the cleaned drawing; the equine draft writes about 0.55 MB): 2,000 views asked for 547 MB (security gate R5.4 round 8) | written | refused before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`, test `review_drawings_past_their_bound_are_refused_before_rendering`) | | an id an `add` writes (the item's, and those nested in its `children` or `categories`) or a `rename` id that is not a stable id (a region code for a region), or a field segment outside `[A-Za-z0-9_-]` (a control character, NUL included, a backtick, a `]`, a space, a non-ASCII letter) | accepted when a later op removed or renamed the item: the raw id stayed in the records and reached the reports' code spans (a line break there forged a section, security gate R5.4 round 2) | refused at the op, the check the final document applies anyway (test `ids_and_field_segments_that_could_forge_report_structure_are_refused`) | | a label, id or path holding a code point XML 1.0 refuses, or a carriage return, in a review drawing | written (a non-well-formed file; a raw CR read back as a line feed) | refused: the drawing is not written | @@ -3246,7 +3246,9 @@ decision; the record that passes the bound has been built before it is refused ( and one clone of the op's value, at most twice the 16 MiB read cap), and a subtree set writes its value into each item before recording it, so the document holds at most as many copies as the records bound admits; the comparison of `--check` looks up each trigger's view linearly (views × -triggers, the views bounded through the drawings bound); and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160, +triggers, the views bounded through the drawings bound); parsing `changes.json` (R5.3 code) holds +about 65 times the file in memory (about 1 GB at the 16 MiB read cap, security gate R5.4 round 9), +before any bound of the applier applies; and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160, U+FFA0, letters Prettier keeps, no structure); the residual risks of §11.13 stand. Conditions for R5.5: the owner's validation of this checkpoint; the upload tool (`ureq`, the