fix(anatomy): R5.4 records weigh every cloned string with its header

This commit is contained in:
bchanot committed 2026-10-23 15:34:38 +02:00
1 parent 8bdb079f70
commit 29eb499539
5 files changed
+133 -15

No files matched your search

@@ -0,0 +1,88 @@
SECURITY — VERDICT: BLOCK(1)
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
SCOPE: 29 files (`git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`, HEAD 8bdb079)
BLOCKING:
1. [HIGH] resource-bound / records weight undercount — crates/core/src/curation/draft/records.rs:176-177 (`parts_weight`) — `change.reference_ids` costs 1 unit per byte, but every record clones the whole `ChangeSummary`, and each 1-char id clone takes about 56 bytes (24-byte String header plus the smallest malloc chunk). The 16 Mi-unit bound therefore lets roughly 0.9 GB of cloned ids through before it trips. — hint: charge each id at 64 + len, as `op_weight` already does for `Cite`, or stop cloning the summary into every record (share it, or index the change)
REPORTED (non-blocking):
- [MEDIUM] resource-bound / parse amplification (pre-existing R5.3 code, outside the diff) — crates/core/src/curation/draft/schema.rs:405-412 (`parse_changeset`: Value tree, `value.clone()`, typed value, `assert_same_shape` serialising it back), plus the Change clones in pipeline.rs:147-159
- [MEDIUM] guardrails SB-POLICY-NOT-OWNER (outside the diff) — supabase/migrations/20261008110000_anatomical_sheets.sql:1177
- [LOW] new lint allowance (clippy, not a security suppression; same convention as the 20 existing integration test files) — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
PROOF: semgrep 162 rules on 29 files → 0 findings; checklist 8 checks → 4 findings (1 HIGH, 2 MEDIUM, 1 LOW)
## Detail
### 1. [HIGH] The records bound does not bound memory for change reference ids
The bound comes from R5.4 (`records::charge`, `MAX_RECORDED_WEIGHT` = 16 Mi units). It is
documented (records.rs:36-41, RUST.md §11.14) as keeping the records, and the reports built
from them, small. `parts_weight` adds `change.reference_ids` as the sum of their byte lengths.
Each record clones the change's summary, so it clones the whole id vector too. Duplicate ids
are accepted (`non_empty_strings` does no dedup), and one declared 1-char reference id is enough.
Measured with a scratch harness that runs `build_equine_draft` + `render_draft_artifacts` on
the seed's legacy sources and frozen drawing (read-only), a crafted `changes.json`, and
`references.json` plus one reference `r`. One level A change with `reference_ids` = `["r"] × N`
and M `set` ops on one finding (`/usr/bin/time -f %M`):
| changes.json | N | M | outcome | peak RSS |
|---|---|---|---|---|
| equine (committed) | — | — | 11 files written | 17.8 MB |
| 1.0 MB | 200,000 | 100 | refused at op 83 by the records bound | 948 MB |
| 2.0 MB | 400,000 | 60 | refused at op 41 by the records bound | 971 MB |
| 8.0 MB | 2,000,000 | 8 | draft built, refused at the table bound | 1.30 GB |
| 16.4 MB (under the 16 MiB cap) | 4,100,000 | 4 | draft built, refused at the table bound | 1.67 GB |
The bound trips only after about 0.9 GB is already allocated: a 1 MB input reaches 948 MB.
That breaks the brief's criterion (peak memory well below the GB range for any input under
the 16 MiB caps), and it breaks the R5.4 claim that the records bound keeps the reports'
inputs small. Other record parts are charged closely enough: JSON values cost 64 units for
roughly 32 to 110 bytes, string bytes cost 1 unit for 1 byte.
Fix direction: charge 64 + len per change reference id, or keep a single copy of each
change's summary and stop cloning it into every record. Add a test with a duplicated
reference-id vector next to `the_records_bound_holds_inside_one_op_and_the_marking`.
### 2. [MEDIUM] Parse-phase amplification (outside the diff)
`parse_changeset` on the 16.4 MB / 4.1 M-id file peaks at 1.075 GB by itself (scratch
`parseonly` binary). That is about 65 times the input: the parsed Value tree, a clone of it
passed to `from_value`, the typed value, and `assert_same_shape` serialising the typed value
back. After parsing, `build_equine_draft` clones the changes again, `apply_change` builds a
summary, and `DraftResult.changes` holds more summaries, each about 230 MB for that input.
This code is R5.3 and does not appear in c6e0d85..HEAD, so it is reported, not blocking.
It falls under the same "any input under the 16 MiB caps" criterion and is worth handling
in the same fix (avoid the clone before `from_value`; compare shapes without re-serialising
everything).
### Checked with no finding
- `--out` / `--check` file safety (import_equine.rs), by reading: refuses an empty path or
`..`; the parent is canonicalised; a destination inside the seed is refused by prefix and by
(dev, inode) of every seed folder; a symlinked or non-empty destination is refused; writes
use `create_dir` + `create_new`, never overwrite, never delete; generated names must be
plain `Normal` components (an absolute path or `..` is refused); reads are canonicalised
into the seed, regular files only, 16 MiB cap; `--check` never lists through a link. The
TOCTOU on a single-user machine is the declared residual (RUST.md §11.13).
- Markdown structure and parity: the scratch differential fuzz of `format_report` against
Prettier 3.8.3 (repo options) ran 33,000 random reports built from admitted and
construct-prone tokens in list items, bibliography lines and table cells. 8,146 were
admitted, 0 differed. Hand-written constructs (`[a] (b)`, `{{ }}`, `[[a]]`, `<`, `|`,
`\"`, `\|`, setext and list markers mid-line) gave no difference. Every authored field is
mid-line behind a generated prefix; line breaks and controls are refused in list items and
turned into spaces in cells as the TypeScript did; code spans only carry stable ids,
targets and field segments limited to `[A-Za-z0-9_-]`.
- Table alignment: every Alphabetic or Numeric code point from U+00A0 to U+3FFFF in a cell
(149,044) is aligned as Prettier aligns it, or refused (emoji candidates, `UnsupportedWidth`).
- SVG well-formedness: attributes and text are escaped (`& < > "`, `\t \n \r` as references),
XML-invalid code points and CR are refused before writing (`drawing::first_unwritable`),
element names come from the reader's allowlist or are constants, and ids are stable ids.
- Existing suites: `cargo test -p faunosteo-core --features curation --release --test
curation_review --test curation_markdown --test curation_draft` → 30 passed.
- Secrets, SQL concatenation, eval/innerHTML, sensitive logging, tracked `.env`: none in the
diff (the only `println!` added prints a generated file name; `.env.example` is the
placeholder file).
### Not covered by this run
I did not finish the bound analysis of the review drawings' markers and labels
(`overlay.rs`, `app_review.rs` through the R3 layer). Nothing is concluded about them here.
@@ -65,6 +65,11 @@ Fresh verifier and security on the fixed commit: below.
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-8.md`): HIGH records bound checked only after a whole op and after the B/C marking: one op records once per item, each record cloning the change texts and the op value (B add of 10,000 children + 128 KiB rationale → 1.37 GB; subtree set 64 KiB → 2.0 GB; subtree cite 4,000 refs → 2.1 GB); RUST.md "tens of MB at most" false. MEDIUM review drawings per view (≈273 KB each view, 2,000 views → 547 MB). LOW `record_weight` ignored the op for cite/rename/remove/move; LOW `TriggerNames::apply` linear view search; LOW test allows, bin allow (context).
- Fixed: the records weight charged at each record before the change and the op are cloned (`records::charge`, `RecordParts`; `Recorder::record` and `mark_to_confirm` return the refusal; op weight for every kind; equine 1,195,000), test `the_records_bound_holds_inside_one_op_and_the_marking`; document bound test from a padded baseline (the records bound now stops a growing changeset first); review drawings projected and bounded at 64 MiB before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`), test `review_drawings_past_their_bound_are_refused_before_rendering`; RUST.md rows and residual risks rewritten exactly.
## Round 9
## Round 9 on 8bdb079: security BLOCK(1) (verifier pending at fix time); `pnpm check:all` green 15:24:59
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-9.md`): HIGH records weight undercount: the change's reference ids at one unit per byte while each cloned id costs ≈56 bytes (200,000 one-character ids × 100 set ops: 948 MB before the bound; 16.4 MB file: 1.67 GB). MEDIUM `parse_changeset` ≈65× the input (1.075 GB at 16.4 MB; R5.3, outside the diff); MEDIUM guardrails WARN in a migration outside the diff; LOW test allows. Not covered: markers and labels bound of the drawings.
- Fixed: every cloned string (texts, ids, op strings, object keys) weighs 64 + its bytes (`records::text_weight`; equine 1,965,944, bound 8.5× above); ids case added to `the_records_bound_holds_inside_one_op_and_the_marking`; parse amplification declared in §11.15 (owner decision on a lower input cap); RUST.md row updated.
## Round 10
Fresh verifier and security on the fixed commit: below.
+21 -11
View File
@@ -147,9 +147,10 @@ impl<'a> Recorder<'a> {
}
}
/// What a record holds, in weight units: 64 per JSON value (a `Value` node costs tens of
/// bytes) plus one per byte of every string: the values, and the change and the op, which each
/// record clones. A rename or a remove records the whole item and one op can record once per
/// What a record holds, in weight units, about its bytes in memory: 64 per JSON value and per
/// string (a `Value` node or a `String` header and its allocation cost tens of bytes, so a run of
/// one-character ids is not free) plus one per byte of text: the values, and the change and the
/// op, which each record clones. A rename or a remove records the whole item and one op can record once per
/// item it reaches, so the records can grow far past the document: `charge` bounds their total.
pub fn record_weight(record: &ChangeRecord) -> usize {
parts_weight(&RecordParts {
@@ -173,22 +174,31 @@ fn parts_weight(parts: &RecordParts<'_>) -> usize {
parts.target,
parts.field.unwrap_or_default(),
];
let text: usize = texts.iter().map(|text| text.len()).sum::<usize>()
+ change.reference_ids.iter().map(String::len).sum::<usize>();
let text: usize = texts.iter().map(|text| text_weight(text)).sum::<usize>()
+ change
.reference_ids
.iter()
.map(|id| text_weight(id))
.sum::<usize>();
text + op_weight(parts.op) + value_weight(parts.before) + value_weight(parts.after)
}
/// The op a record clones: its target, its value or its ids.
fn op_weight(op: &Op) -> usize {
let carried = match op {
Op::Set(op) => op.field.len() + value_weight(&op.value),
Op::Set(op) => text_weight(&op.field) + value_weight(&op.value),
Op::Add(op) => value_weight(&op.value),
Op::Rename(op) => op.id.len(),
Op::Move(op) => op.to.len(),
Op::Cite(op) => op.reference_ids.iter().map(|id| 64 + id.len()).sum(),
Op::Rename(op) => text_weight(&op.id),
Op::Move(op) => text_weight(&op.to),
Op::Cite(op) => op.reference_ids.iter().map(|id| text_weight(id)).sum(),
Op::Remove(_) => 0,
};
64 + op.target().len() + carried
64 + text_weight(op.target()) + carried
}
/// A cloned string: its header and allocation, then its bytes.
fn text_weight(text: &str) -> usize {
64 + text.len()
}
/// The weight of a value, walked with an explicit stack (no recursion).
@@ -200,7 +210,7 @@ fn value_weight(value: &Value) -> usize {
match value {
Value::String(text) => weight += text.len(),
Value::Object(map) => {
weight += map.keys().map(String::len).sum::<usize>();
weight += map.keys().map(|key| text_weight(key)).sum::<usize>();
pending.extend(map.values());
}
Value::Array(items) => pending.extend(items),
+14 -1
View File
@@ -833,11 +833,24 @@ fn the_records_bound_holds_inside_one_op_and_the_marking() {
("B", "r".repeat(8_192), json!([add])),
("A", "Essai".to_owned(), json!([add, subtree])),
];
// A run of one-character reference ids weighs its headers too (security gate R5.4 round 9:
// 200,000 ids cloned into each record reached 948 MB at one unit per byte).
let mut cases = cases.to_vec();
let sets: Vec<Value> = (0..100)
.map(|i| json!({ "op": "set", "target": MOBILITY, "field": "name.en", "value": format!("v{i}") }))
.collect();
cases.push(("ids", "Essai".to_owned(), Value::from(sets)));
for (level, rationale, ops) in cases {
let ids = if level == "ids" {
vec!["atlas"; 200_000]
} else {
Vec::new()
};
let level = if level == "ids" { "A" } else { level };
let file = json!({ "changes": [{
"id": "CHG-001", "level": level, "kind": "technical",
"title": { "fr": "Essai", "en": "Test" }, "rationale": rationale,
"reference_ids": [], "ops": ops,
"reference_ids": ids, "ops": ops,
}]});
let changeset = parse_changeset(&to_js_compact(&file).unwrap()).unwrap();
let error = apply_changeset(&frozen["baseline"], &references, &changeset.changes)
+4 -2
View File
@@ -3204,7 +3204,7 @@ pass the guard unchanged.
| authored text outside the allowlist: Markdown Prettier rewrites (`*x*`, `_x`, `x_**a**`, `**a****b**`, `Voir_https://…`, `a b`, a non-ASCII space, a tab, a line break in a list item, `` `` ``, `a\|b`, `[a](<b>)`, `[[ a ]]`, `&#42;`…), the constructs refused whole (links, HTML tags, autolinks, character references) and any character outside the admitted set (pictographs, symbols, combining marks without the Alphabetic property; an alphabetic mark such as U+0345 is a letter and Prettier aligns it alike) | written, then rewritten by Prettier (escapes, `_emphasis_`, collapsed spaces, indented continuation lines, normalised links; a row with an extra cell), or kept | refused (`UnsupportedMarkdown`), even where Prettier would have kept the text |
| a pictograph in a table cell (an emoji, and ©, ®, ™, ✓, ★, ⚠ and the other characters `text::is_emoji_candidate` lists) | measured with Prettier's tables | refused (`UnsupportedWidth`, R5.2): the set is wider than the emoji, the refusal stays loud |
| an aligned report table past 32 Mi display columns in all (rows × padded row width; the equine CHANGELOG's widest table is about 0.5 MB): one long authored cell pads every row of its level, so an 80 KB rationale over 14,000 records asked for 1 GB (security gate R5.4 round 6) | Prettier aligned it (the same amplification) | refused (`Guard`): the CHANGELOG's tables are projected from their cells (bytes bound the display width) before the raw table is joined (test `a_changelog_whose_alignment_would_explode_is_refused_before_it_is_built`, through the pipeline), every other table in the aligner before any padded row is built (test `an_aligned_table_past_the_bound_is_refused_before_it_is_built`), import report included |
| records weighing more than 16 Mi units in all (`records::record_weight`: 64 per JSON value plus one per string byte of the values, of the change and of the op, which each record clones; the equine records weigh 1,195,000): a rename or a remove records the whole item and one op, or the B/C marking, records once per item it reaches, so 400 renames of a 5,000-child item asked for 4.3 GB (round 7), a level B add of 10,000 children with a 128 KiB rationale 1.37 GB inside one marking, a 64 KiB value set over a subtree 2.0 GB (round 8) | recorded (the same copies) | refused at the record that passes the bound, charged before the change and the op are cloned into it (`records::charge`, tests `the_records_cannot_outgrow_their_bound`, `the_records_bound_holds_inside_one_op_and_the_marking`); the raw report rows, built from the records, are bounded with them; the 200,000-value document bound now holds the baseline (test from a padded baseline) |
| records weighing more than 16 Mi units in all (`records::record_weight`, about the bytes in memory: 64 per JSON value and per string, plus one per byte of text, for the values, the change and the op, which each record clones; the equine records weigh 1,965,944): a rename or a remove records the whole item and one op, or the B/C marking, records once per item it reaches, so 400 renames of a 5,000-child item asked for 4.3 GB (round 7), a level B add of 10,000 children with a 128 KiB rationale 1.37 GB inside one marking, a 64 KiB value set over a subtree 2.0 GB (round 8), 200,000 one-character reference ids cloned into each record 948 MB at one unit per byte (round 9) | recorded (the same copies) | refused at the record that passes the bound, charged before the change and the op are cloned into it (`records::charge`, tests `the_records_cannot_outgrow_their_bound`, `the_records_bound_holds_inside_one_op_and_the_marking`); the raw report rows, built from the records, are bounded with them; the 200,000-value document bound now holds the baseline (test from a padded baseline) |
| review drawings past 64 MiB in all, projected before any is rendered (two per view, each repeating the cleaned drawing; the equine draft writes about 0.55 MB): 2,000 views asked for 547 MB (security gate R5.4 round 8) | written | refused before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`, test `review_drawings_past_their_bound_are_refused_before_rendering`) |
| an id an `add` writes (the item's, and those nested in its `children` or `categories`) or a `rename` id that is not a stable id (a region code for a region), or a field segment outside `[A-Za-z0-9_-]` (a control character, NUL included, a backtick, a `]`, a space, a non-ASCII letter) | accepted when a later op removed or renamed the item: the raw id stayed in the records and reached the reports' code spans (a line break there forged a section, security gate R5.4 round 2) | refused at the op, the check the final document applies anyway (test `ids_and_field_segments_that_could_forge_report_structure_are_refused`) |
| a label, id or path holding a code point XML 1.0 refuses, or a carriage return, in a review drawing | written (a non-well-formed file; a raw CR read back as a line feed) | refused: the drawing is not written |
@@ -3246,7 +3246,9 @@ decision; the record that passes the bound has been built before it is refused (
and one clone of the op's value, at most twice the 16 MiB read cap), and a subtree set writes its
value into each item before recording it, so the document holds at most as many copies as the
records bound admits; the comparison of `--check` looks up each trigger's view linearly (views ×
triggers, the views bounded through the drawings bound); and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160,
triggers, the views bounded through the drawings bound); parsing `changes.json` (R5.3 code) holds
about 65 times the file in memory (about 1 GB at the 16 MiB read cap, security gate R5.4 round 9),
before any bound of the applier applies; and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160,
U+FFA0, letters Prettier keeps, no structure); the residual risks of §11.13 stand.
Conditions for R5.5: the owner's validation of this checkpoint; the upload tool (`ureq`, the