fix(anatomy): R5.4 records weigh every cloned string with its header
This commit is contained in:
1 parent
8bdb079f70
commit
29eb499539
5 files changed
+133
-15
No files matched your search
@@ -0,0 +1,88 @@
|
||||
SECURITY — VERDICT: BLOCK(1)
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 29 files (`git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`, HEAD 8bdb079)
|
||||
BLOCKING:
|
||||
1. [HIGH] resource-bound / records weight undercount — crates/core/src/curation/draft/records.rs:176-177 (`parts_weight`) — `change.reference_ids` costs 1 unit per byte, but every record clones the whole `ChangeSummary`, and each 1-char id clone takes about 56 bytes (24-byte String header plus the smallest malloc chunk). The 16 Mi-unit bound therefore lets roughly 0.9 GB of cloned ids through before it trips. — hint: charge each id at 64 + len, as `op_weight` already does for `Cite`, or stop cloning the summary into every record (share it, or index the change)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] resource-bound / parse amplification (pre-existing R5.3 code, outside the diff) — crates/core/src/curation/draft/schema.rs:405-412 (`parse_changeset`: Value tree, `value.clone()`, typed value, `assert_same_shape` serialising it back), plus the Change clones in pipeline.rs:147-159
|
||||
- [MEDIUM] guardrails SB-POLICY-NOT-OWNER (outside the diff) — supabase/migrations/20261008110000_anatomical_sheets.sql:1177
|
||||
- [LOW] new lint allowance (clippy, not a security suppression; same convention as the 20 existing integration test files) — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
|
||||
PROOF: semgrep 162 rules on 29 files → 0 findings; checklist 8 checks → 4 findings (1 HIGH, 2 MEDIUM, 1 LOW)
|
||||
|
||||
## Detail
|
||||
|
||||
### 1. [HIGH] The records bound does not bound memory for change reference ids
|
||||
|
||||
The bound comes from R5.4 (`records::charge`, `MAX_RECORDED_WEIGHT` = 16 Mi units). It is
|
||||
documented (records.rs:36-41, RUST.md §11.14) as keeping the records, and the reports built
|
||||
from them, small. `parts_weight` adds `change.reference_ids` as the sum of their byte lengths.
|
||||
Each record clones the change's summary, so it clones the whole id vector too. Duplicate ids
|
||||
are accepted (`non_empty_strings` does no dedup), and one declared 1-char reference id is enough.
|
||||
|
||||
Measured with a scratch harness that runs `build_equine_draft` + `render_draft_artifacts` on
|
||||
the seed's legacy sources and frozen drawing (read-only), a crafted `changes.json`, and
|
||||
`references.json` plus one reference `r`. One level A change with `reference_ids` = `["r"] × N`
|
||||
and M `set` ops on one finding (`/usr/bin/time -f %M`):
|
||||
|
||||
| changes.json | N | M | outcome | peak RSS |
|
||||
|---|---|---|---|---|
|
||||
| equine (committed) | — | — | 11 files written | 17.8 MB |
|
||||
| 1.0 MB | 200,000 | 100 | refused at op 83 by the records bound | 948 MB |
|
||||
| 2.0 MB | 400,000 | 60 | refused at op 41 by the records bound | 971 MB |
|
||||
| 8.0 MB | 2,000,000 | 8 | draft built, refused at the table bound | 1.30 GB |
|
||||
| 16.4 MB (under the 16 MiB cap) | 4,100,000 | 4 | draft built, refused at the table bound | 1.67 GB |
|
||||
|
||||
The bound trips only after about 0.9 GB is already allocated: a 1 MB input reaches 948 MB.
|
||||
That breaks the brief's criterion (peak memory well below the GB range for any input under
|
||||
the 16 MiB caps), and it breaks the R5.4 claim that the records bound keeps the reports'
|
||||
inputs small. Other record parts are charged closely enough: JSON values cost 64 units for
|
||||
roughly 32 to 110 bytes, string bytes cost 1 unit for 1 byte.
|
||||
|
||||
Fix direction: charge 64 + len per change reference id, or keep a single copy of each
|
||||
change's summary and stop cloning it into every record. Add a test with a duplicated
|
||||
reference-id vector next to `the_records_bound_holds_inside_one_op_and_the_marking`.
|
||||
|
||||
### 2. [MEDIUM] Parse-phase amplification (outside the diff)
|
||||
|
||||
`parse_changeset` on the 16.4 MB / 4.1 M-id file peaks at 1.075 GB by itself (scratch
|
||||
`parseonly` binary). That is about 65 times the input: the parsed Value tree, a clone of it
|
||||
passed to `from_value`, the typed value, and `assert_same_shape` serialising the typed value
|
||||
back. After parsing, `build_equine_draft` clones the changes again, `apply_change` builds a
|
||||
summary, and `DraftResult.changes` holds more summaries, each about 230 MB for that input.
|
||||
This code is R5.3 and does not appear in c6e0d85..HEAD, so it is reported, not blocking.
|
||||
It falls under the same "any input under the 16 MiB caps" criterion and is worth handling
|
||||
in the same fix (avoid the clone before `from_value`; compare shapes without re-serialising
|
||||
everything).
|
||||
|
||||
### Checked with no finding
|
||||
|
||||
- `--out` / `--check` file safety (import_equine.rs), by reading: refuses an empty path or
|
||||
`..`; the parent is canonicalised; a destination inside the seed is refused by prefix and by
|
||||
(dev, inode) of every seed folder; a symlinked or non-empty destination is refused; writes
|
||||
use `create_dir` + `create_new`, never overwrite, never delete; generated names must be
|
||||
plain `Normal` components (an absolute path or `..` is refused); reads are canonicalised
|
||||
into the seed, regular files only, 16 MiB cap; `--check` never lists through a link. The
|
||||
TOCTOU on a single-user machine is the declared residual (RUST.md §11.13).
|
||||
- Markdown structure and parity: the scratch differential fuzz of `format_report` against
|
||||
Prettier 3.8.3 (repo options) ran 33,000 random reports built from admitted and
|
||||
construct-prone tokens in list items, bibliography lines and table cells. 8,146 were
|
||||
admitted, 0 differed. Hand-written constructs (`[a] (b)`, `{{ }}`, `[[a]]`, `<`, `|`,
|
||||
`\"`, `\|`, setext and list markers mid-line) gave no difference. Every authored field is
|
||||
mid-line behind a generated prefix; line breaks and controls are refused in list items and
|
||||
turned into spaces in cells as the TypeScript did; code spans only carry stable ids,
|
||||
targets and field segments limited to `[A-Za-z0-9_-]`.
|
||||
- Table alignment: every Alphabetic or Numeric code point from U+00A0 to U+3FFFF in a cell
|
||||
(149,044) is aligned as Prettier aligns it, or refused (emoji candidates, `UnsupportedWidth`).
|
||||
- SVG well-formedness: attributes and text are escaped (`& < > "`, `\t \n \r` as references),
|
||||
XML-invalid code points and CR are refused before writing (`drawing::first_unwritable`),
|
||||
element names come from the reader's allowlist or are constants, and ids are stable ids.
|
||||
- Existing suites: `cargo test -p faunosteo-core --features curation --release --test
|
||||
curation_review --test curation_markdown --test curation_draft` → 30 passed.
|
||||
- Secrets, SQL concatenation, eval/innerHTML, sensitive logging, tracked `.env`: none in the
|
||||
diff (the only `println!` added prints a generated file name; `.env.example` is the
|
||||
placeholder file).
|
||||
|
||||
### Not covered by this run
|
||||
|
||||
I did not finish the bound analysis of the review drawings' markers and labels
|
||||
(`overlay.rs`, `app_review.rs` through the R3 layer). Nothing is concluded about them here.
|
||||
@@ -65,6 +65,11 @@ Fresh verifier and security on the fixed commit: below.
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-8.md`): HIGH records bound checked only after a whole op and after the B/C marking: one op records once per item, each record cloning the change texts and the op value (B add of 10,000 children + 128 KiB rationale → 1.37 GB; subtree set 64 KiB → 2.0 GB; subtree cite 4,000 refs → 2.1 GB); RUST.md "tens of MB at most" false. MEDIUM review drawings per view (≈273 KB each view, 2,000 views → 547 MB). LOW `record_weight` ignored the op for cite/rename/remove/move; LOW `TriggerNames::apply` linear view search; LOW test allows, bin allow (context).
|
||||
- Fixed: the records weight charged at each record before the change and the op are cloned (`records::charge`, `RecordParts`; `Recorder::record` and `mark_to_confirm` return the refusal; op weight for every kind; equine 1,195,000), test `the_records_bound_holds_inside_one_op_and_the_marking`; document bound test from a padded baseline (the records bound now stops a growing changeset first); review drawings projected and bounded at 64 MiB before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`), test `review_drawings_past_their_bound_are_refused_before_rendering`; RUST.md rows and residual risks rewritten exactly.
|
||||
|
||||
## Round 9
|
||||
## Round 9 on 8bdb079: security BLOCK(1) (verifier pending at fix time); `pnpm check:all` green 15:24:59
|
||||
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-9.md`): HIGH records weight undercount: the change's reference ids at one unit per byte while each cloned id costs ≈56 bytes (200,000 one-character ids × 100 set ops: 948 MB before the bound; 16.4 MB file: 1.67 GB). MEDIUM `parse_changeset` ≈65× the input (1.075 GB at 16.4 MB; R5.3, outside the diff); MEDIUM guardrails WARN in a migration outside the diff; LOW test allows. Not covered: markers and labels bound of the drawings.
|
||||
- Fixed: every cloned string (texts, ids, op strings, object keys) weighs 64 + its bytes (`records::text_weight`; equine 1,965,944, bound 8.5× above); ids case added to `the_records_bound_holds_inside_one_op_and_the_marking`; parse amplification declared in §11.15 (owner decision on a lower input cap); RUST.md row updated.
|
||||
|
||||
## Round 10
|
||||
|
||||
Fresh verifier and security on the fixed commit: below.
|
||||
@@ -147,9 +147,10 @@ impl<'a> Recorder<'a> {
|
||||
}
|
||||
}
|
||||
|
||||
/// What a record holds, in weight units: 64 per JSON value (a `Value` node costs tens of
|
||||
/// bytes) plus one per byte of every string: the values, and the change and the op, which each
|
||||
/// record clones. A rename or a remove records the whole item and one op can record once per
|
||||
/// What a record holds, in weight units, about its bytes in memory: 64 per JSON value and per
|
||||
/// string (a `Value` node or a `String` header and its allocation cost tens of bytes, so a run of
|
||||
/// one-character ids is not free) plus one per byte of text: the values, and the change and the
|
||||
/// op, which each record clones. A rename or a remove records the whole item and one op can record once per
|
||||
/// item it reaches, so the records can grow far past the document: `charge` bounds their total.
|
||||
pub fn record_weight(record: &ChangeRecord) -> usize {
|
||||
parts_weight(&RecordParts {
|
||||
@@ -173,22 +174,31 @@ fn parts_weight(parts: &RecordParts<'_>) -> usize {
|
||||
parts.target,
|
||||
parts.field.unwrap_or_default(),
|
||||
];
|
||||
let text: usize = texts.iter().map(|text| text.len()).sum::<usize>()
|
||||
+ change.reference_ids.iter().map(String::len).sum::<usize>();
|
||||
let text: usize = texts.iter().map(|text| text_weight(text)).sum::<usize>()
|
||||
+ change
|
||||
.reference_ids
|
||||
.iter()
|
||||
.map(|id| text_weight(id))
|
||||
.sum::<usize>();
|
||||
text + op_weight(parts.op) + value_weight(parts.before) + value_weight(parts.after)
|
||||
}
|
||||
|
||||
/// The op a record clones: its target, its value or its ids.
|
||||
fn op_weight(op: &Op) -> usize {
|
||||
let carried = match op {
|
||||
Op::Set(op) => op.field.len() + value_weight(&op.value),
|
||||
Op::Set(op) => text_weight(&op.field) + value_weight(&op.value),
|
||||
Op::Add(op) => value_weight(&op.value),
|
||||
Op::Rename(op) => op.id.len(),
|
||||
Op::Move(op) => op.to.len(),
|
||||
Op::Cite(op) => op.reference_ids.iter().map(|id| 64 + id.len()).sum(),
|
||||
Op::Rename(op) => text_weight(&op.id),
|
||||
Op::Move(op) => text_weight(&op.to),
|
||||
Op::Cite(op) => op.reference_ids.iter().map(|id| text_weight(id)).sum(),
|
||||
Op::Remove(_) => 0,
|
||||
};
|
||||
64 + op.target().len() + carried
|
||||
64 + text_weight(op.target()) + carried
|
||||
}
|
||||
|
||||
/// A cloned string: its header and allocation, then its bytes.
|
||||
fn text_weight(text: &str) -> usize {
|
||||
64 + text.len()
|
||||
}
|
||||
|
||||
/// The weight of a value, walked with an explicit stack (no recursion).
|
||||
@@ -200,7 +210,7 @@ fn value_weight(value: &Value) -> usize {
|
||||
match value {
|
||||
Value::String(text) => weight += text.len(),
|
||||
Value::Object(map) => {
|
||||
weight += map.keys().map(String::len).sum::<usize>();
|
||||
weight += map.keys().map(|key| text_weight(key)).sum::<usize>();
|
||||
pending.extend(map.values());
|
||||
}
|
||||
Value::Array(items) => pending.extend(items),
|
||||
|
||||
@@ -833,11 +833,24 @@ fn the_records_bound_holds_inside_one_op_and_the_marking() {
|
||||
("B", "r".repeat(8_192), json!([add])),
|
||||
("A", "Essai".to_owned(), json!([add, subtree])),
|
||||
];
|
||||
// A run of one-character reference ids weighs its headers too (security gate R5.4 round 9:
|
||||
// 200,000 ids cloned into each record reached 948 MB at one unit per byte).
|
||||
let mut cases = cases.to_vec();
|
||||
let sets: Vec<Value> = (0..100)
|
||||
.map(|i| json!({ "op": "set", "target": MOBILITY, "field": "name.en", "value": format!("v{i}") }))
|
||||
.collect();
|
||||
cases.push(("ids", "Essai".to_owned(), Value::from(sets)));
|
||||
for (level, rationale, ops) in cases {
|
||||
let ids = if level == "ids" {
|
||||
vec!["atlas"; 200_000]
|
||||
} else {
|
||||
Vec::new()
|
||||
};
|
||||
let level = if level == "ids" { "A" } else { level };
|
||||
let file = json!({ "changes": [{
|
||||
"id": "CHG-001", "level": level, "kind": "technical",
|
||||
"title": { "fr": "Essai", "en": "Test" }, "rationale": rationale,
|
||||
"reference_ids": [], "ops": ops,
|
||||
"reference_ids": ids, "ops": ops,
|
||||
}]});
|
||||
let changeset = parse_changeset(&to_js_compact(&file).unwrap()).unwrap();
|
||||
let error = apply_changeset(&frozen["baseline"], &references, &changeset.changes)
|
||||
|
||||
@@ -3204,7 +3204,7 @@ pass the guard unchanged.
|
||||
| authored text outside the allowlist: Markdown Prettier rewrites (`*x*`, `_x`, `x_**a**`, `**a****b**`, `Voir_https://…`, `a b`, a non-ASCII space, a tab, a line break in a list item, `` `` ``, `a\|b`, `[a](<b>)`, `[[ a ]]`, `*`…), the constructs refused whole (links, HTML tags, autolinks, character references) and any character outside the admitted set (pictographs, symbols, combining marks without the Alphabetic property; an alphabetic mark such as U+0345 is a letter and Prettier aligns it alike) | written, then rewritten by Prettier (escapes, `_emphasis_`, collapsed spaces, indented continuation lines, normalised links; a row with an extra cell), or kept | refused (`UnsupportedMarkdown`), even where Prettier would have kept the text |
|
||||
| a pictograph in a table cell (an emoji, and ©, ®, ™, ✓, ★, ⚠ and the other characters `text::is_emoji_candidate` lists) | measured with Prettier's tables | refused (`UnsupportedWidth`, R5.2): the set is wider than the emoji, the refusal stays loud |
|
||||
| an aligned report table past 32 Mi display columns in all (rows × padded row width; the equine CHANGELOG's widest table is about 0.5 MB): one long authored cell pads every row of its level, so an 80 KB rationale over 14,000 records asked for 1 GB (security gate R5.4 round 6) | Prettier aligned it (the same amplification) | refused (`Guard`): the CHANGELOG's tables are projected from their cells (bytes bound the display width) before the raw table is joined (test `a_changelog_whose_alignment_would_explode_is_refused_before_it_is_built`, through the pipeline), every other table in the aligner before any padded row is built (test `an_aligned_table_past_the_bound_is_refused_before_it_is_built`), import report included |
|
||||
| records weighing more than 16 Mi units in all (`records::record_weight`: 64 per JSON value plus one per string byte of the values, of the change and of the op, which each record clones; the equine records weigh 1,195,000): a rename or a remove records the whole item and one op, or the B/C marking, records once per item it reaches, so 400 renames of a 5,000-child item asked for 4.3 GB (round 7), a level B add of 10,000 children with a 128 KiB rationale 1.37 GB inside one marking, a 64 KiB value set over a subtree 2.0 GB (round 8) | recorded (the same copies) | refused at the record that passes the bound, charged before the change and the op are cloned into it (`records::charge`, tests `the_records_cannot_outgrow_their_bound`, `the_records_bound_holds_inside_one_op_and_the_marking`); the raw report rows, built from the records, are bounded with them; the 200,000-value document bound now holds the baseline (test from a padded baseline) |
|
||||
| records weighing more than 16 Mi units in all (`records::record_weight`, about the bytes in memory: 64 per JSON value and per string, plus one per byte of text, for the values, the change and the op, which each record clones; the equine records weigh 1,965,944): a rename or a remove records the whole item and one op, or the B/C marking, records once per item it reaches, so 400 renames of a 5,000-child item asked for 4.3 GB (round 7), a level B add of 10,000 children with a 128 KiB rationale 1.37 GB inside one marking, a 64 KiB value set over a subtree 2.0 GB (round 8), 200,000 one-character reference ids cloned into each record 948 MB at one unit per byte (round 9) | recorded (the same copies) | refused at the record that passes the bound, charged before the change and the op are cloned into it (`records::charge`, tests `the_records_cannot_outgrow_their_bound`, `the_records_bound_holds_inside_one_op_and_the_marking`); the raw report rows, built from the records, are bounded with them; the 200,000-value document bound now holds the baseline (test from a padded baseline) |
|
||||
| review drawings past 64 MiB in all, projected before any is rendered (two per view, each repeating the cleaned drawing; the equine draft writes about 0.55 MB): 2,000 views asked for 547 MB (security gate R5.4 round 8) | written | refused before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`, test `review_drawings_past_their_bound_are_refused_before_rendering`) |
|
||||
| an id an `add` writes (the item's, and those nested in its `children` or `categories`) or a `rename` id that is not a stable id (a region code for a region), or a field segment outside `[A-Za-z0-9_-]` (a control character, NUL included, a backtick, a `]`, a space, a non-ASCII letter) | accepted when a later op removed or renamed the item: the raw id stayed in the records and reached the reports' code spans (a line break there forged a section, security gate R5.4 round 2) | refused at the op, the check the final document applies anyway (test `ids_and_field_segments_that_could_forge_report_structure_are_refused`) |
|
||||
| a label, id or path holding a code point XML 1.0 refuses, or a carriage return, in a review drawing | written (a non-well-formed file; a raw CR read back as a line feed) | refused: the drawing is not written |
|
||||
@@ -3246,7 +3246,9 @@ decision; the record that passes the bound has been built before it is refused (
|
||||
and one clone of the op's value, at most twice the 16 MiB read cap), and a subtree set writes its
|
||||
value into each item before recording it, so the document holds at most as many copies as the
|
||||
records bound admits; the comparison of `--check` looks up each trigger's view linearly (views ×
|
||||
triggers, the views bounded through the drawings bound); and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160,
|
||||
triggers, the views bounded through the drawings bound); parsing `changes.json` (R5.3 code) holds
|
||||
about 65 times the file in memory (about 1 GB at the 16 MiB read cap, security gate R5.4 round 9),
|
||||
before any bound of the applier applies; and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160,
|
||||
U+FFA0, letters Prettier keeps, no structure); the residual risks of §11.13 stand.
|
||||
|
||||
Conditions for R5.5: the owner's validation of this checkpoint; the upload tool (`ureq`, the
|
||||
|
||||
Reference in new issue
Block a user