fix(anatomy): R5.4 records weigh every cloned string with its header
This commit is contained in:
1 parent
8bdb079f70
commit
29eb499539
5 files changed
+133
-15
No files matched your search
@@ -0,0 +1,88 @@
|
||||
SECURITY — VERDICT: BLOCK(1)
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 29 files (`git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`, HEAD 8bdb079)
|
||||
BLOCKING:
|
||||
1. [HIGH] resource-bound / records weight undercount — crates/core/src/curation/draft/records.rs:176-177 (`parts_weight`) — `change.reference_ids` costs 1 unit per byte, but every record clones the whole `ChangeSummary`, and each 1-char id clone takes about 56 bytes (24-byte String header plus the smallest malloc chunk). The 16 Mi-unit bound therefore lets roughly 0.9 GB of cloned ids through before it trips. — hint: charge each id at 64 + len, as `op_weight` already does for `Cite`, or stop cloning the summary into every record (share it, or index the change)
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] resource-bound / parse amplification (pre-existing R5.3 code, outside the diff) — crates/core/src/curation/draft/schema.rs:405-412 (`parse_changeset`: Value tree, `value.clone()`, typed value, `assert_same_shape` serialising it back), plus the Change clones in pipeline.rs:147-159
|
||||
- [MEDIUM] guardrails SB-POLICY-NOT-OWNER (outside the diff) — supabase/migrations/20261008110000_anatomical_sheets.sql:1177
|
||||
- [LOW] new lint allowance (clippy, not a security suppression; same convention as the 20 existing integration test files) — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
|
||||
PROOF: semgrep 162 rules on 29 files → 0 findings; checklist 8 checks → 4 findings (1 HIGH, 2 MEDIUM, 1 LOW)
|
||||
|
||||
## Detail
|
||||
|
||||
### 1. [HIGH] The records bound does not bound memory for change reference ids
|
||||
|
||||
The bound comes from R5.4 (`records::charge`, `MAX_RECORDED_WEIGHT` = 16 Mi units). It is
|
||||
documented (records.rs:36-41, RUST.md §11.14) as keeping the records, and the reports built
|
||||
from them, small. `parts_weight` adds `change.reference_ids` as the sum of their byte lengths.
|
||||
Each record clones the change's summary, so it clones the whole id vector too. Duplicate ids
|
||||
are accepted (`non_empty_strings` does no dedup), and one declared 1-char reference id is enough.
|
||||
|
||||
Measured with a scratch harness that runs `build_equine_draft` + `render_draft_artifacts` on
|
||||
the seed's legacy sources and frozen drawing (read-only), a crafted `changes.json`, and
|
||||
`references.json` plus one reference `r`. One level A change with `reference_ids` = `["r"] × N`
|
||||
and M `set` ops on one finding (`/usr/bin/time -f %M`):
|
||||
|
||||
| changes.json | N | M | outcome | peak RSS |
|
||||
|---|---|---|---|---|
|
||||
| equine (committed) | — | — | 11 files written | 17.8 MB |
|
||||
| 1.0 MB | 200,000 | 100 | refused at op 83 by the records bound | 948 MB |
|
||||
| 2.0 MB | 400,000 | 60 | refused at op 41 by the records bound | 971 MB |
|
||||
| 8.0 MB | 2,000,000 | 8 | draft built, refused at the table bound | 1.30 GB |
|
||||
| 16.4 MB (under the 16 MiB cap) | 4,100,000 | 4 | draft built, refused at the table bound | 1.67 GB |
|
||||
|
||||
The bound trips only after about 0.9 GB is already allocated: a 1 MB input reaches 948 MB.
|
||||
That breaks the brief's criterion (peak memory well below the GB range for any input under
|
||||
the 16 MiB caps), and it breaks the R5.4 claim that the records bound keeps the reports'
|
||||
inputs small. Other record parts are charged closely enough: JSON values cost 64 units for
|
||||
roughly 32 to 110 bytes, string bytes cost 1 unit for 1 byte.
|
||||
|
||||
Fix direction: charge 64 + len per change reference id, or keep a single copy of each
|
||||
change's summary and stop cloning it into every record. Add a test with a duplicated
|
||||
reference-id vector next to `the_records_bound_holds_inside_one_op_and_the_marking`.
|
||||
|
||||
### 2. [MEDIUM] Parse-phase amplification (outside the diff)
|
||||
|
||||
`parse_changeset` on the 16.4 MB / 4.1 M-id file peaks at 1.075 GB by itself (scratch
|
||||
`parseonly` binary). That is about 65 times the input: the parsed Value tree, a clone of it
|
||||
passed to `from_value`, the typed value, and `assert_same_shape` serialising the typed value
|
||||
back. After parsing, `build_equine_draft` clones the changes again, `apply_change` builds a
|
||||
summary, and `DraftResult.changes` holds more summaries, each about 230 MB for that input.
|
||||
This code is R5.3 and does not appear in c6e0d85..HEAD, so it is reported, not blocking.
|
||||
It falls under the same "any input under the 16 MiB caps" criterion and is worth handling
|
||||
in the same fix (avoid the clone before `from_value`; compare shapes without re-serialising
|
||||
everything).
|
||||
|
||||
### Checked with no finding
|
||||
|
||||
- `--out` / `--check` file safety (import_equine.rs), by reading: refuses an empty path or
|
||||
`..`; the parent is canonicalised; a destination inside the seed is refused by prefix and by
|
||||
(dev, inode) of every seed folder; a symlinked or non-empty destination is refused; writes
|
||||
use `create_dir` + `create_new`, never overwrite, never delete; generated names must be
|
||||
plain `Normal` components (an absolute path or `..` is refused); reads are canonicalised
|
||||
into the seed, regular files only, 16 MiB cap; `--check` never lists through a link. The
|
||||
TOCTOU on a single-user machine is the declared residual (RUST.md §11.13).
|
||||
- Markdown structure and parity: the scratch differential fuzz of `format_report` against
|
||||
Prettier 3.8.3 (repo options) ran 33,000 random reports built from admitted and
|
||||
construct-prone tokens in list items, bibliography lines and table cells. 8,146 were
|
||||
admitted, 0 differed. Hand-written constructs (`[a] (b)`, `{{ }}`, `[[a]]`, `<`, `|`,
|
||||
`\"`, `\|`, setext and list markers mid-line) gave no difference. Every authored field is
|
||||
mid-line behind a generated prefix; line breaks and controls are refused in list items and
|
||||
turned into spaces in cells as the TypeScript did; code spans only carry stable ids,
|
||||
targets and field segments limited to `[A-Za-z0-9_-]`.
|
||||
- Table alignment: every Alphabetic or Numeric code point from U+00A0 to U+3FFFF in a cell
|
||||
(149,044) is aligned as Prettier aligns it, or refused (emoji candidates, `UnsupportedWidth`).
|
||||
- SVG well-formedness: attributes and text are escaped (`& < > "`, `\t \n \r` as references),
|
||||
XML-invalid code points and CR are refused before writing (`drawing::first_unwritable`),
|
||||
element names come from the reader's allowlist or are constants, and ids are stable ids.
|
||||
- Existing suites: `cargo test -p faunosteo-core --features curation --release --test
|
||||
curation_review --test curation_markdown --test curation_draft` → 30 passed.
|
||||
- Secrets, SQL concatenation, eval/innerHTML, sensitive logging, tracked `.env`: none in the
|
||||
diff (the only `println!` added prints a generated file name; `.env.example` is the
|
||||
placeholder file).
|
||||
|
||||
### Not covered by this run
|
||||
|
||||
I did not finish the bound analysis of the review drawings' markers and labels
|
||||
(`overlay.rs`, `app_review.rs` through the R3 layer). Nothing is concluded about them here.
|
||||
@@ -65,6 +65,11 @@ Fresh verifier and security on the fixed commit: below.
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-8.md`): HIGH records bound checked only after a whole op and after the B/C marking: one op records once per item, each record cloning the change texts and the op value (B add of 10,000 children + 128 KiB rationale → 1.37 GB; subtree set 64 KiB → 2.0 GB; subtree cite 4,000 refs → 2.1 GB); RUST.md "tens of MB at most" false. MEDIUM review drawings per view (≈273 KB each view, 2,000 views → 547 MB). LOW `record_weight` ignored the op for cite/rename/remove/move; LOW `TriggerNames::apply` linear view search; LOW test allows, bin allow (context).
|
||||
- Fixed: the records weight charged at each record before the change and the op are cloned (`records::charge`, `RecordParts`; `Recorder::record` and `mark_to_confirm` return the refusal; op weight for every kind; equine 1,195,000), test `the_records_bound_holds_inside_one_op_and_the_marking`; document bound test from a padded baseline (the records bound now stops a growing changeset first); review drawings projected and bounded at 64 MiB before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`), test `review_drawings_past_their_bound_are_refused_before_rendering`; RUST.md rows and residual risks rewritten exactly.
|
||||
|
||||
## Round 9
|
||||
## Round 9 on 8bdb079: security BLOCK(1) (verifier pending at fix time); `pnpm check:all` green 15:24:59
|
||||
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-9.md`): HIGH records weight undercount: the change's reference ids at one unit per byte while each cloned id costs ≈56 bytes (200,000 one-character ids × 100 set ops: 948 MB before the bound; 16.4 MB file: 1.67 GB). MEDIUM `parse_changeset` ≈65× the input (1.075 GB at 16.4 MB; R5.3, outside the diff); MEDIUM guardrails WARN in a migration outside the diff; LOW test allows. Not covered: markers and labels bound of the drawings.
|
||||
- Fixed: every cloned string (texts, ids, op strings, object keys) weighs 64 + its bytes (`records::text_weight`; equine 1,965,944, bound 8.5× above); ids case added to `the_records_bound_holds_inside_one_op_and_the_marking`; parse amplification declared in §11.15 (owner decision on a lower input cap); RUST.md row updated.
|
||||
|
||||
## Round 10
|
||||
|
||||
Fresh verifier and security on the fixed commit: below.
|
||||
Reference in new issue
Block a user