fix(anatomy): R5.4 records charged at each record, review drawings bounded before rendering

This commit is contained in:
bchanot committed 2026-10-23 15:21:31 +02:00
1 parent 5880dfa959
commit 8bdb079f70
10 files changed
+357 -75

No files matched your search

@@ -0,0 +1,148 @@
# Security gate R5.4, round 8 (fresh): 2026-10-23
Scope: `git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`,
HEAD 5880dfa, branch `feature/rust-r5-pipeline-core`. Mode: audit (REPORT path given; this file
is the only write). Blind run: no prior verdict or finding list read.
```
SECURITY — VERDICT: BLOCK(1)
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
SCOPE: 28 files
BLOCKING:
1. [HIGH] resource-bound/records-checked-after-op — crates/core/src/curation/draft/apply.rs:140,145 (sites: apply.rs:66-96 mark_to_confirm, ops.rs:126-168 set_subtree, ops.rs:417-452 apply_cite, records.rs:61-76 Recorder::record) — the document-values and records-weight bounds only run after a whole op (and after the B/C marking); inside one op every record clones the change texts and the op value, once per item, so about 1 MB of changeset allocates GBs before any bound runs (measured: 711 KB → 1.37 GB RSS; 1.1 MB → abort at a 3 GB cap; 646 KB → 2.0 GB; 494 KB → 2.1 GB). RUST.md:3244-3245 says "tens of MB at most", which is false — hint: count the weight as each record is pushed (Recorder::record and mark_to_confirm), refuse before cloning; keep a per-op values budget for subtree writes (set/cite); or share ChangeSummary/Op (Arc) instead of cloning them; include CiteOp in record_weight; correct the RUST.md residual
REPORTED (non-blocking):
- [MEDIUM] resource-bound/review-drawings-per-view (declared residual, now measured) — crates/core/src/curation/draft/pipeline.rs:204-226
- [LOW] record_weight ignores the op cloned into each record for cite/rename/remove/move — crates/core/src/curation/draft/records.rs:125-128
- [LOW] TriggerNames::apply linear view search per trigger element (views × triggers, --check only) — crates/core/src/curation/draft/svg_compare.rs:36-50
- [LOW] new test-file lint allows `#![allow(clippy::disallowed_methods, …)]`, test crates only, same convention as 20 earlier test files, not a security-scanner suppression — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
- [LOW] pre-existing (context) `#![allow(clippy::disallowed_methods)]` in the CLI — crates/core/src/bin/import_equine.rs:17
PROOF: semgrep 162 rules on 28 files → 0 findings; checklist 15 checks → 5 findings (1 HIGH, 1 MEDIUM, 3 LOW) + 1 pre-existing LOW
```
## Blocking finding 1: the records and values bounds run after the op
What the code claims: `apply.rs` bounds the records at 16 Mi weight units, "checked after every
op, before the next one records anything". RUST.md §11.15 states the residual as "one op still
records up to about three copies of an item under the 200,000-value bound before the records
bound is checked (tens of MB at most)".
What happens. `Recorder::record` (records.rs:61-76) clones `self.change` (title, rationale,
question, reference ids) and `self.op` (including a set's or an add's value) into every record.
`assert_depth` and `assert_recorded` (apply.rs:139-140) run only once `apply_op` returns, and
`mark_to_confirm` (apply.rs:66-96) is checked only after it has pushed one record per touched
item (apply.rs:145). One op that reaches N items therefore allocates about N × (author text or
value) before any bound runs. N is bounded only by the 200,000-value document cap; the size of
the text or value is bounded only by the 16 MiB read cap.
Three paths, measured with a scratch probe built on `faunosteo-core` (`--features curation`,
the real legacy import, the committed changes and references, with one or two hostile changes
appended). Each run had `ulimit -v 3000000`:
| Path | changes.json | Peak RSS | Outcome |
|---|---|---|---|
| level B `add` of a node with 10,000 children, rationale 128 KiB → `mark_to_confirm` pushes 10,001 records, each cloning the rationale | 711 KB | 1,372,972 KB | refused afterwards ("records weigh more") |
| same, rationale 512 KiB (about 5 GB needed) | 1.1 MB | 2,996,672 KB | `memory allocation of 524288 bytes failed`, SIGABRT |
| level A add of 10,000 children, then `set` `description` (64 KiB value) with `scope: subtree` → 10,000 writes plus 10,000 records holding the value twice | 646 KB | 2,027,252 KB | refused afterwards |
| add of 2,000 children, then `cite` 4,000 declared references with `scope: subtree` (references.json 150 KB) → 8 M values written before `assert_depth` | 344 KB | 2,112,620 KB | refused afterwards ("more than 200000 values") |
The cost grows linearly in N × size, so a changeset of a few MB asks for tens to hundreds of
GB. The documented "tens of MB at most" is wrong, and the `mark_to_confirm` path is declared
nowhere. Impact is availability only: a hostile or careless `changes.json` run through
`pnpm r5:import-equine --check|--out`, or through the test suites, exhausts the memory of the
developer machine. The OOM killer may then take down other processes, such as the Docker
Supabase stack. No integrity or confidentiality impact. It is blocking because it breaks a
bound that both the code and the threat model of this gate require, and because the
documented residual misstates it.
Fix direction:
- Enforce the budget at push time: route `mark_to_confirm` through the same accounting, and
make `Recorder::record` refuse once the running weight would pass the bound.
- Bound the per-op writes of subtree `set` and `cite` (items × value size, or a running values
budget) before writing.
- Alternatively, stop cloning: share `ChangeSummary` and `Op` across records (`Arc`), so a
record weighs only its own before and after.
- Count the cite op's `reference_ids` in `record_weight`.
- Correct RUST.md §11.15 and add a test per path (marking, subtree set, subtree cite) that
fails before any GB-scale allocation.
Probe source: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/sec8/probe/src/main.rs`
(modes `add <rationale bytes> <children>`, `value2 <value bytes> <children>`,
`cite <references> <children>`).
## Reported findings
MEDIUM, review drawings per view (declared in RUST.md §11.15, now measured). Each view of the
curated document yields `review/annotations-<view>.svg` and `review/app-<view>.svg`. Each file
holds a full copy of the cleaned drawing, and all files sit in memory before the CLI writes
them. Level A adds of a detail view plus its trigger: 500 views → 1,011 files, 138.8 MB,
176 MB RSS; 2,000 views → 4,011 files, 547 MB, 638 MB RSS. That is about 273 KB per view, so
about 1.3 GB at the records bound (about 4,800 views), consistent with the declared "about
2 GB" estimate. The multiplier is the drawing's size. The frozen `horse.svg` is 130 KB, but
the core reader admits up to 2 MiB, so a replaced drawing would scale the same views to about
19 GB. Not blocking because it is declared and the estimate holds. A view-count or output-bytes
bound would close it.
LOW findings:
- `record_weight` counts the op only for `set` and `add` (records.rs:125-128), but every record
clones the op. A cite's `reference_ids` list is copied uncounted into each record, so the
weight under-counts by about a third on that path.
- `TriggerNames::apply` searches the view list linearly for every element carrying
`data-anno-opens-view` (views × triggers). This runs in `--check` only, on committed files,
in seconds at the bounds.
- The two new test files carry the crate's usual test allow (`disallowed_methods`, `unwrap`,
`expect`) for fixture reads. They are test crates only and do not suppress a security scanner.
- Context, outside the diff: `clippy.toml`'s `disallowed-methods` bans `fs::read`,
`fs::write`, `File::open|create` and `env::var`. It does not ban `OpenOptions::open`,
`fs::create_dir`, `fs::read_dir`, `fs::remove_*` or `fs::rename`, so "the core has no I/O"
rests on review for those. A grep of `crates/core/src/curation` finds no `std::fs`,
`std::env`, `std::process` or clock use today.
## What was checked and held
- Markdown guard, differential against Prettier 3.8.3: `report_guard::format_report` against
the repository's Prettier (`parser: markdown`, the repository options), on raw reports in the
generator's three line forms: the list item of `change_line`, the 8-column CHANGELOG row via
`cell()`, and the bibliography line. 16,395 cases (hand-picked cell-start and line constructs,
CJK, Hangul, Arabic with harakat, Devanagari and Thai marks, URLs with `_` and `&name=`,
`{{ }}`, wiki and strong forms, plus three seeded random sets). 4,696 were admitted. The only
3 differences were wiki links opened in one list-item field and closed in another across code
spans. `format_report` was called directly there; in the pipeline,
`review_report::assert_single_line` refuses `[[`/`]]` in every list-item field first. Table
cells parse on their own and showed no difference. No silent difference is reachable in
this sample.
- Structure forging: line breaks never reach a line start. List-item fields refuse control
characters, and `cell()` maps `\n` to a space. `\\|` (an authored `\|` after escaping) is
refused. Code spans are limited to `[A-Za-z0-9_:/. -]`, authored backticks are refused, and
field segments are limited to `[A-Za-z0-9_-]`. Ids written by add and rename must be stable
ids at the op.
- SVG: hostile annotation and view names (`<a href>`, `&amp;`, `&#x41;`, `]]>`, `<!-- -->`,
`<?pi?>`, quotes, U+0085, U+2028, NBSP, tab, newline) and a path `d` holding
`" onload="x` produce four well-formed drawings (xml.dom.minidom). The values stay data,
with no `onload` attribute and no `a` or `script` element. Non-XML characters and CR are
refused (`drawing_text`). The cleaned drawing goes through the core reader first: 2 MiB,
20,000 elements, depth 64, DOCTYPE and ENTITY refused. `compare_drawing` parses the committed
file with the capped reader before roxmltree and requires a byte-exact rebuild.
- CLI file safety: `--check` passes at HEAD (legacy-import 6/0, draft 12/0, app-main declared).
`--out` refusals were run in a scratch folder: empty path, symlink, symlink with a trailing
`/`, non-empty folder, `..`, a destination through a symlink into the seed or into `draft/`,
a missing parent. A fresh folder gets 18 files, and a second run refuses "not empty". Writes
are `create_new`, folders are `create_dir` and components must be `Normal`. Reads are
canonical, confined to the seed, regular files only, capped at 16 MiB. Same-user TOCTOU is
declared. The working tree stayed clean.
- Quadratic scans: the guard is linear (the `&`, `<`, `[[` and URL scans work per segment).
`underscore_pairs_later` rescans only from the few authored plain parts. The renderers are
hashed. No input under the caps reached a measurable guard time (max 0 ms per case).
- Checklist: no hardcoded secret, token or auth URL in the diff or the oracles. No SQL built in
scope; `seed.rs` is unchanged since R5.3. No HTML render, endpoint, authz surface or logging
in scope. The CLI's error messages go to the local operator only. The only tracked env file
is `.env.example`; reading it was refused by the permission rules (rule: Bash `cat .env.example`
denied), it was not retried another way, and it is outside the diff. Guardrails scanner
(`--domain api,supabase`): 0 FAIL, 1 WARN, `SB-POLICY-NOT-OWNER`
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the diff and dropped
per the diff-scope rule, plus 1 MANUAL.
- Anti-gaming: no `nosemgrep`, `nosec`, `eslint-disable` or `unsafe` added. The only new allows
are the two test-file lint allows listed above.
Semgrep note: the three packs carry few Rust rules (162 rules loaded on 28 files, 0 findings),
so semgrep alone says little about this diff. The verdict rests on the checklist and the
probes.
@@ -60,6 +60,11 @@ Fresh verifier and security on the fixed commit: below.
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-7.md`): HIGH output amplification: the CHANGELOG raw table (rows × authored rationale) built whole before the round-6 bound (1.9 MB changes.json aborts at 3 GB), RUST.md "refused before any row is built" false; HIGH record amplification: rename/remove records deep-copy the whole item and clone the change texts, no total bound (400 renames of a 5,000-child item → 4.3 GB). LOW test allows, bin allow (context), error echo size, subfolder race (declared out of model).
- Fixed: records bounded in total (`apply::MAX_RECORDED_WEIGHT` 16 Mi units, `records::record_weight`, checked after every op; equine 1,084,422), test `the_records_cannot_outgrow_their_bound`; CHANGELOG tables projected from their cells before the raw table is joined (`markdown::assert_table_bound`), pipeline-level test `a_changelog_whose_alignment_would_explode_is_refused_before_it_is_built`; RUST.md rows rewritten exactly, residual risk (one op's copies before the check).
## Round 8
## Round 8 on 5880dfa: security BLOCK(1) (verifier pending at fix time); `pnpm check:all` green 15:02:04
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-8.md`): HIGH records bound checked only after a whole op and after the B/C marking: one op records once per item, each record cloning the change texts and the op value (B add of 10,000 children + 128 KiB rationale → 1.37 GB; subtree set 64 KiB → 2.0 GB; subtree cite 4,000 refs → 2.1 GB); RUST.md "tens of MB at most" false. MEDIUM review drawings per view (≈273 KB each view, 2,000 views → 547 MB). LOW `record_weight` ignored the op for cite/rename/remove/move; LOW `TriggerNames::apply` linear view search; LOW test allows, bin allow (context).
- Fixed: the records weight charged at each record before the change and the op are cloned (`records::charge`, `RecordParts`; `Recorder::record` and `mark_to_confirm` return the refusal; op weight for every kind; equine 1,195,000), test `the_records_bound_holds_inside_one_op_and_the_marking`; document bound test from a padded baseline (the records bound now stops a growing changeset first); review drawings projected and bounded at 64 MiB before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`), test `review_drawings_past_their_bound_are_refused_before_rendering`; RUST.md rows and residual risks rewritten exactly.
## Round 9
Fresh verifier and security on the fixed commit: below.
+19 -35
View File
@@ -15,7 +15,7 @@ use serde_json::Value;
use super::ops::{apply_op, assert_declared};
use super::records::{
ChangeRecord, CurationResult, Recorder, record_weight, strip_marks, touch_order,
ChangeRecord, CurationResult, RecordParts, Recorder, charge, strip_marks, touch_order,
};
use super::schema::{Change, ChangeSummary, Op, SetOp};
use super::shape::assert_same_shape;
@@ -67,6 +67,7 @@ fn mark_to_confirm(
doc: &mut Value,
change: &ChangeSummary,
records: &mut Vec<ChangeRecord>,
spent: &mut usize,
) -> Result<(), PipelineError> {
for touched in touched_reviewable(doc)? {
let item = item_at_mut(doc, &touched.path)?;
@@ -85,64 +86,47 @@ fn mark_to_confirm(
value: Value::from("to_confirm"),
scope: None,
});
let after = Value::from("to_confirm");
let parts = RecordParts {
change,
op: &op,
target: &touched.target,
field: Some("review_status"),
before: &before,
after: &after,
};
charge(spent, parts)?;
records.push(ChangeRecord {
change: change.clone(),
op,
target: touched.target,
field: Some("review_status".to_owned()),
before,
after: Value::from("to_confirm"),
after,
});
}
Ok(())
}
/// The total weight of the records (`records::record_weight`): about 16 MiB of text, or a
/// quarter of a million JSON values, far past the equine changeset (test
/// `the_records_cannot_outgrow_their_bound`) and small enough that the reports built from them
/// stay bounded too. Checked after every op, before the next one records anything.
pub const MAX_RECORDED_WEIGHT: usize = 16 * 1024 * 1024;
/// The records counted so far and their weight.
#[derive(Default)]
struct Tally {
counted: usize,
weight: usize,
}
fn assert_recorded(records: &[ChangeRecord], tally: &mut Tally) -> Result<(), PipelineError> {
for record in &records[tally.counted..] {
tally.weight = tally.weight.saturating_add(record_weight(record));
}
tally.counted = records.len();
if tally.weight > MAX_RECORDED_WEIGHT {
return Err(fail(format!(
"the records weigh more than {MAX_RECORDED_WEIGHT} units"
)));
}
Ok(())
}
fn apply_change(
doc: &mut Value,
change: &Change,
records: &mut Vec<ChangeRecord>,
tally: &mut Tally,
spent: &mut usize,
) -> Result<(), PipelineError> {
let summary = change.summary();
assert_declared(doc, &change.reference_ids)
.map_err(|error| fail(format!("{}: {error}", change.id)))?;
let mut next_touch = 0;
for (index, op) in change.ops.iter().enumerate() {
let mut recorder = Recorder::new(&summary, op, records, &mut next_touch);
let mut recorder = Recorder::new(&summary, op, records, (&mut next_touch, &mut *spent));
apply_op(doc, &mut recorder)
.and_then(|()| assert_depth(doc))
.and_then(|()| assert_recorded(records, tally))
.map_err(|error| fail(format!("{} op {index}: {error}", change.id)))?;
}
if change.level.needs_confirmation() {
mark_to_confirm(doc, &summary, records)?;
assert_recorded(records, tally).map_err(|error| fail(format!("{}: {error}", change.id)))?;
mark_to_confirm(doc, &summary, records, spent)
.map_err(|error| fail(format!("{}: {error}", change.id)))?;
}
strip_marks(doc);
Ok(())
@@ -235,9 +219,9 @@ pub fn apply_changeset(
.ok_or_else(|| fail("the baseline is not an object"))?
.insert("references".to_owned(), registry);
let mut records = Vec::new();
let mut tally = Tally::default();
let mut spent = 0;
for change in changes {
apply_change(&mut doc, change, &mut records, &mut tally)?;
apply_change(&mut doc, change, &mut records, &mut spent)?;
}
curated_issues(&doc)?;
Ok(CurationResult {
+7 -7
View File
@@ -118,7 +118,7 @@ fn set_self(
recorder.touch(item);
let before = read_field(item, &op.field)?;
write_field(item, &op.field, &op.value)?;
recorder.record(&op.target, Some(&op.field), before, Some(op.value.clone()));
recorder.record(&op.target, Some(&op.field), before, Some(op.value.clone()))?;
Ok(())
}
@@ -163,7 +163,7 @@ fn set_subtree(
Some(&op.field),
before,
Some(op.value.clone()),
);
)?;
}
Ok(())
}
@@ -199,7 +199,7 @@ fn apply_rename(
item.insert(key.to_owned(), Value::from(op.id.clone()));
recorder.touch(item);
let after = Value::Object(item.clone());
recorder.record(&op.target, None, Some(before), Some(after));
recorder.record(&op.target, None, Some(before), Some(after))?;
let rename = Rename {
kind: located.target.kind,
from,
@@ -331,7 +331,7 @@ fn apply_add(
added.push(Step::Index(index));
recorder.touch_tree(doc, &added)?;
let target = format!("{}{id}", dest.prefix);
recorder.record(&target, None, None, Some(op.value.clone()));
recorder.record(&target, None, None, Some(op.value.clone()))?;
Ok(())
}
@@ -345,7 +345,7 @@ fn apply_remove(
return Err(fail("the document cannot be removed"));
};
let item = list_at_mut(doc, list)?.remove(located.index);
recorder.record(&op.target, None, Some(item), None);
recorder.record(&op.target, None, Some(item), None)?;
Ok(())
}
@@ -383,7 +383,7 @@ fn apply_move(
recorder.touch(item);
let before = Value::Object(item.clone());
let after = format!("{}{id}", child_prefix(&parent));
recorder.record(&op.target, None, Some(before), Some(Value::from(after)));
recorder.record(&op.target, None, Some(before), Some(Value::from(after)))?;
Ok(())
}
@@ -449,7 +449,7 @@ fn apply_cite(
Some("reference_ids"),
Some(Value::Array(before)),
Some(Value::Array(after)),
);
)?;
}
Ok(())
}
+14 -1
View File
@@ -199,13 +199,26 @@ fn view_ids(document: &Value) -> Vec<String> {
.collect()
}
/// The review drawings' bytes, projected before any is rendered: every drawing repeats the
/// whole cleaned drawing, two per view (the equine draft: 2 views, about 0.55 MB). Views added
/// by a changeset multiply it (2,000 views asked for 547 MB, security gate R5.4 round 8).
pub const MAX_REVIEW_DRAWINGS_BYTES: usize = 64 * 1024 * 1024;
/// One review overlay per view (`review/annotations-<view>.svg`), then the application's
/// drawing of each view (`review/app-<view>.svg`).
fn review_drawings(draft: &DraftResult) -> Result<Vec<(String, String)>, PipelineError> {
let views = view_ids(&draft.document);
let projected = draft.cleaned_svg.len().saturating_mul(2 * views.len());
if projected > MAX_REVIEW_DRAWINGS_BYTES {
return Err(fail(format!(
"{} views would write {projected} bytes of review drawings, past the \
{MAX_REVIEW_DRAWINGS_BYTES} bound",
views.len()
)));
}
let drawing = read_cleaned_drawing(&draft.cleaned_svg)?;
let typed: AnatomicalSheetDocument = serde_json::from_value(draft.document.clone())
.map_err(|error| fail(format!("curated document: {error}")))?;
let views = view_ids(&draft.document);
let mut files = Vec::new();
for view_id in &views {
let overlay = render_review_overlay(&OverlayInput {
+78 -17
View File
@@ -33,6 +33,34 @@ pub struct CurationResult {
pub records: Vec<ChangeRecord>,
}
/// The total weight of the records (`record_weight`): about 16 MiB of text, or a quarter of a
/// million JSON values, far past the equine changeset (test `the_records_cannot_outgrow_their_bound`)
/// and small enough that the reports built from them stay bounded too. Charged as each record
/// is made, before the change and the op are cloned into it: one op touching many items is
/// refused at the record that passes the bound.
pub const MAX_RECORDED_WEIGHT: usize = 16 * 1024 * 1024;
/// The records' running weight; refuses the record that would pass `MAX_RECORDED_WEIGHT`.
pub fn charge(spent: &mut usize, record: RecordParts<'_>) -> Result<(), PipelineError> {
*spent = spent.saturating_add(parts_weight(&record));
if *spent > MAX_RECORDED_WEIGHT {
return Err(PipelineError::Guard(format!(
"the records weigh more than {MAX_RECORDED_WEIGHT} units"
)));
}
Ok(())
}
/// What a record is made of, borrowed: weighed before anything is cloned.
pub struct RecordParts<'a> {
pub change: &'a ChangeSummary,
pub op: &'a Op,
pub target: &'a str,
pub field: Option<&'a str>,
pub before: &'a Value,
pub after: &'a Value,
}
/// What the op functions of one change share besides the document; the touch counter spans
/// the ops of the change (the first touch of an item fixes its place in the review marking).
pub struct Recorder<'a> {
@@ -40,6 +68,7 @@ pub struct Recorder<'a> {
pub op: &'a Op,
pub records: &'a mut Vec<ChangeRecord>,
next_touch: &'a mut u64,
spent: &'a mut usize,
}
impl<'a> Recorder<'a> {
@@ -47,32 +76,45 @@ impl<'a> Recorder<'a> {
change: &'a ChangeSummary,
op: &'a Op,
records: &'a mut Vec<ChangeRecord>,
next_touch: &'a mut u64,
counters: (&'a mut u64, &'a mut usize),
) -> Self {
Self {
change,
op,
records,
next_touch,
next_touch: counters.0,
spent: counters.1,
}
}
/// Absent values are recorded as null so records stay plain JSON.
/// Absent values are recorded as null so records stay plain JSON. Refused when the
/// records would pass their bound (`charge`), before the change and the op are cloned.
pub fn record(
&mut self,
target: &str,
field: Option<&str>,
before: Option<Value>,
after: Option<Value>,
) {
) -> Result<(), PipelineError> {
let (before, after) = (unmarked(before), unmarked(after));
let parts = RecordParts {
change: self.change,
op: self.op,
target,
field,
before: &before,
after: &after,
};
charge(self.spent, parts)?;
self.records.push(ChangeRecord {
change: self.change.clone(),
op: self.op.clone(),
target: target.to_owned(),
field: field.map(str::to_owned),
before: unmarked(before),
after: unmarked(after),
before,
after,
});
Ok(())
}
/// Marks an item as touched by the change, once (the first touch keeps its order).
@@ -105,29 +147,48 @@ impl<'a> Recorder<'a> {
}
}
/// What the records hold, in weight units: 64 per JSON value (a `Value` node costs tens of
/// bytes) plus one per byte of every string (the values' and the change's texts, which each
/// record clones). A rename or a remove records the whole item, so the records can grow far
/// past the document; `apply` bounds their total (`MAX_RECORDED_WEIGHT`).
/// What a record holds, in weight units: 64 per JSON value (a `Value` node costs tens of
/// bytes) plus one per byte of every string: the values, and the change and the op, which each
/// record clones. A rename or a remove records the whole item and one op can record once per
/// item it reaches, so the records can grow far past the document: `charge` bounds their total.
pub fn record_weight(record: &ChangeRecord) -> usize {
let change = &record.change;
parts_weight(&RecordParts {
change: &record.change,
op: &record.op,
target: &record.target,
field: record.field.as_deref(),
before: &record.before,
after: &record.after,
})
}
fn parts_weight(parts: &RecordParts<'_>) -> usize {
let change = parts.change;
let texts = [
change.id.as_str(),
&change.title.fr,
&change.title.en,
&change.rationale,
change.question.as_deref().unwrap_or_default(),
&record.target,
record.field.as_deref().unwrap_or_default(),
parts.target,
parts.field.unwrap_or_default(),
];
let text: usize = texts.iter().map(|text| text.len()).sum::<usize>()
+ change.reference_ids.iter().map(String::len).sum::<usize>();
let op_value = match &record.op {
Op::Set(op) => value_weight(&op.value),
text + op_weight(parts.op) + value_weight(parts.before) + value_weight(parts.after)
}
/// The op a record clones: its target, its value or its ids.
fn op_weight(op: &Op) -> usize {
let carried = match op {
Op::Set(op) => op.field.len() + value_weight(&op.value),
Op::Add(op) => value_weight(&op.value),
_ => 0,
Op::Rename(op) => op.id.len(),
Op::Move(op) => op.to.len(),
Op::Cite(op) => op.reference_ids.iter().map(|id| 64 + id.len()).sum(),
Op::Remove(_) => 0,
};
text + op_value + value_weight(&record.before) + value_weight(&record.after)
64 + op.target().len() + carried
}
/// The weight of a value, walked with an explicit stack (no recursion).
+2 -2
View File
@@ -74,7 +74,7 @@ fn follow_link(
Some(link.field),
Some(from.clone()),
Some(Value::from(rename.to.as_str())),
);
)?;
}
Ok(())
}
@@ -109,7 +109,7 @@ fn follow_citations(
Some("reference_ids"),
Some(before),
Some(Value::Array(after.clone())),
);
)?;
item.insert("reference_ids".to_owned(), Value::Array(after));
}
Ok(())
+55 -7
View File
@@ -764,14 +764,26 @@ fn an_integer_beyond_2_53_is_read_as_the_double_json_parse_gives() {
);
}
/// The records bound now stops a growing changeset first (every added value is recorded); the
/// document bound still holds a baseline near it: one small add past 200,000 values is refused.
#[test]
fn the_document_cannot_hold_more_values_than_the_bound() {
let children: Vec<Value> = (0..70_000)
.map(|i| json!({ "id": format!("n{i}"), "name": { "fr": "a", "en": "a" } }))
.collect();
let value = json!({ "id": "many", "name": { "fr": "a", "en": "a" }, "children": children });
let op = json!({ "op": "add", "target": MOBILITY, "value": value });
let error = mini_apply(json!([op])).unwrap_err();
let frozen = oracle("curation-synthetic.json");
let references = parse_references(&to_js_compact(&frozen["references_file"]).unwrap()).unwrap();
let mut baseline = frozen["baseline"].clone();
baseline["padding"] = Value::from(vec![Value::Null; 199_950]);
let value = json!({ "id": "small", "name": { "fr": "a", "en": "a" },
"children": (0..20).map(|i| json!({ "id": format!("n{i}"), "name": { "fr": "a", "en": "a" } }))
.collect::<Vec<_>>() });
let file = json!({ "changes": [{
"id": "CHG-001", "level": "A", "kind": "technical",
"title": { "fr": "Essai", "en": "Test" }, "rationale": "Essai",
"reference_ids": [], "ops": [{ "op": "add", "target": MOBILITY, "value": value }],
}]});
let changeset = parse_changeset(&to_js_compact(&file).unwrap()).unwrap();
let error = apply_changeset(&baseline, &references, &changeset.changes)
.unwrap_err()
.to_string();
assert!(error.contains("more than 200000 values"), "{error}");
}
@@ -780,7 +792,7 @@ fn the_document_cannot_hold_more_values_than_the_bound() {
/// 5,000-child item asked for 4.3 GB). The equine changeset stays far below the bound.
#[test]
fn the_records_cannot_outgrow_their_bound() {
use faunosteo_core::curation::draft::apply::MAX_RECORDED_WEIGHT;
use faunosteo_core::curation::draft::records::MAX_RECORDED_WEIGHT;
use faunosteo_core::curation::draft::records::record_weight;
let equine: usize = equine_draft().records.iter().map(record_weight).sum();
println!("equine records weigh {equine} units");
@@ -802,6 +814,42 @@ fn the_records_cannot_outgrow_their_bound() {
assert!(error.contains("the records weigh more than"), "{error}");
}
/// One op, or the B/C marking, records once per item it reaches: the bound is charged at each
/// record, before the change and the op are cloned into it (security gate R5.4 round 8: a level
/// B add of 10,000 children with a 128 KiB rationale reached 1.37 GB inside one marking).
#[test]
fn the_records_bound_holds_inside_one_op_and_the_marking() {
let children: Vec<Value> = (0..5_000)
.map(|i| json!({ "id": format!("n{i}"), "name": { "fr": "a", "en": "a" } }))
.collect();
let value = json!({ "id": "many", "name": { "fr": "a", "en": "a" }, "children": children });
let add = json!({ "op": "add", "target": MOBILITY, "value": value });
let frozen = oracle("curation-synthetic.json");
let references = parse_references(&to_js_compact(&frozen["references_file"]).unwrap()).unwrap();
let description = json!({ "fr": "d".repeat(4_096), "en": "d" });
let subtree = json!({ "op": "set", "target": format!("{MOBILITY}/many"), "field": "description",
"value": description, "scope": "subtree" });
let cases = [
("B", "r".repeat(8_192), json!([add])),
("A", "Essai".to_owned(), json!([add, subtree])),
];
for (level, rationale, ops) in cases {
let file = json!({ "changes": [{
"id": "CHG-001", "level": level, "kind": "technical",
"title": { "fr": "Essai", "en": "Test" }, "rationale": rationale,
"reference_ids": [], "ops": ops,
}]});
let changeset = parse_changeset(&to_js_compact(&file).unwrap()).unwrap();
let error = apply_changeset(&frozen["baseline"], &references, &changeset.changes)
.unwrap_err()
.to_string();
assert!(
error.contains("the records weigh more than"),
"{level}: {error}"
);
}
}
/// A `__proto__` segment had no faithful reading in the TypeScript (a prototype set, a
/// polluted `Object.prototype`, or nothing written): the port refuses it at the op.
#[test]
+21
View File
@@ -496,3 +496,24 @@ fn ids_and_field_segments_that_could_forge_report_structure_are_refused() {
let rule = TriggerNames::of(&typed(&draft.document));
assert!(compare_drawing("<svg/>", &deep, Some(&rule)).is_err());
}
/// Every review drawing repeats the whole cleaned drawing, two per view: the bytes are
/// projected and bounded before any drawing is rendered (security gate R5.4 round 8).
#[test]
fn review_drawings_past_their_bound_are_refused_before_rendering() {
let (result, mut draft) = equine();
let views = draft.document["views"].as_array_mut().unwrap();
let main = views[0].clone();
for index in 0..300 {
let mut view = main.clone();
view["id"] = Value::from(format!("extra-{index}"));
views.push(view);
}
let error = render_draft_artifacts(&draft, &result.anomalies).unwrap_err();
assert!(
error
.to_string()
.contains("bytes of review drawings, past the"),
"{error}"
);
}
+7 -5
View File
@@ -3204,7 +3204,8 @@ pass the guard unchanged.
| authored text outside the allowlist: Markdown Prettier rewrites (`*x*`, `_x`, `x_**a**`, `**a****b**`, `Voir_https://…`, `a b`, a non-ASCII space, a tab, a line break in a list item, `` `` ``, `a\|b`, `[a](<b>)`, `[[ a ]]`, `&#42;`…), the constructs refused whole (links, HTML tags, autolinks, character references) and any character outside the admitted set (pictographs, symbols, combining marks without the Alphabetic property; an alphabetic mark such as U+0345 is a letter and Prettier aligns it alike) | written, then rewritten by Prettier (escapes, `_emphasis_`, collapsed spaces, indented continuation lines, normalised links; a row with an extra cell), or kept | refused (`UnsupportedMarkdown`), even where Prettier would have kept the text |
| a pictograph in a table cell (an emoji, and ©, ®, ™, ✓, ★, ⚠ and the other characters `text::is_emoji_candidate` lists) | measured with Prettier's tables | refused (`UnsupportedWidth`, R5.2): the set is wider than the emoji, the refusal stays loud |
| an aligned report table past 32 Mi display columns in all (rows × padded row width; the equine CHANGELOG's widest table is about 0.5 MB): one long authored cell pads every row of its level, so an 80 KB rationale over 14,000 records asked for 1 GB (security gate R5.4 round 6) | Prettier aligned it (the same amplification) | refused (`Guard`): the CHANGELOG's tables are projected from their cells (bytes bound the display width) before the raw table is joined (test `a_changelog_whose_alignment_would_explode_is_refused_before_it_is_built`, through the pipeline), every other table in the aligner before any padded row is built (test `an_aligned_table_past_the_bound_is_refused_before_it_is_built`), import report included |
| records weighing more than 16 Mi units in all (`records::record_weight`: 64 per JSON value plus one per string byte of the values and of the change's texts, which each record clones; the equine records weigh 1,084,422): a rename or a remove records the whole item, so 400 renames of a 5,000-child item asked for 4.3 GB, and an 80 KB rationale repeated over 14,000 records for 1.1 GB of raw rows (security gate R5.4 round 7) | recorded (the same copies) | refused at the op that passes the bound, before the next op records anything (test `the_records_cannot_outgrow_their_bound`); the raw report rows, built from the records, are bounded with them |
| records weighing more than 16 Mi units in all (`records::record_weight`: 64 per JSON value plus one per string byte of the values, of the change and of the op, which each record clones; the equine records weigh 1,195,000): a rename or a remove records the whole item and one op, or the B/C marking, records once per item it reaches, so 400 renames of a 5,000-child item asked for 4.3 GB (round 7), a level B add of 10,000 children with a 128 KiB rationale 1.37 GB inside one marking, a 64 KiB value set over a subtree 2.0 GB (round 8) | recorded (the same copies) | refused at the record that passes the bound, charged before the change and the op are cloned into it (`records::charge`, tests `the_records_cannot_outgrow_their_bound`, `the_records_bound_holds_inside_one_op_and_the_marking`); the raw report rows, built from the records, are bounded with them; the 200,000-value document bound now holds the baseline (test from a padded baseline) |
| review drawings past 64 MiB in all, projected before any is rendered (two per view, each repeating the cleaned drawing; the equine draft writes about 0.55 MB): 2,000 views asked for 547 MB (security gate R5.4 round 8) | written | refused before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`, test `review_drawings_past_their_bound_are_refused_before_rendering`) |
| an id an `add` writes (the item's, and those nested in its `children` or `categories`) or a `rename` id that is not a stable id (a region code for a region), or a field segment outside `[A-Za-z0-9_-]` (a control character, NUL included, a backtick, a `]`, a space, a non-ASCII letter) | accepted when a later op removed or renamed the item: the raw id stayed in the records and reached the reports' code spans (a line break there forged a section, security gate R5.4 round 2) | refused at the op, the check the final document applies anyway (test `ids_and_field_segments_that_could_forge_report_structure_are_refused`) |
| a label, id or path holding a code point XML 1.0 refuses, or a carriage return, in a review drawing | written (a non-well-formed file; a raw CR read back as a line feed) | refused: the drawing is not written |
@@ -3241,10 +3242,11 @@ ruled and frozen in the sweep, so a combination of admitted characters no probe
differ; the comparison of the application's drawings accepts
exactly the trigger-name rule, byte for byte; the frozen
`horse.svg` is read, never regenerated, so a future change of the drawing needs its own
decision; one op still records up to about three copies of an item under the 200,000-value bound
before the records bound is checked (tens of MB at most); the review drawings are bounded only through the document's 200,000 values (two drawings
of about 137 KB per view, about 2 GB at the bound, an estimate of the security gate R5.4 round 6,
not measured), and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160,
decision; the record that passes the bound has been built before it is refused (one item copy
and one clone of the op's value, at most twice the 16 MiB read cap), and a subtree set writes its
value into each item before recording it, so the document holds at most as many copies as the
records bound admits; the comparison of `--check` looks up each trigger's view linearly (views ×
triggers, the views bounded through the drawings bound); and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160,
U+FFA0, letters Prettier keeps, no structure); the residual risks of §11.13 stand.
Conditions for R5.5: the owner's validation of this checkpoint; the upload tool (`ureq`, the