diff --git a/.claude/audits/2026-10-23-rust-r5-review-security-8.md b/.claude/audits/2026-10-23-rust-r5-review-security-8.md new file mode 100644 index 00000000..fb5ac9fd --- /dev/null +++ b/.claude/audits/2026-10-23-rust-r5-review-security-8.md @@ -0,0 +1,148 @@ +# Security gate R5.4, round 8 (fresh): 2026-10-23 + +Scope: `git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`, +HEAD 5880dfa, branch `feature/rust-r5-pipeline-core`. Mode: audit (REPORT path given; this file +is the only write). Blind run: no prior verdict or finding list read. + +``` +SECURITY — VERDICT: BLOCK(1) +TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten +SCOPE: 28 files +BLOCKING: + 1. [HIGH] resource-bound/records-checked-after-op — crates/core/src/curation/draft/apply.rs:140,145 (sites: apply.rs:66-96 mark_to_confirm, ops.rs:126-168 set_subtree, ops.rs:417-452 apply_cite, records.rs:61-76 Recorder::record) — the document-values and records-weight bounds only run after a whole op (and after the B/C marking); inside one op every record clones the change texts and the op value, once per item, so about 1 MB of changeset allocates GBs before any bound runs (measured: 711 KB → 1.37 GB RSS; 1.1 MB → abort at a 3 GB cap; 646 KB → 2.0 GB; 494 KB → 2.1 GB). RUST.md:3244-3245 says "tens of MB at most", which is false — hint: count the weight as each record is pushed (Recorder::record and mark_to_confirm), refuse before cloning; keep a per-op values budget for subtree writes (set/cite); or share ChangeSummary/Op (Arc) instead of cloning them; include CiteOp in record_weight; correct the RUST.md residual +REPORTED (non-blocking): + - [MEDIUM] resource-bound/review-drawings-per-view (declared residual, now measured) — crates/core/src/curation/draft/pipeline.rs:204-226 + - [LOW] record_weight ignores the op cloned into each record for cite/rename/remove/move — crates/core/src/curation/draft/records.rs:125-128 + - [LOW] TriggerNames::apply linear view search per trigger element (views × triggers, --check only) — crates/core/src/curation/draft/svg_compare.rs:36-50 + - [LOW] new test-file lint allows `#![allow(clippy::disallowed_methods, …)]`, test crates only, same convention as 20 earlier test files, not a security-scanner suppression — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6 + - [LOW] pre-existing (context) `#![allow(clippy::disallowed_methods)]` in the CLI — crates/core/src/bin/import_equine.rs:17 +PROOF: semgrep 162 rules on 28 files → 0 findings; checklist 15 checks → 5 findings (1 HIGH, 1 MEDIUM, 3 LOW) + 1 pre-existing LOW +``` + +## Blocking finding 1: the records and values bounds run after the op + +What the code claims: `apply.rs` bounds the records at 16 Mi weight units, "checked after every +op, before the next one records anything". RUST.md §11.15 states the residual as "one op still +records up to about three copies of an item under the 200,000-value bound before the records +bound is checked (tens of MB at most)". + +What happens. `Recorder::record` (records.rs:61-76) clones `self.change` (title, rationale, +question, reference ids) and `self.op` (including a set's or an add's value) into every record. +`assert_depth` and `assert_recorded` (apply.rs:139-140) run only once `apply_op` returns, and +`mark_to_confirm` (apply.rs:66-96) is checked only after it has pushed one record per touched +item (apply.rs:145). One op that reaches N items therefore allocates about N × (author text or +value) before any bound runs. N is bounded only by the 200,000-value document cap; the size of +the text or value is bounded only by the 16 MiB read cap. + +Three paths, measured with a scratch probe built on `faunosteo-core` (`--features curation`, +the real legacy import, the committed changes and references, with one or two hostile changes +appended). Each run had `ulimit -v 3000000`: + +| Path | changes.json | Peak RSS | Outcome | +|---|---|---|---| +| level B `add` of a node with 10,000 children, rationale 128 KiB → `mark_to_confirm` pushes 10,001 records, each cloning the rationale | 711 KB | 1,372,972 KB | refused afterwards ("records weigh more") | +| same, rationale 512 KiB (about 5 GB needed) | 1.1 MB | 2,996,672 KB | `memory allocation of 524288 bytes failed`, SIGABRT | +| level A add of 10,000 children, then `set` `description` (64 KiB value) with `scope: subtree` → 10,000 writes plus 10,000 records holding the value twice | 646 KB | 2,027,252 KB | refused afterwards | +| add of 2,000 children, then `cite` 4,000 declared references with `scope: subtree` (references.json 150 KB) → 8 M values written before `assert_depth` | 344 KB | 2,112,620 KB | refused afterwards ("more than 200000 values") | + +The cost grows linearly in N × size, so a changeset of a few MB asks for tens to hundreds of +GB. The documented "tens of MB at most" is wrong, and the `mark_to_confirm` path is declared +nowhere. Impact is availability only: a hostile or careless `changes.json` run through +`pnpm r5:import-equine --check|--out`, or through the test suites, exhausts the memory of the +developer machine. The OOM killer may then take down other processes, such as the Docker +Supabase stack. No integrity or confidentiality impact. It is blocking because it breaks a +bound that both the code and the threat model of this gate require, and because the +documented residual misstates it. + +Fix direction: +- Enforce the budget at push time: route `mark_to_confirm` through the same accounting, and + make `Recorder::record` refuse once the running weight would pass the bound. +- Bound the per-op writes of subtree `set` and `cite` (items × value size, or a running values + budget) before writing. +- Alternatively, stop cloning: share `ChangeSummary` and `Op` across records (`Arc`), so a + record weighs only its own before and after. +- Count the cite op's `reference_ids` in `record_weight`. +- Correct RUST.md §11.15 and add a test per path (marking, subtree set, subtree cite) that + fails before any GB-scale allocation. + +Probe source: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/sec8/probe/src/main.rs` +(modes `add `, `value2 `, +`cite `). + +## Reported findings + +MEDIUM, review drawings per view (declared in RUST.md §11.15, now measured). Each view of the +curated document yields `review/annotations-.svg` and `review/app-.svg`. Each file +holds a full copy of the cleaned drawing, and all files sit in memory before the CLI writes +them. Level A adds of a detail view plus its trigger: 500 views → 1,011 files, 138.8 MB, +176 MB RSS; 2,000 views → 4,011 files, 547 MB, 638 MB RSS. That is about 273 KB per view, so +about 1.3 GB at the records bound (about 4,800 views), consistent with the declared "about +2 GB" estimate. The multiplier is the drawing's size. The frozen `horse.svg` is 130 KB, but +the core reader admits up to 2 MiB, so a replaced drawing would scale the same views to about +19 GB. Not blocking because it is declared and the estimate holds. A view-count or output-bytes +bound would close it. + +LOW findings: +- `record_weight` counts the op only for `set` and `add` (records.rs:125-128), but every record + clones the op. A cite's `reference_ids` list is copied uncounted into each record, so the + weight under-counts by about a third on that path. +- `TriggerNames::apply` searches the view list linearly for every element carrying + `data-anno-opens-view` (views × triggers). This runs in `--check` only, on committed files, + in seconds at the bounds. +- The two new test files carry the crate's usual test allow (`disallowed_methods`, `unwrap`, + `expect`) for fixture reads. They are test crates only and do not suppress a security scanner. +- Context, outside the diff: `clippy.toml`'s `disallowed-methods` bans `fs::read`, + `fs::write`, `File::open|create` and `env::var`. It does not ban `OpenOptions::open`, + `fs::create_dir`, `fs::read_dir`, `fs::remove_*` or `fs::rename`, so "the core has no I/O" + rests on review for those. A grep of `crates/core/src/curation` finds no `std::fs`, + `std::env`, `std::process` or clock use today. + +## What was checked and held + +- Markdown guard, differential against Prettier 3.8.3: `report_guard::format_report` against + the repository's Prettier (`parser: markdown`, the repository options), on raw reports in the + generator's three line forms: the list item of `change_line`, the 8-column CHANGELOG row via + `cell()`, and the bibliography line. 16,395 cases (hand-picked cell-start and line constructs, + CJK, Hangul, Arabic with harakat, Devanagari and Thai marks, URLs with `_` and `&name=`, + `{{ }}`, wiki and strong forms, plus three seeded random sets). 4,696 were admitted. The only + 3 differences were wiki links opened in one list-item field and closed in another across code + spans. `format_report` was called directly there; in the pipeline, + `review_report::assert_single_line` refuses `[[`/`]]` in every list-item field first. Table + cells parse on their own and showed no difference. No silent difference is reachable in + this sample. +- Structure forging: line breaks never reach a line start. List-item fields refuse control + characters, and `cell()` maps `\n` to a space. `\\|` (an authored `\|` after escaping) is + refused. Code spans are limited to `[A-Za-z0-9_:/. -]`, authored backticks are refused, and + field segments are limited to `[A-Za-z0-9_-]`. Ids written by add and rename must be stable + ids at the op. +- SVG: hostile annotation and view names (``, `&`, `A`, `]]>`, ``, + ``, quotes, U+0085, U+2028, NBSP, tab, newline) and a path `d` holding + `" onload="x` produce four well-formed drawings (xml.dom.minidom). The values stay data, + with no `onload` attribute and no `a` or `script` element. Non-XML characters and CR are + refused (`drawing_text`). The cleaned drawing goes through the core reader first: 2 MiB, + 20,000 elements, depth 64, DOCTYPE and ENTITY refused. `compare_drawing` parses the committed + file with the capped reader before roxmltree and requires a byte-exact rebuild. +- CLI file safety: `--check` passes at HEAD (legacy-import 6/0, draft 12/0, app-main declared). + `--out` refusals were run in a scratch folder: empty path, symlink, symlink with a trailing + `/`, non-empty folder, `..`, a destination through a symlink into the seed or into `draft/`, + a missing parent. A fresh folder gets 18 files, and a second run refuses "not empty". Writes + are `create_new`, folders are `create_dir` and components must be `Normal`. Reads are + canonical, confined to the seed, regular files only, capped at 16 MiB. Same-user TOCTOU is + declared. The working tree stayed clean. +- Quadratic scans: the guard is linear (the `&`, `<`, `[[` and URL scans work per segment). + `underscore_pairs_later` rescans only from the few authored plain parts. The renderers are + hashed. No input under the caps reached a measurable guard time (max 0 ms per case). +- Checklist: no hardcoded secret, token or auth URL in the diff or the oracles. No SQL built in + scope; `seed.rs` is unchanged since R5.3. No HTML render, endpoint, authz surface or logging + in scope. The CLI's error messages go to the local operator only. The only tracked env file + is `.env.example`; reading it was refused by the permission rules (rule: Bash `cat .env.example` + denied), it was not retried another way, and it is outside the diff. Guardrails scanner + (`--domain api,supabase`): 0 FAIL, 1 WARN, `SB-POLICY-NOT-OWNER` + `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the diff and dropped + per the diff-scope rule, plus 1 MANUAL. +- Anti-gaming: no `nosemgrep`, `nosec`, `eslint-disable` or `unsafe` added. The only new allows + are the two test-file lint allows listed above. + +Semgrep note: the three packs carry few Rust rules (162 rules loaded on 28 files, 0 findings), +so semgrep alone says little about this diff. The verdict rests on the checklist and the +probes. diff --git a/.claude/audits/2026-10-23-rust-r5-review-verify.md b/.claude/audits/2026-10-23-rust-r5-review-verify.md index cb253884..b0f1e79f 100644 --- a/.claude/audits/2026-10-23-rust-r5-review-verify.md +++ b/.claude/audits/2026-10-23-rust-r5-review-verify.md @@ -60,6 +60,11 @@ Fresh verifier and security on the fixed commit: below. - Security (`.claude/audits/2026-10-23-rust-r5-review-security-7.md`): HIGH output amplification: the CHANGELOG raw table (rows × authored rationale) built whole before the round-6 bound (1.9 MB changes.json aborts at 3 GB), RUST.md "refused before any row is built" false; HIGH record amplification: rename/remove records deep-copy the whole item and clone the change texts, no total bound (400 renames of a 5,000-child item → 4.3 GB). LOW test allows, bin allow (context), error echo size, subfolder race (declared out of model). - Fixed: records bounded in total (`apply::MAX_RECORDED_WEIGHT` 16 Mi units, `records::record_weight`, checked after every op; equine 1,084,422), test `the_records_cannot_outgrow_their_bound`; CHANGELOG tables projected from their cells before the raw table is joined (`markdown::assert_table_bound`), pipeline-level test `a_changelog_whose_alignment_would_explode_is_refused_before_it_is_built`; RUST.md rows rewritten exactly, residual risk (one op's copies before the check). -## Round 8 +## Round 8 on 5880dfa: security BLOCK(1) (verifier pending at fix time); `pnpm check:all` green 15:02:04 + +- Security (`.claude/audits/2026-10-23-rust-r5-review-security-8.md`): HIGH records bound checked only after a whole op and after the B/C marking: one op records once per item, each record cloning the change texts and the op value (B add of 10,000 children + 128 KiB rationale → 1.37 GB; subtree set 64 KiB → 2.0 GB; subtree cite 4,000 refs → 2.1 GB); RUST.md "tens of MB at most" false. MEDIUM review drawings per view (≈273 KB each view, 2,000 views → 547 MB). LOW `record_weight` ignored the op for cite/rename/remove/move; LOW `TriggerNames::apply` linear view search; LOW test allows, bin allow (context). +- Fixed: the records weight charged at each record before the change and the op are cloned (`records::charge`, `RecordParts`; `Recorder::record` and `mark_to_confirm` return the refusal; op weight for every kind; equine 1,195,000), test `the_records_bound_holds_inside_one_op_and_the_marking`; document bound test from a padded baseline (the records bound now stops a growing changeset first); review drawings projected and bounded at 64 MiB before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`), test `review_drawings_past_their_bound_are_refused_before_rendering`; RUST.md rows and residual risks rewritten exactly. + +## Round 9 Fresh verifier and security on the fixed commit: below. diff --git a/crates/core/src/curation/draft/apply.rs b/crates/core/src/curation/draft/apply.rs index 64b31bb8..357789db 100644 --- a/crates/core/src/curation/draft/apply.rs +++ b/crates/core/src/curation/draft/apply.rs @@ -15,7 +15,7 @@ use serde_json::Value; use super::ops::{apply_op, assert_declared}; use super::records::{ - ChangeRecord, CurationResult, Recorder, record_weight, strip_marks, touch_order, + ChangeRecord, CurationResult, RecordParts, Recorder, charge, strip_marks, touch_order, }; use super::schema::{Change, ChangeSummary, Op, SetOp}; use super::shape::assert_same_shape; @@ -67,6 +67,7 @@ fn mark_to_confirm( doc: &mut Value, change: &ChangeSummary, records: &mut Vec, + spent: &mut usize, ) -> Result<(), PipelineError> { for touched in touched_reviewable(doc)? { let item = item_at_mut(doc, &touched.path)?; @@ -85,64 +86,47 @@ fn mark_to_confirm( value: Value::from("to_confirm"), scope: None, }); + let after = Value::from("to_confirm"); + let parts = RecordParts { + change, + op: &op, + target: &touched.target, + field: Some("review_status"), + before: &before, + after: &after, + }; + charge(spent, parts)?; records.push(ChangeRecord { change: change.clone(), op, target: touched.target, field: Some("review_status".to_owned()), before, - after: Value::from("to_confirm"), + after, }); } Ok(()) } -/// The total weight of the records (`records::record_weight`): about 16 MiB of text, or a -/// quarter of a million JSON values, far past the equine changeset (test -/// `the_records_cannot_outgrow_their_bound`) and small enough that the reports built from them -/// stay bounded too. Checked after every op, before the next one records anything. -pub const MAX_RECORDED_WEIGHT: usize = 16 * 1024 * 1024; - -/// The records counted so far and their weight. -#[derive(Default)] -struct Tally { - counted: usize, - weight: usize, -} - -fn assert_recorded(records: &[ChangeRecord], tally: &mut Tally) -> Result<(), PipelineError> { - for record in &records[tally.counted..] { - tally.weight = tally.weight.saturating_add(record_weight(record)); - } - tally.counted = records.len(); - if tally.weight > MAX_RECORDED_WEIGHT { - return Err(fail(format!( - "the records weigh more than {MAX_RECORDED_WEIGHT} units" - ))); - } - Ok(()) -} - fn apply_change( doc: &mut Value, change: &Change, records: &mut Vec, - tally: &mut Tally, + spent: &mut usize, ) -> Result<(), PipelineError> { let summary = change.summary(); assert_declared(doc, &change.reference_ids) .map_err(|error| fail(format!("{}: {error}", change.id)))?; let mut next_touch = 0; for (index, op) in change.ops.iter().enumerate() { - let mut recorder = Recorder::new(&summary, op, records, &mut next_touch); + let mut recorder = Recorder::new(&summary, op, records, (&mut next_touch, &mut *spent)); apply_op(doc, &mut recorder) .and_then(|()| assert_depth(doc)) - .and_then(|()| assert_recorded(records, tally)) .map_err(|error| fail(format!("{} op {index}: {error}", change.id)))?; } if change.level.needs_confirmation() { - mark_to_confirm(doc, &summary, records)?; - assert_recorded(records, tally).map_err(|error| fail(format!("{}: {error}", change.id)))?; + mark_to_confirm(doc, &summary, records, spent) + .map_err(|error| fail(format!("{}: {error}", change.id)))?; } strip_marks(doc); Ok(()) @@ -235,9 +219,9 @@ pub fn apply_changeset( .ok_or_else(|| fail("the baseline is not an object"))? .insert("references".to_owned(), registry); let mut records = Vec::new(); - let mut tally = Tally::default(); + let mut spent = 0; for change in changes { - apply_change(&mut doc, change, &mut records, &mut tally)?; + apply_change(&mut doc, change, &mut records, &mut spent)?; } curated_issues(&doc)?; Ok(CurationResult { diff --git a/crates/core/src/curation/draft/ops.rs b/crates/core/src/curation/draft/ops.rs index 83811c4d..dc82cba5 100644 --- a/crates/core/src/curation/draft/ops.rs +++ b/crates/core/src/curation/draft/ops.rs @@ -118,7 +118,7 @@ fn set_self( recorder.touch(item); let before = read_field(item, &op.field)?; write_field(item, &op.field, &op.value)?; - recorder.record(&op.target, Some(&op.field), before, Some(op.value.clone())); + recorder.record(&op.target, Some(&op.field), before, Some(op.value.clone()))?; Ok(()) } @@ -163,7 +163,7 @@ fn set_subtree( Some(&op.field), before, Some(op.value.clone()), - ); + )?; } Ok(()) } @@ -199,7 +199,7 @@ fn apply_rename( item.insert(key.to_owned(), Value::from(op.id.clone())); recorder.touch(item); let after = Value::Object(item.clone()); - recorder.record(&op.target, None, Some(before), Some(after)); + recorder.record(&op.target, None, Some(before), Some(after))?; let rename = Rename { kind: located.target.kind, from, @@ -331,7 +331,7 @@ fn apply_add( added.push(Step::Index(index)); recorder.touch_tree(doc, &added)?; let target = format!("{}{id}", dest.prefix); - recorder.record(&target, None, None, Some(op.value.clone())); + recorder.record(&target, None, None, Some(op.value.clone()))?; Ok(()) } @@ -345,7 +345,7 @@ fn apply_remove( return Err(fail("the document cannot be removed")); }; let item = list_at_mut(doc, list)?.remove(located.index); - recorder.record(&op.target, None, Some(item), None); + recorder.record(&op.target, None, Some(item), None)?; Ok(()) } @@ -383,7 +383,7 @@ fn apply_move( recorder.touch(item); let before = Value::Object(item.clone()); let after = format!("{}{id}", child_prefix(&parent)); - recorder.record(&op.target, None, Some(before), Some(Value::from(after))); + recorder.record(&op.target, None, Some(before), Some(Value::from(after)))?; Ok(()) } @@ -449,7 +449,7 @@ fn apply_cite( Some("reference_ids"), Some(Value::Array(before)), Some(Value::Array(after)), - ); + )?; } Ok(()) } diff --git a/crates/core/src/curation/draft/pipeline.rs b/crates/core/src/curation/draft/pipeline.rs index 4874933f..db6c5372 100644 --- a/crates/core/src/curation/draft/pipeline.rs +++ b/crates/core/src/curation/draft/pipeline.rs @@ -199,13 +199,26 @@ fn view_ids(document: &Value) -> Vec { .collect() } +/// The review drawings' bytes, projected before any is rendered: every drawing repeats the +/// whole cleaned drawing, two per view (the equine draft: 2 views, about 0.55 MB). Views added +/// by a changeset multiply it (2,000 views asked for 547 MB, security gate R5.4 round 8). +pub const MAX_REVIEW_DRAWINGS_BYTES: usize = 64 * 1024 * 1024; + /// One review overlay per view (`review/annotations-.svg`), then the application's /// drawing of each view (`review/app-.svg`). fn review_drawings(draft: &DraftResult) -> Result, PipelineError> { + let views = view_ids(&draft.document); + let projected = draft.cleaned_svg.len().saturating_mul(2 * views.len()); + if projected > MAX_REVIEW_DRAWINGS_BYTES { + return Err(fail(format!( + "{} views would write {projected} bytes of review drawings, past the \ + {MAX_REVIEW_DRAWINGS_BYTES} bound", + views.len() + ))); + } let drawing = read_cleaned_drawing(&draft.cleaned_svg)?; let typed: AnatomicalSheetDocument = serde_json::from_value(draft.document.clone()) .map_err(|error| fail(format!("curated document: {error}")))?; - let views = view_ids(&draft.document); let mut files = Vec::new(); for view_id in &views { let overlay = render_review_overlay(&OverlayInput { diff --git a/crates/core/src/curation/draft/records.rs b/crates/core/src/curation/draft/records.rs index de50476c..4504fc7f 100644 --- a/crates/core/src/curation/draft/records.rs +++ b/crates/core/src/curation/draft/records.rs @@ -33,6 +33,34 @@ pub struct CurationResult { pub records: Vec, } +/// The total weight of the records (`record_weight`): about 16 MiB of text, or a quarter of a +/// million JSON values, far past the equine changeset (test `the_records_cannot_outgrow_their_bound`) +/// and small enough that the reports built from them stay bounded too. Charged as each record +/// is made, before the change and the op are cloned into it: one op touching many items is +/// refused at the record that passes the bound. +pub const MAX_RECORDED_WEIGHT: usize = 16 * 1024 * 1024; + +/// The records' running weight; refuses the record that would pass `MAX_RECORDED_WEIGHT`. +pub fn charge(spent: &mut usize, record: RecordParts<'_>) -> Result<(), PipelineError> { + *spent = spent.saturating_add(parts_weight(&record)); + if *spent > MAX_RECORDED_WEIGHT { + return Err(PipelineError::Guard(format!( + "the records weigh more than {MAX_RECORDED_WEIGHT} units" + ))); + } + Ok(()) +} + +/// What a record is made of, borrowed: weighed before anything is cloned. +pub struct RecordParts<'a> { + pub change: &'a ChangeSummary, + pub op: &'a Op, + pub target: &'a str, + pub field: Option<&'a str>, + pub before: &'a Value, + pub after: &'a Value, +} + /// What the op functions of one change share besides the document; the touch counter spans /// the ops of the change (the first touch of an item fixes its place in the review marking). pub struct Recorder<'a> { @@ -40,6 +68,7 @@ pub struct Recorder<'a> { pub op: &'a Op, pub records: &'a mut Vec, next_touch: &'a mut u64, + spent: &'a mut usize, } impl<'a> Recorder<'a> { @@ -47,32 +76,45 @@ impl<'a> Recorder<'a> { change: &'a ChangeSummary, op: &'a Op, records: &'a mut Vec, - next_touch: &'a mut u64, + counters: (&'a mut u64, &'a mut usize), ) -> Self { Self { change, op, records, - next_touch, + next_touch: counters.0, + spent: counters.1, } } - /// Absent values are recorded as null so records stay plain JSON. + /// Absent values are recorded as null so records stay plain JSON. Refused when the + /// records would pass their bound (`charge`), before the change and the op are cloned. pub fn record( &mut self, target: &str, field: Option<&str>, before: Option, after: Option, - ) { + ) -> Result<(), PipelineError> { + let (before, after) = (unmarked(before), unmarked(after)); + let parts = RecordParts { + change: self.change, + op: self.op, + target, + field, + before: &before, + after: &after, + }; + charge(self.spent, parts)?; self.records.push(ChangeRecord { change: self.change.clone(), op: self.op.clone(), target: target.to_owned(), field: field.map(str::to_owned), - before: unmarked(before), - after: unmarked(after), + before, + after, }); + Ok(()) } /// Marks an item as touched by the change, once (the first touch keeps its order). @@ -105,29 +147,48 @@ impl<'a> Recorder<'a> { } } -/// What the records hold, in weight units: 64 per JSON value (a `Value` node costs tens of -/// bytes) plus one per byte of every string (the values' and the change's texts, which each -/// record clones). A rename or a remove records the whole item, so the records can grow far -/// past the document; `apply` bounds their total (`MAX_RECORDED_WEIGHT`). +/// What a record holds, in weight units: 64 per JSON value (a `Value` node costs tens of +/// bytes) plus one per byte of every string: the values, and the change and the op, which each +/// record clones. A rename or a remove records the whole item and one op can record once per +/// item it reaches, so the records can grow far past the document: `charge` bounds their total. pub fn record_weight(record: &ChangeRecord) -> usize { - let change = &record.change; + parts_weight(&RecordParts { + change: &record.change, + op: &record.op, + target: &record.target, + field: record.field.as_deref(), + before: &record.before, + after: &record.after, + }) +} + +fn parts_weight(parts: &RecordParts<'_>) -> usize { + let change = parts.change; let texts = [ change.id.as_str(), &change.title.fr, &change.title.en, &change.rationale, change.question.as_deref().unwrap_or_default(), - &record.target, - record.field.as_deref().unwrap_or_default(), + parts.target, + parts.field.unwrap_or_default(), ]; let text: usize = texts.iter().map(|text| text.len()).sum::() + change.reference_ids.iter().map(String::len).sum::(); - let op_value = match &record.op { - Op::Set(op) => value_weight(&op.value), + text + op_weight(parts.op) + value_weight(parts.before) + value_weight(parts.after) +} + +/// The op a record clones: its target, its value or its ids. +fn op_weight(op: &Op) -> usize { + let carried = match op { + Op::Set(op) => op.field.len() + value_weight(&op.value), Op::Add(op) => value_weight(&op.value), - _ => 0, + Op::Rename(op) => op.id.len(), + Op::Move(op) => op.to.len(), + Op::Cite(op) => op.reference_ids.iter().map(|id| 64 + id.len()).sum(), + Op::Remove(_) => 0, }; - text + op_value + value_weight(&record.before) + value_weight(&record.after) + 64 + op.target().len() + carried } /// The weight of a value, walked with an explicit stack (no recursion). diff --git a/crates/core/src/curation/draft/rename.rs b/crates/core/src/curation/draft/rename.rs index 928dec87..ba69f22e 100644 --- a/crates/core/src/curation/draft/rename.rs +++ b/crates/core/src/curation/draft/rename.rs @@ -74,7 +74,7 @@ fn follow_link( Some(link.field), Some(from.clone()), Some(Value::from(rename.to.as_str())), - ); + )?; } Ok(()) } @@ -109,7 +109,7 @@ fn follow_citations( Some("reference_ids"), Some(before), Some(Value::Array(after.clone())), - ); + )?; item.insert("reference_ids".to_owned(), Value::Array(after)); } Ok(()) diff --git a/crates/core/tests/curation_draft.rs b/crates/core/tests/curation_draft.rs index 99dd8e66..aba5c0cc 100644 --- a/crates/core/tests/curation_draft.rs +++ b/crates/core/tests/curation_draft.rs @@ -764,14 +764,26 @@ fn an_integer_beyond_2_53_is_read_as_the_double_json_parse_gives() { ); } +/// The records bound now stops a growing changeset first (every added value is recorded); the +/// document bound still holds a baseline near it: one small add past 200,000 values is refused. #[test] fn the_document_cannot_hold_more_values_than_the_bound() { - let children: Vec = (0..70_000) - .map(|i| json!({ "id": format!("n{i}"), "name": { "fr": "a", "en": "a" } })) - .collect(); - let value = json!({ "id": "many", "name": { "fr": "a", "en": "a" }, "children": children }); - let op = json!({ "op": "add", "target": MOBILITY, "value": value }); - let error = mini_apply(json!([op])).unwrap_err(); + let frozen = oracle("curation-synthetic.json"); + let references = parse_references(&to_js_compact(&frozen["references_file"]).unwrap()).unwrap(); + let mut baseline = frozen["baseline"].clone(); + baseline["padding"] = Value::from(vec![Value::Null; 199_950]); + let value = json!({ "id": "small", "name": { "fr": "a", "en": "a" }, + "children": (0..20).map(|i| json!({ "id": format!("n{i}"), "name": { "fr": "a", "en": "a" } })) + .collect::>() }); + let file = json!({ "changes": [{ + "id": "CHG-001", "level": "A", "kind": "technical", + "title": { "fr": "Essai", "en": "Test" }, "rationale": "Essai", + "reference_ids": [], "ops": [{ "op": "add", "target": MOBILITY, "value": value }], + }]}); + let changeset = parse_changeset(&to_js_compact(&file).unwrap()).unwrap(); + let error = apply_changeset(&baseline, &references, &changeset.changes) + .unwrap_err() + .to_string(); assert!(error.contains("more than 200000 values"), "{error}"); } @@ -780,7 +792,7 @@ fn the_document_cannot_hold_more_values_than_the_bound() { /// 5,000-child item asked for 4.3 GB). The equine changeset stays far below the bound. #[test] fn the_records_cannot_outgrow_their_bound() { - use faunosteo_core::curation::draft::apply::MAX_RECORDED_WEIGHT; + use faunosteo_core::curation::draft::records::MAX_RECORDED_WEIGHT; use faunosteo_core::curation::draft::records::record_weight; let equine: usize = equine_draft().records.iter().map(record_weight).sum(); println!("equine records weigh {equine} units"); @@ -802,6 +814,42 @@ fn the_records_cannot_outgrow_their_bound() { assert!(error.contains("the records weigh more than"), "{error}"); } +/// One op, or the B/C marking, records once per item it reaches: the bound is charged at each +/// record, before the change and the op are cloned into it (security gate R5.4 round 8: a level +/// B add of 10,000 children with a 128 KiB rationale reached 1.37 GB inside one marking). +#[test] +fn the_records_bound_holds_inside_one_op_and_the_marking() { + let children: Vec = (0..5_000) + .map(|i| json!({ "id": format!("n{i}"), "name": { "fr": "a", "en": "a" } })) + .collect(); + let value = json!({ "id": "many", "name": { "fr": "a", "en": "a" }, "children": children }); + let add = json!({ "op": "add", "target": MOBILITY, "value": value }); + let frozen = oracle("curation-synthetic.json"); + let references = parse_references(&to_js_compact(&frozen["references_file"]).unwrap()).unwrap(); + let description = json!({ "fr": "d".repeat(4_096), "en": "d" }); + let subtree = json!({ "op": "set", "target": format!("{MOBILITY}/many"), "field": "description", + "value": description, "scope": "subtree" }); + let cases = [ + ("B", "r".repeat(8_192), json!([add])), + ("A", "Essai".to_owned(), json!([add, subtree])), + ]; + for (level, rationale, ops) in cases { + let file = json!({ "changes": [{ + "id": "CHG-001", "level": level, "kind": "technical", + "title": { "fr": "Essai", "en": "Test" }, "rationale": rationale, + "reference_ids": [], "ops": ops, + }]}); + let changeset = parse_changeset(&to_js_compact(&file).unwrap()).unwrap(); + let error = apply_changeset(&frozen["baseline"], &references, &changeset.changes) + .unwrap_err() + .to_string(); + assert!( + error.contains("the records weigh more than"), + "{level}: {error}" + ); + } +} + /// A `__proto__` segment had no faithful reading in the TypeScript (a prototype set, a /// polluted `Object.prototype`, or nothing written): the port refuses it at the op. #[test] diff --git a/crates/core/tests/curation_review.rs b/crates/core/tests/curation_review.rs index 14fff308..051d3b08 100644 --- a/crates/core/tests/curation_review.rs +++ b/crates/core/tests/curation_review.rs @@ -496,3 +496,24 @@ fn ids_and_field_segments_that_could_forge_report_structure_are_refused() { let rule = TriggerNames::of(&typed(&draft.document)); assert!(compare_drawing("", &deep, Some(&rule)).is_err()); } + +/// Every review drawing repeats the whole cleaned drawing, two per view: the bytes are +/// projected and bounded before any drawing is rendered (security gate R5.4 round 8). +#[test] +fn review_drawings_past_their_bound_are_refused_before_rendering() { + let (result, mut draft) = equine(); + let views = draft.document["views"].as_array_mut().unwrap(); + let main = views[0].clone(); + for index in 0..300 { + let mut view = main.clone(); + view["id"] = Value::from(format!("extra-{index}")); + views.push(view); + } + let error = render_draft_artifacts(&draft, &result.anomalies).unwrap_err(); + assert!( + error + .to_string() + .contains("bytes of review drawings, past the"), + "{error}" + ); +} diff --git a/docs/development/RUST.md b/docs/development/RUST.md index aba017ea..b4892980 100644 --- a/docs/development/RUST.md +++ b/docs/development/RUST.md @@ -3204,7 +3204,8 @@ pass the guard unchanged. | authored text outside the allowlist: Markdown Prettier rewrites (`*x*`, `_x`, `x_**a**`, `**a****b**`, `Voir_https://…`, `a b`, a non-ASCII space, a tab, a line break in a list item, `` `` ``, `a\|b`, `[a]()`, `[[ a ]]`, `*`…), the constructs refused whole (links, HTML tags, autolinks, character references) and any character outside the admitted set (pictographs, symbols, combining marks without the Alphabetic property; an alphabetic mark such as U+0345 is a letter and Prettier aligns it alike) | written, then rewritten by Prettier (escapes, `_emphasis_`, collapsed spaces, indented continuation lines, normalised links; a row with an extra cell), or kept | refused (`UnsupportedMarkdown`), even where Prettier would have kept the text | | a pictograph in a table cell (an emoji, and ©, ®, ™, ✓, ★, ⚠ and the other characters `text::is_emoji_candidate` lists) | measured with Prettier's tables | refused (`UnsupportedWidth`, R5.2): the set is wider than the emoji, the refusal stays loud | | an aligned report table past 32 Mi display columns in all (rows × padded row width; the equine CHANGELOG's widest table is about 0.5 MB): one long authored cell pads every row of its level, so an 80 KB rationale over 14,000 records asked for 1 GB (security gate R5.4 round 6) | Prettier aligned it (the same amplification) | refused (`Guard`): the CHANGELOG's tables are projected from their cells (bytes bound the display width) before the raw table is joined (test `a_changelog_whose_alignment_would_explode_is_refused_before_it_is_built`, through the pipeline), every other table in the aligner before any padded row is built (test `an_aligned_table_past_the_bound_is_refused_before_it_is_built`), import report included | -| records weighing more than 16 Mi units in all (`records::record_weight`: 64 per JSON value plus one per string byte of the values and of the change's texts, which each record clones; the equine records weigh 1,084,422): a rename or a remove records the whole item, so 400 renames of a 5,000-child item asked for 4.3 GB, and an 80 KB rationale repeated over 14,000 records for 1.1 GB of raw rows (security gate R5.4 round 7) | recorded (the same copies) | refused at the op that passes the bound, before the next op records anything (test `the_records_cannot_outgrow_their_bound`); the raw report rows, built from the records, are bounded with them | +| records weighing more than 16 Mi units in all (`records::record_weight`: 64 per JSON value plus one per string byte of the values, of the change and of the op, which each record clones; the equine records weigh 1,195,000): a rename or a remove records the whole item and one op, or the B/C marking, records once per item it reaches, so 400 renames of a 5,000-child item asked for 4.3 GB (round 7), a level B add of 10,000 children with a 128 KiB rationale 1.37 GB inside one marking, a 64 KiB value set over a subtree 2.0 GB (round 8) | recorded (the same copies) | refused at the record that passes the bound, charged before the change and the op are cloned into it (`records::charge`, tests `the_records_cannot_outgrow_their_bound`, `the_records_bound_holds_inside_one_op_and_the_marking`); the raw report rows, built from the records, are bounded with them; the 200,000-value document bound now holds the baseline (test from a padded baseline) | +| review drawings past 64 MiB in all, projected before any is rendered (two per view, each repeating the cleaned drawing; the equine draft writes about 0.55 MB): 2,000 views asked for 547 MB (security gate R5.4 round 8) | written | refused before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`, test `review_drawings_past_their_bound_are_refused_before_rendering`) | | an id an `add` writes (the item's, and those nested in its `children` or `categories`) or a `rename` id that is not a stable id (a region code for a region), or a field segment outside `[A-Za-z0-9_-]` (a control character, NUL included, a backtick, a `]`, a space, a non-ASCII letter) | accepted when a later op removed or renamed the item: the raw id stayed in the records and reached the reports' code spans (a line break there forged a section, security gate R5.4 round 2) | refused at the op, the check the final document applies anyway (test `ids_and_field_segments_that_could_forge_report_structure_are_refused`) | | a label, id or path holding a code point XML 1.0 refuses, or a carriage return, in a review drawing | written (a non-well-formed file; a raw CR read back as a line feed) | refused: the drawing is not written | @@ -3241,10 +3242,11 @@ ruled and frozen in the sweep, so a combination of admitted characters no probe differ; the comparison of the application's drawings accepts exactly the trigger-name rule, byte for byte; the frozen `horse.svg` is read, never regenerated, so a future change of the drawing needs its own -decision; one op still records up to about three copies of an item under the 200,000-value bound -before the records bound is checked (tens of MB at most); the review drawings are bounded only through the document's 200,000 values (two drawings -of about 137 KB per view, about 2 GB at the bound, an estimate of the security gate R5.4 round 6, -not measured), and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160, +decision; the record that passes the bound has been built before it is refused (one item copy +and one clone of the op's value, at most twice the 16 MiB read cap), and a subtree set writes its +value into each item before recording it, so the document holds at most as many copies as the +records bound admits; the comparison of `--check` looks up each trigger's view linearly (views × +triggers, the views bounded through the drawings bound); and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160, U+FFA0, letters Prettier keeps, no structure); the residual risks of §11.13 stand. Conditions for R5.5: the owner's validation of this checkpoint; the upload tool (`ureq`, the