fix(anatomy): R5.4 records charged at each record, review drawings bounded before rendering
This commit is contained in:
1 parent
5880dfa959
commit
8bdb079f70
10 files changed
+357
-75
No files matched your search
@@ -0,0 +1,148 @@
|
||||
# Security gate R5.4, round 8 (fresh): 2026-10-23
|
||||
|
||||
Scope: `git diff c6e0d85..HEAD -- crates/core packages/anatomy/tests docs/development/RUST.md`,
|
||||
HEAD 5880dfa, branch `feature/rust-r5-pipeline-core`. Mode: audit (REPORT path given; this file
|
||||
is the only write). Blind run: no prior verdict or finding list read.
|
||||
|
||||
```
|
||||
SECURITY — VERDICT: BLOCK(1)
|
||||
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
|
||||
SCOPE: 28 files
|
||||
BLOCKING:
|
||||
1. [HIGH] resource-bound/records-checked-after-op — crates/core/src/curation/draft/apply.rs:140,145 (sites: apply.rs:66-96 mark_to_confirm, ops.rs:126-168 set_subtree, ops.rs:417-452 apply_cite, records.rs:61-76 Recorder::record) — the document-values and records-weight bounds only run after a whole op (and after the B/C marking); inside one op every record clones the change texts and the op value, once per item, so about 1 MB of changeset allocates GBs before any bound runs (measured: 711 KB → 1.37 GB RSS; 1.1 MB → abort at a 3 GB cap; 646 KB → 2.0 GB; 494 KB → 2.1 GB). RUST.md:3244-3245 says "tens of MB at most", which is false — hint: count the weight as each record is pushed (Recorder::record and mark_to_confirm), refuse before cloning; keep a per-op values budget for subtree writes (set/cite); or share ChangeSummary/Op (Arc) instead of cloning them; include CiteOp in record_weight; correct the RUST.md residual
|
||||
REPORTED (non-blocking):
|
||||
- [MEDIUM] resource-bound/review-drawings-per-view (declared residual, now measured) — crates/core/src/curation/draft/pipeline.rs:204-226
|
||||
- [LOW] record_weight ignores the op cloned into each record for cite/rename/remove/move — crates/core/src/curation/draft/records.rs:125-128
|
||||
- [LOW] TriggerNames::apply linear view search per trigger element (views × triggers, --check only) — crates/core/src/curation/draft/svg_compare.rs:36-50
|
||||
- [LOW] new test-file lint allows `#![allow(clippy::disallowed_methods, …)]`, test crates only, same convention as 20 earlier test files, not a security-scanner suppression — crates/core/tests/curation_markdown.rs:6, crates/core/tests/curation_review.rs:6
|
||||
- [LOW] pre-existing (context) `#![allow(clippy::disallowed_methods)]` in the CLI — crates/core/src/bin/import_equine.rs:17
|
||||
PROOF: semgrep 162 rules on 28 files → 0 findings; checklist 15 checks → 5 findings (1 HIGH, 1 MEDIUM, 3 LOW) + 1 pre-existing LOW
|
||||
```
|
||||
|
||||
## Blocking finding 1: the records and values bounds run after the op
|
||||
|
||||
What the code claims: `apply.rs` bounds the records at 16 Mi weight units, "checked after every
|
||||
op, before the next one records anything". RUST.md §11.15 states the residual as "one op still
|
||||
records up to about three copies of an item under the 200,000-value bound before the records
|
||||
bound is checked (tens of MB at most)".
|
||||
|
||||
What happens. `Recorder::record` (records.rs:61-76) clones `self.change` (title, rationale,
|
||||
question, reference ids) and `self.op` (including a set's or an add's value) into every record.
|
||||
`assert_depth` and `assert_recorded` (apply.rs:139-140) run only once `apply_op` returns, and
|
||||
`mark_to_confirm` (apply.rs:66-96) is checked only after it has pushed one record per touched
|
||||
item (apply.rs:145). One op that reaches N items therefore allocates about N × (author text or
|
||||
value) before any bound runs. N is bounded only by the 200,000-value document cap; the size of
|
||||
the text or value is bounded only by the 16 MiB read cap.
|
||||
|
||||
Three paths, measured with a scratch probe built on `faunosteo-core` (`--features curation`,
|
||||
the real legacy import, the committed changes and references, with one or two hostile changes
|
||||
appended). Each run had `ulimit -v 3000000`:
|
||||
|
||||
| Path | changes.json | Peak RSS | Outcome |
|
||||
|---|---|---|---|
|
||||
| level B `add` of a node with 10,000 children, rationale 128 KiB → `mark_to_confirm` pushes 10,001 records, each cloning the rationale | 711 KB | 1,372,972 KB | refused afterwards ("records weigh more") |
|
||||
| same, rationale 512 KiB (about 5 GB needed) | 1.1 MB | 2,996,672 KB | `memory allocation of 524288 bytes failed`, SIGABRT |
|
||||
| level A add of 10,000 children, then `set` `description` (64 KiB value) with `scope: subtree` → 10,000 writes plus 10,000 records holding the value twice | 646 KB | 2,027,252 KB | refused afterwards |
|
||||
| add of 2,000 children, then `cite` 4,000 declared references with `scope: subtree` (references.json 150 KB) → 8 M values written before `assert_depth` | 344 KB | 2,112,620 KB | refused afterwards ("more than 200000 values") |
|
||||
|
||||
The cost grows linearly in N × size, so a changeset of a few MB asks for tens to hundreds of
|
||||
GB. The documented "tens of MB at most" is wrong, and the `mark_to_confirm` path is declared
|
||||
nowhere. Impact is availability only: a hostile or careless `changes.json` run through
|
||||
`pnpm r5:import-equine --check|--out`, or through the test suites, exhausts the memory of the
|
||||
developer machine. The OOM killer may then take down other processes, such as the Docker
|
||||
Supabase stack. No integrity or confidentiality impact. It is blocking because it breaks a
|
||||
bound that both the code and the threat model of this gate require, and because the
|
||||
documented residual misstates it.
|
||||
|
||||
Fix direction:
|
||||
- Enforce the budget at push time: route `mark_to_confirm` through the same accounting, and
|
||||
make `Recorder::record` refuse once the running weight would pass the bound.
|
||||
- Bound the per-op writes of subtree `set` and `cite` (items × value size, or a running values
|
||||
budget) before writing.
|
||||
- Alternatively, stop cloning: share `ChangeSummary` and `Op` across records (`Arc`), so a
|
||||
record weighs only its own before and after.
|
||||
- Count the cite op's `reference_ids` in `record_weight`.
|
||||
- Correct RUST.md §11.15 and add a test per path (marking, subtree set, subtree cite) that
|
||||
fails before any GB-scale allocation.
|
||||
|
||||
Probe source: `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/sec8/probe/src/main.rs`
|
||||
(modes `add <rationale bytes> <children>`, `value2 <value bytes> <children>`,
|
||||
`cite <references> <children>`).
|
||||
|
||||
## Reported findings
|
||||
|
||||
MEDIUM, review drawings per view (declared in RUST.md §11.15, now measured). Each view of the
|
||||
curated document yields `review/annotations-<view>.svg` and `review/app-<view>.svg`. Each file
|
||||
holds a full copy of the cleaned drawing, and all files sit in memory before the CLI writes
|
||||
them. Level A adds of a detail view plus its trigger: 500 views → 1,011 files, 138.8 MB,
|
||||
176 MB RSS; 2,000 views → 4,011 files, 547 MB, 638 MB RSS. That is about 273 KB per view, so
|
||||
about 1.3 GB at the records bound (about 4,800 views), consistent with the declared "about
|
||||
2 GB" estimate. The multiplier is the drawing's size. The frozen `horse.svg` is 130 KB, but
|
||||
the core reader admits up to 2 MiB, so a replaced drawing would scale the same views to about
|
||||
19 GB. Not blocking because it is declared and the estimate holds. A view-count or output-bytes
|
||||
bound would close it.
|
||||
|
||||
LOW findings:
|
||||
- `record_weight` counts the op only for `set` and `add` (records.rs:125-128), but every record
|
||||
clones the op. A cite's `reference_ids` list is copied uncounted into each record, so the
|
||||
weight under-counts by about a third on that path.
|
||||
- `TriggerNames::apply` searches the view list linearly for every element carrying
|
||||
`data-anno-opens-view` (views × triggers). This runs in `--check` only, on committed files,
|
||||
in seconds at the bounds.
|
||||
- The two new test files carry the crate's usual test allow (`disallowed_methods`, `unwrap`,
|
||||
`expect`) for fixture reads. They are test crates only and do not suppress a security scanner.
|
||||
- Context, outside the diff: `clippy.toml`'s `disallowed-methods` bans `fs::read`,
|
||||
`fs::write`, `File::open|create` and `env::var`. It does not ban `OpenOptions::open`,
|
||||
`fs::create_dir`, `fs::read_dir`, `fs::remove_*` or `fs::rename`, so "the core has no I/O"
|
||||
rests on review for those. A grep of `crates/core/src/curation` finds no `std::fs`,
|
||||
`std::env`, `std::process` or clock use today.
|
||||
|
||||
## What was checked and held
|
||||
|
||||
- Markdown guard, differential against Prettier 3.8.3: `report_guard::format_report` against
|
||||
the repository's Prettier (`parser: markdown`, the repository options), on raw reports in the
|
||||
generator's three line forms: the list item of `change_line`, the 8-column CHANGELOG row via
|
||||
`cell()`, and the bibliography line. 16,395 cases (hand-picked cell-start and line constructs,
|
||||
CJK, Hangul, Arabic with harakat, Devanagari and Thai marks, URLs with `_` and `&name=`,
|
||||
`{{ }}`, wiki and strong forms, plus three seeded random sets). 4,696 were admitted. The only
|
||||
3 differences were wiki links opened in one list-item field and closed in another across code
|
||||
spans. `format_report` was called directly there; in the pipeline,
|
||||
`review_report::assert_single_line` refuses `[[`/`]]` in every list-item field first. Table
|
||||
cells parse on their own and showed no difference. No silent difference is reachable in
|
||||
this sample.
|
||||
- Structure forging: line breaks never reach a line start. List-item fields refuse control
|
||||
characters, and `cell()` maps `\n` to a space. `\\|` (an authored `\|` after escaping) is
|
||||
refused. Code spans are limited to `[A-Za-z0-9_:/. -]`, authored backticks are refused, and
|
||||
field segments are limited to `[A-Za-z0-9_-]`. Ids written by add and rename must be stable
|
||||
ids at the op.
|
||||
- SVG: hostile annotation and view names (`<a href>`, `&`, `A`, `]]>`, `<!-- -->`,
|
||||
`<?pi?>`, quotes, U+0085, U+2028, NBSP, tab, newline) and a path `d` holding
|
||||
`" onload="x` produce four well-formed drawings (xml.dom.minidom). The values stay data,
|
||||
with no `onload` attribute and no `a` or `script` element. Non-XML characters and CR are
|
||||
refused (`drawing_text`). The cleaned drawing goes through the core reader first: 2 MiB,
|
||||
20,000 elements, depth 64, DOCTYPE and ENTITY refused. `compare_drawing` parses the committed
|
||||
file with the capped reader before roxmltree and requires a byte-exact rebuild.
|
||||
- CLI file safety: `--check` passes at HEAD (legacy-import 6/0, draft 12/0, app-main declared).
|
||||
`--out` refusals were run in a scratch folder: empty path, symlink, symlink with a trailing
|
||||
`/`, non-empty folder, `..`, a destination through a symlink into the seed or into `draft/`,
|
||||
a missing parent. A fresh folder gets 18 files, and a second run refuses "not empty". Writes
|
||||
are `create_new`, folders are `create_dir` and components must be `Normal`. Reads are
|
||||
canonical, confined to the seed, regular files only, capped at 16 MiB. Same-user TOCTOU is
|
||||
declared. The working tree stayed clean.
|
||||
- Quadratic scans: the guard is linear (the `&`, `<`, `[[` and URL scans work per segment).
|
||||
`underscore_pairs_later` rescans only from the few authored plain parts. The renderers are
|
||||
hashed. No input under the caps reached a measurable guard time (max 0 ms per case).
|
||||
- Checklist: no hardcoded secret, token or auth URL in the diff or the oracles. No SQL built in
|
||||
scope; `seed.rs` is unchanged since R5.3. No HTML render, endpoint, authz surface or logging
|
||||
in scope. The CLI's error messages go to the local operator only. The only tracked env file
|
||||
is `.env.example`; reading it was refused by the permission rules (rule: Bash `cat .env.example`
|
||||
denied), it was not retried another way, and it is outside the diff. Guardrails scanner
|
||||
(`--domain api,supabase`): 0 FAIL, 1 WARN, `SB-POLICY-NOT-OWNER`
|
||||
`supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the diff and dropped
|
||||
per the diff-scope rule, plus 1 MANUAL.
|
||||
- Anti-gaming: no `nosemgrep`, `nosec`, `eslint-disable` or `unsafe` added. The only new allows
|
||||
are the two test-file lint allows listed above.
|
||||
|
||||
Semgrep note: the three packs carry few Rust rules (162 rules loaded on 28 files, 0 findings),
|
||||
so semgrep alone says little about this diff. The verdict rests on the checklist and the
|
||||
probes.
|
||||
@@ -60,6 +60,11 @@ Fresh verifier and security on the fixed commit: below.
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-7.md`): HIGH output amplification: the CHANGELOG raw table (rows × authored rationale) built whole before the round-6 bound (1.9 MB changes.json aborts at 3 GB), RUST.md "refused before any row is built" false; HIGH record amplification: rename/remove records deep-copy the whole item and clone the change texts, no total bound (400 renames of a 5,000-child item → 4.3 GB). LOW test allows, bin allow (context), error echo size, subfolder race (declared out of model).
|
||||
- Fixed: records bounded in total (`apply::MAX_RECORDED_WEIGHT` 16 Mi units, `records::record_weight`, checked after every op; equine 1,084,422), test `the_records_cannot_outgrow_their_bound`; CHANGELOG tables projected from their cells before the raw table is joined (`markdown::assert_table_bound`), pipeline-level test `a_changelog_whose_alignment_would_explode_is_refused_before_it_is_built`; RUST.md rows rewritten exactly, residual risk (one op's copies before the check).
|
||||
|
||||
## Round 8
|
||||
## Round 8 on 5880dfa: security BLOCK(1) (verifier pending at fix time); `pnpm check:all` green 15:02:04
|
||||
|
||||
- Security (`.claude/audits/2026-10-23-rust-r5-review-security-8.md`): HIGH records bound checked only after a whole op and after the B/C marking: one op records once per item, each record cloning the change texts and the op value (B add of 10,000 children + 128 KiB rationale → 1.37 GB; subtree set 64 KiB → 2.0 GB; subtree cite 4,000 refs → 2.1 GB); RUST.md "tens of MB at most" false. MEDIUM review drawings per view (≈273 KB each view, 2,000 views → 547 MB). LOW `record_weight` ignored the op for cite/rename/remove/move; LOW `TriggerNames::apply` linear view search; LOW test allows, bin allow (context).
|
||||
- Fixed: the records weight charged at each record before the change and the op are cloned (`records::charge`, `RecordParts`; `Recorder::record` and `mark_to_confirm` return the refusal; op weight for every kind; equine 1,195,000), test `the_records_bound_holds_inside_one_op_and_the_marking`; document bound test from a padded baseline (the records bound now stops a growing changeset first); review drawings projected and bounded at 64 MiB before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`), test `review_drawings_past_their_bound_are_refused_before_rendering`; RUST.md rows and residual risks rewritten exactly.
|
||||
|
||||
## Round 9
|
||||
|
||||
Fresh verifier and security on the fixed commit: below.
|
||||
@@ -15,7 +15,7 @@ use serde_json::Value;
|
||||
|
||||
use super::ops::{apply_op, assert_declared};
|
||||
use super::records::{
|
||||
ChangeRecord, CurationResult, Recorder, record_weight, strip_marks, touch_order,
|
||||
ChangeRecord, CurationResult, RecordParts, Recorder, charge, strip_marks, touch_order,
|
||||
};
|
||||
use super::schema::{Change, ChangeSummary, Op, SetOp};
|
||||
use super::shape::assert_same_shape;
|
||||
@@ -67,6 +67,7 @@ fn mark_to_confirm(
|
||||
doc: &mut Value,
|
||||
change: &ChangeSummary,
|
||||
records: &mut Vec<ChangeRecord>,
|
||||
spent: &mut usize,
|
||||
) -> Result<(), PipelineError> {
|
||||
for touched in touched_reviewable(doc)? {
|
||||
let item = item_at_mut(doc, &touched.path)?;
|
||||
@@ -85,64 +86,47 @@ fn mark_to_confirm(
|
||||
value: Value::from("to_confirm"),
|
||||
scope: None,
|
||||
});
|
||||
let after = Value::from("to_confirm");
|
||||
let parts = RecordParts {
|
||||
change,
|
||||
op: &op,
|
||||
target: &touched.target,
|
||||
field: Some("review_status"),
|
||||
before: &before,
|
||||
after: &after,
|
||||
};
|
||||
charge(spent, parts)?;
|
||||
records.push(ChangeRecord {
|
||||
change: change.clone(),
|
||||
op,
|
||||
target: touched.target,
|
||||
field: Some("review_status".to_owned()),
|
||||
before,
|
||||
after: Value::from("to_confirm"),
|
||||
after,
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The total weight of the records (`records::record_weight`): about 16 MiB of text, or a
|
||||
/// quarter of a million JSON values, far past the equine changeset (test
|
||||
/// `the_records_cannot_outgrow_their_bound`) and small enough that the reports built from them
|
||||
/// stay bounded too. Checked after every op, before the next one records anything.
|
||||
pub const MAX_RECORDED_WEIGHT: usize = 16 * 1024 * 1024;
|
||||
|
||||
/// The records counted so far and their weight.
|
||||
#[derive(Default)]
|
||||
struct Tally {
|
||||
counted: usize,
|
||||
weight: usize,
|
||||
}
|
||||
|
||||
fn assert_recorded(records: &[ChangeRecord], tally: &mut Tally) -> Result<(), PipelineError> {
|
||||
for record in &records[tally.counted..] {
|
||||
tally.weight = tally.weight.saturating_add(record_weight(record));
|
||||
}
|
||||
tally.counted = records.len();
|
||||
if tally.weight > MAX_RECORDED_WEIGHT {
|
||||
return Err(fail(format!(
|
||||
"the records weigh more than {MAX_RECORDED_WEIGHT} units"
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn apply_change(
|
||||
doc: &mut Value,
|
||||
change: &Change,
|
||||
records: &mut Vec<ChangeRecord>,
|
||||
tally: &mut Tally,
|
||||
spent: &mut usize,
|
||||
) -> Result<(), PipelineError> {
|
||||
let summary = change.summary();
|
||||
assert_declared(doc, &change.reference_ids)
|
||||
.map_err(|error| fail(format!("{}: {error}", change.id)))?;
|
||||
let mut next_touch = 0;
|
||||
for (index, op) in change.ops.iter().enumerate() {
|
||||
let mut recorder = Recorder::new(&summary, op, records, &mut next_touch);
|
||||
let mut recorder = Recorder::new(&summary, op, records, (&mut next_touch, &mut *spent));
|
||||
apply_op(doc, &mut recorder)
|
||||
.and_then(|()| assert_depth(doc))
|
||||
.and_then(|()| assert_recorded(records, tally))
|
||||
.map_err(|error| fail(format!("{} op {index}: {error}", change.id)))?;
|
||||
}
|
||||
if change.level.needs_confirmation() {
|
||||
mark_to_confirm(doc, &summary, records)?;
|
||||
assert_recorded(records, tally).map_err(|error| fail(format!("{}: {error}", change.id)))?;
|
||||
mark_to_confirm(doc, &summary, records, spent)
|
||||
.map_err(|error| fail(format!("{}: {error}", change.id)))?;
|
||||
}
|
||||
strip_marks(doc);
|
||||
Ok(())
|
||||
@@ -235,9 +219,9 @@ pub fn apply_changeset(
|
||||
.ok_or_else(|| fail("the baseline is not an object"))?
|
||||
.insert("references".to_owned(), registry);
|
||||
let mut records = Vec::new();
|
||||
let mut tally = Tally::default();
|
||||
let mut spent = 0;
|
||||
for change in changes {
|
||||
apply_change(&mut doc, change, &mut records, &mut tally)?;
|
||||
apply_change(&mut doc, change, &mut records, &mut spent)?;
|
||||
}
|
||||
curated_issues(&doc)?;
|
||||
Ok(CurationResult {
|
||||
|
||||
@@ -118,7 +118,7 @@ fn set_self(
|
||||
recorder.touch(item);
|
||||
let before = read_field(item, &op.field)?;
|
||||
write_field(item, &op.field, &op.value)?;
|
||||
recorder.record(&op.target, Some(&op.field), before, Some(op.value.clone()));
|
||||
recorder.record(&op.target, Some(&op.field), before, Some(op.value.clone()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -163,7 +163,7 @@ fn set_subtree(
|
||||
Some(&op.field),
|
||||
before,
|
||||
Some(op.value.clone()),
|
||||
);
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -199,7 +199,7 @@ fn apply_rename(
|
||||
item.insert(key.to_owned(), Value::from(op.id.clone()));
|
||||
recorder.touch(item);
|
||||
let after = Value::Object(item.clone());
|
||||
recorder.record(&op.target, None, Some(before), Some(after));
|
||||
recorder.record(&op.target, None, Some(before), Some(after))?;
|
||||
let rename = Rename {
|
||||
kind: located.target.kind,
|
||||
from,
|
||||
@@ -331,7 +331,7 @@ fn apply_add(
|
||||
added.push(Step::Index(index));
|
||||
recorder.touch_tree(doc, &added)?;
|
||||
let target = format!("{}{id}", dest.prefix);
|
||||
recorder.record(&target, None, None, Some(op.value.clone()));
|
||||
recorder.record(&target, None, None, Some(op.value.clone()))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -345,7 +345,7 @@ fn apply_remove(
|
||||
return Err(fail("the document cannot be removed"));
|
||||
};
|
||||
let item = list_at_mut(doc, list)?.remove(located.index);
|
||||
recorder.record(&op.target, None, Some(item), None);
|
||||
recorder.record(&op.target, None, Some(item), None)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -383,7 +383,7 @@ fn apply_move(
|
||||
recorder.touch(item);
|
||||
let before = Value::Object(item.clone());
|
||||
let after = format!("{}{id}", child_prefix(&parent));
|
||||
recorder.record(&op.target, None, Some(before), Some(Value::from(after)));
|
||||
recorder.record(&op.target, None, Some(before), Some(Value::from(after)))?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -449,7 +449,7 @@ fn apply_cite(
|
||||
Some("reference_ids"),
|
||||
Some(Value::Array(before)),
|
||||
Some(Value::Array(after)),
|
||||
);
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -199,13 +199,26 @@ fn view_ids(document: &Value) -> Vec<String> {
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The review drawings' bytes, projected before any is rendered: every drawing repeats the
|
||||
/// whole cleaned drawing, two per view (the equine draft: 2 views, about 0.55 MB). Views added
|
||||
/// by a changeset multiply it (2,000 views asked for 547 MB, security gate R5.4 round 8).
|
||||
pub const MAX_REVIEW_DRAWINGS_BYTES: usize = 64 * 1024 * 1024;
|
||||
|
||||
/// One review overlay per view (`review/annotations-<view>.svg`), then the application's
|
||||
/// drawing of each view (`review/app-<view>.svg`).
|
||||
fn review_drawings(draft: &DraftResult) -> Result<Vec<(String, String)>, PipelineError> {
|
||||
let views = view_ids(&draft.document);
|
||||
let projected = draft.cleaned_svg.len().saturating_mul(2 * views.len());
|
||||
if projected > MAX_REVIEW_DRAWINGS_BYTES {
|
||||
return Err(fail(format!(
|
||||
"{} views would write {projected} bytes of review drawings, past the \
|
||||
{MAX_REVIEW_DRAWINGS_BYTES} bound",
|
||||
views.len()
|
||||
)));
|
||||
}
|
||||
let drawing = read_cleaned_drawing(&draft.cleaned_svg)?;
|
||||
let typed: AnatomicalSheetDocument = serde_json::from_value(draft.document.clone())
|
||||
.map_err(|error| fail(format!("curated document: {error}")))?;
|
||||
let views = view_ids(&draft.document);
|
||||
let mut files = Vec::new();
|
||||
for view_id in &views {
|
||||
let overlay = render_review_overlay(&OverlayInput {
|
||||
|
||||
@@ -33,6 +33,34 @@ pub struct CurationResult {
|
||||
pub records: Vec<ChangeRecord>,
|
||||
}
|
||||
|
||||
/// The total weight of the records (`record_weight`): about 16 MiB of text, or a quarter of a
|
||||
/// million JSON values, far past the equine changeset (test `the_records_cannot_outgrow_their_bound`)
|
||||
/// and small enough that the reports built from them stay bounded too. Charged as each record
|
||||
/// is made, before the change and the op are cloned into it: one op touching many items is
|
||||
/// refused at the record that passes the bound.
|
||||
pub const MAX_RECORDED_WEIGHT: usize = 16 * 1024 * 1024;
|
||||
|
||||
/// The records' running weight; refuses the record that would pass `MAX_RECORDED_WEIGHT`.
|
||||
pub fn charge(spent: &mut usize, record: RecordParts<'_>) -> Result<(), PipelineError> {
|
||||
*spent = spent.saturating_add(parts_weight(&record));
|
||||
if *spent > MAX_RECORDED_WEIGHT {
|
||||
return Err(PipelineError::Guard(format!(
|
||||
"the records weigh more than {MAX_RECORDED_WEIGHT} units"
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// What a record is made of, borrowed: weighed before anything is cloned.
|
||||
pub struct RecordParts<'a> {
|
||||
pub change: &'a ChangeSummary,
|
||||
pub op: &'a Op,
|
||||
pub target: &'a str,
|
||||
pub field: Option<&'a str>,
|
||||
pub before: &'a Value,
|
||||
pub after: &'a Value,
|
||||
}
|
||||
|
||||
/// What the op functions of one change share besides the document; the touch counter spans
|
||||
/// the ops of the change (the first touch of an item fixes its place in the review marking).
|
||||
pub struct Recorder<'a> {
|
||||
@@ -40,6 +68,7 @@ pub struct Recorder<'a> {
|
||||
pub op: &'a Op,
|
||||
pub records: &'a mut Vec<ChangeRecord>,
|
||||
next_touch: &'a mut u64,
|
||||
spent: &'a mut usize,
|
||||
}
|
||||
|
||||
impl<'a> Recorder<'a> {
|
||||
@@ -47,32 +76,45 @@ impl<'a> Recorder<'a> {
|
||||
change: &'a ChangeSummary,
|
||||
op: &'a Op,
|
||||
records: &'a mut Vec<ChangeRecord>,
|
||||
next_touch: &'a mut u64,
|
||||
counters: (&'a mut u64, &'a mut usize),
|
||||
) -> Self {
|
||||
Self {
|
||||
change,
|
||||
op,
|
||||
records,
|
||||
next_touch,
|
||||
next_touch: counters.0,
|
||||
spent: counters.1,
|
||||
}
|
||||
}
|
||||
|
||||
/// Absent values are recorded as null so records stay plain JSON.
|
||||
/// Absent values are recorded as null so records stay plain JSON. Refused when the
|
||||
/// records would pass their bound (`charge`), before the change and the op are cloned.
|
||||
pub fn record(
|
||||
&mut self,
|
||||
target: &str,
|
||||
field: Option<&str>,
|
||||
before: Option<Value>,
|
||||
after: Option<Value>,
|
||||
) {
|
||||
) -> Result<(), PipelineError> {
|
||||
let (before, after) = (unmarked(before), unmarked(after));
|
||||
let parts = RecordParts {
|
||||
change: self.change,
|
||||
op: self.op,
|
||||
target,
|
||||
field,
|
||||
before: &before,
|
||||
after: &after,
|
||||
};
|
||||
charge(self.spent, parts)?;
|
||||
self.records.push(ChangeRecord {
|
||||
change: self.change.clone(),
|
||||
op: self.op.clone(),
|
||||
target: target.to_owned(),
|
||||
field: field.map(str::to_owned),
|
||||
before: unmarked(before),
|
||||
after: unmarked(after),
|
||||
before,
|
||||
after,
|
||||
});
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Marks an item as touched by the change, once (the first touch keeps its order).
|
||||
@@ -105,29 +147,48 @@ impl<'a> Recorder<'a> {
|
||||
}
|
||||
}
|
||||
|
||||
/// What the records hold, in weight units: 64 per JSON value (a `Value` node costs tens of
|
||||
/// bytes) plus one per byte of every string (the values' and the change's texts, which each
|
||||
/// record clones). A rename or a remove records the whole item, so the records can grow far
|
||||
/// past the document; `apply` bounds their total (`MAX_RECORDED_WEIGHT`).
|
||||
/// What a record holds, in weight units: 64 per JSON value (a `Value` node costs tens of
|
||||
/// bytes) plus one per byte of every string: the values, and the change and the op, which each
|
||||
/// record clones. A rename or a remove records the whole item and one op can record once per
|
||||
/// item it reaches, so the records can grow far past the document: `charge` bounds their total.
|
||||
pub fn record_weight(record: &ChangeRecord) -> usize {
|
||||
let change = &record.change;
|
||||
parts_weight(&RecordParts {
|
||||
change: &record.change,
|
||||
op: &record.op,
|
||||
target: &record.target,
|
||||
field: record.field.as_deref(),
|
||||
before: &record.before,
|
||||
after: &record.after,
|
||||
})
|
||||
}
|
||||
|
||||
fn parts_weight(parts: &RecordParts<'_>) -> usize {
|
||||
let change = parts.change;
|
||||
let texts = [
|
||||
change.id.as_str(),
|
||||
&change.title.fr,
|
||||
&change.title.en,
|
||||
&change.rationale,
|
||||
change.question.as_deref().unwrap_or_default(),
|
||||
&record.target,
|
||||
record.field.as_deref().unwrap_or_default(),
|
||||
parts.target,
|
||||
parts.field.unwrap_or_default(),
|
||||
];
|
||||
let text: usize = texts.iter().map(|text| text.len()).sum::<usize>()
|
||||
+ change.reference_ids.iter().map(String::len).sum::<usize>();
|
||||
let op_value = match &record.op {
|
||||
Op::Set(op) => value_weight(&op.value),
|
||||
text + op_weight(parts.op) + value_weight(parts.before) + value_weight(parts.after)
|
||||
}
|
||||
|
||||
/// The op a record clones: its target, its value or its ids.
|
||||
fn op_weight(op: &Op) -> usize {
|
||||
let carried = match op {
|
||||
Op::Set(op) => op.field.len() + value_weight(&op.value),
|
||||
Op::Add(op) => value_weight(&op.value),
|
||||
_ => 0,
|
||||
Op::Rename(op) => op.id.len(),
|
||||
Op::Move(op) => op.to.len(),
|
||||
Op::Cite(op) => op.reference_ids.iter().map(|id| 64 + id.len()).sum(),
|
||||
Op::Remove(_) => 0,
|
||||
};
|
||||
text + op_value + value_weight(&record.before) + value_weight(&record.after)
|
||||
64 + op.target().len() + carried
|
||||
}
|
||||
|
||||
/// The weight of a value, walked with an explicit stack (no recursion).
|
||||
|
||||
@@ -74,7 +74,7 @@ fn follow_link(
|
||||
Some(link.field),
|
||||
Some(from.clone()),
|
||||
Some(Value::from(rename.to.as_str())),
|
||||
);
|
||||
)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -109,7 +109,7 @@ fn follow_citations(
|
||||
Some("reference_ids"),
|
||||
Some(before),
|
||||
Some(Value::Array(after.clone())),
|
||||
);
|
||||
)?;
|
||||
item.insert("reference_ids".to_owned(), Value::Array(after));
|
||||
}
|
||||
Ok(())
|
||||
|
||||
@@ -764,14 +764,26 @@ fn an_integer_beyond_2_53_is_read_as_the_double_json_parse_gives() {
|
||||
);
|
||||
}
|
||||
|
||||
/// The records bound now stops a growing changeset first (every added value is recorded); the
|
||||
/// document bound still holds a baseline near it: one small add past 200,000 values is refused.
|
||||
#[test]
|
||||
fn the_document_cannot_hold_more_values_than_the_bound() {
|
||||
let children: Vec<Value> = (0..70_000)
|
||||
.map(|i| json!({ "id": format!("n{i}"), "name": { "fr": "a", "en": "a" } }))
|
||||
.collect();
|
||||
let value = json!({ "id": "many", "name": { "fr": "a", "en": "a" }, "children": children });
|
||||
let op = json!({ "op": "add", "target": MOBILITY, "value": value });
|
||||
let error = mini_apply(json!([op])).unwrap_err();
|
||||
let frozen = oracle("curation-synthetic.json");
|
||||
let references = parse_references(&to_js_compact(&frozen["references_file"]).unwrap()).unwrap();
|
||||
let mut baseline = frozen["baseline"].clone();
|
||||
baseline["padding"] = Value::from(vec![Value::Null; 199_950]);
|
||||
let value = json!({ "id": "small", "name": { "fr": "a", "en": "a" },
|
||||
"children": (0..20).map(|i| json!({ "id": format!("n{i}"), "name": { "fr": "a", "en": "a" } }))
|
||||
.collect::<Vec<_>>() });
|
||||
let file = json!({ "changes": [{
|
||||
"id": "CHG-001", "level": "A", "kind": "technical",
|
||||
"title": { "fr": "Essai", "en": "Test" }, "rationale": "Essai",
|
||||
"reference_ids": [], "ops": [{ "op": "add", "target": MOBILITY, "value": value }],
|
||||
}]});
|
||||
let changeset = parse_changeset(&to_js_compact(&file).unwrap()).unwrap();
|
||||
let error = apply_changeset(&baseline, &references, &changeset.changes)
|
||||
.unwrap_err()
|
||||
.to_string();
|
||||
assert!(error.contains("more than 200000 values"), "{error}");
|
||||
}
|
||||
|
||||
@@ -780,7 +792,7 @@ fn the_document_cannot_hold_more_values_than_the_bound() {
|
||||
/// 5,000-child item asked for 4.3 GB). The equine changeset stays far below the bound.
|
||||
#[test]
|
||||
fn the_records_cannot_outgrow_their_bound() {
|
||||
use faunosteo_core::curation::draft::apply::MAX_RECORDED_WEIGHT;
|
||||
use faunosteo_core::curation::draft::records::MAX_RECORDED_WEIGHT;
|
||||
use faunosteo_core::curation::draft::records::record_weight;
|
||||
let equine: usize = equine_draft().records.iter().map(record_weight).sum();
|
||||
println!("equine records weigh {equine} units");
|
||||
@@ -802,6 +814,42 @@ fn the_records_cannot_outgrow_their_bound() {
|
||||
assert!(error.contains("the records weigh more than"), "{error}");
|
||||
}
|
||||
|
||||
/// One op, or the B/C marking, records once per item it reaches: the bound is charged at each
|
||||
/// record, before the change and the op are cloned into it (security gate R5.4 round 8: a level
|
||||
/// B add of 10,000 children with a 128 KiB rationale reached 1.37 GB inside one marking).
|
||||
#[test]
|
||||
fn the_records_bound_holds_inside_one_op_and_the_marking() {
|
||||
let children: Vec<Value> = (0..5_000)
|
||||
.map(|i| json!({ "id": format!("n{i}"), "name": { "fr": "a", "en": "a" } }))
|
||||
.collect();
|
||||
let value = json!({ "id": "many", "name": { "fr": "a", "en": "a" }, "children": children });
|
||||
let add = json!({ "op": "add", "target": MOBILITY, "value": value });
|
||||
let frozen = oracle("curation-synthetic.json");
|
||||
let references = parse_references(&to_js_compact(&frozen["references_file"]).unwrap()).unwrap();
|
||||
let description = json!({ "fr": "d".repeat(4_096), "en": "d" });
|
||||
let subtree = json!({ "op": "set", "target": format!("{MOBILITY}/many"), "field": "description",
|
||||
"value": description, "scope": "subtree" });
|
||||
let cases = [
|
||||
("B", "r".repeat(8_192), json!([add])),
|
||||
("A", "Essai".to_owned(), json!([add, subtree])),
|
||||
];
|
||||
for (level, rationale, ops) in cases {
|
||||
let file = json!({ "changes": [{
|
||||
"id": "CHG-001", "level": level, "kind": "technical",
|
||||
"title": { "fr": "Essai", "en": "Test" }, "rationale": rationale,
|
||||
"reference_ids": [], "ops": ops,
|
||||
}]});
|
||||
let changeset = parse_changeset(&to_js_compact(&file).unwrap()).unwrap();
|
||||
let error = apply_changeset(&frozen["baseline"], &references, &changeset.changes)
|
||||
.unwrap_err()
|
||||
.to_string();
|
||||
assert!(
|
||||
error.contains("the records weigh more than"),
|
||||
"{level}: {error}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A `__proto__` segment had no faithful reading in the TypeScript (a prototype set, a
|
||||
/// polluted `Object.prototype`, or nothing written): the port refuses it at the op.
|
||||
#[test]
|
||||
|
||||
@@ -496,3 +496,24 @@ fn ids_and_field_segments_that_could_forge_report_structure_are_refused() {
|
||||
let rule = TriggerNames::of(&typed(&draft.document));
|
||||
assert!(compare_drawing("<svg/>", &deep, Some(&rule)).is_err());
|
||||
}
|
||||
|
||||
/// Every review drawing repeats the whole cleaned drawing, two per view: the bytes are
|
||||
/// projected and bounded before any drawing is rendered (security gate R5.4 round 8).
|
||||
#[test]
|
||||
fn review_drawings_past_their_bound_are_refused_before_rendering() {
|
||||
let (result, mut draft) = equine();
|
||||
let views = draft.document["views"].as_array_mut().unwrap();
|
||||
let main = views[0].clone();
|
||||
for index in 0..300 {
|
||||
let mut view = main.clone();
|
||||
view["id"] = Value::from(format!("extra-{index}"));
|
||||
views.push(view);
|
||||
}
|
||||
let error = render_draft_artifacts(&draft, &result.anomalies).unwrap_err();
|
||||
assert!(
|
||||
error
|
||||
.to_string()
|
||||
.contains("bytes of review drawings, past the"),
|
||||
"{error}"
|
||||
);
|
||||
}
|
||||
@@ -3204,7 +3204,8 @@ pass the guard unchanged.
|
||||
| authored text outside the allowlist: Markdown Prettier rewrites (`*x*`, `_x`, `x_**a**`, `**a****b**`, `Voir_https://…`, `a b`, a non-ASCII space, a tab, a line break in a list item, `` `` ``, `a\|b`, `[a](<b>)`, `[[ a ]]`, `*`…), the constructs refused whole (links, HTML tags, autolinks, character references) and any character outside the admitted set (pictographs, symbols, combining marks without the Alphabetic property; an alphabetic mark such as U+0345 is a letter and Prettier aligns it alike) | written, then rewritten by Prettier (escapes, `_emphasis_`, collapsed spaces, indented continuation lines, normalised links; a row with an extra cell), or kept | refused (`UnsupportedMarkdown`), even where Prettier would have kept the text |
|
||||
| a pictograph in a table cell (an emoji, and ©, ®, ™, ✓, ★, ⚠ and the other characters `text::is_emoji_candidate` lists) | measured with Prettier's tables | refused (`UnsupportedWidth`, R5.2): the set is wider than the emoji, the refusal stays loud |
|
||||
| an aligned report table past 32 Mi display columns in all (rows × padded row width; the equine CHANGELOG's widest table is about 0.5 MB): one long authored cell pads every row of its level, so an 80 KB rationale over 14,000 records asked for 1 GB (security gate R5.4 round 6) | Prettier aligned it (the same amplification) | refused (`Guard`): the CHANGELOG's tables are projected from their cells (bytes bound the display width) before the raw table is joined (test `a_changelog_whose_alignment_would_explode_is_refused_before_it_is_built`, through the pipeline), every other table in the aligner before any padded row is built (test `an_aligned_table_past_the_bound_is_refused_before_it_is_built`), import report included |
|
||||
| records weighing more than 16 Mi units in all (`records::record_weight`: 64 per JSON value plus one per string byte of the values and of the change's texts, which each record clones; the equine records weigh 1,084,422): a rename or a remove records the whole item, so 400 renames of a 5,000-child item asked for 4.3 GB, and an 80 KB rationale repeated over 14,000 records for 1.1 GB of raw rows (security gate R5.4 round 7) | recorded (the same copies) | refused at the op that passes the bound, before the next op records anything (test `the_records_cannot_outgrow_their_bound`); the raw report rows, built from the records, are bounded with them |
|
||||
| records weighing more than 16 Mi units in all (`records::record_weight`: 64 per JSON value plus one per string byte of the values, of the change and of the op, which each record clones; the equine records weigh 1,195,000): a rename or a remove records the whole item and one op, or the B/C marking, records once per item it reaches, so 400 renames of a 5,000-child item asked for 4.3 GB (round 7), a level B add of 10,000 children with a 128 KiB rationale 1.37 GB inside one marking, a 64 KiB value set over a subtree 2.0 GB (round 8) | recorded (the same copies) | refused at the record that passes the bound, charged before the change and the op are cloned into it (`records::charge`, tests `the_records_cannot_outgrow_their_bound`, `the_records_bound_holds_inside_one_op_and_the_marking`); the raw report rows, built from the records, are bounded with them; the 200,000-value document bound now holds the baseline (test from a padded baseline) |
|
||||
| review drawings past 64 MiB in all, projected before any is rendered (two per view, each repeating the cleaned drawing; the equine draft writes about 0.55 MB): 2,000 views asked for 547 MB (security gate R5.4 round 8) | written | refused before rendering (`pipeline::MAX_REVIEW_DRAWINGS_BYTES`, test `review_drawings_past_their_bound_are_refused_before_rendering`) |
|
||||
| an id an `add` writes (the item's, and those nested in its `children` or `categories`) or a `rename` id that is not a stable id (a region code for a region), or a field segment outside `[A-Za-z0-9_-]` (a control character, NUL included, a backtick, a `]`, a space, a non-ASCII letter) | accepted when a later op removed or renamed the item: the raw id stayed in the records and reached the reports' code spans (a line break there forged a section, security gate R5.4 round 2) | refused at the op, the check the final document applies anyway (test `ids_and_field_segments_that_could_forge_report_structure_are_refused`) |
|
||||
| a label, id or path holding a code point XML 1.0 refuses, or a carriage return, in a review drawing | written (a non-well-formed file; a raw CR read back as a line feed) | refused: the drawing is not written |
|
||||
|
||||
@@ -3241,10 +3242,11 @@ ruled and frozen in the sweep, so a combination of admitted characters no probe
|
||||
differ; the comparison of the application's drawings accepts
|
||||
exactly the trigger-name rule, byte for byte; the frozen
|
||||
`horse.svg` is read, never regenerated, so a future change of the drawing needs its own
|
||||
decision; one op still records up to about three copies of an item under the 200,000-value bound
|
||||
before the records bound is checked (tens of MB at most); the review drawings are bounded only through the document's 200,000 values (two drawings
|
||||
of about 137 KB per view, about 2 GB at the bound, an estimate of the security gate R5.4 round 6,
|
||||
not measured), and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160,
|
||||
decision; the record that passes the bound has been built before it is refused (one item copy
|
||||
and one clone of the op's value, at most twice the 16 MiB read cap), and a subtree set writes its
|
||||
value into each item before recording it, so the document holds at most as many copies as the
|
||||
records bound admits; the comparison of `--check` looks up each trigger's view linearly (views ×
|
||||
triggers, the views bounded through the drawings bound); and the allowlist admits the invisible Hangul fillers (U+3164, U+115F, U+1160,
|
||||
U+FFA0, letters Prettier keeps, no structure); the residual risks of §11.13 stand.
|
||||
|
||||
Conditions for R5.5: the owner's validation of this checkpoint; the upload tool (`ureq`, the
|
||||
|
||||
Reference in new issue
Block a user