From 6c0fe88725b7ca349fdb62ec762eb72ad17985e1 Mon Sep 17 00:00:00 2001 From: bchanot Date: Fri, 23 Oct 2026 10:11:51 +0200 Subject: [PATCH] =?UTF-8?q?docs(docs):=20R5.3=20third=20final=20round=20de?= =?UTF-8?q?clared=20(RUST.md=20=C2=A711.12-11.13),=20security=20final3,=20?= =?UTF-8?q?verify=20record?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../audits/2026-10-22-rust-r5-draft-verify.md | 6 + ...026-10-23-rust-r5-draft-security-final3.md | 188 ++++++++++++++++++ docs/development/RUST.md | 32 ++- 3 files changed, 217 insertions(+), 9 deletions(-) create mode 100644 .claude/audits/2026-10-23-rust-r5-draft-security-final3.md diff --git a/.claude/audits/2026-10-22-rust-r5-draft-verify.md b/.claude/audits/2026-10-22-rust-r5-draft-verify.md index c924ed1f..68ec1d1b 100644 --- a/.claude/audits/2026-10-22-rust-r5-draft-verify.md +++ b/.claude/audits/2026-10-22-rust-r5-draft-verify.md @@ -46,3 +46,9 @@ Contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`. Fresh verifier, - Verifier ECARTS(3): C2, C3 (one family) and C13. serde's derives accept a struct written as a positional array and a unit enum written as a one-key map, in changes.json, references.json and the curated document (gate through the R2 derives); Zod refused all. 55,000-changeset random differential (TS generator, Rust probe): 7,436 accepted on both sides byte-identical, 335 mismatches, all this family but one message (`document has no "categories"` vs `document has no list "categories"`). C1, C4 to C12 MET (18 tests, 203 crate tests, `--check` 0/12, refusal probes, floor clean). Note: the same derives sit in the R2 report and archive contracts (unverified there). - Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final2.md`): MEDIUM memory amplification (no cap); LOW quadratic depth walk, `--check` and reads following symlinks, residual TOCTOU, `write_exclusive` any component, lossy normalisation, test aimed at the real seed, suppressions context, lint list gaps. - Fixed: `draft/shape.rs` (typed reading serialised back, compared by shape) on both authored files and the gate; 10 changesets + 2 references files frozen from TS (all refused by TS, refused alike); missing-list message aligned; reads: regular file, 16 MiB cap; `listed` does not follow links; `write_exclusive` plain names, root re-checked; seed guard as pure `inside` + unit test, integration test no longer targets the seed. Kept, declared in §11.13: memory amplification, quadratic walk, single-user TOCTOU residual, lossy UTF-8 and repeated keys. Raised to the owner: the R2 derive leniency in the gated readers. + +## Third final round on 42bbc48 + +- Verifier ECARTS(2), C3 + C13, one family: a null or primitive among the children of an added item. TS: TypeError where walked (refused), skipped by a reference rename's citation walk (accepted if later removed), gate if kept. Rust: tolerated by touch/walk (accepted), refused by rename's item_at_mut. Three splits; §11.12 claimed "both refuse" for the null child, false. Differential: 82,000 random changesets, 13,758 accepted identical (document, records, stats, map), 63 schema + 34 references probes agree. C1, C2, C4 to C12 MET (19 tests). +- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final3.md`), LOW only: reads follow folder links out of the seed; shape comparison ignores array length; a folder appearing after the claim accepted; seed guard wiring untested; `set name.fr.x` records `before: null`; lint allows (context). +- Fixed: non-object children refused at the add (3 TS-frozen cases, 1 declared refusal); reads confined to the canonical seed; array length compared; only folders this run created; `claim_destination(typed, seed)` + fixture unit test. Declared: replaced-string `before` (TS same), memory factor ~2x TS. diff --git a/.claude/audits/2026-10-23-rust-r5-draft-security-final3.md b/.claude/audits/2026-10-23-rust-r5-draft-security-final3.md new file mode 100644 index 00000000..d8efb9a5 --- /dev/null +++ b/.claude/audits/2026-10-23-rust-r5-draft-security-final3.md @@ -0,0 +1,188 @@ +# Security gate (fresh, audit mode): R5.3 curated draft, third final round + +Branch `feature/rust-r5-pipeline-core`, HEAD `42bbc48`. Scope: `git diff 5e114da..HEAD`, 34 files, +with the fix commit `776c709` read whole (shape.rs, schema.rs, apply.rs, targets.rs, +bin/import_equine.rs, tests/curation_draft.rs), plus every other changed product file under +`crates/core/src/curation/draft/` and `scripts/import-equine.sh`. + +``` +SECURITY — VERDICT: PASS +TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten +SCOPE: 34 files +BLOCKING: + (none) +REPORTED (non-blocking): + - [LOW] reads-follow-directory-symlinks — crates/core/src/bin/import_equine.rs:74 + - [LOW] shape-array-length-not-enforced — crates/core/src/curation/draft/shape.rs:47 + - [LOW] write-exclusive-accepts-a-new-folder — crates/core/src/bin/import_equine.rs:282 + - [LOW] seed-guard-wiring-untested — crates/core/src/bin/import_equine.rs:223 + - [LOW] write-field-before-null-on-replaced-holder — crates/core/src/curation/draft/fields.rs:131 + - [LOW] clippy-allow-new-in-test (context) — crates/core/tests/curation_draft.rs:7 + - [LOW] clippy-allow-pre-existing (context) — crates/core/src/bin/import_equine.rs:15 +PROOF: semgrep 163 rules on 34 files → 0 findings; checklist 14 checks → 7 findings +``` + +## Method + +- semgrep 1.168.0, the three pinned packs, `--metrics=off --quiet --json`, on the 34 existing + files of the diff (16 `.rs`, 13 `.md`, 2 `.json`, 1 `.ts`, 1 `.sh`, 1 `.toml`). Exit 0, 0 + errors, 0 results; the `--time` rerun counts 163 rules. No `EIO_BACKEND=posix` was needed. + These packs carry few Rust rules, so the verdict rests on reading the code and on the probes. +- Checklist (14 checks): hardcoded secrets, SQL built by formatting, unsafe render, authz, + errors to the user, secrets in logs, tracked `.env` / credentials, guardrails scanner, + new suppressions, `--out` destination containment, read containment, shape and strictness + gate, depth and resource caps, author data loss and audit trail. +- Probes, all under + `/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/probe.m1yh/`: + a copy of the seed (`seedroot/sheets/equine`), a throwaway crate linking `faunosteo-core` + with `curation`: `ie` = `import_equine.rs` with only the `SEED` constant pointed at the + copy (diff = those lines only), and `shape` = direct calls to `parse_changeset` and + `assert_same_shape`. No `--out` ever targeted the repository. Nothing was deleted; the + scratch artifacts stay in place. + +| Probe | Result | +|---|---| +| `ie --check` on the unmodified copy | legacy-import 6 files 0 differences, draft 6 files 0 differences, 6 pending R5.4, exit 0 | +| `--out` parent = symlink into the seed (`seedlink/new`, `//seedlink//new`) | refused, inside the seed folder | +| `--out` = symlink to the seed (`seedlink`, `seedlink/`) | refused, a symbolic link | +| `--out` = seed itself / empty folder inside seed / new folder inside seed / `sheets/./equine/new` | refused (not empty / inside the seed folder) | +| `--out /`, `--out ""` | refused | +| `INIT_CWD=` with a relative `--out` | refused, inside the seed folder | +| FIFO at a committed file name in `draft/` | not opened, reported, exit 1, no hang | +| 17 MiB sparse input in `legacy-sources/` | refused, larger than 16777216 bytes, exit 2 | +| `add` of a value nested 118 levels; `add` of a 60-level children chain (level B) | refused, nests more than 64 levels, exit 2, no stack overflow | +| changeset value nested 120 / 140 levels | parsed / refused by serde's recursion limit | +| title as array, level as map, scope as map, change as array, position as array | all refused (shape gate or untagged enum) | +| `draft/assets` and `curation` replaced by directory symlinks to outside folders | see L1: read through both links | + +## Findings (all LOW, non-blocking) + +**L1. Reads follow a symbolic link in any folder component. `import_equine.rs:64-75` +(read at :74), callers :84, :96-97, :123-127, :178.** +`read` checks the final component with `symlink_metadata` (:67), then `fs::read(path)` (:74) +resolves the whole path again. A directory symlink inside the seed is followed. Measured: with +`draft/assets` and `curation` swapped for directory symlinks to folders outside the seed copy, +`--out` wrote its 12 files, exit 0, no warning, built from the changeset and the cleaned drawing +read through the links. `--check` read `assets/horse.svg` through the link without calling it +missing, and only flagged `not produced: assets`, exit 1. Between :67 and :74 a swap of the +final component is also followed, and a FIFO swapped in would block the read. Impact is small: +read-only, regular files under 16 MiB, output only to a fresh folder or a comparison, and the +symlink must already sit in the user's own repository. But the docstring at :64 ("never a +symbolic link") and RUST.md:3017 ("every read a regular file under 16 MiB, never a link") +claim more than the code does, and the owner requirement says no unintended access through +symbolic links. The §11.13 single-user residual names `--out` only. +Fix: open once and check the handle. Either canonicalise the path and require it to equal the +lexical path under the canonical seed (no link in any component), then `File::open`, +`file.metadata()` for `is_file` and length, and read through `take(MAX_FILE_BYTES + 1)`. Or +open with `O_NOFOLLOW` per component. Otherwise, correct :64 and RUST.md:3017 to "final +component only" and add reads to the §11.13 residual. + +**L2. The shape gate does not enforce array length. `shape.rs:47-53`.** +When two arrays differ in length, the length guard on the array arm (:47) fails, and the +fall-through at :52 returns `None` because both are arrays. Probe: +`assert_same_shape({"a":[1,2,3]}, {"a":[1]})` returns `Ok(())`. The module doc (:5) and +RUST.md:2969 promise "an array an array of the same length". This is unreachable today. No +deserializer in `draft/schema.rs` or `domain/sheet/schema.rs` changes an array's length: +`non_empty_strings`, `stable_ids` and `at_least_three` keep every element, and `Point` reads +an exact 2-tuple. So it is a latent gap: a future filtering or deduplicating deserializer +would drop elements without the gate noticing. +Fix: `(Value::Array(w), Value::Array(t)) if w.len() != t.len() => Some(format!("expected +{} elements at {path}, received {}", t.len(), w.len()))` ahead of the equal-length arm. + +**L3. `write_exclusive` accepts a folder that appeared after the claim. +`import_equine.rs:280-284`, docstring :261-263, RUST.md:2902.** +The docs say an entry that appeared since the claim, "file, folder or symbolic link, is +refused". Line 282 accepts any existing real directory. It has to, for the folders this run +created itself (`legacy-import/` for its second file, `assets/`). This is harmless: it stays +inside the claimed root, a symlink is still refused (`symlink_metadata`), and files stay +`create_new`, so nothing is overwritten. It falls within the declared single-user residual. +Fix: correct the docs to "a file or a symbolic link". Or record the folders this run created +and refuse any other one that already exists. + +**L4. The seed guard is tested as a predicate, never through its call site. +`import_equine.rs:223` (wiring), :365-380 (test).** +The unit test pins `inside()` on literal paths. No test runs `claim_destination` against a +seed. A regression at the call site would pass the whole suite: comparing the lexical `out` +instead of the canonical `parent`, or dropping :222-225. That guard backs the owner's first +requirement. The probes above show today's wiring is correct. +Fix: make the seed a parameter of `claim_destination` (the binary passes `SEED`). Then test it +on a temporary "seed" folder: a parent symlinked into it, the seed itself, a new folder under +it. This never touches the real seed. + +**L5. Replacing a non-object intermediate records `before: null`. +`fields.rs:126-139` (replace at :131-133), `read_field` :111-122, used at `ops.rs:116-118`.** +`set name.fr.x` turns the string `name.fr` into `{x: …}`. `read_field` returns `None` for +that path, so the record says `before: null` and the original string appears in no record. +The TypeScript does the same (`fields.ts:62-80`). The final document is safe: the strict gate +refuses the stray object unless a later op overwrites the field. In that case the CHANGELOG +trail lacks the original value. This is the family of the declared inherited-member row +(§11.12) and is not declared. +Fix: declare it in the §11.12 table, or refuse a write through an existing non-object +intermediate (always an author error). + +**L6. New lint allow in a test (context). `crates/core/tests/curation_draft.rs:7`.** +`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]` on an +integration test that reads fixtures and runs the built binary. This is not a security-scanner +suppression, and product code keeps the lint. No `nosemgrep`, `nosec`, `eslint-disable`, +`ts-ignore` or `noqa` was added anywhere in the diff. + +**L7. Pre-existing lint allow (context). `crates/core/src/bin/import_equine.rs:15`.** +`#![allow(clippy::disallowed_methods)]` was already present at `5e114da` (line 11). This file +is the one place of the tool allowed the file system. + +## Checklist results + +| # | Check | Result | +|---|---|---| +| 1 | Hardcoded secret, token, key, auth URL (diff incl. docs, `.claude`, oracles) | none | +| 2 | SQL by formatting | `seed.rs`: fixed and pipeline values through `sql_text` (quotes doubled), the document in `$equine_doc$` with the tag refused (:91); no injection path. Nothing new | +| 3 | innerHTML / eval / unsafe render | none in code (prose in memory files only) | +| 4 | Endpoint without authz | N/A (local tool, no endpoint) | +| 5 | Internals surfaced to a user | N/A (the operator's own CLI) | +| 6 | Secret / PII in a log | none | +| 7 | Tracked `.env` / credentials | `.env.example` only | +| 8 | guardrails `scan.py --domain api,supabase` | 0 FAIL; 1 WARN `SB-POLICY-NOT-OWNER` at `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the delta, not counted; 1 MANUAL | +| 9 | New suppressions (anti-gaming) | none un-gated; L6, L7 context | +| 10 | `--out` containment | passes every probe; L3, L4 | +| 11 | Read containment | L1 | +| 12 | Shape and strictness gate | Zod's two refusals reproduced; L2 latent | +| 13 | Depth and resource caps | 16 MiB reads, serde depth 128, 8 field segments, document depth 64: all hold under probes | +| 14 | Author data loss, audit trail | NUL keys refused, marks stripped; L5 | + +## Owner requirements + +- Official seed never modified automatically: met. `--out` under or at the seed is refused + on canonical paths (8 probe paths), `--check` writes nothing, the binary test aims only at + `CARGO_TARGET_TMPDIR`, and the TypeScript generator writes oracles only under + `MAKE_R5_ORACLES=1`. Test gap: L4. +- `--check` writes nothing: met. Its code path holds only `symlink_metadata`, `read`, + `read_dir`. +- `--out` never deletes or overwrites: met. No remove call; `create_dir` and `create_new`. +- Dangerous paths and invalid destinations refused: met (empty, `..`, missing parent, `/`, + symlink with `/` or `/.`, file, non-empty folder, inside the seed). +- No unintended access through symbolic links: met on the write side (outside the declared + residual window). Partly met on the read side: L1. +- Strict changeset validation: met. L2 is latent. +- Depth and resource caps: met. +- Author data never dropped silently: met for the document. L5 concerns the record trail only. + +## Declared residuals (RUST.md §11.13) checked + +- Memory: no new cap. One correction: the Rust records clone the whole op per record + (`records.rs:70`), while the TypeScript shared `ctx.op` (`types.ts:41-48`). A subtree `set` + therefore holds its value twice per record, against once in the TypeScript: about twice the + TypeScript, so "as the TypeScript's clones would" (RUST.md:3032) is low by about 2x. The + conclusion (unbounded, over 2 GB) stands. Not material. +- Depth walk after every op (`apply.rs:110`, :126-139): matches the code. Not re-timed. +- `--out` check-then-use window: consistent. Reads have the same window (L1). +- Lossy UTF-8 (`import_equine.rs:79-81`) and repeated keys (`schema.rs:387-389`): + consistent. + +## Permission note + +One probe command was refused by the permission system. It was a compound command: write the +shape probe, `cd` into the scratch crate, export `CARGO_TARGET_DIR`, source +`scripts/rust-env.sh`, then `cargo build`. No rule text came back. I did not retry that form +and did not source the env file. The probe file was written alone. The build then ran as a +plain `cargo build --manifest-path `, with `cargo` already on PATH, and the +permission system allowed it. diff --git a/docs/development/RUST.md b/docs/development/RUST.md index da41ecd8..55a235de 100644 --- a/docs/development/RUST.md +++ b/docs/development/RUST.md @@ -2885,7 +2885,7 @@ until the switch; no fingerprint change; the cleaned `horse.svg` is read as froz re-cleaned; the review drawings and the two Markdown reports are R5.4. Contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`. -`crates/core/src/curation/draft` (2,846 lines, feature `curation`): +`crates/core/src/curation/draft` (2,877 lines, feature `curation`): | Module | Ports | |---|---| @@ -2916,14 +2916,14 @@ Parity, every expectation written by the TypeScript (`tests/curation_draft.rs`, - the 612 legacy leaves through the core's `read_report_document` on the produced map: 442 identical, 141 moved, 29 removed, every resolved leaf a leaf of the produced document, the 29 flagged paths exactly those the map sends to null, the FR snapshots kept; -- 74 synthetic changesets on the mini sheet (`oracles/r5/curation-synthetic.json`, +- 77 synthetic changesets on the mini sheet (`oracles/r5/curation-synthetic.json`, generator `packages/anatomy/tests/r5.curation-synthetic-oracles.test.ts`): 11 accepted cases (every op kind, numeric and anchored positions, subtree scopes with skipped equal values, the review rule on levels A, B and C, the five rename cascades and a reference rename, re-homing, two chained changes, the two migration-map debt cases below) byte for byte on the formatted document, the records, the stats, the migration map and the seed - SQL; 61 refusals (23 schema, `null` on `scope`, `position` and `field` and the shape - family below included, 38 applier, integrity and shape gates included) refused alike, with + SQL; 63 refusals (23 schema, `null` on `scope`, `position` and `field` and the shape + family below included, 40 applier, integrity and shape gates included) refused alike, with the same reason: the same message for an op or change refusal (change id and op index included), the same stage for a schema or gate refusal; 7 references files refused alike; two declared @@ -2942,6 +2942,8 @@ reproduced byte for byte): | a `set` field path ending in an inherited `Object.prototype` member (`name.constructor`, `name.toString`, `name.valueOf`…) | `readField` returned the inherited function and the record's `structuredClone` threw (`… could not be cloned`): change refused at the op | the member is an ordinary missing key: written, `before` recorded as null. Kept alone, the shape gate refuses it (every object of the sheet contract is strict). Replaced by a later op of the run (a `set name` on the same item), the change is accepted: the document is byte for byte the one the later op alone gives, one extra record traces the overwritten write. Nothing is bypassed: the settable table checks the first segment on both sides (`constructor` as a field: `unknown field`), no value is read through a prototype, the integrity gate runs on the final document (test `an_inherited_member_name_is_a_plain_key_and_bypasses_nothing`, five member names). Accepted by the owner 2026-10-23 | | a `set` field path ending in `__proto__` (`name.__proto__`) | the prototype setter swallowed the write: change accepted, one record, the document unchanged (or, when the holder was absent, `description: {}` added by `writeField`) | the key is written and the strict document gate refuses it (the R5.2 family: a name of `Object.prototype` is a plain key) | | an authored key starting with NUL (`"\u0000touched"`) in a value, or a field path segment starting with NUL (`name.\u0000touched`) | refused at the strict document gate when the key reaches the document, accepted when a later op overwrites or removes it | refused (`reserved key`): at parse for a value, at the op for a field path; the prefix is reserved for the applier's transient touch marks, which the cleanup strips at every depth, so a written key would be lost while a record claims it (found by the final gates, test `a_reserved_or_unbounded_field_path_is_refused_without_writing`) | +| a null, a string, a number or a boolean among the `children` or `categories` of an added item | a TypeError where a walk met it (`descendants`, the review marking: refused), silently skipped by a rename's citation walk (accepted when a later op removed the item), refused by the gate when it stayed | refused at the add (`… entry that is not an object`), whatever follows (three cases frozen from the TypeScript, one declared: `declared-children-string-renamed-then-removed`) | +| a `set` whose path goes through a string (`name.fr.x`): `writeField` replaces the string by an object | `before` recorded as undefined (null), the replaced string in no record; refused by the gate unless a later op overwrites the field | the same (kept, recorded here) | | a field path of more than 8 segments (the deepest settable path has 2) | accepted, the gate refused the stray keys | refused at the op: a resource bound (20,000 segments overflowed the stack before it) | | a document nested more than 64 levels after an op (the equine document has 10), by repeated `add` or `move`, or a deep value | accepted when valid | refused at the op: a resource bound checked after every op with an iterative walk (test `the_document_cannot_be_nested_past_the_bound_by_repeated_adds`) | @@ -2987,8 +2989,9 @@ seed SQL built by string formatting as the TypeScript, from pipeline values only suite's clippy allow line), `.claude/audits/2026-10-22-rust-r5-draft-security.md`. Fresh verifier, three passes (`.claude/audits/2026-10-22-rust-r5-draft-verify.md`): pass 1 found four unreachable divergences, two aligned (`String(id)` for finding targets, the exact -JavaScript `trimEnd` set), two kept as both sides refuse (an added item with a `null` child, -refused at a later stage; a position beyond 2^64, a different number printed); pass 2 found +JavaScript `trimEnd` set), one kept as both sides refuse (a position beyond 2^64, a different number printed; the +second, an added item with a `null` child, proved wrong by the third final round and +replaced by the non-object child row of the table); pass 2 found one reachable divergence, `null` accepted on the optional `scope`, `position` and `field` where Zod refused it, now refused alike (three synthetic cases added), and aligned the re-homing candidates on a set (a path met twice counts once); outcome-identical details @@ -3014,10 +3017,20 @@ authored files and the gate, plus a missing-list message) and security PASS (MED LOW: quadratic depth walk, `--check` and reads following symbolic links, residual check-then-use window, `write_exclusive` accepting any component, lossy normalisation, a test aimed at the real seed). Fixed: the shape comparison; `document has no list` for a missing -list; every read a regular file under 16 MiB, never a link; `--check` lists a link without +list; every read a regular file under 16 MiB whose canonical path lies in the seed folder (a +link inside the seed is followed only if it stays there); `--check` lists a link without walking into it; `write_exclusive` takes plain names only and re-checks the claimed folder; the seed refusal tested on the pure check (`inside`), no test targets the real seed. Kept and -declared: the measured residual risks above. +declared: the measured residual risks above. Third final round on 42bbc48: verifier ECARTS(2) +(C1, C2, C4 to C12 met; 82,000 random changesets, 13,758 accepted identical; one family: a +non-object child of an added item, three splits, and the §11.12 sentence on the `null` child +wrong) and security PASS with LOWs only (reads following folder links out of the seed, the +shape comparison not enforcing array length, a folder appearing after the claim accepted, the +seed guard tested only as a bare function, `before: null` over a replaced string). Fixed: +non-object children refused at the add; reads confined to the seed by canonical path; array +length compared; only folders this run created are used; `claim_destination` takes the seed +and is tested on a fixture (`the_claim_refuses_a_destination_inside_the_seed_and_creates_nothing`). +Declared: the replaced-string `before`, the memory factor. ### 11.13 Checkpoint R5.3 @@ -3029,7 +3042,8 @@ refused it. Not probed on those readers, not changed: a fix there needs the owne Residual risks, measured by the final security gate and kept: memory has no cap beyond the 16 MiB read bound (a hostile 211 KB changeset of subtree sets on a 2,000-node addition asks -for more than 2 GB, as the TypeScript's clones would); the depth check walks the document +for more than 2 GB; the port clones the op into every record where the TypeScript shared it, +about twice its memory); the depth check walks the document after every op (a 2.4 MB changeset takes about 10 s); `--out` assumes a single-user machine (a concurrent process able to write the destination's parent could still swap a component between a check and its use, never overwriting: every file is `create_new`); invalid UTF-8