diff --git a/.claude/audits/2026-10-22-rust-r5-draft-verify.md b/.claude/audits/2026-10-22-rust-r5-draft-verify.md index 33ec6636..c924ed1f 100644 --- a/.claude/audits/2026-10-22-rust-r5-draft-verify.md +++ b/.claude/audits/2026-10-22-rust-r5-draft-verify.md @@ -40,3 +40,9 @@ Contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`. Fresh verifier, - Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final.md`): MEDIUM-1 a 20,000-segment field path (40 KB changes.json) overflowed the stack; MEDIUM-2 `--out` writes not exclusive after the claim; LOW-1 the NUL field path (same as the verifier); LOW-2 `..` and parents created before a failed claim; LOW-3 a destination inside the seed accepted; LOW-4 INIT_CWD and direct-run base; LOW-5 non-UTF-8 argument panic; LOW-6 clippy disallowed list partial (pre-existing); LOW-7 the test allow line (convention). - Fixed: NUL segment refused at the op (`reserved key`); field paths capped at 8 segments; document depth capped at 64 levels after every op (iterative walk); `--out`: `..` refused, parent must exist, destination inside the seed refused, destination printed before writing, every folder `create_dir`, every file `create_new`; `args_os`. Documented: direct-run base (LOW-4). Kept: LOW-6 (pre-existing lint scope, out of R5.3), LOW-7 (test convention). Tests added: field-path refusals and 20,000 segments refused without crash, 40 nested adds refused at the bound, 8 `--out` refusals against the built binary with nothing written. - Next: a fresh verifier and a fresh security audit on the fixed commit. + +## Second final round on 314cf04 + +- Verifier ECARTS(3): C2, C3 (one family) and C13. serde's derives accept a struct written as a positional array and a unit enum written as a one-key map, in changes.json, references.json and the curated document (gate through the R2 derives); Zod refused all. 55,000-changeset random differential (TS generator, Rust probe): 7,436 accepted on both sides byte-identical, 335 mismatches, all this family but one message (`document has no "categories"` vs `document has no list "categories"`). C1, C4 to C12 MET (18 tests, 203 crate tests, `--check` 0/12, refusal probes, floor clean). Note: the same derives sit in the R2 report and archive contracts (unverified there). +- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final2.md`): MEDIUM memory amplification (no cap); LOW quadratic depth walk, `--check` and reads following symlinks, residual TOCTOU, `write_exclusive` any component, lossy normalisation, test aimed at the real seed, suppressions context, lint list gaps. +- Fixed: `draft/shape.rs` (typed reading serialised back, compared by shape) on both authored files and the gate; 10 changesets + 2 references files frozen from TS (all refused by TS, refused alike); missing-list message aligned; reads: regular file, 16 MiB cap; `listed` does not follow links; `write_exclusive` plain names, root re-checked; seed guard as pure `inside` + unit test, integration test no longer targets the seed. Kept, declared in §11.13: memory amplification, quadratic walk, single-user TOCTOU residual, lossy UTF-8 and repeated keys. Raised to the owner: the R2 derive leniency in the gated readers. diff --git a/.claude/audits/2026-10-23-rust-r5-draft-security-final2.md b/.claude/audits/2026-10-23-rust-r5-draft-security-final2.md new file mode 100644 index 00000000..14acb635 --- /dev/null +++ b/.claude/audits/2026-10-23-rust-r5-draft-security-final2.md @@ -0,0 +1,211 @@ +# Security gate (fresh): R5.3 draft port, final 2 + +Branch `feature/rust-r5-pipeline-core`, HEAD `314cf04`, scope `git diff 5e114da..HEAD` +(32 files), focus on `9f11499`. Fresh run: no earlier verdict consulted. + +``` +SECURITY — VERDICT: PASS +TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten +SCOPE: 32 files +BLOCKING: + (none) +REPORTED (non-blocking): + - [MEDIUM] resource-cap/memory-amplification — crates/core/src/curation/draft/records.rs:68 + - [LOW] resource-cap/cpu-quadratic — crates/core/src/curation/draft/apply.rs:109 + - [LOW] check-follows-symlinks — crates/core/src/bin/import_equine.rs:148 + - [LOW] out-toctou-residual — crates/core/src/bin/import_equine.rs:203 + - [LOW] write-exclusive-components — crates/core/src/bin/import_equine.rs:246 + - [LOW] silent-input-normalisation — crates/core/src/bin/import_equine.rs:66 + - [LOW] test-targets-real-seed — crates/core/tests/curation_draft.rs:681 + - [LOW] suppression-context (new, test crate, documented policy) — crates/core/tests/curation_draft.rs:7 + - [LOW] suppression-context (pre-existing) + lint list gaps — crates/core/src/bin/import_equine.rs:15 +PROOF: semgrep 129 rules on 32 files → 0 findings; checklist 10 checks → 9 findings +``` + +## Tool notes + +- semgrep 1.168.0, pinned packs only, `--metrics=off`. The first run died inside semgrep-core + (`io_uring_queue_init: Cannot allocate memory`, sandbox memlock limit). The rerun with + `EIO_BACKEND=posix` scanned all 32 files: 0 errors, 129 rules run (163 loaded), 0 findings. +- These packs carry almost no Rust rules. The Rust analysis below therefore rests on a full + read of every changed product file plus runtime probes, not on semgrep. +- Guardrails scanner (`api,supabase`): 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER` in + `supabase/migrations/20261008110000_anatomical_sheets.sql`, outside the diff, dropped). + +## Checklist + +| # | Check | Result | +|---|---|---| +| 1 | Hardcoded secret, token, key, auth URL (diff incl. docs, `.claude`, oracle JSON) | none (only `https://example.org/a`) | +| 2 | SQL by concatenation / interpolation | none: `seed.rs` interpolates constants only, `sql_text` doubles quotes, the document goes into `$equine_doc$…$equine_doc$`, and a document holding the tag is refused. The trimmed document always ends with `}`, so no prefix of the tag can join the closing tag. `config.toml` `sql_paths` lists literal files, so an `--out` under `supabase/seed/` is never executed by `db reset` | +| 3 | Unsanitised render / `eval` | none (no rendering; the TS oracle test has no sink) | +| 4 | Sensitive endpoint without authz | N/A (dev tool, no endpoint) | +| 5 | Stack trace / internal path to end users | none (operator CLI only) | +| 6 | Secret / PII in a log | none | +| 7 | Tracked `.env` / credential file | none (`.env.example` only, outside the diff) | +| 8 | Guardrails scanner | 0 findings in diff files | +| 9 | New suppressions (anti-gaming) | no `nosemgrep` / `nosec` / `eslint-disable` / `ts-ignore` / `noqa`; one clippy allow (L7) | +| 10 | Brief-specific: seed protection, `--check` read-only, `--out` semantics, caps, strict validation, author data | M1, L1 to L6 | + +## Findings + +### M1 [MEDIUM] No memory cap: hostile `changes.json` amplifies about 10,000× (non-blocking) + +`records.rs:68-75`: every record deep-clones the whole `Op`, value included, plus +`before`/`after`. `ops.rs:145-164`: a subtree `set` writes the value into every entry and +records it once per entry. `ops.rs:193,197`: a `rename` records the whole item twice, subtree +included. `import_equine.rs:60-68`: input reads have no size bound. The depth and segment +caps of `9f11499` bound the stack. Nothing bounds memory, the document size or the record +volume. + +Measured with the release probe under `ulimit -v 3000000`: + +| input | size | peak RSS | outcome | +|---|---|---|---| +| real `changes.json` | 227 KB | 14 MB | OK, 825 records | +| add 1,000 nodes + subtree `set` of a 100 KB description | 154 KB | 1.10 GB | OK, accepted and rendered | +| same, 2,000 nodes | 211 KB | > 2.38 GB | abort: `memory allocation of 804175406 bytes failed` | +| 5,000-node category + 200 rename ping-pongs | 300 KB | 2.13 GB | OK | +| same, 500 renames | 326 KB | > 3 GB | abort | + +Growth is linear in (ops × item size): one 90-byte rename op on that category costs about +10 MB. A 1 MB hostile file asks for tens of GB. This host has 14 GB of RAM with its 3 GB of +swap already full, so the result is an OOM kill or thrashing, during `--check` as well. No +data is lost and nothing is written. The input is an authored repository file, which can come +from a contributor PR on an MIT project. The TypeScript cloned the same way, so this is a +parity issue. It still falls short of the owner's explicit "depth and resource caps" +requirement. + +Fix direction: +- Cap the input files in the binary (for example `changes.json` ≤ 4 MiB; the real file is + 227 KB). +- Bound the curated document next to `assert_depth`: node count or serialised size, plus + string length. +- Optionally, records reference the op (index or `Rc`) instead of cloning it. + +### L1 [LOW] Quadratic CPU: whole-document depth walk after every op + +`apply.rs:109,125-147`: `depth_of` (new in `9f11499`) walks the whole document after each op. +A 2.4 MB input (20,000-node added subtree, then 10,000 one-field `set` ops) takes 9.9 s. The +cost is O(ops × document). Fix direction: the input cap of M1. Otherwise compute depth +incrementally (the depth of the touched item's path plus the depth of the written or inserted +value). + +### L2 [LOW] `--check` and input reads follow symbolic links of the committed seed + +`import_equine.rs:148`: `listed` decides recursion with `entry.path().is_dir()`, which follows +links. `import_equine.rs:60-62,163`: `fs::read` follows links and has no size bound. Two +consequences: +- A committed `draft/x -> /` makes `--check` walk the filesystem and print file names to + stderr, bounded only by ELOOP and time. +- `curation/changes.json -> /dev/zero`, or a FIFO, ends in OOM or a hang. + +Both stay read-only and local, with no exfiltration. Traced by reading: no symlink was planted +in the repository. Fix direction: use `entry.file_type()` (no follow) and report a symlink as a +difference; read inputs only after a `symlink_metadata` regular-file check and under a size +cap. + +### L3 [LOW] Residual check-then-use window in `--out` + +The path is canonicalised at `import_equine.rs:203`, then the folder is created at `:216`. +Each intermediate folder is checked with `symlink_metadata` at `:252`, then the file is opened +at `:257-261`. All of these are path-based. A concurrent local actor with write access to the +destination's parent, or the owner of a pre-existing empty destination, could swap a +component for a symlink between check and use. The impact is bounded: every file is +`create_new` (O_EXCL) with a fixed name, so nothing is overwritten, truncated or deleted. The +worst case is new fixed-name files in another folder the user can write. Fix direction, only +if wanted: hold the claimed root as a directory fd and create children with +`openat`/`mkdirat` + `O_NOFOLLOW` (rustix or cap-std). Otherwise document the single-user +machine assumption. + +### L4 [LOW] `write_exclusive` accepts any path component + +`import_equine.rs:244-255` pushes each component of `relative` without refusing `RootDir`, +`ParentDir` or `Prefix`. Today every name is a compile-time constant (`render_artifacts`, +`render_draft_artifacts`, `assets/Horse.svg`), so this is not exploitable. It becomes a +traversal if R5.4 derives a file name from data, for example review drawings named after view +ids. Fix direction: refuse any component other than `Component::Normal`. + +### L5 [LOW] Silent input normalisation, against "author data never dropped silently" + +- `import_equine.rs:66-68`: invalid UTF-8 in `changes.json` or `references.json` silently + becomes U+FFFD. +- `schema.rs:382-384` (`js::parse_json`): a duplicated key keeps its last value, and the first + is dropped without notice. + +Both are declared parity with `readFile(..., 'utf8')` and `JSON.parse`. Fix direction: refuse +invalid UTF-8 (`String::from_utf8`) and duplicate keys in the authored files, or warn. + +### L6 [LOW] A test runs the real binary against the official seed path + +`curation_draft.rs:676-701` runs `import-equine --out /new-folder`. If the seed guard +regressed, the test itself would create `new-folder/` with 12 files inside the official seed +before its assertion fails (no overwrite). For the same reason this gate did not run +`cargo test --test curation_draft`: the brief forbids `--out` against a repository path. Fix +direction: assert the seed refusal through the pure claim logic with an injectable seed root, +or keep it and accept the regression risk knowingly. + +### L7 [LOW, context] New lint allow in a test crate: not a security suppression + +`curation_draft.rs:7` adds +`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. It sits on an +integration-test crate, uses the same line as 10 existing test files, and follows the policy +written in `clippy.toml` ("Tests … allow these lints locally where they own I/O"). It silences +no security scanner and no security finding, so it is not blocking. + +### L8 [LOW, context] Pre-existing allow, and gaps in the I/O ban list + +`import_equine.rs:15` `#![allow(clippy::disallowed_methods)]` was already present at +`5e114da`. Separately, `clippy.toml` bans only `fs::read`, `read_to_string`, `write`, +`File::open`/`create`, `env::var`/`var_os` and the clocks. `fs::create_dir*`, +`OpenOptions::open`, `read_dir`, `canonicalize`, `remove_*`, `env::current_dir`, +`env::args_os` and `process::Command` would pass the lint inside the library. A grep confirms +that `crates/core/src` outside `src/bin` uses none of them today. Fix direction: extend the +list. + +## Verified, no finding + +Stack and validation. The probe linked `faunosteo-core` with the `curation` feature and ran +in debug on a 2 MiB thread, in debug on the main thread, and in release: +- a 20,000-segment field and a 9-segment field are refused; the 8-segment field is refused + later by the strict shape check; +- `name.\0touched` is refused; a `\0touched` key inside a value is refused at parse; +- a value at serde's 128 nesting limit is refused with "recursion limit exceeded"; +- a 122-deep object value, a 58-level children chain and 40 chained adds (refused at op 29) + all end with "nests more than 64 levels"; +- moving an item into itself is refused; +- no crash, no panic in any mode. + +Margins on real data: document depth 10 against the cap of 64; field segments at most 2 +against the cap of 8. + +`--check`, as briefed: `legacy-import` 6 files, 0 differences; `draft` 6 files, 0 differences +(6 pending R5.4). `git status` clean afterwards. + +`--out`, 20 cases in a fresh `mktemp -d`. A copy of the binary was built there; the +repository stayed clean. +- Written: + - an absent folder: 12 files, `document.json` byte-identical to the seed; + - an existing empty folder; + - a parent symlink: the files land at the canonical path; + - an interior `./`; + - a relative path with an absolute `INIT_CWD`; a relative `INIT_CWD` is ignored. +- Refused with exit code 2 and nothing created: + - an existing non-empty folder; a file; + - a symlink, `symlink/`, `symlink/.`, a dangling symlink (its target is not created); + - a missing parent; `..`; `""`; `/`; an extra argument; a non-UTF-8 argument. + +Scratch total: 74 files = 6 × 12 + 2 fixtures. + +Seed protection, traced by reading and not run against the repository: +- the parent is canonicalised, then compared with `starts_with` component by component + against the canonical seed; +- a root equal to the seed, or to any ancestor, is non-empty and refused; +- a final-component symlink is refused; +- `folders()` is computed before the claim, so a refused changeset creates nothing. + +The library performs no file system, environment or process access. The Rust ownership model +rules out reference cycles from `move`. + +Probe artefacts (scratch only): +`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/secfinal2.SbMNbq` diff --git a/.claude/tasks/TODO.md b/.claude/tasks/TODO.md index 8b506a0c..00701af7 100644 --- a/.claude/tasks/TODO.md +++ b/.claude/tasks/TODO.md @@ -317,6 +317,8 @@ feat/findings-tree-v2 merged (FindingsTreePicker tracked in HEAD); all work belo - [x] R5.3e (security PASS 6 LOW / 3 fixed; verifier 4 passes, last C1-C12 met + C13 docs fixed after; check:all green 00:10:03) gates: security auditor (diff), independent verifier on the contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`; docs RUST.md §11.12-11.13, CHANGELOG, CLAUDE.md commands; memory → checkpoint report, STOP before R5.4 - [x] R5.3f owner 2026-10-23: Object.prototype member characterised (safe, accepted, test), migration-map debt kept + characterisation test, refusal reasons compared by stage; final fresh verifier + security on the R5.3 state - [ ] debt (owner 2026-10-23, after R5): migration map replays cascade records by their op (`applyRecord` / `apply_record`): a view, zone, region, category or cited-reference rename corrupts the map; fix both sides as a declared change before any changeset uses such a rename (RUST.md §11.12) + - [ ] OWNER DECISION (found 2026-10-23 by the R5.3 final verifier): serde derives of the R2 contracts read a struct from an array and a unit enum from a one-key map, where Zod refused; likely reachable by the gated readers of imported PDFs (`archive/read.rs`, `report/read.rs`, `pdf/import.rs`); probe the readers, then a per-file go for a strict deserialisation (RUST.md §11.13) + - [ ] chore (pre-existing, security gates R5.3): `clippy.toml` disallowed list misses `fs::create_dir*`, `OpenOptions::open`, `read_dir`, `canonicalize`, `remove_*`, `env::current_dir`, `env::args_os`, `process::Command`; extend so the core's no-I/O rule is enforced, not only observed - [ ] R5.4 review overlays + app drawings on the R3 layer/compose (parsed trees), CHANGELOG + REVIEW-REPORT - [ ] R5.5 `upload-sheet-assets` bin + runbook (local stack run) - [ ] R5.6 D-R5 switch (convention, aliases, membership test in gated readers with per-file go, seed regenerated once by the Rust tool, `.prettierignore`, CHANGELOG) diff --git a/docs/development/RUST.md b/docs/development/RUST.md index be1de795..da41ecd8 100644 --- a/docs/development/RUST.md +++ b/docs/development/RUST.md @@ -2885,7 +2885,7 @@ until the switch; no fingerprint change; the cleaned `horse.svg` is read as froz re-cleaned; the review drawings and the two Markdown reports are R5.4. Contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`. -`crates/core/src/curation/draft` (2,718 lines, feature `curation`): +`crates/core/src/curation/draft` (2,846 lines, feature `curation`): | Module | Ports | |---|---| @@ -2916,16 +2916,17 @@ Parity, every expectation written by the TypeScript (`tests/curation_draft.rs`, - the 612 legacy leaves through the core's `read_report_document` on the produced map: 442 identical, 141 moved, 29 removed, every resolved leaf a leaf of the produced document, the 29 flagged paths exactly those the map sends to null, the FR snapshots kept; -- 64 synthetic changesets on the mini sheet (`oracles/r5/curation-synthetic.json`, +- 74 synthetic changesets on the mini sheet (`oracles/r5/curation-synthetic.json`, generator `packages/anatomy/tests/r5.curation-synthetic-oracles.test.ts`): 11 accepted cases (every op kind, numeric and anchored positions, subtree scopes with skipped equal values, the review rule on levels A, B and C, the five rename cascades and a reference rename, re-homing, two chained changes, the two migration-map debt cases below) byte for byte on the formatted document, the records, the stats, the migration map and the seed - SQL; 51 refusals (19 schema, `null` on `scope`, `position` and `field` included, 32 - applier, integrity and shape gates included) refused alike, with the same reason: the - same message for an op or change refusal (change id and op index included), the same - stage for a schema or gate refusal; 5 references files refused alike; two declared + SQL; 61 refusals (23 schema, `null` on `scope`, `position` and `field` and the shape + family below included, 38 applier, integrity and shape gates included) refused alike, with + the same reason: the same message for an op or change refusal (change id and op index + included), the same stage for a schema or gate refusal; 7 references files refused alike; + two declared differences below (one refusal, one acceptance), each held by its own test. Differences declared (none reachable by the committed files, since the folders are @@ -2960,6 +2961,15 @@ findings only (33) and no reference, so the equine map is untouched. A future ch renaming a view, zone, region, category or a cited reference must not be published before the fix (a declared behaviour change on both sides, after R5). +Aligned on Zod, not a difference: serde's derives read a struct from a positional array +(`"title": ["Titre", "Title"]`, a whole change or reference as an array) and a unit enum +variant from a one-key map (`"level": {"A": null}`, `scope`, `review_status`, `laterality`, +the reference `type`); Zod refused both. Each authored file and the curated document are +compared, by shape, with their typed reading serialised back (`draft/shape.rs`): an object +stays an object, an array an array of the same length, a scalar a scalar of its kind (found by +the final verifier's 55,000-changeset differential; 10 changesets and 2 references files frozen +from the TypeScript). The R2 contracts carry the same derives (open item below). + Kept as the TypeScript: invalid UTF-8 in the two curation files becomes U+FFFD (`readFile(path, 'utf8')`, as R5.2 kept for the catalog); a repeated key in either file keeps its first position and its last value (`JSON.parse`, through the core's `parse_json`; tested on a change and a reference); @@ -2999,10 +3009,32 @@ checked after every op); `--out` writes were not exclusive after the claim (fixe `create_dir` and `create_new`); `..`, a missing parent and a destination inside the seed are refused, the destination printed before writing, a non-UTF-8 argument refused; the binary's refusals are tested against the built binary (`out_refuses_every_unsafe_destination_and_writes_nothing`). +Second final round on 314cf04: verifier ECARTS(3) (the serde shape family above, in the +authored files and the gate, plus a missing-list message) and security PASS (MEDIUM memory, +LOW: quadratic depth walk, `--check` and reads following symbolic links, residual +check-then-use window, `write_exclusive` accepting any component, lossy normalisation, a test +aimed at the real seed). Fixed: the shape comparison; `document has no list` for a missing +list; every read a regular file under 16 MiB, never a link; `--check` lists a link without +walking into it; `write_exclusive` takes plain names only and re-checks the claimed folder; +the seed refusal tested on the pure check (`inside`), no test targets the real seed. Kept and +declared: the measured residual risks above. ### 11.13 Checkpoint R5.3 -Residual risks: the `--check` allowance for the six R5.4 files hides nothing about the +Open item for the owner (outside R5.3, gated readers): the serde leniency above is in the R2 +derives of `domain/sheet/schema.rs` and very likely of the report and archive contracts, so +the readers of imported PDFs (`archive/read.rs`, `report/read.rs`, `pdf/import.rs`) would +accept an object written as an array or an enum written as a map where the frozen Zod reader +refused it. Not probed on those readers, not changed: a fix there needs the owner's per-file go. + +Residual risks, measured by the final security gate and kept: memory has no cap beyond the +16 MiB read bound (a hostile 211 KB changeset of subtree sets on a 2,000-node addition asks +for more than 2 GB, as the TypeScript's clones would); the depth check walks the document +after every op (a 2.4 MB changeset takes about 10 s); `--out` assumes a single-user machine +(a concurrent process able to write the destination's parent could still swap a component +between a check and its use, never overwriting: every file is `create_new`); invalid UTF-8 +and repeated keys are read as the TypeScript read them. Residual risks of the slice: the +`--check` allowance for the six R5.4 files hides nothing about the produced files but means the draft folder is not yet compared whole (R5.4 removes it); the touch marks are a reserved prefix (a NUL character starts it), refused in any authored value at parse and in any field path segment at the op, and removed after each change, so no mark