docs(docs): R5.3 second final round: shape alignment, residual risks measured, R2 derive leniency raised to the owner; audits, TODO

This commit is contained in:
bchanot committed 2026-10-23 01:33:02 +02:00
1 parent 776c7095b3
commit 42bbc48fcc
4 files changed
+258 -7

No files matched your search

@@ -40,3 +40,9 @@ Contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`. Fresh verifier,
- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final.md`): MEDIUM-1 a 20,000-segment field path (40 KB changes.json) overflowed the stack; MEDIUM-2 `--out` writes not exclusive after the claim; LOW-1 the NUL field path (same as the verifier); LOW-2 `..` and parents created before a failed claim; LOW-3 a destination inside the seed accepted; LOW-4 INIT_CWD and direct-run base; LOW-5 non-UTF-8 argument panic; LOW-6 clippy disallowed list partial (pre-existing); LOW-7 the test allow line (convention).
- Fixed: NUL segment refused at the op (`reserved key`); field paths capped at 8 segments; document depth capped at 64 levels after every op (iterative walk); `--out`: `..` refused, parent must exist, destination inside the seed refused, destination printed before writing, every folder `create_dir`, every file `create_new`; `args_os`. Documented: direct-run base (LOW-4). Kept: LOW-6 (pre-existing lint scope, out of R5.3), LOW-7 (test convention). Tests added: field-path refusals and 20,000 segments refused without crash, 40 nested adds refused at the bound, 8 `--out` refusals against the built binary with nothing written.
- Next: a fresh verifier and a fresh security audit on the fixed commit.
## Second final round on 314cf04
- Verifier ECARTS(3): C2, C3 (one family) and C13. serde's derives accept a struct written as a positional array and a unit enum written as a one-key map, in changes.json, references.json and the curated document (gate through the R2 derives); Zod refused all. 55,000-changeset random differential (TS generator, Rust probe): 7,436 accepted on both sides byte-identical, 335 mismatches, all this family but one message (`document has no "categories"` vs `document has no list "categories"`). C1, C4 to C12 MET (18 tests, 203 crate tests, `--check` 0/12, refusal probes, floor clean). Note: the same derives sit in the R2 report and archive contracts (unverified there).
- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final2.md`): MEDIUM memory amplification (no cap); LOW quadratic depth walk, `--check` and reads following symlinks, residual TOCTOU, `write_exclusive` any component, lossy normalisation, test aimed at the real seed, suppressions context, lint list gaps.
- Fixed: `draft/shape.rs` (typed reading serialised back, compared by shape) on both authored files and the gate; 10 changesets + 2 references files frozen from TS (all refused by TS, refused alike); missing-list message aligned; reads: regular file, 16 MiB cap; `listed` does not follow links; `write_exclusive` plain names, root re-checked; seed guard as pure `inside` + unit test, integration test no longer targets the seed. Kept, declared in §11.13: memory amplification, quadratic walk, single-user TOCTOU residual, lossy UTF-8 and repeated keys. Raised to the owner: the R2 derive leniency in the gated readers.
@@ -0,0 +1,211 @@
# Security gate (fresh): R5.3 draft port, final 2
Branch `feature/rust-r5-pipeline-core`, HEAD `314cf04`, scope `git diff 5e114da..HEAD`
(32 files), focus on `9f11499`. Fresh run: no earlier verdict consulted.
```
SECURITY — VERDICT: PASS
TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten
SCOPE: 32 files
BLOCKING:
(none)
REPORTED (non-blocking):
- [MEDIUM] resource-cap/memory-amplification — crates/core/src/curation/draft/records.rs:68
- [LOW] resource-cap/cpu-quadratic — crates/core/src/curation/draft/apply.rs:109
- [LOW] check-follows-symlinks — crates/core/src/bin/import_equine.rs:148
- [LOW] out-toctou-residual — crates/core/src/bin/import_equine.rs:203
- [LOW] write-exclusive-components — crates/core/src/bin/import_equine.rs:246
- [LOW] silent-input-normalisation — crates/core/src/bin/import_equine.rs:66
- [LOW] test-targets-real-seed — crates/core/tests/curation_draft.rs:681
- [LOW] suppression-context (new, test crate, documented policy) — crates/core/tests/curation_draft.rs:7
- [LOW] suppression-context (pre-existing) + lint list gaps — crates/core/src/bin/import_equine.rs:15
PROOF: semgrep 129 rules on 32 files → 0 findings; checklist 10 checks → 9 findings
```
## Tool notes
- semgrep 1.168.0, pinned packs only, `--metrics=off`. The first run died inside semgrep-core
(`io_uring_queue_init: Cannot allocate memory`, sandbox memlock limit). The rerun with
`EIO_BACKEND=posix` scanned all 32 files: 0 errors, 129 rules run (163 loaded), 0 findings.
- These packs carry almost no Rust rules. The Rust analysis below therefore rests on a full
read of every changed product file plus runtime probes, not on semgrep.
- Guardrails scanner (`api,supabase`): 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER` in
`supabase/migrations/20261008110000_anatomical_sheets.sql`, outside the diff, dropped).
## Checklist
| # | Check | Result |
|---|---|---|
| 1 | Hardcoded secret, token, key, auth URL (diff incl. docs, `.claude`, oracle JSON) | none (only `https://example.org/a`) |
| 2 | SQL by concatenation / interpolation | none: `seed.rs` interpolates constants only, `sql_text` doubles quotes, the document goes into `$equine_doc$…$equine_doc$`, and a document holding the tag is refused. The trimmed document always ends with `}`, so no prefix of the tag can join the closing tag. `config.toml` `sql_paths` lists literal files, so an `--out` under `supabase/seed/` is never executed by `db reset` |
| 3 | Unsanitised render / `eval` | none (no rendering; the TS oracle test has no sink) |
| 4 | Sensitive endpoint without authz | N/A (dev tool, no endpoint) |
| 5 | Stack trace / internal path to end users | none (operator CLI only) |
| 6 | Secret / PII in a log | none |
| 7 | Tracked `.env` / credential file | none (`.env.example` only, outside the diff) |
| 8 | Guardrails scanner | 0 findings in diff files |
| 9 | New suppressions (anti-gaming) | no `nosemgrep` / `nosec` / `eslint-disable` / `ts-ignore` / `noqa`; one clippy allow (L7) |
| 10 | Brief-specific: seed protection, `--check` read-only, `--out` semantics, caps, strict validation, author data | M1, L1 to L6 |
## Findings
### M1 [MEDIUM] No memory cap: hostile `changes.json` amplifies about 10,000× (non-blocking)
`records.rs:68-75`: every record deep-clones the whole `Op`, value included, plus
`before`/`after`. `ops.rs:145-164`: a subtree `set` writes the value into every entry and
records it once per entry. `ops.rs:193,197`: a `rename` records the whole item twice, subtree
included. `import_equine.rs:60-68`: input reads have no size bound. The depth and segment
caps of `9f11499` bound the stack. Nothing bounds memory, the document size or the record
volume.
Measured with the release probe under `ulimit -v 3000000`:
| input | size | peak RSS | outcome |
|---|---|---|---|
| real `changes.json` | 227 KB | 14 MB | OK, 825 records |
| add 1,000 nodes + subtree `set` of a 100 KB description | 154 KB | 1.10 GB | OK, accepted and rendered |
| same, 2,000 nodes | 211 KB | > 2.38 GB | abort: `memory allocation of 804175406 bytes failed` |
| 5,000-node category + 200 rename ping-pongs | 300 KB | 2.13 GB | OK |
| same, 500 renames | 326 KB | > 3 GB | abort |
Growth is linear in (ops × item size): one 90-byte rename op on that category costs about
10 MB. A 1 MB hostile file asks for tens of GB. This host has 14 GB of RAM with its 3 GB of
swap already full, so the result is an OOM kill or thrashing, during `--check` as well. No
data is lost and nothing is written. The input is an authored repository file, which can come
from a contributor PR on an MIT project. The TypeScript cloned the same way, so this is a
parity issue. It still falls short of the owner's explicit "depth and resource caps"
requirement.
Fix direction:
- Cap the input files in the binary (for example `changes.json` ≤ 4 MiB; the real file is
227 KB).
- Bound the curated document next to `assert_depth`: node count or serialised size, plus
string length.
- Optionally, records reference the op (index or `Rc<Op>`) instead of cloning it.
### L1 [LOW] Quadratic CPU: whole-document depth walk after every op
`apply.rs:109,125-147`: `depth_of` (new in `9f11499`) walks the whole document after each op.
A 2.4 MB input (20,000-node added subtree, then 10,000 one-field `set` ops) takes 9.9 s. The
cost is O(ops × document). Fix direction: the input cap of M1. Otherwise compute depth
incrementally (the depth of the touched item's path plus the depth of the written or inserted
value).
### L2 [LOW] `--check` and input reads follow symbolic links of the committed seed
`import_equine.rs:148`: `listed` decides recursion with `entry.path().is_dir()`, which follows
links. `import_equine.rs:60-62,163`: `fs::read` follows links and has no size bound. Two
consequences:
- A committed `draft/x -> /` makes `--check` walk the filesystem and print file names to
stderr, bounded only by ELOOP and time.
- `curation/changes.json -> /dev/zero`, or a FIFO, ends in OOM or a hang.
Both stay read-only and local, with no exfiltration. Traced by reading: no symlink was planted
in the repository. Fix direction: use `entry.file_type()` (no follow) and report a symlink as a
difference; read inputs only after a `symlink_metadata` regular-file check and under a size
cap.
### L3 [LOW] Residual check-then-use window in `--out`
The path is canonicalised at `import_equine.rs:203`, then the folder is created at `:216`.
Each intermediate folder is checked with `symlink_metadata` at `:252`, then the file is opened
at `:257-261`. All of these are path-based. A concurrent local actor with write access to the
destination's parent, or the owner of a pre-existing empty destination, could swap a
component for a symlink between check and use. The impact is bounded: every file is
`create_new` (O_EXCL) with a fixed name, so nothing is overwritten, truncated or deleted. The
worst case is new fixed-name files in another folder the user can write. Fix direction, only
if wanted: hold the claimed root as a directory fd and create children with
`openat`/`mkdirat` + `O_NOFOLLOW` (rustix or cap-std). Otherwise document the single-user
machine assumption.
### L4 [LOW] `write_exclusive` accepts any path component
`import_equine.rs:244-255` pushes each component of `relative` without refusing `RootDir`,
`ParentDir` or `Prefix`. Today every name is a compile-time constant (`render_artifacts`,
`render_draft_artifacts`, `assets/Horse.svg`), so this is not exploitable. It becomes a
traversal if R5.4 derives a file name from data, for example review drawings named after view
ids. Fix direction: refuse any component other than `Component::Normal`.
### L5 [LOW] Silent input normalisation, against "author data never dropped silently"
- `import_equine.rs:66-68`: invalid UTF-8 in `changes.json` or `references.json` silently
becomes U+FFFD.
- `schema.rs:382-384` (`js::parse_json`): a duplicated key keeps its last value, and the first
is dropped without notice.
Both are declared parity with `readFile(..., 'utf8')` and `JSON.parse`. Fix direction: refuse
invalid UTF-8 (`String::from_utf8`) and duplicate keys in the authored files, or warn.
### L6 [LOW] A test runs the real binary against the official seed path
`curation_draft.rs:676-701` runs `import-equine --out <seed>/new-folder`. If the seed guard
regressed, the test itself would create `new-folder/` with 12 files inside the official seed
before its assertion fails (no overwrite). For the same reason this gate did not run
`cargo test --test curation_draft`: the brief forbids `--out` against a repository path. Fix
direction: assert the seed refusal through the pure claim logic with an injectable seed root,
or keep it and accept the regression risk knowingly.
### L7 [LOW, context] New lint allow in a test crate: not a security suppression
`curation_draft.rs:7` adds
`#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. It sits on an
integration-test crate, uses the same line as 10 existing test files, and follows the policy
written in `clippy.toml` ("Tests … allow these lints locally where they own I/O"). It silences
no security scanner and no security finding, so it is not blocking.
### L8 [LOW, context] Pre-existing allow, and gaps in the I/O ban list
`import_equine.rs:15` `#![allow(clippy::disallowed_methods)]` was already present at
`5e114da`. Separately, `clippy.toml` bans only `fs::read`, `read_to_string`, `write`,
`File::open`/`create`, `env::var`/`var_os` and the clocks. `fs::create_dir*`,
`OpenOptions::open`, `read_dir`, `canonicalize`, `remove_*`, `env::current_dir`,
`env::args_os` and `process::Command` would pass the lint inside the library. A grep confirms
that `crates/core/src` outside `src/bin` uses none of them today. Fix direction: extend the
list.
## Verified, no finding
Stack and validation. The probe linked `faunosteo-core` with the `curation` feature and ran
in debug on a 2 MiB thread, in debug on the main thread, and in release:
- a 20,000-segment field and a 9-segment field are refused; the 8-segment field is refused
later by the strict shape check;
- `name.\0touched` is refused; a `\0touched` key inside a value is refused at parse;
- a value at serde's 128 nesting limit is refused with "recursion limit exceeded";
- a 122-deep object value, a 58-level children chain and 40 chained adds (refused at op 29)
all end with "nests more than 64 levels";
- moving an item into itself is refused;
- no crash, no panic in any mode.
Margins on real data: document depth 10 against the cap of 64; field segments at most 2
against the cap of 8.
`--check`, as briefed: `legacy-import` 6 files, 0 differences; `draft` 6 files, 0 differences
(6 pending R5.4). `git status` clean afterwards.
`--out`, 20 cases in a fresh `mktemp -d`. A copy of the binary was built there; the
repository stayed clean.
- Written:
- an absent folder: 12 files, `document.json` byte-identical to the seed;
- an existing empty folder;
- a parent symlink: the files land at the canonical path;
- an interior `./`;
- a relative path with an absolute `INIT_CWD`; a relative `INIT_CWD` is ignored.
- Refused with exit code 2 and nothing created:
- an existing non-empty folder; a file;
- a symlink, `symlink/`, `symlink/.`, a dangling symlink (its target is not created);
- a missing parent; `..`; `""`; `/`; an extra argument; a non-UTF-8 argument.
Scratch total: 74 files = 6 × 12 + 2 fixtures.
Seed protection, traced by reading and not run against the repository:
- the parent is canonicalised, then compared with `starts_with` component by component
against the canonical seed;
- a root equal to the seed, or to any ancestor, is non-empty and refused;
- a final-component symlink is refused;
- `folders()` is computed before the claim, so a refused changeset creates nothing.
The library performs no file system, environment or process access. The Rust ownership model
rules out reference cycles from `move`.
Probe artefacts (scratch only):
`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/secfinal2.SbMNbq`
+2
View File
@@ -317,6 +317,8 @@ feat/findings-tree-v2 merged (FindingsTreePicker tracked in HEAD); all work belo
- [x] R5.3e (security PASS 6 LOW / 3 fixed; verifier 4 passes, last C1-C12 met + C13 docs fixed after; check:all green 00:10:03) gates: security auditor (diff), independent verifier on the contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`; docs RUST.md §11.12-11.13, CHANGELOG, CLAUDE.md commands; memory → checkpoint report, STOP before R5.4
- [x] R5.3f owner 2026-10-23: Object.prototype member characterised (safe, accepted, test), migration-map debt kept + characterisation test, refusal reasons compared by stage; final fresh verifier + security on the R5.3 state
- [ ] debt (owner 2026-10-23, after R5): migration map replays cascade records by their op (`applyRecord` / `apply_record`): a view, zone, region, category or cited-reference rename corrupts the map; fix both sides as a declared change before any changeset uses such a rename (RUST.md §11.12)
- [ ] OWNER DECISION (found 2026-10-23 by the R5.3 final verifier): serde derives of the R2 contracts read a struct from an array and a unit enum from a one-key map, where Zod refused; likely reachable by the gated readers of imported PDFs (`archive/read.rs`, `report/read.rs`, `pdf/import.rs`); probe the readers, then a per-file go for a strict deserialisation (RUST.md §11.13)
- [ ] chore (pre-existing, security gates R5.3): `clippy.toml` disallowed list misses `fs::create_dir*`, `OpenOptions::open`, `read_dir`, `canonicalize`, `remove_*`, `env::current_dir`, `env::args_os`, `process::Command`; extend so the core's no-I/O rule is enforced, not only observed
- [ ] R5.4 review overlays + app drawings on the R3 layer/compose (parsed trees), CHANGELOG + REVIEW-REPORT
- [ ] R5.5 `upload-sheet-assets` bin + runbook (local stack run)
- [ ] R5.6 D-R5 switch (convention, aliases, membership test in gated readers with per-file go, seed regenerated once by the Rust tool, `.prettierignore`, CHANGELOG)
+39 -7
View File
@@ -2885,7 +2885,7 @@ until the switch; no fingerprint change; the cleaned `horse.svg` is read as froz
re-cleaned; the review drawings and the two Markdown reports are R5.4. Contract
`.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`.
`crates/core/src/curation/draft` (2,718 lines, feature `curation`):
`crates/core/src/curation/draft` (2,846 lines, feature `curation`):
| Module | Ports |
|---|---|
@@ -2916,16 +2916,17 @@ Parity, every expectation written by the TypeScript (`tests/curation_draft.rs`,
- the 612 legacy leaves through the core's `read_report_document` on the produced map: 442
identical, 141 moved, 29 removed, every resolved leaf a leaf of the produced document, the
29 flagged paths exactly those the map sends to null, the FR snapshots kept;
- 64 synthetic changesets on the mini sheet (`oracles/r5/curation-synthetic.json`,
- 74 synthetic changesets on the mini sheet (`oracles/r5/curation-synthetic.json`,
generator `packages/anatomy/tests/r5.curation-synthetic-oracles.test.ts`): 11 accepted
cases (every op kind, numeric and anchored positions, subtree scopes with skipped equal
values, the review rule on levels A, B and C, the five rename cascades and a reference
rename, re-homing, two chained changes, the two migration-map debt cases below) byte for
byte on the formatted document, the records, the stats, the migration map and the seed
SQL; 51 refusals (19 schema, `null` on `scope`, `position` and `field` included, 32
applier, integrity and shape gates included) refused alike, with the same reason: the
same message for an op or change refusal (change id and op index included), the same
stage for a schema or gate refusal; 5 references files refused alike; two declared
SQL; 61 refusals (23 schema, `null` on `scope`, `position` and `field` and the shape
family below included, 38 applier, integrity and shape gates included) refused alike, with
the same reason: the same message for an op or change refusal (change id and op index
included), the same stage for a schema or gate refusal; 7 references files refused alike;
two declared
differences below (one refusal, one acceptance), each held by its own test.
Differences declared (none reachable by the committed files, since the folders are
@@ -2960,6 +2961,15 @@ findings only (33) and no reference, so the equine map is untouched. A future ch
renaming a view, zone, region, category or a cited reference must not be published before
the fix (a declared behaviour change on both sides, after R5).
Aligned on Zod, not a difference: serde's derives read a struct from a positional array
(`"title": ["Titre", "Title"]`, a whole change or reference as an array) and a unit enum
variant from a one-key map (`"level": {"A": null}`, `scope`, `review_status`, `laterality`,
the reference `type`); Zod refused both. Each authored file and the curated document are
compared, by shape, with their typed reading serialised back (`draft/shape.rs`): an object
stays an object, an array an array of the same length, a scalar a scalar of its kind (found by
the final verifier's 55,000-changeset differential; 10 changesets and 2 references files frozen
from the TypeScript). The R2 contracts carry the same derives (open item below).
Kept as the TypeScript: invalid UTF-8 in the two curation files becomes U+FFFD
(`readFile(path, 'utf8')`, as R5.2 kept for the catalog); a repeated key in either file keeps its first position and its last
value (`JSON.parse`, through the core's `parse_json`; tested on a change and a reference);
@@ -2999,10 +3009,32 @@ checked after every op); `--out` writes were not exclusive after the claim (fixe
`create_dir` and `create_new`); `..`, a missing parent and a destination inside the seed are
refused, the destination printed before writing, a non-UTF-8 argument refused; the binary's
refusals are tested against the built binary (`out_refuses_every_unsafe_destination_and_writes_nothing`).
Second final round on 314cf04: verifier ECARTS(3) (the serde shape family above, in the
authored files and the gate, plus a missing-list message) and security PASS (MEDIUM memory,
LOW: quadratic depth walk, `--check` and reads following symbolic links, residual
check-then-use window, `write_exclusive` accepting any component, lossy normalisation, a test
aimed at the real seed). Fixed: the shape comparison; `document has no list` for a missing
list; every read a regular file under 16 MiB, never a link; `--check` lists a link without
walking into it; `write_exclusive` takes plain names only and re-checks the claimed folder;
the seed refusal tested on the pure check (`inside`), no test targets the real seed. Kept and
declared: the measured residual risks above.
### 11.13 Checkpoint R5.3
Residual risks: the `--check` allowance for the six R5.4 files hides nothing about the
Open item for the owner (outside R5.3, gated readers): the serde leniency above is in the R2
derives of `domain/sheet/schema.rs` and very likely of the report and archive contracts, so
the readers of imported PDFs (`archive/read.rs`, `report/read.rs`, `pdf/import.rs`) would
accept an object written as an array or an enum written as a map where the frozen Zod reader
refused it. Not probed on those readers, not changed: a fix there needs the owner's per-file go.
Residual risks, measured by the final security gate and kept: memory has no cap beyond the
16 MiB read bound (a hostile 211 KB changeset of subtree sets on a 2,000-node addition asks
for more than 2 GB, as the TypeScript's clones would); the depth check walks the document
after every op (a 2.4 MB changeset takes about 10 s); `--out` assumes a single-user machine
(a concurrent process able to write the destination's parent could still swap a component
between a check and its use, never overwriting: every file is `create_new`); invalid UTF-8
and repeated keys are read as the TypeScript read them. Residual risks of the slice: the
`--check` allowance for the six R5.4 files hides nothing about the
produced files but means the draft folder is not yet compared whole (R5.4 removes it); the
touch marks are a reserved prefix (a NUL character starts it), refused in any authored value
at parse and in any field path segment at the op, and removed after each change, so no mark