diff --git a/.claude/audits/2026-10-22-rust-r5-draft-verify.md b/.claude/audits/2026-10-22-rust-r5-draft-verify.md index 68ec1d1b..cfc709fe 100644 --- a/.claude/audits/2026-10-22-rust-r5-draft-verify.md +++ b/.claude/audits/2026-10-22-rust-r5-draft-verify.md @@ -52,3 +52,9 @@ Contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`. Fresh verifier, - Verifier ECARTS(2), C3 + C13, one family: a null or primitive among the children of an added item. TS: TypeError where walked (refused), skipped by a reference rename's citation walk (accepted if later removed), gate if kept. Rust: tolerated by touch/walk (accepted), refused by rename's item_at_mut. Three splits; §11.12 claimed "both refuse" for the null child, false. Differential: 82,000 random changesets, 13,758 accepted identical (document, records, stats, map), 63 schema + 34 references probes agree. C1, C2, C4 to C12 MET (19 tests). - Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final3.md`), LOW only: reads follow folder links out of the seed; shape comparison ignores array length; a folder appearing after the claim accepted; seed guard wiring untested; `set name.fr.x` records `before: null`; lint allows (context). - Fixed: non-object children refused at the add (3 TS-frozen cases, 1 declared refusal); reads confined to the canonical seed; array length compared; only folders this run created; `claim_destination(typed, seed)` + fixture unit test. Declared: replaced-string `before` (TS same), memory factor ~2x TS. + +## Fourth final round on 6c0fe88 + +- Verifier ECARTS(2), C3 + C13: records differ (document identical) on array index keys in authored values or field segments (JSON.parse puts them first) and on integers beyond 2^53 (JSON.parse rounds; a Rust record held 9007199254740993 while document.json printed ...992); §11.12 declared-case count wrong. C1, C2, C4 to C12 MET; 25,000 random changesets: 0 accept/refuse splits, 4,331 accepted identical. +- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final4.md`): MEDIUM a present non-list `children` replaced by `[]` (author data lost); LOW `--check` lists through a top-level folder link, mount alias of the seed, walk cost understated (8.2 MB file ~36 s, cap-bound worst case ~45 min), fixture in shared temp, context items. +- Fixed: `text::js_normalize` (key order + big integers at parse) and `text::js_insert` (field path writes); non-list `children`/`categories` refused at add, `children_path` never replaces; `--check` requires each generated folder to be the seed's own; `inside_by_identity` (dev, ino of every ancestor); `MAX_DOCUMENT_VALUES` 200,000; fixture under target/. Cases frozen from TS: 2 key-order, 1 integer, 1 declared non-list children; Rust tests: big integer from raw text, value bound. diff --git a/.claude/audits/2026-10-23-rust-r5-draft-security-final4.md b/.claude/audits/2026-10-23-rust-r5-draft-security-final4.md new file mode 100644 index 00000000..7f310562 --- /dev/null +++ b/.claude/audits/2026-10-23-rust-r5-draft-security-final4.md @@ -0,0 +1,170 @@ +``` +SECURITY — VERDICT: PASS +TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten +SCOPE: 35 files +BLOCKING: + (none) +REPORTED (non-blocking): + - [MEDIUM] author-data-silently-replaced (non-array `children` of an added item) — crates/core/src/curation/draft/ops.rs:252 + crates/core/src/curation/draft/targets.rs:246 + - [LOW] check-lists-through-top-level-folder-link — crates/core/src/bin/import_equine.rs:193 (listed, :164) + - [LOW] seed-containment-is-path-based (bind-mount alias) — crates/core/src/bin/import_equine.rs:230 (inside, :253) + - [LOW] declared-depth-walk-cost-understates-cap-bound-worst-case — crates/core/src/curation/draft/apply.rs:110 / docs/development/RUST.md:3046 + - [LOW] unit-test-fixture-in-shared-temp-with-fixed-name — crates/core/src/bin/import_equine.rs:383 + - [LOW] context: clippy purity lint allowed in a new test file (project convention, not a scanner suppression) — crates/core/tests/curation_draft.rs:7 + - [LOW] context: seed SQL rendered by string formatting from pipeline values — crates/core/src/curation/draft/seed.rs:29 +PROOF: semgrep 129 rules on 35 files → 0 findings; checklist 9 checks → 7 findings (1 MEDIUM, 6 LOW) +``` + +# R5.3 draft, security gate (fresh, final 4) + +Branch `feature/rust-r5-pipeline-core`, HEAD `6c0fe88`, scope `git diff 5e114da..HEAD` +(35 files, 11 commits), with attention to `86273b6`. Read whole: `bin/import_equine.rs`, +`curation/draft/{shape,ops,apply,schema,targets,fields,records,rename,stats,migration_map,seed,pipeline,mod}.rs`, +`curation/mod.rs`, `scripts/import-equine.sh`, the `package.json` line, the binary and +integration tests that touch the file system, the oracle generator's write guard +(`MAKE_R5_ORACLES=1` only). Every probe ran in +`/tmp/claude-1000/-home-bchanot-Documents-faunosteo/9b0edf59-ccd2-4662-b90c-e473dd78028e/scratchpad/sec4.j5fliv` +on a copy of the seed, through a probe binary that is `import_equine.rs` byte for byte except +the `SEED` constant (pointed at the copy). Nothing ran `--out` against the repository, nothing +was deleted, `git status` is clean after the run. + +## Semgrep + +`semgrep scan --config p/security-audit --config p/secrets --config p/owasp-top-ten +--metrics=off` on the 35 existing files of the diff: 129 rules run, 0 findings, 0 errors. +The rulesets carry almost no Rust rules; the Rust code was covered by reading and probing. + +## Checklist + +| Check | Result | +|---|---| +| Hardcoded secret / token / key / auth URL | none (grep over the 35 files) | +| SQL by string interpolation | `seed.rs` renders the seed file by formatting: LOW context below, not a runtime query | +| Unsanitized render (innerHTML, eval…) | not applicable (Rust CLI and library, no DOM) | +| Sensitive endpoint without authz | not applicable (no endpoint) | +| Stack trace / internal path to an end user | not applicable (errors go to the local developer's stderr) | +| Secret / PII in a log | none (stdout/stderr carry paths, counts and refusals) | +| Tracked `.env` / credential file | only `.env.example` tracked | +| Guardrails scanner (`--domain api,supabase`) | 0 FAIL, 1 WARN (`SB-POLICY-NOT-OWNER`, `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the diff, dropped), 1 MANUAL | +| New suppressions (anti-gaming) | no `nosemgrep`, `nosec`, `eslint-disable`, `ts-ignore`, `noqa`; one clippy allow in a new test file (LOW context below) | + +## Owner requirements, verified + +- Seed never modified automatically: every read goes through `read()` (canonical path inside + the canonical seed, regular file, 16 MiB cap). Probed: `curation/` as a folder link out of + the seed and `changes.json` as a file link out of it are both refused (`leads outside the + seed folder`); a folder link that stays inside the seed is accepted. +- `--check` writes nothing: only `canonicalize`, `metadata`, `read`, `read_dir` on that path. + Probe copy: 12 files, 0 differences, nothing new under the fixture seed (`find -newer`). +- `--out` refusals, probed on the fixture seed: a link to the seed and to `draft/` (as the + destination and as its parent), a link chain `p/q` ending in a seed link, the seed path + itself plus `/new`, `legacy-import/new`, `INIT_CWD` set inside the seed, `..`, an empty path, + `/`: all refused, nothing created. A relative `INIT_CWD` is ignored. A valid destination + gets 12 files identical to the seed (`diff -rq`), a second run on it is refused `not empty`. + Files are `create_new`, folders `create_dir` tracked in `created`, so an entry that appears + after the claim is refused. File names are pipeline constants. +- Strict changeset validation, depth and resource caps: `deny_unknown_fields`, the `op` + discriminator, non-empty strings, the shape comparison (now with array length), reserved + NUL keys at parse and in field paths, 8 field segments, serde_json's 128 parse depth, 64 + document levels after each op (iterative walk), 16 MiB per file. Recursive walks + (`reserved_key`, `first_mismatch`, `strip_marks`, `walk`, `touch_tree`, `js_deep_equal`) are + all bounded by the 128 parse depth or the 64-level check. +- `86273b6`: `assert_object_children` refuses a non-object entry of a `children` or + `categories` list (explicit stack); the length arm of `first_mismatch` sits before the + element arm; `claim_destination(typed, seed)` with its fixture test refuses before any + `create_dir`. All correct. One gap left next to the first: MEDIUM below. + +## Findings + +### MEDIUM: a non-array `children` on an added item is silently replaced by `[]` + +`crates/core/src/curation/draft/ops.rs:252` (`assert_object_children` skips a `children` +value that is not an array) and `crates/core/src/curation/draft/targets.rs:246` +(`children_path` overwrites any present non-array `children` with `[]`). An `add` whose value +carries `"children"` as an object, a string or `null`, followed by an `add` (or a `move`) +into that item, is accepted: the strict gate never sees the author's value, which is gone +from `document.json` and the migration map. Probed on the real equine changeset plus one +level-A change: `"children": {"author-note": {"id": "kept", ...}}` (a whole nested finding +written as an object instead of an array), `"children": "author text"` and +`"children": null` all give `probe-node.children = [probe-child]`, exit 0, 12 files written. +Only the add record's `after` keeps the value; no R5.3 output shows it (the CHANGELOG is +R5.4). Parity with the TypeScript `childrenOf`, and not declared: §11.12 says "a lean leaf +gets `children: []` appended on demand", which covers an absent key, not a present value +replaced. This breaks the owner's "author data never dropped silently" through a plausible +typo (object in place of array). Non-blocking: no security boundary is crossed. +Hint: in `assert_object_children`, refuse a present `children` or `categories` key whose +value is not an array (same message family, same place as the entry check), or make +`children_path` refuse a present non-array value and append `[]` only when the key is absent. +Declare the split in the §11.12 table with a frozen TypeScript case. + +### LOW: `--check` lists through a top-level folder link + +`crates/core/src/bin/import_equine.rs:193` calls `listed(&target, ...)` on +`SEED/legacy-import` or `SEED/draft` without checking the folder itself: `read_dir` follows +it when it is a link. Probed with `legacy-import` pointing out of the seed: the reads are +refused (`missing: ...`), but `not produced: private-name.txt` and `not produced: +sub/deep.txt` print the names of files outside the seed, subfolders included. Read-only, +names only, same user. §11.12's "`--check` lists a link without walking into it" holds for +entries, not for the generated folder itself. +Hint: canonicalize the target and require `inside(&canonical, &seed)` before listing, or +refuse a target whose `symlink_metadata` is a link. + +### LOW: the seed guard is path-based, a mount alias passes it + +`crates/core/src/bin/import_equine.rs:230` compares canonical paths (`starts_with`). +`canonicalize` resolves links, not mounts: if the seed (or the repository) is reachable +through a second path (bind mount, the same NAS share mounted twice), `--out +/.../equine/new` passes `inside` and creates a new folder inside the official seed. +Only new files (never an overwrite, never a delete); the user must type the alias. Reasoned, +not probed (a mount needs privileges). +Hint: also compare `(dev, ino)` (`std::os::unix::fs::MetadataExt`) of the canonical parent and +each of its ancestors with the seed folder's; a bind mount root shares the source's pair. + +### LOW: the declared depth-walk cost understates the cap-bound worst case + +`crates/core/src/curation/draft/apply.rs:110` runs `assert_depth` over the whole document +after every op; RUST.md:3046 declares it with one sample ("a 2.4 MB changeset takes about +10 s"). The sample is right (probe: 2.3 MB, one 1 M-node add then 2,000 sets, 12.4 s, +455 MB). The worst case inside the 16 MiB read cap is two orders larger: 8.2 MB, one 4 M-node +add then 1,000 sets, 35.7 s and 1.7 GB, so about 30 ms per op, and the remaining 8 MB holds +about 94,000 more ops: roughly 45 minutes of CPU before the final gate refuses. Twenty +thousand plain sets on the real document take 0.9 s, so ordinary changesets are not affected. +Local developer tool, interruptible. The declaration is not wrong, it is incomplete. +Hint: state the cap-bound figure in §11.13, or bound the work: a total node cap on the +document after each op (also bounds memory), or a depth check limited to the subtree the op +wrote. + +### LOW: unit-test fixture at a fixed name in the shared temp folder + +`crates/core/src/bin/import_equine.rs:383`: `env::temp_dir().join("faunosteo-import-equine-claim")` +with `create_dir_all`. On a multi-user host another account can pre-create or link that +name. The test only runs refusals, so the worst outcome is a failed test or an empty folder +created where the link points. The integration test already uses the private +`CARGO_TARGET_TMPDIR`. +Hint: put the fixture under the target folder (for example +`concat!(env!("CARGO_MANIFEST_DIR"), "/../../target/...")`) or a per-run name created +exclusively. + +### LOW (context): clippy purity lint allowed in a new test file + +`crates/core/tests/curation_draft.rs:7` `#![allow(clippy::disallowed_methods, +clippy::unwrap_used, clippy::expect_used)]`. A determinism and robustness lint (no I/O, no +clock in the core), not a security scanner suppression. The same line heads the ten other +core test files. The binary's `#![allow(clippy::disallowed_methods)]` was already present at +`5e114da`. Not blocking. + +### LOW (context): seed SQL rendered by string formatting + +`crates/core/src/curation/draft/seed.rs:29` (`sql_text`) and `:90` (`document_literal`). Every +interpolated value is a pipeline constant, a sha256 hex, a derived uuid or a byte count; +quotes are doubled (correct with `standard_conforming_strings` on, the PostgreSQL default); +the document is embedded in a `$equine_doc$` literal and refused if it holds the tag; the +formatted JSON always ends with `}`, so no tag can form at the closing boundary. This +generates a reviewed seed file, it is not a runtime query. No action. + +## Residual risks declared in RUST.md §11.13 + +Memory without a cap beyond 16 MiB per file: confirmed (1.7 GB from an 8.2 MB file), in line +with the declaration. Single-user check-then-use window on reads and `--out`: confirmed as +described, every file stays `create_new`. Lossy UTF-8 and repeated keys: as declared. The +quadratic walk: LOW above. diff --git a/docs/development/RUST.md b/docs/development/RUST.md index 55a235de..58c501ec 100644 --- a/docs/development/RUST.md +++ b/docs/development/RUST.md @@ -2885,7 +2885,7 @@ until the switch; no fingerprint change; the cleaned `horse.svg` is read as froz re-cleaned; the review drawings and the two Markdown reports are R5.4. Contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`. -`crates/core/src/curation/draft` (2,877 lines, feature `curation`): +`crates/core/src/curation/draft` (2,912 lines, feature `curation`): | Module | Ports | |---|---| @@ -2916,18 +2916,19 @@ Parity, every expectation written by the TypeScript (`tests/curation_draft.rs`, - the 612 legacy leaves through the core's `read_report_document` on the produced map: 442 identical, 141 moved, 29 removed, every resolved leaf a leaf of the produced document, the 29 flagged paths exactly those the map sends to null, the FR snapshots kept; -- 77 synthetic changesets on the mini sheet (`oracles/r5/curation-synthetic.json`, - generator `packages/anatomy/tests/r5.curation-synthetic-oracles.test.ts`): 11 accepted +- 81 synthetic changesets on the mini sheet (`oracles/r5/curation-synthetic.json`, + generator `packages/anatomy/tests/r5.curation-synthetic-oracles.test.ts`): 14 accepted cases (every op kind, numeric and anchored positions, subtree scopes with skipped equal values, the review rule on levels A, B and C, the five rename cascades and a reference - rename, re-homing, two chained changes, the two migration-map debt cases below) byte for + rename, re-homing, two chained changes, the two migration-map debt cases below, the three + `JSON.parse` cases of the aligned paragraph) byte for byte on the formatted document, the records, the stats, the migration map and the seed SQL; 63 refusals (23 schema, `null` on `scope`, `position` and `field` and the shape family below included, 40 applier, integrity and shape gates included) refused alike, with the same reason: the same message for an op or change refusal (change id and op index included), the same stage for a schema or gate refusal; 7 references files refused alike; - two declared - differences below (one refusal, one acceptance), each held by its own test. + four declared synthetic cases below (three refusals, one acceptance), each held by its own + test (14 compared + 63 refused alike + 4 declared = 81). Differences declared (none reachable by the committed files, since the folders are reproduced byte for byte): @@ -2943,6 +2944,7 @@ reproduced byte for byte): | a `set` field path ending in `__proto__` (`name.__proto__`) | the prototype setter swallowed the write: change accepted, one record, the document unchanged (or, when the holder was absent, `description: {}` added by `writeField`) | the key is written and the strict document gate refuses it (the R5.2 family: a name of `Object.prototype` is a plain key) | | an authored key starting with NUL (`"\u0000touched"`) in a value, or a field path segment starting with NUL (`name.\u0000touched`) | refused at the strict document gate when the key reaches the document, accepted when a later op overwrites or removes it | refused (`reserved key`): at parse for a value, at the op for a field path; the prefix is reserved for the applier's transient touch marks, which the cleanup strips at every depth, so a written key would be lost while a record claims it (found by the final gates, test `a_reserved_or_unbounded_field_path_is_refused_without_writing`) | | a null, a string, a number or a boolean among the `children` or `categories` of an added item | a TypeError where a walk met it (`descendants`, the review marking: refused), silently skipped by a rename's citation walk (accepted when a later op removed the item), refused by the gate when it stayed | refused at the add (`… entry that is not an object`), whatever follows (three cases frozen from the TypeScript, one declared: `declared-children-string-renamed-then-removed`) | +| a `children` or `categories` of an added item that is present but not a list (an object, a string, `null`) | replaced by `[]` when a later op added under the item (`childrenOf`): the author's value lost without a word | refused at the add (`… that is not a list`); `children_path` never replaces a present value (declared case `declared-children-object-then-added-to`) | | a `set` whose path goes through a string (`name.fr.x`): `writeField` replaces the string by an object | `before` recorded as undefined (null), the replaced string in no record; refused by the gate unless a later op overwrites the field | the same (kept, recorded here) | | a field path of more than 8 segments (the deepest settable path has 2) | accepted, the gate refused the stray keys | refused at the op: a resource bound (20,000 segments overflowed the stack before it) | | a document nested more than 64 levels after an op (the equine document has 10), by repeated `add` or `move`, or a deep value | accepted when valid | refused at the op: a resource bound checked after every op with an iterative walk (test `the_document_cannot_be_nested_past_the_bound_by_repeated_adds`) | @@ -2963,6 +2965,14 @@ findings only (33) and no reference, so the equine map is untouched. A future ch renaming a view, zone, region, category or a cited reference must not be published before the fix (a declared behaviour change on both sides, after R5). +Aligned on `JSON.parse`, not a difference: an authored value keeps JavaScript's key order (array +index keys such as `"1"` first, ascending, then the others in insertion order) and an integer +beyond 2^53 is read as the double JavaScript holds (`9007199254740993` → `9007199254740992`), +at parse (`text::js_normalize`) and when a field path writes a key (`text::js_insert`), so a +record never holds a value its own document lacks (found by the fourth final verifier; cases +`js-index-key-in-value-overwritten`, `js-index-segment-overwritten`, test +`an_integer_beyond_2_53_is_read_as_the_double_json_parse_gives`). + Aligned on Zod, not a difference: serde's derives read a struct from a positional array (`"title": ["Titre", "Title"]`, a whole change or reference as an array) and a unit enum variant from a one-key map (`"level": {"A": null}`, `scope`, `review_status`, `laterality`, @@ -3030,7 +3040,15 @@ seed guard tested only as a bare function, `before: null` over a replaced string non-object children refused at the add; reads confined to the seed by canonical path; array length compared; only folders this run created are used; `claim_destination` takes the seed and is tested on a fixture (`the_claim_refuses_a_destination_inside_the_seed_and_creates_nothing`). -Declared: the replaced-string `before`, the memory factor. +Declared: the replaced-string `before`, the memory factor. Fourth final round on 6c0fe88: +verifier ECARTS(2) (C1, C2, C4 to C12 met, 0 accept/refuse split in 25,000 random changesets; +records differing from the TypeScript on array index keys and on integers beyond 2^53; the +declared-case count wrong) and security PASS with one MEDIUM (a present non-list `children` +silently replaced by `[]`) and LOWs (`--check` listing through a generated folder that is a +link, a mount alias of the seed, the walk cost understated, the claim test fixture in the +shared temporary folder). Fixed: `js_normalize` and `js_insert`; non-list `children` and +`categories` refused at the add and never replaced; `--check` confined to the seed's folders; +the identity guard; a 200,000-value bound; the fixture under `target/`. ### 11.13 Checkpoint R5.3 @@ -3040,16 +3058,21 @@ the readers of imported PDFs (`archive/read.rs`, `report/read.rs`, `pdf/import.r accept an object written as an array or an enum written as a map where the frozen Zod reader refused it. Not probed on those readers, not changed: a fix there needs the owner's per-file go. -Residual risks, measured by the final security gate and kept: memory has no cap beyond the -16 MiB read bound (a hostile 211 KB changeset of subtree sets on a 2,000-node addition asks -for more than 2 GB; the port clones the op into every record where the TypeScript shared it, -about twice its memory); the depth check walks the document -after every op (a 2.4 MB changeset takes about 10 s); `--out` assumes a single-user machine +Residual risks, measured by the final security gates and kept: the document is bounded (64 +levels, 200,000 values, checked after every op; the equine document has 10 and 6,374), but +the records are not: a hostile changeset of subtree sets with a large value still costs +memory in proportion to ops × items × value size (211 KB asked for more than 2 GB before the +value bound; the port clones the op into every record where the TypeScript shared it, about +twice its memory); the bound walk runs after every op (at most 200,000 values each, a few +milliseconds; a 16 MiB changeset of tiny ops could take minutes); `--out` assumes a single-user machine (a concurrent process able to write the destination's parent could still swap a component between a check and its use, never overwriting: every file is `create_new`); invalid UTF-8 and repeated keys are read as the TypeScript read them. Residual risks of the slice: the `--check` allowance for the six R5.4 files hides nothing about the produced files but means the draft folder is not yet compared whole (R5.4 removes it); the +seed guard of `--out` compares canonical paths and file identity (device, inode) of every +ancestor, so a second mount path of the seed is refused too; `--check` refuses a generated +folder that is not itself a folder of the seed (a link there is never listed through); the touch marks are a reserved prefix (a NUL character starts it), refused in any authored value at parse and in any field path segment at the op, and removed after each change, so no mark reaches a record or the document; the migration-map replay quirk above