From 0a78492d64fd20fccec53f04ab78a3001d12fa30 Mon Sep 17 00:00:00 2001 From: bchanot Date: Fri, 23 Oct 2026 11:12:22 +0200 Subject: [PATCH] chore(memory): R5.3 closed on c6e0d85 (verifier CONFORME 13/13, security PASS), journal, TODO, security final6 --- .../audits/2026-10-22-rust-r5-draft-verify.md | 7 + ...026-10-23-rust-r5-draft-security-final6.md | 151 ++++++++++++++++++ .claude/memory/journal.md | 4 + .claude/tasks/TODO.md | 3 +- 4 files changed, 164 insertions(+), 1 deletion(-) create mode 100644 .claude/audits/2026-10-23-rust-r5-draft-security-final6.md diff --git a/.claude/audits/2026-10-22-rust-r5-draft-verify.md b/.claude/audits/2026-10-22-rust-r5-draft-verify.md index 1d9f2664..2734cc69 100644 --- a/.claude/audits/2026-10-22-rust-r5-draft-verify.md +++ b/.claude/audits/2026-10-22-rust-r5-draft-verify.md @@ -64,3 +64,10 @@ Contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`. Fresh verifier, - Verifier ECARTS(2), C3 + C13: a `__proto__` segment in a field path: at the end, TS records `before: {}` (setter ignored the primitive) where Rust recorded null and kept a plain key; in the middle, TS polluted Object.prototype and refused, Rust accepted when overwritten; RUST.md claimed the inherited-member family refused by TS at the op, false for `__proto__`; test count stale (15 vs 13). C1, C2, C4 to C12 MET; 120,000 random changesets: 40,494 accepted identical, no other undeclared split. - Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final5.md`): LOW identity guard covers the seed folder only (a subfolder alias passes), quadratic id comparisons (parse_references 31 s at 160,000), seed SQL formatting (checked), test allow (context). - Fixed: `__proto__` segment refused at the op (reserved key), declared case frozen from TS (the non-polluting one; the polluting one tested in Rust only, it would corrupt the generator's process); identity guard over every folder of the seed tree; HashSet in assert_declared, distinct, duplicates; docs rows and counts. + +## Sixth final round on c6e0d85: CONFORME 13/13 (R5.3 closed) + +- Verifier CONFORME: C1 to C13 MET on the committed state c6e0d85 (23 curation tests, `--check` 12/0 via cargo and pnpm, `--out` + `diff -r`, 21 destination probes, anatomy 142, floor clean; TS-vs-Rust differential on 70,073 changesets: 9,397 accepted identical, 0 Rust-only accepts, TS-only accepts all declared families). +- Security PASS (`.claude/audits/2026-10-23-rust-r5-draft-security-final6.md`), LOW only: destination folder itself not identity-checked (needs an empty seed folder + a hand mount; none exists), one linear membership in cite (doc precision), seed SQL formatting (checked), test allow (context). +- check:all green 10:54:50 on c6e0d85. +- Carried into R5.4 (not applied to the verified state): destination identity check, cite membership hashed, "seed folder unreadable" message, stale test comment, doc nits (inherited row: remove form too; 200,000-value row in the table; line count 2,908; set-without-value counted among schema refusals). diff --git a/.claude/audits/2026-10-23-rust-r5-draft-security-final6.md b/.claude/audits/2026-10-23-rust-r5-draft-security-final6.md new file mode 100644 index 00000000..93a4165f --- /dev/null +++ b/.claude/audits/2026-10-23-rust-r5-draft-security-final6.md @@ -0,0 +1,151 @@ +SECURITY — VERDICT: PASS +TOOL: semgrep 1.168.0 — p/security-audit, p/secrets, p/owasp-top-ten +SCOPE: 38 files +BLOCKING: + (none) +REPORTED (non-blocking): + - [LOW] seed-guard-destination-identity — crates/core/src/bin/import_equine.rs:236 (destination branch :243-253) + - [LOW] cite-membership-linear (doc precision, RUST.md §11.13) — crates/core/src/curation/draft/ops.rs:401-405 + - [LOW] sql-built-by-formatting (checked, not injectable) — crates/core/src/curation/draft/seed.rs:29-135 + - [LOW] new-lint-allow-in-test (context, not a scanner suppression) — crates/core/tests/curation_draft.rs:7 +PROOF: semgrep 163 rules on 38 files → 0 findings; checklist 14 checks → 4 findings + +# Security gate, R5.3 curated draft, sixth final round (fresh, blind) + +- Repository: /home/bchanot/Documents/faunosteo, branch `feature/rust-r5-pipeline-core`, HEAD `c6e0d85`. +- Scope: `git diff 5e114da..HEAD`, 38 files, all present. Focus on `f245b3e`. +- Product files read in full: `crates/core/src/bin/import_equine.rs`, + `crates/core/src/curation/draft/{apply,fields,migration_map,mod,ops,pipeline,records,rename,schema,seed,shape,stats,targets}.rs`, + `crates/core/src/curation/{mod,text}.rs`, `scripts/import-equine.sh`, and the diffs of `crates/core/Cargo.toml` and + `package.json`. The test files and the TypeScript oracle generator were checked for file system and process calls. + +## Semgrep + +`EIO_BACKEND=posix semgrep scan --config p/security-audit --config p/secrets --config p/owasp-top-ten --metrics=off --quiet --json <38 files>`: +163 rules run (javascript 73, generic 57, typescript 11, python 6, package managers 5, json 3, java 3, scala 3, bash 1, +ruby 1), 38 files scanned, 0 findings, 0 errors. The pinned packs hold no Rust rule, so the Rust code is covered by the +reading, the checklist and the probes below. + +## Checklist (14 checks) + +| # | Check | Result | +|---|---|---| +| 1 | Hardcoded secret, token, key, auth URL (whole diff incl. docs, `.claude`, oracle JSON) | none | +| 2 | SQL built by string formatting | `seed.rs` generator, quoting verified: LOW 3 | +| 3 | Unsanitized render, `eval`, `unsafe`, process spawn | none in scope | +| 4 | Sensitive endpoint without authz | not applicable (dev CLI and library, no endpoint) | +| 5 | Stack trace, internal path, DB error to an end user | none (CLI messages go to the developer running the tool) | +| 6 | Secret or PII in a log | none (stdout and stderr carry paths, counts, refusals) | +| 7 | Tracked `.env` or credential file | none (`.env.example` only, pre-existing, outside the diff) | +| 8 | Guardrails scanner (`--domain api,supabase`) | 0 FAIL, 1 WARN `SB-POLICY-NOT-OWNER` at `supabase/migrations/20261008110000_anatomical_sheets.sql:1177`, outside the diff, filtered out; 1 MANUAL not reported | +| 9 | New suppressions (anti-gaming) | no `nosemgrep`, `nosec`, `eslint-disable`, `ts-ignore`, `noqa`; one clippy allow in a test file: LOW 4 | +| 10 | `--check` writes nothing | built binary run: 12 files, 0 differences, 6 R5.4 files listed as pending; seed file list and mtimes identical before and after; `git status` clean | +| 11 | `--out` destinations | 11 probes in a fresh `mktemp -d` under the scratchpad, all as designed (below) | +| 12 | Seed guard of `--out` (canonical path, then device/inode over the seed tree) | holds for every parent path; the destination folder itself is not identity-checked: LOW 1 | +| 13 | `__proto__` field segments | refused at every position, before any write (probe below) | +| 14 | Resource caps and id comparisons | parser depth, 8 segments, 64 levels, 200,000 values hold; hash sets in `distinct`, `assert_declared`, `duplicates` keep the old semantics; one linear membership left in `cite`: LOW 2 | + +### Probes (scratchpad only; no `--out` into the repository; no recursive delete) + +The binary was built with `CARGO_TARGET_DIR` in the scratchpad (`cargo build --release --locked --offline -p faunosteo-core --features curation --bin import-equine`). +Two throwaway probes ran there: one linking `faunosteo-core` with `curation`, and one compiling lines 213-293 of +`import_equine.rs` verbatim against a scratch seed tree. + +- `split_field`: `__proto__`, `name.__proto__`, `name.__proto__.x` and `name.fr.__proto__` are all refused with + `reserved key "__proto__"`. `name.__proto__ ` (trailing space), `name.__PROTO__`, `name.constructor` and `name.prototype` are + plain keys, as in JavaScript on a plain object. A NUL segment, 9 segments and an empty segment are refused. + `apply_changeset` with `set name.__proto__…` followed by an overwriting `set name` refuses at `CHG-001 op 0` in all four + variants. `is_settable` calls `split_field` before `item_at_mut`, so nothing is touched or written first. +- Identity guard (verbatim copy, scratch tree `seed/{a/b/c,d}` plus a link to `/` and a link loop inside it): the walk + records 5 identities and follows neither link. `seed/a/b/new`, `seed/new`, a parent link into `seed/a/b` and a parent link + to `seed` are all refused as `inside the seed folder`, with nothing created. A final link, a trailing `/` and a trailing + `/.` are refused as `a symbolic link`. A sibling folder is accepted. With one seed folder made unreadable (`chmod 000`, + restored afterwards), every claim is refused: the walk fails closed. +- Real binary `--out`: absent folder (12 files written, then `diff -r` against the seed shows only the 6 R5.4 files); + the same folder again (`not empty`, exit 2); an empty folder (written); a regular file (`not a folder`); a final link, with + `/` and `/.` (`a symbolic link`, exit 2, link target left empty); `empty/../x` (`..` refused); a missing parent; `""` + (usage error); a relative path with `INIT_CWD` unset (typed from the current folder, the canonical path printed before the + first write). No entry appeared outside the probe folder. +- Cite cost (release build): citing 28,000 declared ids twice on one finding takes 0.92 s, four times 2.7 s. The value bound + then refuses a larger registry plus its citations. + +## Findings (all non-blocking) + +### LOW 1: the destination folder itself is never identity-checked + +`crates/core/src/bin/import_equine.rs:236` runs `inside_by_identity(&parent, seed)`, which checks the ancestors of the +canonical parent. When the destination already exists (`:247-253`), only its emptiness is checked, never its own +(device, inode). Suppose an existing empty folder is a second mount path (a bind mount, or a share mounted twice on the +same device) of an empty folder of the seed. The claim accepts it, and `write` then creates `legacy-import/…` and +`draft/…` inside the seed tree. A mount alias of a non-empty seed folder is refused as `not empty`. + +Why only LOW: this needs an empty folder under the seed (none in the checkout: `find … -type d -empty` returns nothing, +and git tracks no empty folder) plus a mount made by hand at the destination. Every write is `create_dir` or `create_new`, +so nothing is overwritten or deleted. Also, the §11.13 sentence "a second mount path of the seed or of any of its folders +on the same device is refused too" holds for every parent, but not for the destination itself. Traced by reading, not +reproduced: a mount needs privileges and is the user's to run. + +Suggested fix: in the existing-folder branch, also refuse when `inside_by_identity(&root, seed)` is true (`root.ancestors()` +starts with `root` itself). Alternatively, run the identity check on `root` once it exists, created or not. Or narrow the +§11.13 sentence to the ancestors of the destination. + +### LOW 2: one cite comparison is still linear, while §11.13 says cite uses hash sets + +`crates/core/src/curation/draft/ops.rs:401-405`: for each scope entry, every distinct cited id is checked with +`before.contains(&Value::from(id))` on the item's existing citations. That costs O(distinct × existing) per item. `distinct` +and `assert_declared` are hashed now, but this membership is not, and §11.13 reads "id comparisons use hash sets (cite, +declared references, duplicate ids)". Measured: about 0.9 s per repeated op at the largest registry the 200,000-value bound +admits (about 28,000 references, each serialised as 6 values). Spread over a subtree, the worst case is about twice that. +A 16 MiB changeset therefore stays in minutes, the same order as the declared CPU residual. So this is a precision issue +in the declaration, not a new risk. + +Suggested fix: build a `HashSet<&str>` of the string entries of `before` once per item and keep the output order by +walking `distinct`. Or reword §11.13 ("cite deduplication and the declared check are hashed; the check against existing +citations is linear, bounded by the value bound"). + +### LOW 3: seed SQL built by string formatting, checked and not injectable + +`crates/core/src/curation/draft/seed.rs:29-135`. This is a checklist hit on "SQL built by interpolation", but the code +writes a generated seed file, not a runtime query. The `sql_text` values are pipeline values: the sheet id from the legacy +import, sha256 hex, the asset id derived from it, the fixed file name and the constant change note. All of them are quoted +by doubling `'`. The authored document goes into a `$equine_doc$` dollar-quoted literal, and a document text that holds the +tag is refused (`:91`). The text is `format_json` of an object, so it begins with `{` and ends with `}` after `trimEnd`, and +no tag can form at a junction. Same construction as the TypeScript. No action needed. If this is touched later, keep the +tag check on the final text. + +### LOW 4 (context): lint allow in a new integration test + +`crates/core/tests/curation_draft.rs:7` `#![allow(clippy::disallowed_methods, clippy::unwrap_used, clippy::expect_used)]`. +Every other core test file carries the same allow, and `clippy.toml` documents it ("Tests and the worker binary allow these +lints locally where they own I/O"). It silences no security scanner, so it is not a gated suppression. The binary's +`#![allow(clippy::disallowed_methods)]` (`import_equine.rs:15`) was already present at `5e114da`. The core library itself has +no file system, environment, clock or process call (grep over `crates/core/src` outside `bin/`). + +## Owner requirements, checked + +- The official seed is never modified automatically. `--check` was run and is read-only. `--out` refuses every parent + inside the seed by canonical path and by identity over the whole seed tree (exception: LOW 1, the destination itself). +- `--out` never deletes (the binary has no remove call) and never overwrites (`create_dir`, `create_new`, the claimed + folder re-checked before each file). It refuses an empty path, `..`, a missing parent, a symbolic link (trailing `/` and + `/.` included), a non-folder and a non-empty folder. +- Strict changeset validation: strict shapes plus the shape comparison; NUL keys refused at parse and in field segments; + `__proto__` segments refused at the op; non-list or non-object children refused at the add. +- Depth and resource caps: parser 128 levels, field path 8 segments, document 64 levels and 200,000 values after each op, + source files under 16 MiB. The seed walk is iterative and follows no link. +- Author data is not dropped silently. An overwritten value is recorded as `before`, and the cases with no faithful reading + (`__proto__`, NUL keys, non-list children) refuse. +- The residual risks declared in §11.13 (record memory, the per-op bound walk, the single-user check-then-use window, lossy + UTF-8 and repeated keys, a share mounted twice on distinct devices) match the code. The exceptions are the two wording + points in LOW 1 and LOW 2. + +## Notes (no severity) + +- When the seed walk fails, the refusal reads `inside the seed folder` (`import_equine.rs:287-289`). That fails closed, but + the message is misleading. Something like "seed folder unreadable" would point at the real cause. +- The doc comment of `crates/core/tests/curation_draft.rs:520-526` lists `__proto__` among the ordinary keys, then says it + is refused. It is stale wording in a test. + +## Not done + +- The pinned semgrep packs have no Rust rules: Rust coverage comes from the reading, the checklist and the probes. +- LOW 1 was traced by reading. No mount was made. diff --git a/.claude/memory/journal.md b/.claude/memory/journal.md index b4f35fd9..f5e4aa85 100644 --- a/.claude/memory/journal.md +++ b/.claude/memory/journal.md @@ -330,3 +330,7 @@ Owner validated R5.1, decided D-R5 = B `sha256-rfc8785-validated-v1` + aliases, ## 2026-10-22/23 (R5.3 curated draft in Rust) Owner validated R5.1b/R5.2, go R5.3, named `pnpm r5:import-equine`. e0f149b: `crates/core/src/curation/draft` (2,718 lines: schema, targets, fields, records, ops, rename, apply, migration_map, stats, seed, pipeline) + `import-equine` both folders + `scripts/import-equine.sh`; draft 5 texts + frozen horse.svg byte-exact (`--check` 12/0), 825 records, 46 changes, stats, migration stats 792/571/180/7/34, 612 leaves 442/141/29, 58 synthetic TS cases (`oracles/r5/curation-synthetic.json`, 9 byte-exact, 49 refused alike). Security PASS (6 LOW, 3 fixed). Verifier 4 passes: found null-on-optional, repeated keys, NUL touch-mark drop, symlink trailing slash, lossy UTF-8; all fixed or declared (RUST.md §11.12); pass 4 C1-C12 met, C13 doc gaps fixed after, not re-verified. check:all green 00:10:03. documentSha256 unchanged. STOP at R5.3 checkpoint; R5.4 on owner go. + +## 2026-10-23 (R5.3 closed after 6 final gate rounds) + +Owner validated R5.3 functionally, asked a fresh verifier on the final state, Object.prototype characterisation, migration-map debt kept + characterised, LRN/EVAL. 9901106/8654905: Object.prototype member = plain key, safe, test; debt test from TS-frozen cases; refusal reasons compared by stage; LRN-072, LRN-073, EVAL-028. Then 6 rounds (verifier + security + check:all each) on the final state, each finding real edge families via TS-vs-Rust fuzzing (up to 150,000 changesets): NUL field segment silent drop, stack overflow (8 segments, 64 levels, 200,000 values), non-exclusive --out, serde struct-as-array / enum-as-map (draft/shape.rs; R2 derives raised to owner, gated readers), non-list/non-object children, JSON.parse key order + big ints, __proto__ segments, seed identity guard, reads confined to seed. Sixth round c6e0d85: CONFORME 13/13, security PASS LOW only, check:all green. LOW leftovers carried into R5.4. Next: R5.4 (owner go conditional on these checks: met). diff --git a/.claude/tasks/TODO.md b/.claude/tasks/TODO.md index 00701af7..ea5acc8a 100644 --- a/.claude/tasks/TODO.md +++ b/.claude/tasks/TODO.md @@ -309,12 +309,13 @@ feat/findings-tree-v2 merged (FindingsTreePicker tracked in HEAD); all work belo - [x] owner decisions (2026-10-22, BDR-068): D-R5 = B `sha256-rfc8785-validated-v1` (applied at R5.6 only), aliases from trusted frozen documents only, F1 go, byte-exact by default + declared differences, historical SVG bytes kept, `.prettierignore` on Rust-written paths + reproducible check, `ureq` preferred at R5.5; stop after R5.2 - [x] R5.1b F1 fix (aed6b1e): `zmij` digits; V8 oracle `oracles/r5/numbers.json` 44,322 rows (1,552 mismatches before, 0 after); 840.1692504882812 test; `zod_oracle` checks oracle bytes vs core compact print (red on old printer); consumers audited R2-R4: rust:check + wasm + check green, no oracle/golden/digest moved - [x] R5.2 legacy importer in Rust (feature `curation`, `crates/core/src/curation/**`, 3,187 lines) + `import-equine --check | --out`: legacy-import/ 4 texts + 2 SVG byte-exact, pipeline.json legacy parts equal, raw report = pre-Prettier digest, determinism, 10 synthetic TS cases (2 byte-exact, 8 refused alike), format probes (968 JSON, 10 tables, slugs, uuids, Number, Math.round, key order); declared differences RUST.md §11.10; contract `.claude/tasks/contracts/2026-10-22-rust-r5-f1-importer.md` → checkpoint, STOP before R5.3 - - [x] R5.3 changeset applier, migration map, stats, seed SQL (done e0f149b, checkpoint delivered, STOP before R5.4; owner go 2026-10-22 « Validation R5.1b / R5.2 et autorisation R5.3 »: `pnpm r5:import-equine --check | --out ` names the Rust binary, `pnpm import:equine` kept; no fingerprint switch; `horse.svg` frozen bytes read, never re-cleaned; R5.4 files (review SVGs, CHANGELOG, REVIEW-REPORT) out of scope; stop at the checkpoint) + - [x] R5.3 changeset applier, migration map, stats, seed SQL (CLOSED 2026-10-23 on c6e0d85: fresh verifier CONFORME 13/13 after 6 final rounds, security PASS LOW only, check:all green; owner go 2026-10-22 « Validation R5.1b / R5.2 et autorisation R5.3 »: `pnpm r5:import-equine --check | --out ` names the Rust binary, `pnpm import:equine` kept; no fingerprint switch; `horse.svg` frozen bytes read, never re-cleaned; R5.4 files (review SVGs, CHANGELOG, REVIEW-REPORT) out of scope; stop at the checkpoint) - [x] R5.3a `crates/core/src/curation/draft/{schema,targets,fields,ops,rename,apply}.rs`: changeset + references files parsed strictly (Zod shape order on re-emit, `priority` default, level C question, duplicate ids), target grammar, settable fields, the six ops on the plain JSON document (`serde_json::Value`, insertion order), rename propagation, review rule (touched items → `to_confirm`, identity through a transient marker key), shape + integrity gate through the R2 core (`AnatomicalSheetDocument` + `validate_sheet_document`) - [x] R5.3b `draft/{migration_map,stats,seed,pipeline}.rs`: replay of rename/remove/move records over the 792 legacy paths, re-homing, stats; `documentStats` (17 fields, JS key order); `renderDraftSeedSql` byte-exact; `build_equine_draft` (CHG-000 from the frozen `horse.svg` bytes and the head view box) + `render_draft_artifacts` (document, assets, sheet-ref with sha256 of the document bytes, migration map, seed SQL) - [x] R5.3c `import-equine`: both folders (`/legacy-import`, `/draft`), `--check` compares the produced files and lists the R5.4 files as pending (not counted), `--out` refuses a non-empty folder, a symlink, a file, an empty path; `scripts/import-equine.sh` + root script `r5:import-equine` - [x] R5.3d proofs (draft 5 files byte-exact, 12/12 `--check`, 46 changes + 825 records + stats equal, 612 leaves 442/141/29, 58 synthetic cases: 9 byte-exact + 49 refused alike + 5 references files, rust:check green): `tests/curation_draft.rs` (draft files byte-exact vs committed, `pipeline.json` draft section: 46 changes, 825 records, stats, migration stats, assets, issues; determinism; 612-leaf walk through the core's migration on the produced map); TS generator `r5.curation-synthetic-oracles.test.ts` → `oracles/r5/curation-synthetic.json` (mini sheet: every op kind, positions, subtree scopes, review rule B/C, renames with cascades, re-homing, refusals) run by the Rust suite; `pnpm rust:check`, `pnpm check:all` - [x] R5.3e (security PASS 6 LOW / 3 fixed; verifier 4 passes, last C1-C12 met + C13 docs fixed after; check:all green 00:10:03) gates: security auditor (diff), independent verifier on the contract `.claude/tasks/contracts/2026-10-22-rust-r5-draft.md`; docs RUST.md §11.12-11.13, CHANGELOG, CLAUDE.md commands; memory → checkpoint report, STOP before R5.4 + - [x] R5.3g final rounds 1-6 (verifier + security + check:all each): NUL field segments, depth/segment/value bounds, exclusive --out writes, shape family, JSON.parse normalisation, non-list/non-object children, __proto__ segments, seed identity guard, reads confined to the seed; all fixed or declared (RUST.md §11.12-11.13) - [x] R5.3f owner 2026-10-23: Object.prototype member characterised (safe, accepted, test), migration-map debt kept + characterisation test, refusal reasons compared by stage; final fresh verifier + security on the R5.3 state - [ ] debt (owner 2026-10-23, after R5): migration map replays cascade records by their op (`applyRecord` / `apply_record`): a view, zone, region, category or cited-reference rename corrupts the map; fix both sides as a declared change before any changeset uses such a rename (RUST.md §11.12) - [ ] OWNER DECISION (found 2026-10-23 by the R5.3 final verifier): serde derives of the R2 contracts read a struct from an array and a unit enum from a one-key map, where Zod refused; likely reachable by the gated readers of imported PDFs (`archive/read.rs`, `report/read.rs`, `pdf/import.rs`); probe the readers, then a per-file go for a strict deserialisation (RUST.md §11.13)